Objectives

  • Discuss count normalizations
  • Execute model fitting for differential expression comparisons

Differential Expression Workflow

Here we will proceed with count normalizations and fit our DESeq2 model.


Count normalizations

Since counts of mapped reads for each gene is proportional to the expression of RNA in addition to many “uninteresting” other factors, normalization is the process of scaling raw count values to account for the “uninteresting” factors and ensure expression levels are more comparable.

Normalization goals

Two common factors that need to be accounted for during normalization are sequencing depth and gene length. Common normalization approaches (such as FPKM, RPKM, CPM, TPM, etc.) account for one or both of these factors.

  • Sequencing depth normalization is necessary to account for the proportion of reads per gene expected for more deeply sequenced samples (like in pink below) versus a less deeply sequenced sample (like in green below).

Note that each pink or green rectangle represents an aligned read, with reads spanning an intron connected by a dashed line. Figure from HBC training materials

  • Gene length normalization may also be necessary if comparing between different genes, since genes of different lengths have different probabilities of generating fragments that end up in the library.

Note: The above figure was originally from a HBC tutorial that also includes a detailed comparison of different normalization (CPM, TPM, FPKM) approaches and their best uses.

Check-in: Questions about normalizations?

DESeq2 normalizations

DESeq2 has an internal normalization process that accounts for RNA composition. A few highly differentially expressed genes, differences in the number of genes expressed between samples, or contamination are not accounted for by depth or gene length normalization methods. Accounting for RNA composition is particularly important for differential expression analyses, regardless of the tool used.

For data exploration and visualizations, it is helpful to generate an object of independently normalized counts. We will use the rlog transformation from DESeq2 that accounts for sequencing depth for each sample and RNA composition for the downstream quality control visualizations.

The rlog transformation produces log2 scaled data that has also been normalized to overall library size as well as variance across genes at different mean expression levels. For larger numbers of samples, there is an alternative transformation method, vst that can be used instead for count normalizations.

The command to generate the normalized count object has a few parts, including dds as an input and providing a value to the option blind. For our purposes, we set blind = TRUE because we want to compare samples in downstream QC plots in an unbiased manner.

rld = rlog(dds, blind = TRUE)

Note: We might see a message that our data did not fit the default ‘parametric’ dispersion model so a local regression was substituted. If we had more time, we might look at a dispersion plot with the plotDispEsts(dds) function, but as this bioconductor thread discusses, other visualizations of our data might be more helpful and/or easier to interpret.

Next, we’ll look at the results of the transformation by extracting the values with the assay() function.

head(assay(rld), 2)
                   sample_A sample_B sample_C sample_D sample_E sample_F
ENSMUSG00000000001 10.51481 10.36671 10.41946 10.84037 10.41045 10.57877
ENSMUSG00000000028 10.60446 10.73451 10.73503 10.68271 10.82094 10.99100

Looking at the rld values, we can see that they are now in log scale. Since we set blind=TRUE, the transformation is blind to the sample information we specified in the design formula. The normalized counts are helpful for visualization methods during expression-level quality assessment but aren’t used in the model fitting.

DESeq2 Model Fitting

Next, we’ll fit our standard model using the DESeq function and take a look at the objects we generate. This command applies the model to our data, using the sample information supplied when generating the dds object so it takes some time to run.

dds = DESeq(dds)
dds
class: DESeqDataSet 
dim: 16249 6 
metadata(1): version
assays(4): counts mu H cooks
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ...
  ENSMUSG00000118651 ENSMUSG00000118653
rowData names(22): baseMean baseVar ... deviance maxCooks
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(3): genotype condition sizeFactor

Notice that there is now more information in the DESeqDataSet object than there was prior to our normalization. There is information about the model fit and about the library size normalization. DESeq2 will use this information when we perform the test for differential expression.

The DESeq() function is actually doing three things automatically for us. It calculates:

  1. The size factors to normalize for library size with estimateSizeFactors(dds),
  2. Dispersion estimates to shrink the dispersions with estimateDispersions(dds), and
  3. The Wald test statistics with nbinomWaldTest(dds).

The resultsNames() function returns the names of the estimated effects of the model.

resultsNames(dds)
[1] "Intercept"               "condition_minus_vs_plus"

The results include the single comparison representing the two levels of condition. If there were more levels in the condition column, there would be more results listed here because DESeq2 would implicitly compare all other levels to the reference level.

Checkpoint: If you see the same results when you execute resultsNames(dds), please indicate with the green ‘yes’ button. Otherwise, please use the red ‘x’ button to get help before the break

Optional content

Click for fitting a model that includes a covariate

If you executed the commands in the additional content section from Module 07, you can fit a separate DESeq2 model for the batch example.

dds_batch = DESeq(dds_batch)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resultsNames(dds_batch)
[1] "Intercept"               "batch_Day2_vs_Day1"     
[3] "batch_Day3_vs_Day1"      "condition_minus_vs_plus"
If you run through the optional exercises, you can explore the impact of adding a covariate by substituting dds_patient for dds and re-running those commands since both DESeq2 objects have their data organized in the same way.


Summary

In this section, we:

  • Learned about count normalizations and uses
  • Generated a normalized count table
  • Fit two DESeq2 models for our data
  • Saw the impact of including a covariate in our model

Sources

Training resources used to develop materials

These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

LS0tCnRpdGxlOiAiTW9kdWxlIDA4OiBERSBOb3JtYWxpemF0aW9uIGFuZCBNb2RlbGluZyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA4LSIpCmBgYAoKPiAjIE9iamVjdGl2ZXMgey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiAqIERpc2N1c3MgY291bnQgbm9ybWFsaXphdGlvbnMKPiAqIEV4ZWN1dGUgbW9kZWwgZml0dGluZyBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIHByb2NlZWQgd2l0aCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgZml0IG91ciBERVNlcTIgbW9kZWwuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERVNlcTJERS5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBDb3VudCBub3JtYWxpemF0aW9ucwoKU2luY2UgY291bnRzIG9mIG1hcHBlZCByZWFkcyBmb3IgZWFjaCBnZW5lIGlzIHByb3BvcnRpb25hbCB0byB0aGUgZXhwcmVzc2lvbiBvZiBSTkEgaW4gYWRkaXRpb24gdG8gbWFueSDigJx1bmludGVyZXN0aW5n4oCdIG90aGVyIGZhY3RvcnMsIG5vcm1hbGl6YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2Ygc2NhbGluZyByYXcgY291bnQgdmFsdWVzIHRvIGFjY291bnQgZm9yIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIGZhY3RvcnMgYW5kIGVuc3VyZSBleHByZXNzaW9uIGxldmVscyBhcmUgbW9yZSBjb21wYXJhYmxlLgoKIyMgTm9ybWFsaXphdGlvbiBnb2FscwoKVHdvIGNvbW1vbiBmYWN0b3JzIHRoYXQgbmVlZCB0byBiZSBhY2NvdW50ZWQgZm9yIGR1cmluZyBub3JtYWxpemF0aW9uIGFyZSAqKnNlcXVlbmNpbmcgZGVwdGgqKiBhbmQgKipnZW5lIGxlbmd0aCoqLiBDb21tb24gbm9ybWFsaXphdGlvbiBhcHByb2FjaGVzIChzdWNoIGFzIEZQS00sIFJQS00sIENQTSwgVFBNLCBldGMuKSBhY2NvdW50IGZvciBvbmUgb3IgYm90aCBvZiB0aGVzZSBmYWN0b3JzLgoKKiAqKlNlcXVlbmNpbmcgZGVwdGgqKiBub3JtYWxpemF0aW9uIGlzIG5lY2Vzc2FyeSB0byBhY2NvdW50IGZvciB0aGUgcHJvcG9ydGlvbiBvZiByZWFkcyBwZXIgZ2VuZSBleHBlY3RlZCBmb3IgbW9yZSBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZXMgKGxpa2UgaW4gcGluayBiZWxvdykgdmVyc3VzIGEgbGVzcyBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZSAobGlrZSBpbiBncmVlbiBiZWxvdykuCgohWypOb3RlIHRoYXQgZWFjaCBwaW5rIG9yIGdyZWVuIHJlY3RhbmdsZSByZXByZXNlbnRzIGFuIGFsaWduZWQgcmVhZCwgd2l0aCByZWFkcyBzcGFubmluZyBhbiBpbnRyb24gY29ubmVjdGVkIGJ5IGEgZGFzaGVkIGxpbmUuIEZpZ3VyZSBmcm9tIFtIQkMgIHRyYWluaW5nIG1hdGVyaWFsc10oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkqXSguL2ltYWdlcy9ub3JtYWxpemF0aW9uX21ldGhvZHNfZGVwdGgucG5nKXt3aWR0aD03NSV9CgoqICoqR2VuZSBsZW5ndGgqKiBub3JtYWxpemF0aW9uIG1heSBhbHNvIGJlIG5lY2Vzc2FyeSBpZiBjb21wYXJpbmcgYmV0d2VlbiBkaWZmZXJlbnQgZ2VuZXMsIHNpbmNlIGdlbmVzIG9mIGRpZmZlcmVudCBsZW5ndGhzIGhhdmUgZGlmZmVyZW50IHByb2JhYmlsaXRpZXMgb2YgZ2VuZXJhdGluZyBmcmFnbWVudHMgdGhhdCBlbmQgdXAgaW4gdGhlIGxpYnJhcnkuCgo+ICoqTm90ZSoqOiBUaGUgYWJvdmUgZmlndXJlIHdhcyBvcmlnaW5hbGx5IGZyb20gYSBbSEJDICB0dXRvcmlhbF0oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkgdGhhdCBhbHNvIGluY2x1ZGVzIGEgZGV0YWlsZWQgY29tcGFyaXNvbiBvZiBkaWZmZXJlbnQgbm9ybWFsaXphdGlvbiAoQ1BNLCBUUE0sIEZQS00pIGFwcHJvYWNoZXMgYW5kIHRoZWlyIGJlc3QgdXNlcy4KCioqQ2hlY2staW46IFF1ZXN0aW9ucyBhYm91dCBub3JtYWxpemF0aW9ucz8qKgoKIyMgREVTZXEyIG5vcm1hbGl6YXRpb25zCgpERVNlcTIgaGFzIGFuIFtpbnRlcm5hbCBub3JtYWxpemF0aW9uIHByb2Nlc3NdKGh0dHBzOi8vZ2Vub21lYmlvbG9neS5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2L2diLTIwMTAtMTEtMTAtcjEwNikgdGhhdCBhY2NvdW50cyBmb3IgKipSTkEgY29tcG9zaXRpb24qKi4gQSBmZXcgaGlnaGx5IGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcywgZGlmZmVyZW5jZXMgaW4gdGhlIG51bWJlciBvZiBnZW5lcyBleHByZXNzZWQgYmV0d2VlbiBzYW1wbGVzLCBvciBjb250YW1pbmF0aW9uIGFyZSBub3QgYWNjb3VudGVkIGZvciBieSBkZXB0aCBvciBnZW5lIGxlbmd0aCBub3JtYWxpemF0aW9uIG1ldGhvZHMuIEFjY291bnRpbmcgZm9yIFJOQSBjb21wb3NpdGlvbiBpcyBwYXJ0aWN1bGFybHkgaW1wb3J0YW50IGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNlcywgcmVnYXJkbGVzcyBvZiB0aGUgdG9vbCB1c2VkLgoKRm9yIGRhdGEgZXhwbG9yYXRpb24gYW5kIHZpc3VhbGl6YXRpb25zLCBpdCBpcyBoZWxwZnVsIHRvIGdlbmVyYXRlIGFuIG9iamVjdCBvZiBpbmRlcGVuZGVudGx5IG5vcm1hbGl6ZWQgY291bnRzLiBXZSB3aWxsIHVzZSB0aGUgW3Jsb2cgdHJhbnNmb3JtYXRpb25dKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNjb3VudC1kYXRhLXRyYW5zZm9ybWF0aW9ucykgZnJvbSBERVNlcTIgdGhhdCBhY2NvdW50cyBmb3Igc2VxdWVuY2luZyBkZXB0aCBmb3IgZWFjaCBzYW1wbGUgYW5kIFJOQSBjb21wb3NpdGlvbiBmb3IgdGhlIGRvd25zdHJlYW0gcXVhbGl0eSBjb250cm9sIHZpc3VhbGl6YXRpb25zLgoKVGhlIHJsb2cgdHJhbnNmb3JtYXRpb24gcHJvZHVjZXMgbG9nMiBzY2FsZWQgZGF0YSB0aGF0IGhhcyBhbHNvIGJlZW4gbm9ybWFsaXplZCB0byBvdmVyYWxsIGxpYnJhcnkgc2l6ZSBhcyB3ZWxsIGFzIHZhcmlhbmNlIGFjcm9zcyBnZW5lcyBhdCBkaWZmZXJlbnQgbWVhbiBleHByZXNzaW9uIGxldmVscy4gRm9yIGxhcmdlciBudW1iZXJzIG9mIHNhbXBsZXMsIHRoZXJlIGlzIGFuIGFsdGVybmF0aXZlIHRyYW5zZm9ybWF0aW9uIG1ldGhvZCwgW3ZzdF0oaHR0cDovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2NvdW50LWRhdGEtdHJhbnNmb3JtYXRpb25zKSB0aGF0IGNhbiBiZSB1c2VkIGluc3RlYWQgZm9yIGNvdW50IG5vcm1hbGl6YXRpb25zLgoKVGhlIGNvbW1hbmQgdG8gZ2VuZXJhdGUgdGhlIG5vcm1hbGl6ZWQgY291bnQgb2JqZWN0IGhhcyBhIGZldyBwYXJ0cywgaW5jbHVkaW5nIGBkZHNgIGFzIGFuIGlucHV0IGFuZCBwcm92aWRpbmcgYSB2YWx1ZSB0byB0aGUgb3B0aW9uIGBibGluZGAuIEZvciBvdXIgcHVycG9zZXMsIHdlIHNldCBgYmxpbmQgPSBUUlVFYCBiZWNhdXNlIHdlIHdhbnQgdG8gY29tcGFyZSBzYW1wbGVzIGluIGRvd25zdHJlYW0gUUMgcGxvdHMgaW4gYW4gdW5iaWFzZWQgbWFubmVyLgoKYGBge3IgQ291bnROb3JtfQpybGQgPSBybG9nKGRkcywgYmxpbmQgPSBUUlVFKQpgYGAKCk5vdGU6IFdlIG1pZ2h0IHNlZSBhIG1lc3NhZ2UgdGhhdCBvdXIgZGF0YSBkaWQgbm90IGZpdCB0aGUgZGVmYXVsdCAncGFyYW1ldHJpYycgZGlzcGVyc2lvbiBtb2RlbCBzbyBhIGxvY2FsIHJlZ3Jlc3Npb24gd2FzIHN1YnN0aXR1dGVkLiBJZiB3ZSBoYWQgbW9yZSB0aW1lLCB3ZSBtaWdodCBsb29rIGF0IGEgZGlzcGVyc2lvbiBwbG90IHdpdGggdGhlIGBwbG90RGlzcEVzdHMoZGRzKWAgZnVuY3Rpb24sIGJ1dCBhcyBbdGhpcyBiaW9jb25kdWN0b3IgdGhyZWFkXShodHRwczovL3N1cHBvcnQuYmlvY29uZHVjdG9yLm9yZy9wLzEwNzkzNy8pIGRpc2N1c3Nlcywgb3RoZXIgdmlzdWFsaXphdGlvbnMgb2Ygb3VyIGRhdGEgbWlnaHQgYmUgbW9yZSBoZWxwZnVsIGFuZC9vciBlYXNpZXIgdG8gaW50ZXJwcmV0LgoKTmV4dCwgd2UnbGwgbG9vayBhdCB0aGUgcmVzdWx0cyBvZiB0aGUgdHJhbnNmb3JtYXRpb24gYnkgZXh0cmFjdGluZyB0aGUgdmFsdWVzIHdpdGggdGhlIGBhc3NheSgpYCBmdW5jdGlvbi4KCmBgYHtyIENvdW50Tm9ybUNoZWNrfQpoZWFkKGFzc2F5KHJsZCksIDIpCmBgYAoKTG9va2luZyBhdCB0aGUgcmxkIHZhbHVlcywgd2UgY2FuIHNlZSB0aGF0IHRoZXkgYXJlIG5vdyBpbiBsb2cgc2NhbGUuIFNpbmNlIHdlIHNldCBgYmxpbmQ9VFJVRWAsIHRoZSB0cmFuc2Zvcm1hdGlvbiBpcyBibGluZCB0byB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIHdlIHNwZWNpZmllZCBpbiB0aGUgZGVzaWduIGZvcm11bGEuIFRoZSBub3JtYWxpemVkIGNvdW50cyBhcmUgaGVscGZ1bCBmb3IgdmlzdWFsaXphdGlvbiBtZXRob2RzIGR1cmluZyBleHByZXNzaW9uLWxldmVsIHF1YWxpdHkgYXNzZXNzbWVudCBidXQgKiphcmVuJ3QgdXNlZCBpbiB0aGUgbW9kZWwgZml0dGluZyoqLgoKIyBERVNlcTIgTW9kZWwgRml0dGluZwoKTmV4dCwgd2UnbGwgZml0IG91ciBzdGFuZGFyZCBtb2RlbCB1c2luZyB0aGUgYERFU2VxYCBmdW5jdGlvbiBhbmQgdGFrZSBhIGxvb2sgYXQgdGhlIG9iamVjdHMgd2UgZ2VuZXJhdGUuIFRoaXMgY29tbWFuZCBhcHBsaWVzIHRoZSBtb2RlbCB0byBvdXIgZGF0YSwgdXNpbmcgdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiBzdXBwbGllZCB3aGVuIGdlbmVyYXRpbmcgdGhlIGBkZHNgIG9iamVjdCBzbyBpdCB0YWtlcyBzb21lIHRpbWUgdG8gcnVuLgoKYGBge3IgRml0TW9kZWxTdGFuZGFyZCwgbWVzc2FnZT1GQUxTRX0KZGRzID0gREVTZXEoZGRzKQpkZHMKYGBgCgpOb3RpY2UgdGhhdCB0aGVyZSBpcyBub3cgbW9yZSBpbmZvcm1hdGlvbiBpbiB0aGUgYERFU2VxRGF0YVNldGAgb2JqZWN0IHRoYW4gdGhlcmUgd2FzIHByaW9yIHRvIG91ciBub3JtYWxpemF0aW9uLiBUaGVyZSBpcyBpbmZvcm1hdGlvbiBhYm91dCB0aGUgbW9kZWwgZml0IGFuZCBhYm91dCB0aGUgbGlicmFyeSBzaXplIG5vcm1hbGl6YXRpb24uIERFU2VxMiB3aWxsIHVzZSB0aGlzIGluZm9ybWF0aW9uIHdoZW4gd2UgcGVyZm9ybSB0aGUgdGVzdCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24uCgpUaGUgYERFU2VxKClgIGZ1bmN0aW9uIGlzIGFjdHVhbGx5IGRvaW5nIHRocmVlIHRoaW5ncyBhdXRvbWF0aWNhbGx5IGZvciB1cy4gSXQgY2FsY3VsYXRlczoKCjEuIFRoZSBzaXplIGZhY3RvcnMgdG8gbm9ybWFsaXplIGZvciBsaWJyYXJ5IHNpemUgd2l0aCBgZXN0aW1hdGVTaXplRmFjdG9ycyhkZHMpYCwKMi4gRGlzcGVyc2lvbiBlc3RpbWF0ZXMgdG8gc2hyaW5rIHRoZSBkaXNwZXJzaW9ucyB3aXRoIGBlc3RpbWF0ZURpc3BlcnNpb25zKGRkcylgLCBhbmQKMy4gVGhlIFdhbGQgdGVzdCBzdGF0aXN0aWNzIHdpdGggYG5iaW5vbVdhbGRUZXN0KGRkcylgLgoKVGhlIGByZXN1bHRzTmFtZXMoKWAgZnVuY3Rpb24gcmV0dXJucyB0aGUgbmFtZXMgb2YgdGhlIGVzdGltYXRlZCBlZmZlY3RzIG9mIHRoZSBtb2RlbC4KCmBgYHtyIEZpdE1vZGVsU3RhbmRhcmRDaGVjazEsIGV2YWw9VFJVRX0KcmVzdWx0c05hbWVzKGRkcykKYGBgCgpUaGUgcmVzdWx0cyBpbmNsdWRlIHRoZSBzaW5nbGUgY29tcGFyaXNvbiByZXByZXNlbnRpbmcgdGhlIHR3byBsZXZlbHMgb2YgYGNvbmRpdGlvbmAuIElmIHRoZXJlIHdlcmUgbW9yZSBsZXZlbHMgaW4gdGhlIGBjb25kaXRpb25gIGNvbHVtbiwgdGhlcmUgd291bGQgYmUgbW9yZSByZXN1bHRzIGxpc3RlZCBoZXJlIGJlY2F1c2UgREVTZXEyIHdvdWxkIGltcGxpY2l0bHkgY29tcGFyZSBhbGwgb3RoZXIgbGV2ZWxzIHRvIHRoZSByZWZlcmVuY2UgbGV2ZWwuCgoqKkNoZWNrcG9pbnQqKjogKklmIHlvdSBzZWUgdGhlIHNhbWUgcmVzdWx0cyB3aGVuIHlvdSBleGVjdXRlIGByZXN1bHRzTmFtZXMoZGRzKWAsIHBsZWFzZSBpbmRpY2F0ZSB3aXRoIHRoZSBncmVlbiAneWVzJyBidXR0b24uIE90aGVyd2lzZSwgcGxlYXNlIHVzZSB0aGUgcmVkICd4JyBidXR0b24gdG8gZ2V0IGhlbHAgYmVmb3JlIHRoZSBicmVhayoKCiMgT3B0aW9uYWwgY29udGVudAoKPGRldGFpbHM+CjxzdW1tYXJ5PipDbGljayBmb3IgZml0dGluZyBhIG1vZGVsIHRoYXQgaW5jbHVkZXMgYSBjb3ZhcmlhdGUqPC9zdW1tYXJ5PgoKSWYgeW91IGV4ZWN1dGVkIHRoZSBjb21tYW5kcyBpbiB0aGUgYWRkaXRpb25hbCBjb250ZW50IHNlY3Rpb24gZnJvbSBNb2R1bGUgMDcsIHlvdSBjYW4gZml0IGEgc2VwYXJhdGUgREVTZXEyIG1vZGVsIGZvciB0aGUgYmF0Y2ggZXhhbXBsZS4KCmBgYHtyIEZpdE1vZGVsQ292YXJpYXRlfQpkZHNfYmF0Y2ggPSBERVNlcShkZHNfYmF0Y2gpCnJlc3VsdHNOYW1lcyhkZHNfYmF0Y2gpCmBgYAoKSWYgeW91IHJ1biB0aHJvdWdoIHRoZSBvcHRpb25hbCBleGVyY2lzZXMsIHlvdSBjYW4gZXhwbG9yZSB0aGUgaW1wYWN0IG9mIGFkZGluZyBhIGNvdmFyaWF0ZSBieSBzdWJzdGl0dXRpbmcgYGRkc19wYXRpZW50YCBmb3IgYGRkc2AgYW5kIHJlLXJ1bm5pbmcgdGhvc2UgY29tbWFuZHMgc2luY2UgYm90aCBERVNlcTIgb2JqZWN0cyBoYXZlIHRoZWlyIGRhdGEgb3JnYW5pemVkIGluIHRoZSBzYW1lIHdheS4KPC9kZXRhaWxzPgoKPGJyPgoKIyBTdW1tYXJ5CgpJbiB0aGlzIHNlY3Rpb24sIHdlOgoKKiBMZWFybmVkIGFib3V0IGNvdW50IG5vcm1hbGl6YXRpb25zIGFuZCB1c2VzCiogR2VuZXJhdGVkIGEgbm9ybWFsaXplZCBjb3VudCB0YWJsZQoqIEZpdCB0d28gREVTZXEyIG1vZGVscyBmb3Igb3VyIGRhdGEKKiBTYXcgdGhlIGltcGFjdCBvZiBpbmNsdWRpbmcgYSBjb3ZhcmlhdGUgaW4gb3VyIG1vZGVsCgotLS0KCiMgU291cmNlcwoKVHJhaW5pbmcgcmVzb3VyY2VzIHVzZWQgdG8gZGV2ZWxvcCBtYXRlcmlhbHMKCiogSEJDIERHRSBzZXR1cDogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDFfREdFX3NldHVwX2FuZF9vdmVydmlldy5odG1sCiogSEJDIENvdW50IE5vcm1hbGl6YXRpb246IGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzAyX0RHRV9jb3VudF9ub3JtYWxpemF0aW9uLmh0bWwKKiBERVNlcTIgc3RhbmRhcmQgdmlnbmV0dGU6IGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbAoqIERFU2VxMiBiZWdpbm5lcnMgdmlnbmV0dGU6IGh0dHBzOi8vYmlvYy5pc20uYWMuanAvcGFja2FnZXMvMi4xNC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvYmVnaW5uZXIucGRmCiogQmlvY29uZHVjdG9yIFJOQS1zZXEgV29ya2Zsb3dzOiBodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL2hlbHAvY291cnNlLW1hdGVyaWFscy8yMDE1L0xlYXJuQmlvY29uZHVjdG9yRmViMjAxNS9CMDIuMV9STkFTZXEuaHRtbAoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQpgYGAKClRoZXNlIG1hdGVyaWFscyBoYXZlIGJlZW4gYWRhcHRlZCBhbmQgZXh0ZW5kZWQgZnJvbSBtYXRlcmlhbHMgbGlzdGVkIGFib3ZlLiBUaGVzZSBhcmUgb3BlbiBhY2Nlc3MgbWF0ZXJpYWxzIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgW0NyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gbGljZW5zZSAoQ0MgQlkgNC4wKV0oaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyksIHdoaWNoIHBlcm1pdHMgdW5yZXN0cmljdGVkIHVzZSwgZGlzdHJpYnV0aW9uLCBhbmQgcmVwcm9kdWN0aW9uIGluIGFueSBtZWRpdW0sIHByb3ZpZGVkIHRoZSBvcmlnaW5hbCBhdXRob3IgYW5kIHNvdXJjZSBhcmUgY3JlZGl0ZWQuCg==