Objectives

  • Discuss count normalizations
  • Execute model fitting for differential expression comparisons

Differential Expression Workflow

Here we will proceed with count normalizations and fit our DESeq2 model.


Count normalizations

Since counts of mapped reads for each gene is proportional to the expression of RNA in addition to many “uninteresting” other factors, normalization is the process of scaling raw count values to account for the “uninteresting” factors and ensure expression levels are more comparable.

Normalization goals

Two common factors that need to be accounted for during normalization are sequencing depth and gene length. Common normalization approaches (such as FPKM, RPKM, CPM, TPM, etc.) account for one or both of these factors.

  • Sequencing depth normalization is necessary to account for the proportion of reads per gene expected for more deeply sequenced samples (like in pink below) versus a less deeply sequenced sample (like in green below).

Note that each pink or green rectangle represents an aligned read, with reads spanning an intron connected by a dashed line. Figure from HBC training materials

  • Gene length normalization may also be necessary if comparing between different genes, since genes of different lengths have different probabilities of generating fragments that end up in the library.

Note: The above figure was originally from a HBC tutorial that also includes a detailed comparison of different normalization (CPM, TPM, FPKM) approaches and their best uses.

Check-in: Questions about normalizations?

DESeq2 normalizations

DESeq2 has an internal normalization process that accounts for RNA composition. A few highly differentially expressed genes, differences in the number of genes expressed between samples, or contamination are not accounted for by depth or gene length normalization methods. Accounting for RNA composition is particularly important for differential expression analyses, regardless of the tool used.

For data exploration and visualizations, it is helpful to generate an object of independently normalized counts. We will use the rlog transformation from DESeq2 that accounts for sequencing depth for each sample and RNA composition for the downstream quality control visualizations.

The rlog transformation produces log2 scaled data that has also been normalized to overall library size as well as variance across genes at different mean expression levels. For larger numbers of samples, there is an alternative transformation method, vst that can be used instead for count normalizations.

The command to generate the normalized count object has a few parts, including dds as an input and providing a value to the option blind. For our purposes, we set blind = TRUE because we want to compare samples in downstream QC plots in an unbiased manner.

rld = rlog(dds, blind = TRUE)

Next, we’ll look at the results of the transformation by extracting the values with the assay() function.

head(assay(rld), 2)
                   sample_A sample_B sample_C sample_D sample_E sample_F
ENSMUSG00000000001 10.51433  10.3622 10.41675 10.84821 10.40736 10.58012
ENSMUSG00000000003  0.00000   0.0000  0.00000  0.00000  0.00000  0.00000

Looking at the rld values, we can see that they are now in log scale. Since we set blind=TRUE, the transformation is blind to the sample information we specified in the design formula. The normalized counts are helpful for visualization methods during expression-level quality assessment but aren’t used in the model fitting.

We’ll come back to these normalized data, but first let’s write out both the raw and normalized count tables to file.

Output count tables

To output the raw counts, we will need to use the counts function to access the count table from within its larger DESeqDataSet object.

write.csv(counts(dds, normalized = FALSE), file="outputs/tables/DESeq2_raw_counts.csv")

Then we’ll output the rlog count table, using the assay function to access the normalized count table from within its larger DESeqDataSet object.

write.csv(assay(rld), file="outputs/tables/DESeq2_rlog_normalized_counts.csv")

DESeq2 Model Fitting

Next, we’ll fit our standard model using the DESeq function and take a look at the objects we generate. This command applies the model to our data, using the sample information supplied when generating the initial dds object so can take some time to run.

dds_fitted = DESeq(dds_filtered)
dds_fitted
class: DESeqDataSet 
dim: 16249 6 
metadata(1): version
assays(4): counts mu H cooks
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ... ENSMUSG00000118651
  ENSMUSG00000118653
rowData names(22): baseMean baseVar ... deviance maxCooks
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(3): genotype condition sizeFactor
Dispersion models and possible warning messages

Depending on the data set you are analyzing, you may see a warning that the default ‘parametric’ dispersion model so a local regression was substituted. When seeing this warning, we recommend looking at a dispersion plot with the plotDispEsts(dds) function, but as this bioconductor thread discusses, other visualizations of our data might be more helpful and/or easier to interpret why the data doesn’t fit the default parametric model.


Notice that there is now more information in the DESeqDataSet object than there was prior to our normalization. There is information about the model fit and about the library size normalization. DESeq2 will use this information when we perform the test for differential expression.

The DESeq() function is actually doing three things automatically for us. It calculates:

  1. The size factors to normalize for library size with estimateSizeFactors(dds_filtered),
  2. Dispersion estimates to shrink the dispersions with estimateDispersions(dds_filtered), and
  3. The Wald test statistics with nbinomWaldTest(dds_filtered).

The resultsNames() function returns the names of the estimated effects of the model.

resultsNames(dds_fitted)
[1] "Intercept"               "condition_minus_vs_plus"

The results include the single comparison representing the two levels of condition. If there were more levels in the condition column, there would be more results listed here because DESeq2 would implicitly compare all other levels to the reference level.

Checkpoint: If you see the same results when you execute resultsNames(dds_filtered), please indicate with the green ‘yes’ button. Otherwise, please use the red ‘x’ button to get help before the break

Save fitted model and data in Robj

It can be useful to save key R objects to file as we proceed through our analysis - before we do that, let’s look at the documentation for the save function to see if it does what we want.

?save

As we can see from the documentation the function save writes an “external representation” of R objects that can be read back from the file at a later date by using the function load or attach in most cases. We’ll proceed with saving our dds_fitted object, creating a subdirectory first.

dir.create("outputs/Robjs", recursive=TRUE)
Warning in dir.create("outputs/Robjs", recursive = TRUE): 'outputs/Robjs' already exists
save(dds_fitted,
          file="outputs/Robjs/dds_fitted.Robj")

Optional content

Click to fit a model that includes a covariate

If you executed the commands in the additional content section from Module 07, you can fit a separate DESeq2 model for the batch example.

dds_batch_fitted = DESeq(dds_batch)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resultsNames(dds_batch_fitted)
[1] "Intercept"               "batch_Day2_vs_Day1"      "batch_Day3_vs_Day1"     
[4] "condition_minus_vs_plus"


Summary

In this section, we:

  • Learned about count normalizations and uses
  • Generated a normalized count table
  • Fit two DESeq2 models for our data
  • (Optional) - saw the impact of including a covariate in our model
  • Wrote intermediate data to file

Sources

Training resources used to develop materials


These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

LS0tCnRpdGxlOiAiTW9kdWxlIDA4OiBERSBOb3JtYWxpemF0aW9uIGFuZCBNb2RlbGluZyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA4LSIpCmBgYAoKPiAjIE9iamVjdGl2ZXMgey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiAqIERpc2N1c3MgY291bnQgbm9ybWFsaXphdGlvbnMKPiAqIEV4ZWN1dGUgbW9kZWwgZml0dGluZyBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIHByb2NlZWQgd2l0aCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgZml0IG91ciBERVNlcTIgbW9kZWwuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERVNlcTJERS5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBDb3VudCBub3JtYWxpemF0aW9ucwoKU2luY2UgY291bnRzIG9mIG1hcHBlZCByZWFkcyBmb3IgZWFjaCBnZW5lIGlzIHByb3BvcnRpb25hbCB0byB0aGUgZXhwcmVzc2lvbiBvZiBSTkEgaW4gYWRkaXRpb24gdG8gbWFueSDigJx1bmludGVyZXN0aW5n4oCdIG90aGVyIGZhY3RvcnMsIG5vcm1hbGl6YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2Ygc2NhbGluZyByYXcgY291bnQgdmFsdWVzIHRvIGFjY291bnQgZm9yIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIGZhY3RvcnMgYW5kIGVuc3VyZSBleHByZXNzaW9uIGxldmVscyBhcmUgbW9yZSBjb21wYXJhYmxlLgoKIyMgTm9ybWFsaXphdGlvbiBnb2FscwoKVHdvIGNvbW1vbiBmYWN0b3JzIHRoYXQgbmVlZCB0byBiZSBhY2NvdW50ZWQgZm9yIGR1cmluZyBub3JtYWxpemF0aW9uIGFyZSAqKnNlcXVlbmNpbmcgZGVwdGgqKiBhbmQgKipnZW5lIGxlbmd0aCoqLiBDb21tb24gbm9ybWFsaXphdGlvbiBhcHByb2FjaGVzIChzdWNoIGFzIEZQS00sIFJQS00sIENQTSwgVFBNLCBldGMuKSBhY2NvdW50IGZvciBvbmUgb3IgYm90aCBvZiB0aGVzZSBmYWN0b3JzLgoKKiAqKlNlcXVlbmNpbmcgZGVwdGgqKiBub3JtYWxpemF0aW9uIGlzIG5lY2Vzc2FyeSB0byBhY2NvdW50IGZvciB0aGUgcHJvcG9ydGlvbiBvZiByZWFkcyBwZXIgZ2VuZSBleHBlY3RlZCBmb3IgbW9yZSBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZXMgKGxpa2UgaW4gcGluayBiZWxvdykgdmVyc3VzIGEgbGVzcyBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZSAobGlrZSBpbiBncmVlbiBiZWxvdykuCgohWypOb3RlIHRoYXQgZWFjaCBwaW5rIG9yIGdyZWVuIHJlY3RhbmdsZSByZXByZXNlbnRzIGFuIGFsaWduZWQgcmVhZCwgd2l0aCByZWFkcyBzcGFubmluZyBhbiBpbnRyb24gY29ubmVjdGVkIGJ5IGEgZGFzaGVkIGxpbmUuIEZpZ3VyZSBmcm9tIFtIQkMgIHRyYWluaW5nIG1hdGVyaWFsc10oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkqXSguL2ltYWdlcy9ub3JtYWxpemF0aW9uX21ldGhvZHNfZGVwdGgucG5nKXt3aWR0aD03NSV9CgoqICoqR2VuZSBsZW5ndGgqKiBub3JtYWxpemF0aW9uIG1heSBhbHNvIGJlIG5lY2Vzc2FyeSBpZiBjb21wYXJpbmcgYmV0d2VlbiBkaWZmZXJlbnQgZ2VuZXMsIHNpbmNlIGdlbmVzIG9mIGRpZmZlcmVudCBsZW5ndGhzIGhhdmUgZGlmZmVyZW50IHByb2JhYmlsaXRpZXMgb2YgZ2VuZXJhdGluZyBmcmFnbWVudHMgdGhhdCBlbmQgdXAgaW4gdGhlIGxpYnJhcnkuCgo+ICoqTm90ZSoqOiBUaGUgYWJvdmUgZmlndXJlIHdhcyBvcmlnaW5hbGx5IGZyb20gYSBbSEJDICB0dXRvcmlhbF0oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkgdGhhdCBhbHNvIGluY2x1ZGVzIGEgZGV0YWlsZWQgY29tcGFyaXNvbiBvZiBkaWZmZXJlbnQgbm9ybWFsaXphdGlvbiAoQ1BNLCBUUE0sIEZQS00pIGFwcHJvYWNoZXMgYW5kIHRoZWlyIGJlc3QgdXNlcy4KCioqQ2hlY2staW46IFF1ZXN0aW9ucyBhYm91dCBub3JtYWxpemF0aW9ucz8qKgoKIyMgREVTZXEyIG5vcm1hbGl6YXRpb25zCgpERVNlcTIgaGFzIGFuIFtpbnRlcm5hbCBub3JtYWxpemF0aW9uIHByb2Nlc3NdKGh0dHBzOi8vZ2Vub21lYmlvbG9neS5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2L2diLTIwMTAtMTEtMTAtcjEwNikgdGhhdCBhY2NvdW50cyBmb3IgKipSTkEgY29tcG9zaXRpb24qKi4gQSBmZXcgaGlnaGx5IGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcywgZGlmZmVyZW5jZXMgaW4gdGhlIG51bWJlciBvZiBnZW5lcyBleHByZXNzZWQgYmV0d2VlbiBzYW1wbGVzLCBvciBjb250YW1pbmF0aW9uIGFyZSBub3QgYWNjb3VudGVkIGZvciBieSBkZXB0aCBvciBnZW5lIGxlbmd0aCBub3JtYWxpemF0aW9uIG1ldGhvZHMuIEFjY291bnRpbmcgZm9yIFJOQSBjb21wb3NpdGlvbiBpcyBwYXJ0aWN1bGFybHkgaW1wb3J0YW50IGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNlcywgcmVnYXJkbGVzcyBvZiB0aGUgdG9vbCB1c2VkLgoKRm9yIGRhdGEgZXhwbG9yYXRpb24gYW5kIHZpc3VhbGl6YXRpb25zLCBpdCBpcyBoZWxwZnVsIHRvIGdlbmVyYXRlIGFuIG9iamVjdCBvZiBpbmRlcGVuZGVudGx5IG5vcm1hbGl6ZWQgY291bnRzLiBXZSB3aWxsIHVzZSB0aGUgW3Jsb2cgdHJhbnNmb3JtYXRpb25dKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNjb3VudC1kYXRhLXRyYW5zZm9ybWF0aW9ucykgZnJvbSBERVNlcTIgdGhhdCBhY2NvdW50cyBmb3Igc2VxdWVuY2luZyBkZXB0aCBmb3IgZWFjaCBzYW1wbGUgYW5kIFJOQSBjb21wb3NpdGlvbiBmb3IgdGhlIGRvd25zdHJlYW0gcXVhbGl0eSBjb250cm9sIHZpc3VhbGl6YXRpb25zLgoKVGhlIHJsb2cgdHJhbnNmb3JtYXRpb24gcHJvZHVjZXMgbG9nMiBzY2FsZWQgZGF0YSB0aGF0IGhhcyBhbHNvIGJlZW4gbm9ybWFsaXplZCB0byBvdmVyYWxsIGxpYnJhcnkgc2l6ZSBhcyB3ZWxsIGFzIHZhcmlhbmNlIGFjcm9zcyBnZW5lcyBhdCBkaWZmZXJlbnQgbWVhbiBleHByZXNzaW9uIGxldmVscy4gRm9yIGxhcmdlciBudW1iZXJzIG9mIHNhbXBsZXMsIHRoZXJlIGlzIGFuIGFsdGVybmF0aXZlIHRyYW5zZm9ybWF0aW9uIG1ldGhvZCwgW3ZzdF0oaHR0cDovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2NvdW50LWRhdGEtdHJhbnNmb3JtYXRpb25zKSB0aGF0IGNhbiBiZSB1c2VkIGluc3RlYWQgZm9yIGNvdW50IG5vcm1hbGl6YXRpb25zLgoKVGhlIGNvbW1hbmQgdG8gZ2VuZXJhdGUgdGhlIG5vcm1hbGl6ZWQgY291bnQgb2JqZWN0IGhhcyBhIGZldyBwYXJ0cywgaW5jbHVkaW5nIGBkZHNgIGFzIGFuIGlucHV0IGFuZCBwcm92aWRpbmcgYSB2YWx1ZSB0byB0aGUgb3B0aW9uIGBibGluZGAuIEZvciBvdXIgcHVycG9zZXMsIHdlIHNldCBgYmxpbmQgPSBUUlVFYCBiZWNhdXNlIHdlIHdhbnQgdG8gY29tcGFyZSBzYW1wbGVzIGluIGRvd25zdHJlYW0gUUMgcGxvdHMgaW4gYW4gdW5iaWFzZWQgbWFubmVyLgoKYGBge3IgQ291bnROb3JtfQpybGQgPSBybG9nKGRkcywgYmxpbmQgPSBUUlVFKQpgYGAKCk5leHQsIHdlJ2xsIGxvb2sgYXQgdGhlIHJlc3VsdHMgb2YgdGhlIHRyYW5zZm9ybWF0aW9uIGJ5IGV4dHJhY3RpbmcgdGhlIHZhbHVlcyB3aXRoIHRoZSBgYXNzYXkoKWAgZnVuY3Rpb24uCgpgYGB7ciBDb3VudE5vcm1DaGVja30KaGVhZChhc3NheShybGQpLCAyKQpgYGAKCkxvb2tpbmcgYXQgdGhlIHJsZCB2YWx1ZXMsIHdlIGNhbiBzZWUgdGhhdCB0aGV5IGFyZSBub3cgaW4gbG9nIHNjYWxlLiBTaW5jZSB3ZSBzZXQgYGJsaW5kPVRSVUVgLCB0aGUgdHJhbnNmb3JtYXRpb24gaXMgYmxpbmQgdG8gdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiB3ZSBzcGVjaWZpZWQgaW4gdGhlIGRlc2lnbiBmb3JtdWxhLiBUaGUgbm9ybWFsaXplZCBjb3VudHMgYXJlIGhlbHBmdWwgZm9yIHZpc3VhbGl6YXRpb24gbWV0aG9kcyBkdXJpbmcgZXhwcmVzc2lvbi1sZXZlbCBxdWFsaXR5IGFzc2Vzc21lbnQgYnV0ICoqYXJlbid0IHVzZWQgaW4gdGhlIG1vZGVsIGZpdHRpbmcqKi4KCldlJ2xsIGNvbWUgYmFjayB0byB0aGVzZSBub3JtYWxpemVkIGRhdGEsIGJ1dCBmaXJzdCBsZXQncyB3cml0ZSBvdXQgYm90aCB0aGUgcmF3IGFuZCBub3JtYWxpemVkIGNvdW50IHRhYmxlcyB0byBmaWxlLgoKIyMjIE91dHB1dCBjb3VudCB0YWJsZXMKClRvIG91dHB1dCB0aGUgcmF3IGNvdW50cywgd2Ugd2lsbCBuZWVkIHRvIHVzZSB0aGUgYGNvdW50c2AgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBjb3VudCB0YWJsZSBmcm9tIHdpdGhpbiBpdHMgbGFyZ2VyIGBERVNlcURhdGFTZXRgIG9iamVjdC4KCmBgYHtyIE91dHB1dENvdW50c1JhdywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoY291bnRzKGRkcywgbm9ybWFsaXplZCA9IEZBTFNFKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jhd19jb3VudHMuY3N2IikKYGBgCgpUaGVuIHdlJ2xsIG91dHB1dCB0aGUgcmxvZyBjb3VudCB0YWJsZSwgdXNpbmcgdGhlIGBhc3NheWAgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBub3JtYWxpemVkIGNvdW50IHRhYmxlIGZyb20gd2l0aGluIGl0cyBsYXJnZXIgYERFU2VxRGF0YVNldGAgb2JqZWN0LgoKYGBge3IgT3V0cHV0Q291bnRzUmxvZywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoYXNzYXkocmxkKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jsb2dfbm9ybWFsaXplZF9jb3VudHMuY3N2IikKYGBgCgoKIyBERVNlcTIgTW9kZWwgRml0dGluZwoKTmV4dCwgd2UnbGwgZml0IG91ciBzdGFuZGFyZCBtb2RlbCB1c2luZyB0aGUgYERFU2VxYCBmdW5jdGlvbiBhbmQgdGFrZSBhIGxvb2sgYXQgdGhlIG9iamVjdHMgd2UgZ2VuZXJhdGUuIFRoaXMgY29tbWFuZCBhcHBsaWVzIHRoZSBtb2RlbCB0byBvdXIgZGF0YSwgdXNpbmcgdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiBzdXBwbGllZCB3aGVuIGdlbmVyYXRpbmcgdGhlIGluaXRpYWwgYGRkc2Agb2JqZWN0IHNvIGNhbiB0YWtlIHNvbWUgdGltZSB0byBydW4uCgpgYGB7ciBGaXRNb2RlbFN0YW5kYXJkLCBtZXNzYWdlPUZBTFNFfQpkZHNfZml0dGVkID0gREVTZXEoZGRzX2ZpbHRlcmVkKQpkZHNfZml0dGVkCmBgYAoKPGRldGFpbHM+CiAgICA8c3VtbWFyeT4qRGlzcGVyc2lvbiBtb2RlbHMgYW5kIHBvc3NpYmxlIHdhcm5pbmcgbWVzc2FnZXMqPC9zdW1tYXJ5PgoKICAgIERlcGVuZGluZyBvbiB0aGUgZGF0YSBzZXQgeW91IGFyZSBhbmFseXppbmcsIHlvdSBtYXkgc2VlIGEgd2FybmluZyB0aGF0IHRoZSBkZWZhdWx0ICdwYXJhbWV0cmljJyBkaXNwZXJzaW9uIG1vZGVsIHNvIGEgbG9jYWwgcmVncmVzc2lvbiB3YXMgc3Vic3RpdHV0ZWQuIFdoZW4gc2VlaW5nIHRoaXMgd2FybmluZywgd2UgcmVjb21tZW5kIGxvb2tpbmcgYXQgYSBkaXNwZXJzaW9uIHBsb3Qgd2l0aCB0aGUgYHBsb3REaXNwRXN0cyhkZHMpYCBmdW5jdGlvbiwgYnV0IGFzIFt0aGlzIGJpb2NvbmR1Y3RvciB0aHJlYWRdKGh0dHBzOi8vc3VwcG9ydC5iaW9jb25kdWN0b3Iub3JnL3AvMTA3OTM3LykgZGlzY3Vzc2VzLCBvdGhlciB2aXN1YWxpemF0aW9ucyBvZiBvdXIgZGF0YSBtaWdodCBiZSBtb3JlIGhlbHBmdWwgYW5kL29yIGVhc2llciB0byBpbnRlcnByZXQgd2h5IHRoZSBkYXRhIGRvZXNuJ3QgZml0IHRoZSBkZWZhdWx0IHBhcmFtZXRyaWMgbW9kZWwuCgo8L2RldGFpbHM+Cjxicj4KCgpOb3RpY2UgdGhhdCB0aGVyZSBpcyBub3cgbW9yZSBpbmZvcm1hdGlvbiBpbiB0aGUgYERFU2VxRGF0YVNldGAgb2JqZWN0IHRoYW4gdGhlcmUgd2FzIHByaW9yIHRvIG91ciBub3JtYWxpemF0aW9uLiBUaGVyZSBpcyBpbmZvcm1hdGlvbiBhYm91dCB0aGUgbW9kZWwgZml0IGFuZCBhYm91dCB0aGUgbGlicmFyeSBzaXplIG5vcm1hbGl6YXRpb24uIERFU2VxMiB3aWxsIHVzZSB0aGlzIGluZm9ybWF0aW9uIHdoZW4gd2UgcGVyZm9ybSB0aGUgdGVzdCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24uCgpUaGUgYERFU2VxKClgIGZ1bmN0aW9uIGlzIGFjdHVhbGx5IGRvaW5nIHRocmVlIHRoaW5ncyBhdXRvbWF0aWNhbGx5IGZvciB1cy4gSXQgY2FsY3VsYXRlczoKCjEuIFRoZSBzaXplIGZhY3RvcnMgdG8gbm9ybWFsaXplIGZvciBsaWJyYXJ5IHNpemUgd2l0aCBgZXN0aW1hdGVTaXplRmFjdG9ycyhkZHNfZmlsdGVyZWQpYCwKMi4gRGlzcGVyc2lvbiBlc3RpbWF0ZXMgdG8gc2hyaW5rIHRoZSBkaXNwZXJzaW9ucyB3aXRoIGBlc3RpbWF0ZURpc3BlcnNpb25zKGRkc19maWx0ZXJlZClgLCBhbmQKMy4gVGhlIFdhbGQgdGVzdCBzdGF0aXN0aWNzIHdpdGggYG5iaW5vbVdhbGRUZXN0KGRkc19maWx0ZXJlZClgLgoKVGhlIGByZXN1bHRzTmFtZXMoKWAgZnVuY3Rpb24gcmV0dXJucyB0aGUgbmFtZXMgb2YgdGhlIGVzdGltYXRlZCBlZmZlY3RzIG9mIHRoZSBtb2RlbC4KCmBgYHtyIEZpdE1vZGVsU3RhbmRhcmRDaGVjazEsIGV2YWw9VFJVRX0KcmVzdWx0c05hbWVzKGRkc19maXR0ZWQpCmBgYAoKVGhlIHJlc3VsdHMgaW5jbHVkZSB0aGUgc2luZ2xlIGNvbXBhcmlzb24gcmVwcmVzZW50aW5nIHRoZSB0d28gbGV2ZWxzIG9mIGBjb25kaXRpb25gLiBJZiB0aGVyZSB3ZXJlIG1vcmUgbGV2ZWxzIGluIHRoZSBgY29uZGl0aW9uYCBjb2x1bW4sIHRoZXJlIHdvdWxkIGJlIG1vcmUgcmVzdWx0cyBsaXN0ZWQgaGVyZSBiZWNhdXNlIERFU2VxMiB3b3VsZCBpbXBsaWNpdGx5IGNvbXBhcmUgYWxsIG90aGVyIGxldmVscyB0byB0aGUgcmVmZXJlbmNlIGxldmVsLgoKCioqQ2hlY2twb2ludCoqOiAqSWYgeW91IHNlZSB0aGUgc2FtZSByZXN1bHRzIHdoZW4geW91IGV4ZWN1dGUgYHJlc3VsdHNOYW1lcyhkZHNfZmlsdGVyZWQpYCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICd5ZXMnIGJ1dHRvbi4gT3RoZXJ3aXNlLCBwbGVhc2UgdXNlIHRoZSByZWQgJ3gnIGJ1dHRvbiB0byBnZXQgaGVscCBiZWZvcmUgdGhlIGJyZWFrKgoKIyMgU2F2ZSBmaXR0ZWQgbW9kZWwgYW5kIGRhdGEgaW4gUm9iagoKSXQgY2FuIGJlIHVzZWZ1bCB0byBzYXZlIGtleSBSIG9iamVjdHMgdG8gZmlsZSBhcyB3ZSBwcm9jZWVkIHRocm91Z2ggb3VyIGFuYWx5c2lzIC0gYmVmb3JlIHdlIGRvIHRoYXQsIGxldCdzIGxvb2sgYXQgdGhlIGRvY3VtZW50YXRpb24gZm9yIHRoZSBgc2F2ZWAgZnVuY3Rpb24gdG8gc2VlIGlmIGl0IGRvZXMgd2hhdCB3ZSB3YW50LgoKYGBge3IgU2F2ZURvY3VtZW50YXRpb259Cj9zYXZlCmBgYAoKQXMgd2UgY2FuIHNlZSBmcm9tIHRoZSBkb2N1bWVudGF0aW9uIHRoZSBmdW5jdGlvbiBgc2F2ZWAgd3JpdGVzIGFuICJleHRlcm5hbCByZXByZXNlbnRhdGlvbiIgb2YgUiBvYmplY3RzIHRoYXQgY2FuIGJlIHJlYWQgYmFjayBmcm9tIHRoZSBmaWxlIGF0IGEgbGF0ZXIgZGF0ZSBieSB1c2luZyB0aGUgZnVuY3Rpb24gYGxvYWRgIG9yIGBhdHRhY2hgIGluIG1vc3QgY2FzZXMuIFdlJ2xsIHByb2NlZWQgd2l0aCBzYXZpbmcgb3VyIGBkZHNfZml0dGVkYCBvYmplY3QsIGNyZWF0aW5nIGEgc3ViZGlyZWN0b3J5IGZpcnN0LgoKYGBge3IgV3JpdGVGaXRNb2RlbE9iamVjdH0KZGlyLmNyZWF0ZSgib3V0cHV0cy9Sb2JqcyIsIHJlY3Vyc2l2ZT1UUlVFKQpzYXZlKGRkc19maXR0ZWQsCiAgICAgICAgICBmaWxlPSJvdXRwdXRzL1JvYmpzL2Rkc19maXR0ZWQuUm9iaiIpCmBgYAoKCiMgT3B0aW9uYWwgY29udGVudAoKPGRldGFpbHM+CjxzdW1tYXJ5PipDbGljayB0byBmaXQgYSBtb2RlbCB0aGF0IGluY2x1ZGVzIGEgY292YXJpYXRlKjwvc3VtbWFyeT4KCklmIHlvdSBleGVjdXRlZCB0aGUgY29tbWFuZHMgaW4gdGhlIGFkZGl0aW9uYWwgY29udGVudCBzZWN0aW9uIGZyb20gTW9kdWxlIDA3LCB5b3UgY2FuIGZpdCBhIHNlcGFyYXRlIERFU2VxMiBtb2RlbCBmb3IgdGhlIGJhdGNoIGV4YW1wbGUuCgpgYGB7ciBGaXRNb2RlbENvdmFyaWF0ZX0KZGRzX2JhdGNoX2ZpdHRlZCA9IERFU2VxKGRkc19iYXRjaCkKcmVzdWx0c05hbWVzKGRkc19iYXRjaF9maXR0ZWQpCmBgYAoKPC9kZXRhaWxzPgoKPGJyPgoKIyBTdW1tYXJ5CgpJbiB0aGlzIHNlY3Rpb24sIHdlOgoKKiBMZWFybmVkIGFib3V0IGNvdW50IG5vcm1hbGl6YXRpb25zIGFuZCB1c2VzCiogR2VuZXJhdGVkIGEgbm9ybWFsaXplZCBjb3VudCB0YWJsZQoqIEZpdCB0d28gREVTZXEyIG1vZGVscyBmb3Igb3VyIGRhdGEKKiAoT3B0aW9uYWwpIC0gc2F3IHRoZSBpbXBhY3Qgb2YgaW5jbHVkaW5nIGEgY292YXJpYXRlIGluIG91ciBtb2RlbAoqIFdyb3RlIGludGVybWVkaWF0ZSBkYXRhIHRvIGZpbGUKCi0tLQoKIyBTb3VyY2VzCgpUcmFpbmluZyByZXNvdXJjZXMgdXNlZCB0byBkZXZlbG9wIG1hdGVyaWFscwoKKiBIQkMgREdFIHNldHVwOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wMV9ER0Vfc2V0dXBfYW5kX292ZXJ2aWV3Lmh0bWwKKiBIQkMgQ291bnQgTm9ybWFsaXphdGlvbjogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbAoqIERFU2VxMiBzdGFuZGFyZCB2aWduZXR0ZTogaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sCiogREVTZXEyIGJlZ2lubmVycyB2aWduZXR0ZTogaHR0cHM6Ly9iaW9jLmlzbS5hYy5qcC9wYWNrYWdlcy8yLjE0L2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9iZWdpbm5lci5wZGYKKiBCaW9jb25kdWN0b3IgUk5BLXNlcSBXb3JrZmxvd3M6IGh0dHBzOi8vd3d3LmJpb2NvbmR1Y3Rvci5vcmcvaGVscC9jb3Vyc2UtbWF0ZXJpYWxzLzIwMTUvTGVhcm5CaW9jb25kdWN0b3JGZWIyMDE1L0IwMi4xX1JOQVNlcS5odG1sCgpgYGB7ciBXcml0ZU91dC5SRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQojIEhpZGRlbiBjb2RlIGJsb2NrIHRvIHdyaXRlIG91dCBkYXRhIGZvciBrbml0dGluZwojIHNhdmUuaW1hZ2UoZmlsZSA9ICJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKLS0tCgpUaGVzZSBtYXRlcmlhbHMgaGF2ZSBiZWVuIGFkYXB0ZWQgYW5kIGV4dGVuZGVkIGZyb20gbWF0ZXJpYWxzIGxpc3RlZCBhYm92ZS4gVGhlc2UgYXJlIG9wZW4gYWNjZXNzIG1hdGVyaWFscyBkaXN0cmlidXRlZCB1bmRlciB0aGUgdGVybXMgb2YgdGhlIFtDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uIGxpY2Vuc2UgKENDIEJZIDQuMCldKGh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LzQuMC8pLCB3aGljaCBwZXJtaXRzIHVucmVzdHJpY3RlZCB1c2UsIGRpc3RyaWJ1dGlvbiwgYW5kIHJlcHJvZHVjdGlvbiBpbiBhbnkgbWVkaXVtLCBwcm92aWRlZCB0aGUgb3JpZ2luYWwgYXV0aG9yIGFuZCBzb3VyY2UgYXJlIGNyZWRpdGVkLgo=