In this module, we will learn:

  • Different count normalizations and their uses
  • How to fit a model for differential expression comparisons


Differential Expression Workflow

Here we will proceed with count normalizations and fit our DESeq2 model.


Count normalizations

Since counts of mapped reads for each gene is proportional to the expression of RNA in addition to many “uninteresting” other factors, normalization is the process of scaling raw count values to account for the “uninteresting” factors and ensure expression levels are more comparable.

Normalization goals

Two common factors that need to be accounted for during normalization are sequencing depth and gene length. Common normalization approaches (such as FPKM, RPKM, CPM, TPM, etc.) account for one or both of these factors.

  • Sequencing depth normalization is necessary to account for the proportion of reads per gene expected for more deeply sequenced samples (like in pink below) versus a less deeply sequenced sample (like in green below).

Note that each pink or green rectangle represents an aligned read, with reads spanning an intron connected by a dashed line. Figure from HBC training materials

  • Gene length normalization may also be necessary if comparing between different genes, since genes of different lengths have different probabilities of generating fragments that end up in the library.

Note: The above figure was originally from a HBC tutorial that also includes a detailed comparison of different normalization (CPM, TPM, FPKM) approaches and their best uses, which are summarized below.

Common normalization methods

Several common normalization methods exist to account for these differences:

Normalization method Description Accounted factors Recommendations for use
CPM (counts per million) counts scaled by total number of reads sequencing depth gene count comparisons between replicates of the same samplegroup; NOT for within sample comparisons or DE analysis
TPM (transcripts per kilobase million) counts per length of transcript (kb) per million reads mapped sequencing depth and gene length gene count comparisons within a sample or between samples of the same sample group; NOT for DE analysis
RPKM/FPKM (reads/fragments per kilobase of exon per million reads/fragments mapped) similar to TPM sequencing depth and gene length gene count comparisons between genes within a sample; NOT for between sample comparisons or DE analysis
DESeq2’s median of ratios [1] counts divided by sample-specific size factors determined by median ratio of gene counts relative to geometric mean per gene sequencing depth and RNA composition gene count comparisons between samples and for DE analysis; NOT for within sample comparisons
EdgeR’s trimmed mean of M values (TMM) [2] uses a weighted trimmed mean of the log expression ratios between samples sequencing depth, RNA composition gene count comparisons between samples and for DE analysis; NOT for within sample comparisons

Check-in: Questions about normalizations?

DESeq2 normalizations

DESeq2 has an internal normalization process that accounts for RNA composition. A few highly differentially expressed genes, differences in the number of genes expressed between samples, or contamination are not accounted for by depth or gene length normalization methods. Accounting for RNA composition is particularly important for differential expression analyses, regardless of the tool used.

For data exploration and visualizations, it is helpful to generate an object of independently normalized counts. We will use the rlog transformation from DESeq2 that accounts for sequencing depth for each sample and RNA composition for the downstream quality control visualizations.

The rlog transformation produces log2 scaled data that has also been normalized to overall library size as well as variance across genes at different mean expression levels. For larger numbers of samples, there is an alternative transformation method, vst that can be used instead for count normalization.

The command to generate the normalized count object has a few parts, including dds as an input and providing a value to the option blind.

rld = rlog(dds_filtered, blind = TRUE)

Next, we’ll look at the results of the transformation by extracting the values with the assay() function.

head(assay(rld), 2)
                   sample_A sample_B sample_C sample_D sample_E sample_F
ENSMUSG00000000001 10.51481 10.36671 10.41946 10.84037 10.41045 10.57877
ENSMUSG00000000028 10.60446 10.73451 10.73503 10.68271 10.82094 10.99100

Looking at the rld values, we can see that they are now in log scale. Since we set blind=TRUE, the transformation is blind to the sample information we specified in the design formula. The normalized counts are helpful for visualization methods during expression-level quality assessment but aren’t used in the model fitting.

We’ll come back to these normalized data, but first let’s write out both the raw and normalized count tables to file.

Output count tables

To output the raw counts, we will need to use the counts function to access the count table from within its larger DESeqDataSet object.

write.csv(counts(dds, normalized = FALSE), file="outputs/tables/DESeq2_raw_counts.csv")

Then we’ll output the rlog count table, using the assay function to access the normalized count table from within its larger DESeqDataSet object.

write.csv(assay(rld), file="outputs/tables/DESeq2_rlog_normalized_counts.csv")

Summary

In this section, we:

  • Learned about count normalizations and uses
  • Generated a normalized count table
  • Wrote intermediate data to file

Sources

Training resources used to develop materials


These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.




Previous lesson Top of this lesson Next lesson
LS0tCnRpdGxlOiAiREUgTm9ybWFsaXphdGlvbiIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA4LSIpCmBgYAoKSW4gdGhpcyBtb2R1bGUsIHdlIHdpbGwgbGVhcm46CgoqIERpZmZlcmVudCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgdGhlaXIgdXNlcwoqIEhvdyB0byBmaXQgYSBtb2RlbCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKCjxicj4KCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIHByb2NlZWQgd2l0aCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgZml0IG91ciBERVNlcTIgbW9kZWwuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERVNlcTJERS5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBDb3VudCBub3JtYWxpemF0aW9ucwoKU2luY2UgY291bnRzIG9mIG1hcHBlZCByZWFkcyBmb3IgZWFjaCBnZW5lIGlzIHByb3BvcnRpb25hbCB0byB0aGUgZXhwcmVzc2lvbiBvZiBSTkEgaW4gYWRkaXRpb24gdG8gbWFueSDigJx1bmludGVyZXN0aW5n4oCdIG90aGVyIGZhY3RvcnMsIG5vcm1hbGl6YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2Ygc2NhbGluZyByYXcgY291bnQgdmFsdWVzIHRvIGFjY291bnQgZm9yIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIGZhY3RvcnMgYW5kIGVuc3VyZSBleHByZXNzaW9uIGxldmVscyBhcmUgbW9yZSBjb21wYXJhYmxlLgoKIyMgTm9ybWFsaXphdGlvbiBnb2FscwoKVHdvIGNvbW1vbiBmYWN0b3JzIHRoYXQgbmVlZCB0byBiZSBhY2NvdW50ZWQgZm9yIGR1cmluZyBub3JtYWxpemF0aW9uIGFyZSAqKnNlcXVlbmNpbmcgZGVwdGgqKiBhbmQgKipnZW5lIGxlbmd0aCoqLiBDb21tb24gbm9ybWFsaXphdGlvbiBhcHByb2FjaGVzIChzdWNoIGFzIEZQS00sIFJQS00sIENQTSwgVFBNLCBldGMuKSBhY2NvdW50IGZvciBvbmUgb3IgYm90aCBvZiB0aGVzZSBmYWN0b3JzLgoKKiAqKlNlcXVlbmNpbmcgZGVwdGgqKiBub3JtYWxpemF0aW9uIGlzIG5lY2Vzc2FyeSB0byBhY2NvdW50IGZvciB0aGUgcHJvcG9ydGlvbiBvZiByZWFkcyBwZXIgZ2VuZSBleHBlY3RlZCBmb3IgbW9yZSBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZXMgKGxpa2UgaW4gcGluayBiZWxvdykgdmVyc3VzIGEgbGVzcyBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZSAobGlrZSBpbiBncmVlbiBiZWxvdykuCgohWypOb3RlIHRoYXQgZWFjaCBwaW5rIG9yIGdyZWVuIHJlY3RhbmdsZSByZXByZXNlbnRzIGFuIGFsaWduZWQgcmVhZCwgd2l0aCByZWFkcyBzcGFubmluZyBhbiBpbnRyb24gY29ubmVjdGVkIGJ5IGEgZGFzaGVkIGxpbmUuIEZpZ3VyZSBmcm9tIFtIQkMgIHRyYWluaW5nIG1hdGVyaWFsc10oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkqXSguL2ltYWdlcy9ub3JtYWxpemF0aW9uX21ldGhvZHNfZGVwdGgucG5nKXt3aWR0aD03NSV9CgoqICoqR2VuZSBsZW5ndGgqKiBub3JtYWxpemF0aW9uIG1heSBhbHNvIGJlIG5lY2Vzc2FyeSBpZiBjb21wYXJpbmcgYmV0d2VlbiBkaWZmZXJlbnQgZ2VuZXMsIHNpbmNlIGdlbmVzIG9mIGRpZmZlcmVudCBsZW5ndGhzIGhhdmUgZGlmZmVyZW50IHByb2JhYmlsaXRpZXMgb2YgZ2VuZXJhdGluZyBmcmFnbWVudHMgdGhhdCBlbmQgdXAgaW4gdGhlIGxpYnJhcnkuCgo+ICoqTm90ZSoqOiBUaGUgYWJvdmUgZmlndXJlIHdhcyBvcmlnaW5hbGx5IGZyb20gYSBbSEJDICB0dXRvcmlhbF0oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkgdGhhdCBhbHNvIGluY2x1ZGVzIGEgZGV0YWlsZWQgY29tcGFyaXNvbiBvZiBkaWZmZXJlbnQgbm9ybWFsaXphdGlvbiAoQ1BNLCBUUE0sIEZQS00pIGFwcHJvYWNoZXMgYW5kIHRoZWlyIGJlc3QgdXNlcywgd2hpY2ggYXJlIHN1bW1hcml6ZWQgYmVsb3cuCgo8RURJVCB0byBiZSBhIGRyb3Bkb3duIGZvciB0YWJsZSBvZiBjb3VudHM+CgojIyMgQ29tbW9uIG5vcm1hbGl6YXRpb24gbWV0aG9kcwoKU2V2ZXJhbCBjb21tb24gbm9ybWFsaXphdGlvbiBtZXRob2RzIGV4aXN0IHRvIGFjY291bnQgZm9yIHRoZXNlIGRpZmZlcmVuY2VzOgoJCnwgTm9ybWFsaXphdGlvbiBtZXRob2QgfCBEZXNjcmlwdGlvbiB8IEFjY291bnRlZCBmYWN0b3JzIHwgUmVjb21tZW5kYXRpb25zIGZvciB1c2UgfAp8IC0tLS0gfCAtLS0tIHwgLS0tLSB8IC0tLS0gfAp8ICoqQ1BNKiogKGNvdW50cyBwZXIgbWlsbGlvbikgfCBjb3VudHMgc2NhbGVkIGJ5IHRvdGFsIG51bWJlciBvZiByZWFkcyB8IHNlcXVlbmNpbmcgZGVwdGggfCBnZW5lIGNvdW50IGNvbXBhcmlzb25zIGJldHdlZW4gcmVwbGljYXRlcyBvZiB0aGUgc2FtZSBzYW1wbGVncm91cDsgKipOT1QgZm9yIHdpdGhpbiBzYW1wbGUgY29tcGFyaXNvbnMgb3IgREUgYW5hbHlzaXMqKiAgfAp8ICoqVFBNKiogKHRyYW5zY3JpcHRzIHBlciBraWxvYmFzZSBtaWxsaW9uKSB8IGNvdW50cyBwZXIgbGVuZ3RoIG9mIHRyYW5zY3JpcHQgKGtiKSBwZXIgbWlsbGlvbiByZWFkcyBtYXBwZWQgfCBzZXF1ZW5jaW5nIGRlcHRoIGFuZCBnZW5lIGxlbmd0aCB8IGdlbmUgY291bnQgY29tcGFyaXNvbnMgd2l0aGluIGEgc2FtcGxlIG9yIGJldHdlZW4gc2FtcGxlcyBvZiB0aGUgc2FtZSBzYW1wbGUgZ3JvdXA7ICoqTk9UIGZvciBERSBhbmFseXNpcyoqIHwKfCAqKlJQS00vRlBLTSoqIChyZWFkcy9mcmFnbWVudHMgcGVyIGtpbG9iYXNlIG9mIGV4b24gcGVyIG1pbGxpb24gcmVhZHMvZnJhZ21lbnRzIG1hcHBlZCkgfCBzaW1pbGFyIHRvIFRQTSB8IHNlcXVlbmNpbmcgZGVwdGggYW5kIGdlbmUgbGVuZ3RoIHwgZ2VuZSBjb3VudCBjb21wYXJpc29ucyBiZXR3ZWVuIGdlbmVzIHdpdGhpbiBhIHNhbXBsZTsgKipOT1QgZm9yIGJldHdlZW4gc2FtcGxlIGNvbXBhcmlzb25zIG9yIERFIGFuYWx5c2lzKiogfAp8IERFU2VxMidzICoqbWVkaWFuIG9mIHJhdGlvcyoqIFtbMV0oaHR0cHM6Ly9nZW5vbWViaW9sb2d5LmJpb21lZGNlbnRyYWwuY29tL2FydGljbGVzLzEwLjExODYvZ2ItMjAxMC0xMS0xMC1yMTA2KV0gfCBjb3VudHMgZGl2aWRlZCBieSBzYW1wbGUtc3BlY2lmaWMgc2l6ZSBmYWN0b3JzIGRldGVybWluZWQgYnkgbWVkaWFuIHJhdGlvIG9mIGdlbmUgY291bnRzIHJlbGF0aXZlIHRvIGdlb21ldHJpYyBtZWFuIHBlciBnZW5lIHwgc2VxdWVuY2luZyBkZXB0aCBhbmQgUk5BIGNvbXBvc2l0aW9uIHwgZ2VuZSBjb3VudCBjb21wYXJpc29ucyBiZXR3ZWVuIHNhbXBsZXMgYW5kIGZvciAqKkRFIGFuYWx5c2lzKio7ICoqTk9UIGZvciB3aXRoaW4gc2FtcGxlIGNvbXBhcmlzb25zKiogfAp8IEVkZ2VSJ3MgKip0cmltbWVkIG1lYW4gb2YgTSB2YWx1ZXMgKFRNTSkqKiBbWzJdKGh0dHBzOi8vZ2Vub21lYmlvbG9neS5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2L2diLTIwMTAtMTEtMy1yMjUpXSB8IHVzZXMgYSB3ZWlnaHRlZCB0cmltbWVkIG1lYW4gb2YgdGhlIGxvZyBleHByZXNzaW9uIHJhdGlvcyBiZXR3ZWVuIHNhbXBsZXMgfCBzZXF1ZW5jaW5nIGRlcHRoLCBSTkEgY29tcG9zaXRpb24gfCBnZW5lIGNvdW50IGNvbXBhcmlzb25zIGJldHdlZW4gc2FtcGxlcyBhbmQgZm9yICoqREUgYW5hbHlzaXMqKjsgKipOT1QgZm9yIHdpdGhpbiBzYW1wbGUgY29tcGFyaXNvbnMqKiB8CgoKKipDaGVjay1pbjogUXVlc3Rpb25zIGFib3V0IG5vcm1hbGl6YXRpb25zPyoqCgoKCiMjIERFU2VxMiBub3JtYWxpemF0aW9ucwoKREVTZXEyIGhhcyBhbiBbaW50ZXJuYWwgbm9ybWFsaXphdGlvbiBwcm9jZXNzXShodHRwczovL2dlbm9tZWJpb2xvZ3kuYmlvbWVkY2VudHJhbC5jb20vYXJ0aWNsZXMvMTAuMTE4Ni9nYi0yMDEwLTExLTEwLXIxMDYpIHRoYXQgYWNjb3VudHMgZm9yICoqUk5BIGNvbXBvc2l0aW9uKiouIEEgZmV3IGhpZ2hseSBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMsIGRpZmZlcmVuY2VzIGluIHRoZSBudW1iZXIgb2YgZ2VuZXMgZXhwcmVzc2VkIGJldHdlZW4gc2FtcGxlcywgb3IgY29udGFtaW5hdGlvbiBhcmUgbm90IGFjY291bnRlZCBmb3IgYnkgZGVwdGggb3IgZ2VuZSBsZW5ndGggbm9ybWFsaXphdGlvbiBtZXRob2RzLiBBY2NvdW50aW5nIGZvciBSTkEgY29tcG9zaXRpb24gaXMgcGFydGljdWxhcmx5IGltcG9ydGFudCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzZXMsIHJlZ2FyZGxlc3Mgb2YgdGhlIHRvb2wgdXNlZC4KCkZvciBkYXRhIGV4cGxvcmF0aW9uIGFuZCB2aXN1YWxpemF0aW9ucywgaXQgaXMgaGVscGZ1bCB0byBnZW5lcmF0ZSBhbiBvYmplY3Qgb2YgaW5kZXBlbmRlbnRseSBub3JtYWxpemVkIGNvdW50cy4gV2Ugd2lsbCB1c2UgdGhlIFtybG9nIHRyYW5zZm9ybWF0aW9uXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjY291bnQtZGF0YS10cmFuc2Zvcm1hdGlvbnMpIGZyb20gREVTZXEyIHRoYXQgYWNjb3VudHMgZm9yIHNlcXVlbmNpbmcgZGVwdGggZm9yIGVhY2ggc2FtcGxlIGFuZCBSTkEgY29tcG9zaXRpb24gZm9yIHRoZSBkb3duc3RyZWFtIHF1YWxpdHkgY29udHJvbCB2aXN1YWxpemF0aW9ucy4KClRoZSBybG9nIHRyYW5zZm9ybWF0aW9uIHByb2R1Y2VzIGxvZzIgc2NhbGVkIGRhdGEgdGhhdCBoYXMgYWxzbyBiZWVuIG5vcm1hbGl6ZWQgdG8gb3ZlcmFsbCBsaWJyYXJ5IHNpemUgYXMgd2VsbCBhcyB2YXJpYW5jZSBhY3Jvc3MgZ2VuZXMgYXQgZGlmZmVyZW50IG1lYW4gZXhwcmVzc2lvbiBsZXZlbHMuIEZvciBsYXJnZXIgbnVtYmVycyBvZiBzYW1wbGVzLCB0aGVyZSBpcyBhbiBhbHRlcm5hdGl2ZSB0cmFuc2Zvcm1hdGlvbiBtZXRob2QsIFt2c3RdKGh0dHA6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNjb3VudC1kYXRhLXRyYW5zZm9ybWF0aW9ucykgdGhhdCBjYW4gYmUgdXNlZCBpbnN0ZWFkIGZvciBjb3VudCBub3JtYWxpemF0aW9uLgoKVGhlIGNvbW1hbmQgdG8gZ2VuZXJhdGUgdGhlIG5vcm1hbGl6ZWQgY291bnQgb2JqZWN0IGhhcyBhIGZldyBwYXJ0cywgaW5jbHVkaW5nIGBkZHNgIGFzIGFuIGlucHV0IGFuZCBwcm92aWRpbmcgYSB2YWx1ZSB0byB0aGUgb3B0aW9uIGBibGluZGAuIAoKYGBge3IgQ291bnROb3JtfQpybGQgPSBybG9nKGRkc19maWx0ZXJlZCwgYmxpbmQgPSBUUlVFKQpgYGAKCk5leHQsIHdlJ2xsIGxvb2sgYXQgdGhlIHJlc3VsdHMgb2YgdGhlIHRyYW5zZm9ybWF0aW9uIGJ5IGV4dHJhY3RpbmcgdGhlIHZhbHVlcyB3aXRoIHRoZSBgYXNzYXkoKWAgZnVuY3Rpb24uCgpgYGB7ciBDb3VudE5vcm1DaGVja30KaGVhZChhc3NheShybGQpLCAyKQpgYGAKCkxvb2tpbmcgYXQgdGhlIHJsZCB2YWx1ZXMsIHdlIGNhbiBzZWUgdGhhdCB0aGV5IGFyZSBub3cgaW4gbG9nIHNjYWxlLiBTaW5jZSB3ZSBzZXQgYGJsaW5kPVRSVUVgLCB0aGUgdHJhbnNmb3JtYXRpb24gaXMgYmxpbmQgdG8gdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiB3ZSBzcGVjaWZpZWQgaW4gdGhlIGRlc2lnbiBmb3JtdWxhLiBUaGUgbm9ybWFsaXplZCBjb3VudHMgYXJlIGhlbHBmdWwgZm9yIHZpc3VhbGl6YXRpb24gbWV0aG9kcyBkdXJpbmcgZXhwcmVzc2lvbi1sZXZlbCBxdWFsaXR5IGFzc2Vzc21lbnQgYnV0ICoqYXJlbid0IHVzZWQgaW4gdGhlIG1vZGVsIGZpdHRpbmcqKi4KCldlJ2xsIGNvbWUgYmFjayB0byB0aGVzZSBub3JtYWxpemVkIGRhdGEsIGJ1dCBmaXJzdCBsZXQncyB3cml0ZSBvdXQgYm90aCB0aGUgcmF3IGFuZCBub3JtYWxpemVkIGNvdW50IHRhYmxlcyB0byBmaWxlLgoKIyMjIE91dHB1dCBjb3VudCB0YWJsZXMKClRvIG91dHB1dCB0aGUgcmF3IGNvdW50cywgd2Ugd2lsbCBuZWVkIHRvIHVzZSB0aGUgYGNvdW50c2AgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBjb3VudCB0YWJsZSBmcm9tIHdpdGhpbiBpdHMgbGFyZ2VyIGBERVNlcURhdGFTZXRgIG9iamVjdC4KCmBgYHtyIE91dHB1dENvdW50c1JhdywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoY291bnRzKGRkcywgbm9ybWFsaXplZCA9IEZBTFNFKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jhd19jb3VudHMuY3N2IikKYGBgCgpUaGVuIHdlJ2xsIG91dHB1dCB0aGUgcmxvZyBjb3VudCB0YWJsZSwgdXNpbmcgdGhlIGBhc3NheWAgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBub3JtYWxpemVkIGNvdW50IHRhYmxlIGZyb20gd2l0aGluIGl0cyBsYXJnZXIgYERFU2VxRGF0YVNldGAgb2JqZWN0LgoKYGBge3IgT3V0cHV0Q291bnRzUmxvZywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoYXNzYXkocmxkKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jsb2dfbm9ybWFsaXplZF9jb3VudHMuY3N2IikKYGBgCgoKCiMgU3VtbWFyeQoKSW4gdGhpcyBzZWN0aW9uLCB3ZToKCiogTGVhcm5lZCBhYm91dCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgdXNlcwoqIEdlbmVyYXRlZCBhIG5vcm1hbGl6ZWQgY291bnQgdGFibGUKKiBXcm90ZSBpbnRlcm1lZGlhdGUgZGF0YSB0byBmaWxlCgotLS0KCiMgU291cmNlcwoKVHJhaW5pbmcgcmVzb3VyY2VzIHVzZWQgdG8gZGV2ZWxvcCBtYXRlcmlhbHMKCiogSEJDIERHRSBzZXR1cDogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDFfREdFX3NldHVwX2FuZF9vdmVydmlldy5odG1sCiogSEJDIENvdW50IE5vcm1hbGl6YXRpb246IGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzAyX0RHRV9jb3VudF9ub3JtYWxpemF0aW9uLmh0bWwKKiBERVNlcTIgc3RhbmRhcmQgdmlnbmV0dGU6IGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbAoqIERFU2VxMiBiZWdpbm5lcnMgdmlnbmV0dGU6IGh0dHBzOi8vYmlvYy5pc20uYWMuanAvcGFja2FnZXMvMi4xNC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvYmVnaW5uZXIucGRmCiogQmlvY29uZHVjdG9yIFJOQS1zZXEgV29ya2Zsb3dzOiBodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL2hlbHAvY291cnNlLW1hdGVyaWFscy8yMDE1L0xlYXJuQmlvY29uZHVjdG9yRmViMjAxNS9CMDIuMV9STkFTZXEuaHRtbAoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQpgYGAKCi0tLQoKVGhlc2UgbWF0ZXJpYWxzIGhhdmUgYmVlbiBhZGFwdGVkIGFuZCBleHRlbmRlZCBmcm9tIG1hdGVyaWFscyBsaXN0ZWQgYWJvdmUuIFRoZXNlIGFyZSBvcGVuIGFjY2VzcyBtYXRlcmlhbHMgZGlzdHJpYnV0ZWQgdW5kZXIgdGhlIHRlcm1zIG9mIHRoZSBbQ3JlYXRpdmUgQ29tbW9ucyBBdHRyaWJ1dGlvbiBsaWNlbnNlIChDQyBCWSA0LjApXShodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS80LjAvKSwgd2hpY2ggcGVybWl0cyB1bnJlc3RyaWN0ZWQgdXNlLCBkaXN0cmlidXRpb24sIGFuZCByZXByb2R1Y3Rpb24gaW4gYW55IG1lZGl1bSwgcHJvdmlkZWQgdGhlIG9yaWdpbmFsIGF1dGhvciBhbmQgc291cmNlIGFyZSBjcmVkaXRlZC4KCjxici8+Cjxici8+Cjxoci8+CnwgW1ByZXZpb3VzIGxlc3Nvbl0oTW9kdWxlMDdfREVTZXEySW5pdC5odG1sKSB8IFtUb3Agb2YgdGhpcyBsZXNzb25dKCN0b3ApIHwgW05leHQgbGVzc29uXShNb2R1bGUwOV9TYW1wbGVRQ1Zpei5odG1sKSB8CnwgOi0tLSB8IDotLS0tOiB8IC0tLTogfA==