Objectives
- Discuss count normalizations
- Execute model fitting for differential expression comparisons
Differential Expression Workflow
Here we will proceed with count normalizations and fit our DESeq2
model.
Count normalizations
Since counts of mapped reads for each gene is proportional to the
expression of RNA in addition to many “uninteresting” other factors,
normalization is the process of scaling raw count values to account for
the “uninteresting” factors and ensure expression levels are more
comparable.
Normalization goals
Two common factors that need to be accounted for during normalization
are sequencing depth and gene length.
Common normalization approaches (such as FPKM, RPKM, CPM, TPM, etc.)
account for one or both of these factors.
- Sequencing depth normalization is necessary to
account for the proportion of reads per gene expected for more deeply
sequenced samples (like in pink below) versus a less deeply sequenced
sample (like in green below).
- Gene length normalization may also be necessary if
comparing between different genes, since genes of different lengths have
different probabilities of generating fragments that end up in the
library.
Note: The above figure was originally from a HBC
tutorial that also includes a detailed comparison of different
normalization (CPM, TPM, FPKM) approaches and their best uses.
Check-in: Questions about normalizations?
DESeq2 normalizations
DESeq2 has an internal
normalization process that accounts for RNA
composition. A few highly differentially expressed genes,
differences in the number of genes expressed between samples, or
contamination are not accounted for by depth or gene length
normalization methods. Accounting for RNA composition is particularly
important for differential expression analyses, regardless of the tool
used.
For data exploration and visualizations, it is helpful to generate an
object of independently normalized counts. We will use the rlog
transformation from DESeq2 that accounts for sequencing depth for
each sample and RNA composition for the downstream quality control
visualizations.
The rlog transformation produces log2 scaled data that has also been
normalized to overall library size as well as variance across genes at
different mean expression levels. For larger numbers of samples, there
is an alternative transformation method, vst
that can be used instead for count normalizations.
The command to generate the normalized count object has a few parts,
including dds
as an input and providing a value to the
option blind
. For our purposes, we set
blind = TRUE
because we want to compare samples in
downstream QC plots in an unbiased manner.
rld = rlog(dds, blind = TRUE)
Note: We might see a message that our data did not fit the default
‘parametric’ dispersion model so a local regression was substituted. If
we had more time, we might look at a dispersion plot with the
plotDispEsts(dds)
function, but as this bioconductor
thread discusses, other visualizations of our data might be more
helpful and/or easier to interpret.
Next, we’ll look at the results of the transformation by extracting
the values with the assay()
function.
head(assay(rld), 2)
sample_A sample_B sample_C sample_D sample_E sample_F
ENSMUSG00000000001 10.51481 10.36671 10.41946 10.84037 10.41045 10.57877
ENSMUSG00000000028 10.60446 10.73451 10.73503 10.68271 10.82094 10.99100
Looking at the rld values, we can see that they are now in log scale.
Since we set blind=TRUE
, the transformation is blind to the
sample information we specified in the design formula. The normalized
counts are helpful for visualization methods during expression-level
quality assessment but aren’t used in the model
fitting.
DESeq2 Model Fitting
Next, we’ll fit our standard model using the DESeq
function and take a look at the objects we generate. This command
applies the model to our data, using the sample information supplied
when generating the dds
object so it takes some time to
run.
dds = DESeq(dds)
dds
class: DESeqDataSet
dim: 16249 6
metadata(1): version
assays(4): counts mu H cooks
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ...
ENSMUSG00000118651 ENSMUSG00000118653
rowData names(22): baseMean baseVar ... deviance maxCooks
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(3): genotype condition sizeFactor
Notice that there is now more information in the
DESeqDataSet
object than there was prior to our
normalization. There is information about the model fit and about the
library size normalization. DESeq2 will use this information when we
perform the test for differential expression.
The DESeq()
function is actually doing three things
automatically for us. It calculates:
- The size factors to normalize for library size with
estimateSizeFactors(dds)
,
- Dispersion estimates to shrink the dispersions with
estimateDispersions(dds)
, and
- The Wald test statistics with
nbinomWaldTest(dds)
.
The resultsNames()
function returns the names of the
estimated effects of the model.
resultsNames(dds)
[1] "Intercept" "condition_minus_vs_plus"
The results include the single comparison representing the two levels
of condition
. If there were more levels in the
condition
column, there would be more results listed here
because DESeq2 would implicitly compare all other levels to the
reference level.
Checkpoint: If you see the same results when you
execute resultsNames(dds)
, please indicate with the green
‘yes’ button. Otherwise, please use the red ‘x’ button to get help
before the break
Optional content
Click for fitting a model that includes a covariate
If you executed the commands in the additional content section from
Module 07, you can fit a separate DESeq2 model for the batch
example.
dds_batch = DESeq(dds_batch)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resultsNames(dds_batch)
[1] "Intercept" "batch_Day2_vs_Day1"
[3] "batch_Day3_vs_Day1" "condition_minus_vs_plus"
If you run through the optional exercises, you can explore the impact of
adding a covariate by substituting dds_patient
for
dds
and re-running those commands since both DESeq2 objects
have their data organized in the same way.
Summary
In this section, we:
- Learned about count normalizations and uses
- Generated a normalized count table
- Fit two DESeq2 models for our data
- Saw the impact of including a covariate in our model
Sources
Training resources used to develop materials
These materials have been adapted and extended from materials listed
above. These are open access materials distributed under the terms of
the Creative
Commons Attribution license (CC BY 4.0), which permits unrestricted
use, distribution, and reproduction in any medium, provided the original
author and source are credited.
LS0tCnRpdGxlOiAiTW9kdWxlIDA4OiBERSBOb3JtYWxpemF0aW9uIGFuZCBNb2RlbGluZyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA4LSIpCmBgYAoKPiAjIE9iamVjdGl2ZXMgey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiAqIERpc2N1c3MgY291bnQgbm9ybWFsaXphdGlvbnMKPiAqIEV4ZWN1dGUgbW9kZWwgZml0dGluZyBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIHByb2NlZWQgd2l0aCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgZml0IG91ciBERVNlcTIgbW9kZWwuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERVNlcTJERS5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBDb3VudCBub3JtYWxpemF0aW9ucwoKU2luY2UgY291bnRzIG9mIG1hcHBlZCByZWFkcyBmb3IgZWFjaCBnZW5lIGlzIHByb3BvcnRpb25hbCB0byB0aGUgZXhwcmVzc2lvbiBvZiBSTkEgaW4gYWRkaXRpb24gdG8gbWFueSDigJx1bmludGVyZXN0aW5n4oCdIG90aGVyIGZhY3RvcnMsIG5vcm1hbGl6YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2Ygc2NhbGluZyByYXcgY291bnQgdmFsdWVzIHRvIGFjY291bnQgZm9yIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIGZhY3RvcnMgYW5kIGVuc3VyZSBleHByZXNzaW9uIGxldmVscyBhcmUgbW9yZSBjb21wYXJhYmxlLgoKIyMgTm9ybWFsaXphdGlvbiBnb2FscwoKVHdvIGNvbW1vbiBmYWN0b3JzIHRoYXQgbmVlZCB0byBiZSBhY2NvdW50ZWQgZm9yIGR1cmluZyBub3JtYWxpemF0aW9uIGFyZSAqKnNlcXVlbmNpbmcgZGVwdGgqKiBhbmQgKipnZW5lIGxlbmd0aCoqLiBDb21tb24gbm9ybWFsaXphdGlvbiBhcHByb2FjaGVzIChzdWNoIGFzIEZQS00sIFJQS00sIENQTSwgVFBNLCBldGMuKSBhY2NvdW50IGZvciBvbmUgb3IgYm90aCBvZiB0aGVzZSBmYWN0b3JzLgoKKiAqKlNlcXVlbmNpbmcgZGVwdGgqKiBub3JtYWxpemF0aW9uIGlzIG5lY2Vzc2FyeSB0byBhY2NvdW50IGZvciB0aGUgcHJvcG9ydGlvbiBvZiByZWFkcyBwZXIgZ2VuZSBleHBlY3RlZCBmb3IgbW9yZSBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZXMgKGxpa2UgaW4gcGluayBiZWxvdykgdmVyc3VzIGEgbGVzcyBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZSAobGlrZSBpbiBncmVlbiBiZWxvdykuCgohWypOb3RlIHRoYXQgZWFjaCBwaW5rIG9yIGdyZWVuIHJlY3RhbmdsZSByZXByZXNlbnRzIGFuIGFsaWduZWQgcmVhZCwgd2l0aCByZWFkcyBzcGFubmluZyBhbiBpbnRyb24gY29ubmVjdGVkIGJ5IGEgZGFzaGVkIGxpbmUuIEZpZ3VyZSBmcm9tIFtIQkMgIHRyYWluaW5nIG1hdGVyaWFsc10oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkqXSguL2ltYWdlcy9ub3JtYWxpemF0aW9uX21ldGhvZHNfZGVwdGgucG5nKXt3aWR0aD03NSV9CgoqICoqR2VuZSBsZW5ndGgqKiBub3JtYWxpemF0aW9uIG1heSBhbHNvIGJlIG5lY2Vzc2FyeSBpZiBjb21wYXJpbmcgYmV0d2VlbiBkaWZmZXJlbnQgZ2VuZXMsIHNpbmNlIGdlbmVzIG9mIGRpZmZlcmVudCBsZW5ndGhzIGhhdmUgZGlmZmVyZW50IHByb2JhYmlsaXRpZXMgb2YgZ2VuZXJhdGluZyBmcmFnbWVudHMgdGhhdCBlbmQgdXAgaW4gdGhlIGxpYnJhcnkuCgo+ICoqTm90ZSoqOiBUaGUgYWJvdmUgZmlndXJlIHdhcyBvcmlnaW5hbGx5IGZyb20gYSBbSEJDICB0dXRvcmlhbF0oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkgdGhhdCBhbHNvIGluY2x1ZGVzIGEgZGV0YWlsZWQgY29tcGFyaXNvbiBvZiBkaWZmZXJlbnQgbm9ybWFsaXphdGlvbiAoQ1BNLCBUUE0sIEZQS00pIGFwcHJvYWNoZXMgYW5kIHRoZWlyIGJlc3QgdXNlcy4KCioqQ2hlY2staW46IFF1ZXN0aW9ucyBhYm91dCBub3JtYWxpemF0aW9ucz8qKgoKIyMgREVTZXEyIG5vcm1hbGl6YXRpb25zCgpERVNlcTIgaGFzIGFuIFtpbnRlcm5hbCBub3JtYWxpemF0aW9uIHByb2Nlc3NdKGh0dHBzOi8vZ2Vub21lYmlvbG9neS5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2L2diLTIwMTAtMTEtMTAtcjEwNikgdGhhdCBhY2NvdW50cyBmb3IgKipSTkEgY29tcG9zaXRpb24qKi4gQSBmZXcgaGlnaGx5IGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcywgZGlmZmVyZW5jZXMgaW4gdGhlIG51bWJlciBvZiBnZW5lcyBleHByZXNzZWQgYmV0d2VlbiBzYW1wbGVzLCBvciBjb250YW1pbmF0aW9uIGFyZSBub3QgYWNjb3VudGVkIGZvciBieSBkZXB0aCBvciBnZW5lIGxlbmd0aCBub3JtYWxpemF0aW9uIG1ldGhvZHMuIEFjY291bnRpbmcgZm9yIFJOQSBjb21wb3NpdGlvbiBpcyBwYXJ0aWN1bGFybHkgaW1wb3J0YW50IGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNlcywgcmVnYXJkbGVzcyBvZiB0aGUgdG9vbCB1c2VkLgoKRm9yIGRhdGEgZXhwbG9yYXRpb24gYW5kIHZpc3VhbGl6YXRpb25zLCBpdCBpcyBoZWxwZnVsIHRvIGdlbmVyYXRlIGFuIG9iamVjdCBvZiBpbmRlcGVuZGVudGx5IG5vcm1hbGl6ZWQgY291bnRzLiBXZSB3aWxsIHVzZSB0aGUgW3Jsb2cgdHJhbnNmb3JtYXRpb25dKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNjb3VudC1kYXRhLXRyYW5zZm9ybWF0aW9ucykgZnJvbSBERVNlcTIgdGhhdCBhY2NvdW50cyBmb3Igc2VxdWVuY2luZyBkZXB0aCBmb3IgZWFjaCBzYW1wbGUgYW5kIFJOQSBjb21wb3NpdGlvbiBmb3IgdGhlIGRvd25zdHJlYW0gcXVhbGl0eSBjb250cm9sIHZpc3VhbGl6YXRpb25zLgoKVGhlIHJsb2cgdHJhbnNmb3JtYXRpb24gcHJvZHVjZXMgbG9nMiBzY2FsZWQgZGF0YSB0aGF0IGhhcyBhbHNvIGJlZW4gbm9ybWFsaXplZCB0byBvdmVyYWxsIGxpYnJhcnkgc2l6ZSBhcyB3ZWxsIGFzIHZhcmlhbmNlIGFjcm9zcyBnZW5lcyBhdCBkaWZmZXJlbnQgbWVhbiBleHByZXNzaW9uIGxldmVscy4gRm9yIGxhcmdlciBudW1iZXJzIG9mIHNhbXBsZXMsIHRoZXJlIGlzIGFuIGFsdGVybmF0aXZlIHRyYW5zZm9ybWF0aW9uIG1ldGhvZCwgW3ZzdF0oaHR0cDovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2NvdW50LWRhdGEtdHJhbnNmb3JtYXRpb25zKSB0aGF0IGNhbiBiZSB1c2VkIGluc3RlYWQgZm9yIGNvdW50IG5vcm1hbGl6YXRpb25zLgoKVGhlIGNvbW1hbmQgdG8gZ2VuZXJhdGUgdGhlIG5vcm1hbGl6ZWQgY291bnQgb2JqZWN0IGhhcyBhIGZldyBwYXJ0cywgaW5jbHVkaW5nIGBkZHNgIGFzIGFuIGlucHV0IGFuZCBwcm92aWRpbmcgYSB2YWx1ZSB0byB0aGUgb3B0aW9uIGBibGluZGAuIEZvciBvdXIgcHVycG9zZXMsIHdlIHNldCBgYmxpbmQgPSBUUlVFYCBiZWNhdXNlIHdlIHdhbnQgdG8gY29tcGFyZSBzYW1wbGVzIGluIGRvd25zdHJlYW0gUUMgcGxvdHMgaW4gYW4gdW5iaWFzZWQgbWFubmVyLgoKYGBge3IgQ291bnROb3JtfQpybGQgPSBybG9nKGRkcywgYmxpbmQgPSBUUlVFKQpgYGAKCk5vdGU6IFdlIG1pZ2h0IHNlZSBhIG1lc3NhZ2UgdGhhdCBvdXIgZGF0YSBkaWQgbm90IGZpdCB0aGUgZGVmYXVsdCAncGFyYW1ldHJpYycgZGlzcGVyc2lvbiBtb2RlbCBzbyBhIGxvY2FsIHJlZ3Jlc3Npb24gd2FzIHN1YnN0aXR1dGVkLiBJZiB3ZSBoYWQgbW9yZSB0aW1lLCB3ZSBtaWdodCBsb29rIGF0IGEgZGlzcGVyc2lvbiBwbG90IHdpdGggdGhlIGBwbG90RGlzcEVzdHMoZGRzKWAgZnVuY3Rpb24sIGJ1dCBhcyBbdGhpcyBiaW9jb25kdWN0b3IgdGhyZWFkXShodHRwczovL3N1cHBvcnQuYmlvY29uZHVjdG9yLm9yZy9wLzEwNzkzNy8pIGRpc2N1c3Nlcywgb3RoZXIgdmlzdWFsaXphdGlvbnMgb2Ygb3VyIGRhdGEgbWlnaHQgYmUgbW9yZSBoZWxwZnVsIGFuZC9vciBlYXNpZXIgdG8gaW50ZXJwcmV0LgoKTmV4dCwgd2UnbGwgbG9vayBhdCB0aGUgcmVzdWx0cyBvZiB0aGUgdHJhbnNmb3JtYXRpb24gYnkgZXh0cmFjdGluZyB0aGUgdmFsdWVzIHdpdGggdGhlIGBhc3NheSgpYCBmdW5jdGlvbi4KCmBgYHtyIENvdW50Tm9ybUNoZWNrfQpoZWFkKGFzc2F5KHJsZCksIDIpCmBgYAoKTG9va2luZyBhdCB0aGUgcmxkIHZhbHVlcywgd2UgY2FuIHNlZSB0aGF0IHRoZXkgYXJlIG5vdyBpbiBsb2cgc2NhbGUuIFNpbmNlIHdlIHNldCBgYmxpbmQ9VFJVRWAsIHRoZSB0cmFuc2Zvcm1hdGlvbiBpcyBibGluZCB0byB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIHdlIHNwZWNpZmllZCBpbiB0aGUgZGVzaWduIGZvcm11bGEuIFRoZSBub3JtYWxpemVkIGNvdW50cyBhcmUgaGVscGZ1bCBmb3IgdmlzdWFsaXphdGlvbiBtZXRob2RzIGR1cmluZyBleHByZXNzaW9uLWxldmVsIHF1YWxpdHkgYXNzZXNzbWVudCBidXQgKiphcmVuJ3QgdXNlZCBpbiB0aGUgbW9kZWwgZml0dGluZyoqLgoKIyBERVNlcTIgTW9kZWwgRml0dGluZwoKTmV4dCwgd2UnbGwgZml0IG91ciBzdGFuZGFyZCBtb2RlbCB1c2luZyB0aGUgYERFU2VxYCBmdW5jdGlvbiBhbmQgdGFrZSBhIGxvb2sgYXQgdGhlIG9iamVjdHMgd2UgZ2VuZXJhdGUuIFRoaXMgY29tbWFuZCBhcHBsaWVzIHRoZSBtb2RlbCB0byBvdXIgZGF0YSwgdXNpbmcgdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiBzdXBwbGllZCB3aGVuIGdlbmVyYXRpbmcgdGhlIGBkZHNgIG9iamVjdCBzbyBpdCB0YWtlcyBzb21lIHRpbWUgdG8gcnVuLgoKYGBge3IgRml0TW9kZWxTdGFuZGFyZCwgbWVzc2FnZT1GQUxTRX0KZGRzID0gREVTZXEoZGRzKQpkZHMKYGBgCgpOb3RpY2UgdGhhdCB0aGVyZSBpcyBub3cgbW9yZSBpbmZvcm1hdGlvbiBpbiB0aGUgYERFU2VxRGF0YVNldGAgb2JqZWN0IHRoYW4gdGhlcmUgd2FzIHByaW9yIHRvIG91ciBub3JtYWxpemF0aW9uLiBUaGVyZSBpcyBpbmZvcm1hdGlvbiBhYm91dCB0aGUgbW9kZWwgZml0IGFuZCBhYm91dCB0aGUgbGlicmFyeSBzaXplIG5vcm1hbGl6YXRpb24uIERFU2VxMiB3aWxsIHVzZSB0aGlzIGluZm9ybWF0aW9uIHdoZW4gd2UgcGVyZm9ybSB0aGUgdGVzdCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24uCgpUaGUgYERFU2VxKClgIGZ1bmN0aW9uIGlzIGFjdHVhbGx5IGRvaW5nIHRocmVlIHRoaW5ncyBhdXRvbWF0aWNhbGx5IGZvciB1cy4gSXQgY2FsY3VsYXRlczoKCjEuIFRoZSBzaXplIGZhY3RvcnMgdG8gbm9ybWFsaXplIGZvciBsaWJyYXJ5IHNpemUgd2l0aCBgZXN0aW1hdGVTaXplRmFjdG9ycyhkZHMpYCwKMi4gRGlzcGVyc2lvbiBlc3RpbWF0ZXMgdG8gc2hyaW5rIHRoZSBkaXNwZXJzaW9ucyB3aXRoIGBlc3RpbWF0ZURpc3BlcnNpb25zKGRkcylgLCBhbmQKMy4gVGhlIFdhbGQgdGVzdCBzdGF0aXN0aWNzIHdpdGggYG5iaW5vbVdhbGRUZXN0KGRkcylgLgoKVGhlIGByZXN1bHRzTmFtZXMoKWAgZnVuY3Rpb24gcmV0dXJucyB0aGUgbmFtZXMgb2YgdGhlIGVzdGltYXRlZCBlZmZlY3RzIG9mIHRoZSBtb2RlbC4KCmBgYHtyIEZpdE1vZGVsU3RhbmRhcmRDaGVjazEsIGV2YWw9VFJVRX0KcmVzdWx0c05hbWVzKGRkcykKYGBgCgpUaGUgcmVzdWx0cyBpbmNsdWRlIHRoZSBzaW5nbGUgY29tcGFyaXNvbiByZXByZXNlbnRpbmcgdGhlIHR3byBsZXZlbHMgb2YgYGNvbmRpdGlvbmAuIElmIHRoZXJlIHdlcmUgbW9yZSBsZXZlbHMgaW4gdGhlIGBjb25kaXRpb25gIGNvbHVtbiwgdGhlcmUgd291bGQgYmUgbW9yZSByZXN1bHRzIGxpc3RlZCBoZXJlIGJlY2F1c2UgREVTZXEyIHdvdWxkIGltcGxpY2l0bHkgY29tcGFyZSBhbGwgb3RoZXIgbGV2ZWxzIHRvIHRoZSByZWZlcmVuY2UgbGV2ZWwuCgoqKkNoZWNrcG9pbnQqKjogKklmIHlvdSBzZWUgdGhlIHNhbWUgcmVzdWx0cyB3aGVuIHlvdSBleGVjdXRlIGByZXN1bHRzTmFtZXMoZGRzKWAsIHBsZWFzZSBpbmRpY2F0ZSB3aXRoIHRoZSBncmVlbiAneWVzJyBidXR0b24uIE90aGVyd2lzZSwgcGxlYXNlIHVzZSB0aGUgcmVkICd4JyBidXR0b24gdG8gZ2V0IGhlbHAgYmVmb3JlIHRoZSBicmVhayoKCiMgT3B0aW9uYWwgY29udGVudAoKPGRldGFpbHM+CjxzdW1tYXJ5PipDbGljayBmb3IgZml0dGluZyBhIG1vZGVsIHRoYXQgaW5jbHVkZXMgYSBjb3ZhcmlhdGUqPC9zdW1tYXJ5PgoKSWYgeW91IGV4ZWN1dGVkIHRoZSBjb21tYW5kcyBpbiB0aGUgYWRkaXRpb25hbCBjb250ZW50IHNlY3Rpb24gZnJvbSBNb2R1bGUgMDcsIHlvdSBjYW4gZml0IGEgc2VwYXJhdGUgREVTZXEyIG1vZGVsIGZvciB0aGUgYmF0Y2ggZXhhbXBsZS4KCmBgYHtyIEZpdE1vZGVsQ292YXJpYXRlfQpkZHNfYmF0Y2ggPSBERVNlcShkZHNfYmF0Y2gpCnJlc3VsdHNOYW1lcyhkZHNfYmF0Y2gpCmBgYAoKSWYgeW91IHJ1biB0aHJvdWdoIHRoZSBvcHRpb25hbCBleGVyY2lzZXMsIHlvdSBjYW4gZXhwbG9yZSB0aGUgaW1wYWN0IG9mIGFkZGluZyBhIGNvdmFyaWF0ZSBieSBzdWJzdGl0dXRpbmcgYGRkc19wYXRpZW50YCBmb3IgYGRkc2AgYW5kIHJlLXJ1bm5pbmcgdGhvc2UgY29tbWFuZHMgc2luY2UgYm90aCBERVNlcTIgb2JqZWN0cyBoYXZlIHRoZWlyIGRhdGEgb3JnYW5pemVkIGluIHRoZSBzYW1lIHdheS4KPC9kZXRhaWxzPgoKPGJyPgoKIyBTdW1tYXJ5CgpJbiB0aGlzIHNlY3Rpb24sIHdlOgoKKiBMZWFybmVkIGFib3V0IGNvdW50IG5vcm1hbGl6YXRpb25zIGFuZCB1c2VzCiogR2VuZXJhdGVkIGEgbm9ybWFsaXplZCBjb3VudCB0YWJsZQoqIEZpdCB0d28gREVTZXEyIG1vZGVscyBmb3Igb3VyIGRhdGEKKiBTYXcgdGhlIGltcGFjdCBvZiBpbmNsdWRpbmcgYSBjb3ZhcmlhdGUgaW4gb3VyIG1vZGVsCgotLS0KCiMgU291cmNlcwoKVHJhaW5pbmcgcmVzb3VyY2VzIHVzZWQgdG8gZGV2ZWxvcCBtYXRlcmlhbHMKCiogSEJDIERHRSBzZXR1cDogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDFfREdFX3NldHVwX2FuZF9vdmVydmlldy5odG1sCiogSEJDIENvdW50IE5vcm1hbGl6YXRpb246IGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzAyX0RHRV9jb3VudF9ub3JtYWxpemF0aW9uLmh0bWwKKiBERVNlcTIgc3RhbmRhcmQgdmlnbmV0dGU6IGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbAoqIERFU2VxMiBiZWdpbm5lcnMgdmlnbmV0dGU6IGh0dHBzOi8vYmlvYy5pc20uYWMuanAvcGFja2FnZXMvMi4xNC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvYmVnaW5uZXIucGRmCiogQmlvY29uZHVjdG9yIFJOQS1zZXEgV29ya2Zsb3dzOiBodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL2hlbHAvY291cnNlLW1hdGVyaWFscy8yMDE1L0xlYXJuQmlvY29uZHVjdG9yRmViMjAxNS9CMDIuMV9STkFTZXEuaHRtbAoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQpgYGAKClRoZXNlIG1hdGVyaWFscyBoYXZlIGJlZW4gYWRhcHRlZCBhbmQgZXh0ZW5kZWQgZnJvbSBtYXRlcmlhbHMgbGlzdGVkIGFib3ZlLiBUaGVzZSBhcmUgb3BlbiBhY2Nlc3MgbWF0ZXJpYWxzIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgW0NyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gbGljZW5zZSAoQ0MgQlkgNC4wKV0oaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyksIHdoaWNoIHBlcm1pdHMgdW5yZXN0cmljdGVkIHVzZSwgZGlzdHJpYnV0aW9uLCBhbmQgcmVwcm9kdWN0aW9uIGluIGFueSBtZWRpdW0sIHByb3ZpZGVkIHRoZSBvcmlnaW5hbCBhdXRob3IgYW5kIHNvdXJjZSBhcmUgY3JlZGl0ZWQuCg==