Objectives
- Discuss count normalizations
- Execute model fitting for differential expression comparisons
Differential Expression Workflow
Here we will proceed with count normalizations and fit our DESeq2
model.
Count normalizations
Since counts of mapped reads for each gene is proportional to the
expression of RNA in addition to many “uninteresting” other factors,
normalization is the process of scaling raw count values to account for
the “uninteresting” factors and ensure expression levels are more
comparable.
Normalization goals
Two common factors that need to be accounted for during normalization
are sequencing depth and gene length.
Common normalization approaches (such as FPKM, RPKM, CPM, TPM, etc.)
account for one or both of these factors.
- Sequencing depth normalization is necessary to
account for the proportion of reads per gene expected for more deeply
sequenced samples (like in pink below) versus a less deeply sequenced
sample (like in green below).
- Gene length normalization may also be necessary if
comparing between different genes, since genes of different lengths have
different probabilities of generating fragments that end up in the
library.
Note: The above figure was originally from a HBC
tutorial that also includes a detailed comparison of different
normalization (CPM, TPM, FPKM) approaches and their best uses.
Check-in: Questions about normalizations?
DESeq2 normalizations
DESeq2 has an internal
normalization process that accounts for RNA
composition. A few highly differentially expressed genes,
differences in the number of genes expressed between samples, or
contamination are not accounted for by depth or gene length
normalization methods. Accounting for RNA composition is particularly
important for differential expression analyses, regardless of the tool
used.
For data exploration and visualizations, it is helpful to generate an
object of independently normalized counts. We will use the rlog
transformation from DESeq2 that accounts for sequencing depth for
each sample and RNA composition for the downstream quality control
visualizations.
The rlog transformation produces log2 scaled data that has also been
normalized to overall library size as well as variance across genes at
different mean expression levels. For larger numbers of samples, there
is an alternative transformation method, vst
that can be used instead for count normalizations.
The command to generate the normalized count object has a few parts,
including dds
as an input and providing a value to the
option blind
. For our purposes, we set
blind = TRUE
because we want to compare samples in
downstream QC plots in an unbiased manner.
rld = rlog(dds, blind = TRUE)
Next, we’ll look at the results of the transformation by extracting
the values with the assay()
function.
head(assay(rld), 2)
sample_A sample_B sample_C sample_D sample_E sample_F
ENSMUSG00000000001 10.51433 10.3622 10.41675 10.84821 10.40736 10.58012
ENSMUSG00000000003 0.00000 0.0000 0.00000 0.00000 0.00000 0.00000
Looking at the rld values, we can see that they are now in log scale.
Since we set blind=TRUE
, the transformation is blind to the
sample information we specified in the design formula. The normalized
counts are helpful for visualization methods during expression-level
quality assessment but aren’t used in the model
fitting.
We’ll come back to these normalized data, but first let’s write out
both the raw and normalized count tables to file.
Output count tables
To output the raw counts, we will need to use the counts
function to access the count table from within its larger
DESeqDataSet
object.
write.csv(counts(dds, normalized = FALSE), file="outputs/tables/DESeq2_raw_counts.csv")
Then we’ll output the rlog count table, using the assay
function to access the normalized count table from within its larger
DESeqDataSet
object.
write.csv(assay(rld), file="outputs/tables/DESeq2_rlog_normalized_counts.csv")
DESeq2 Model Fitting
Next, we’ll fit our standard model using the DESeq
function and take a look at the objects we generate. This command
applies the model to our data, using the sample information supplied
when generating the initial dds
object so can take some
time to run.
dds_fitted = DESeq(dds_filtered)
dds_fitted
class: DESeqDataSet
dim: 16249 6
metadata(1): version
assays(4): counts mu H cooks
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ... ENSMUSG00000118651
ENSMUSG00000118653
rowData names(22): baseMean baseVar ... deviance maxCooks
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(3): genotype condition sizeFactor
Dispersion models and possible warning messages
Depending on the data set you are analyzing, you may see a warning
that the default ‘parametric’ dispersion model so a local regression was
substituted. When seeing this warning, we recommend looking at a
dispersion plot with the plotDispEsts(dds)
function, but as
this bioconductor
thread discusses, other visualizations of our data might be more
helpful and/or easier to interpret why the data doesn’t fit the default
parametric model.
Notice that there is now more information in the
DESeqDataSet
object than there was prior to our
normalization. There is information about the model fit and about the
library size normalization. DESeq2 will use this information when we
perform the test for differential expression.
The DESeq()
function is actually doing three things
automatically for us. It calculates:
- The size factors to normalize for library size with
estimateSizeFactors(dds_filtered)
,
- Dispersion estimates to shrink the dispersions with
estimateDispersions(dds_filtered)
, and
- The Wald test statistics with
nbinomWaldTest(dds_filtered)
.
The resultsNames()
function returns the names of the
estimated effects of the model.
resultsNames(dds_fitted)
[1] "Intercept" "condition_minus_vs_plus"
The results include the single comparison representing the two levels
of condition
. If there were more levels in the
condition
column, there would be more results listed here
because DESeq2 would implicitly compare all other levels to the
reference level.
Checkpoint: If you see the same results when you
execute resultsNames(dds_filtered)
, please indicate with
the green ‘yes’ button. Otherwise, please use the red ‘x’ button to get
help before the break
Save fitted model and data in Robj
It can be useful to save key R objects to file as we proceed through
our analysis - before we do that, let’s look at the documentation for
the save
function to see if it does what we want.
?save
As we can see from the documentation the function save
writes an “external representation” of R objects that can be read back
from the file at a later date by using the function load
or
attach
in most cases. We’ll proceed with saving our
dds_fitted
object, creating a subdirectory first.
dir.create("outputs/Robjs", recursive=TRUE)
Warning in dir.create("outputs/Robjs", recursive = TRUE): 'outputs/Robjs' already exists
save(dds_fitted,
file="outputs/Robjs/dds_fitted.Robj")
Optional content
Click to fit a model that includes a covariate
If you executed the commands in the additional content section from
Module 07, you can fit a separate DESeq2 model for the batch
example.
dds_batch_fitted = DESeq(dds_batch)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resultsNames(dds_batch_fitted)
[1] "Intercept" "batch_Day2_vs_Day1" "batch_Day3_vs_Day1"
[4] "condition_minus_vs_plus"
Summary
In this section, we:
- Learned about count normalizations and uses
- Generated a normalized count table
- Fit two DESeq2 models for our data
- (Optional) - saw the impact of including a covariate in our
model
- Wrote intermediate data to file
Sources
Training resources used to develop materials
These materials have been adapted and extended from materials listed
above. These are open access materials distributed under the terms of
the Creative
Commons Attribution license (CC BY 4.0), which permits unrestricted
use, distribution, and reproduction in any medium, provided the original
author and source are credited.
LS0tCnRpdGxlOiAiTW9kdWxlIDA4OiBERSBOb3JtYWxpemF0aW9uIGFuZCBNb2RlbGluZyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA4LSIpCmBgYAoKPiAjIE9iamVjdGl2ZXMgey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiAqIERpc2N1c3MgY291bnQgbm9ybWFsaXphdGlvbnMKPiAqIEV4ZWN1dGUgbW9kZWwgZml0dGluZyBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIHByb2NlZWQgd2l0aCBjb3VudCBub3JtYWxpemF0aW9ucyBhbmQgZml0IG91ciBERVNlcTIgbW9kZWwuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERVNlcTJERS5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBDb3VudCBub3JtYWxpemF0aW9ucwoKU2luY2UgY291bnRzIG9mIG1hcHBlZCByZWFkcyBmb3IgZWFjaCBnZW5lIGlzIHByb3BvcnRpb25hbCB0byB0aGUgZXhwcmVzc2lvbiBvZiBSTkEgaW4gYWRkaXRpb24gdG8gbWFueSDigJx1bmludGVyZXN0aW5n4oCdIG90aGVyIGZhY3RvcnMsIG5vcm1hbGl6YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2Ygc2NhbGluZyByYXcgY291bnQgdmFsdWVzIHRvIGFjY291bnQgZm9yIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIGZhY3RvcnMgYW5kIGVuc3VyZSBleHByZXNzaW9uIGxldmVscyBhcmUgbW9yZSBjb21wYXJhYmxlLgoKIyMgTm9ybWFsaXphdGlvbiBnb2FscwoKVHdvIGNvbW1vbiBmYWN0b3JzIHRoYXQgbmVlZCB0byBiZSBhY2NvdW50ZWQgZm9yIGR1cmluZyBub3JtYWxpemF0aW9uIGFyZSAqKnNlcXVlbmNpbmcgZGVwdGgqKiBhbmQgKipnZW5lIGxlbmd0aCoqLiBDb21tb24gbm9ybWFsaXphdGlvbiBhcHByb2FjaGVzIChzdWNoIGFzIEZQS00sIFJQS00sIENQTSwgVFBNLCBldGMuKSBhY2NvdW50IGZvciBvbmUgb3IgYm90aCBvZiB0aGVzZSBmYWN0b3JzLgoKKiAqKlNlcXVlbmNpbmcgZGVwdGgqKiBub3JtYWxpemF0aW9uIGlzIG5lY2Vzc2FyeSB0byBhY2NvdW50IGZvciB0aGUgcHJvcG9ydGlvbiBvZiByZWFkcyBwZXIgZ2VuZSBleHBlY3RlZCBmb3IgbW9yZSBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZXMgKGxpa2UgaW4gcGluayBiZWxvdykgdmVyc3VzIGEgbGVzcyBkZWVwbHkgc2VxdWVuY2VkIHNhbXBsZSAobGlrZSBpbiBncmVlbiBiZWxvdykuCgohWypOb3RlIHRoYXQgZWFjaCBwaW5rIG9yIGdyZWVuIHJlY3RhbmdsZSByZXByZXNlbnRzIGFuIGFsaWduZWQgcmVhZCwgd2l0aCByZWFkcyBzcGFubmluZyBhbiBpbnRyb24gY29ubmVjdGVkIGJ5IGEgZGFzaGVkIGxpbmUuIEZpZ3VyZSBmcm9tIFtIQkMgIHRyYWluaW5nIG1hdGVyaWFsc10oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkqXSguL2ltYWdlcy9ub3JtYWxpemF0aW9uX21ldGhvZHNfZGVwdGgucG5nKXt3aWR0aD03NSV9CgoqICoqR2VuZSBsZW5ndGgqKiBub3JtYWxpemF0aW9uIG1heSBhbHNvIGJlIG5lY2Vzc2FyeSBpZiBjb21wYXJpbmcgYmV0d2VlbiBkaWZmZXJlbnQgZ2VuZXMsIHNpbmNlIGdlbmVzIG9mIGRpZmZlcmVudCBsZW5ndGhzIGhhdmUgZGlmZmVyZW50IHByb2JhYmlsaXRpZXMgb2YgZ2VuZXJhdGluZyBmcmFnbWVudHMgdGhhdCBlbmQgdXAgaW4gdGhlIGxpYnJhcnkuCgo+ICoqTm90ZSoqOiBUaGUgYWJvdmUgZmlndXJlIHdhcyBvcmlnaW5hbGx5IGZyb20gYSBbSEJDICB0dXRvcmlhbF0oaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbCkgdGhhdCBhbHNvIGluY2x1ZGVzIGEgZGV0YWlsZWQgY29tcGFyaXNvbiBvZiBkaWZmZXJlbnQgbm9ybWFsaXphdGlvbiAoQ1BNLCBUUE0sIEZQS00pIGFwcHJvYWNoZXMgYW5kIHRoZWlyIGJlc3QgdXNlcy4KCioqQ2hlY2staW46IFF1ZXN0aW9ucyBhYm91dCBub3JtYWxpemF0aW9ucz8qKgoKIyMgREVTZXEyIG5vcm1hbGl6YXRpb25zCgpERVNlcTIgaGFzIGFuIFtpbnRlcm5hbCBub3JtYWxpemF0aW9uIHByb2Nlc3NdKGh0dHBzOi8vZ2Vub21lYmlvbG9neS5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2L2diLTIwMTAtMTEtMTAtcjEwNikgdGhhdCBhY2NvdW50cyBmb3IgKipSTkEgY29tcG9zaXRpb24qKi4gQSBmZXcgaGlnaGx5IGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcywgZGlmZmVyZW5jZXMgaW4gdGhlIG51bWJlciBvZiBnZW5lcyBleHByZXNzZWQgYmV0d2VlbiBzYW1wbGVzLCBvciBjb250YW1pbmF0aW9uIGFyZSBub3QgYWNjb3VudGVkIGZvciBieSBkZXB0aCBvciBnZW5lIGxlbmd0aCBub3JtYWxpemF0aW9uIG1ldGhvZHMuIEFjY291bnRpbmcgZm9yIFJOQSBjb21wb3NpdGlvbiBpcyBwYXJ0aWN1bGFybHkgaW1wb3J0YW50IGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNlcywgcmVnYXJkbGVzcyBvZiB0aGUgdG9vbCB1c2VkLgoKRm9yIGRhdGEgZXhwbG9yYXRpb24gYW5kIHZpc3VhbGl6YXRpb25zLCBpdCBpcyBoZWxwZnVsIHRvIGdlbmVyYXRlIGFuIG9iamVjdCBvZiBpbmRlcGVuZGVudGx5IG5vcm1hbGl6ZWQgY291bnRzLiBXZSB3aWxsIHVzZSB0aGUgW3Jsb2cgdHJhbnNmb3JtYXRpb25dKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNjb3VudC1kYXRhLXRyYW5zZm9ybWF0aW9ucykgZnJvbSBERVNlcTIgdGhhdCBhY2NvdW50cyBmb3Igc2VxdWVuY2luZyBkZXB0aCBmb3IgZWFjaCBzYW1wbGUgYW5kIFJOQSBjb21wb3NpdGlvbiBmb3IgdGhlIGRvd25zdHJlYW0gcXVhbGl0eSBjb250cm9sIHZpc3VhbGl6YXRpb25zLgoKVGhlIHJsb2cgdHJhbnNmb3JtYXRpb24gcHJvZHVjZXMgbG9nMiBzY2FsZWQgZGF0YSB0aGF0IGhhcyBhbHNvIGJlZW4gbm9ybWFsaXplZCB0byBvdmVyYWxsIGxpYnJhcnkgc2l6ZSBhcyB3ZWxsIGFzIHZhcmlhbmNlIGFjcm9zcyBnZW5lcyBhdCBkaWZmZXJlbnQgbWVhbiBleHByZXNzaW9uIGxldmVscy4gRm9yIGxhcmdlciBudW1iZXJzIG9mIHNhbXBsZXMsIHRoZXJlIGlzIGFuIGFsdGVybmF0aXZlIHRyYW5zZm9ybWF0aW9uIG1ldGhvZCwgW3ZzdF0oaHR0cDovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2NvdW50LWRhdGEtdHJhbnNmb3JtYXRpb25zKSB0aGF0IGNhbiBiZSB1c2VkIGluc3RlYWQgZm9yIGNvdW50IG5vcm1hbGl6YXRpb25zLgoKVGhlIGNvbW1hbmQgdG8gZ2VuZXJhdGUgdGhlIG5vcm1hbGl6ZWQgY291bnQgb2JqZWN0IGhhcyBhIGZldyBwYXJ0cywgaW5jbHVkaW5nIGBkZHNgIGFzIGFuIGlucHV0IGFuZCBwcm92aWRpbmcgYSB2YWx1ZSB0byB0aGUgb3B0aW9uIGBibGluZGAuIEZvciBvdXIgcHVycG9zZXMsIHdlIHNldCBgYmxpbmQgPSBUUlVFYCBiZWNhdXNlIHdlIHdhbnQgdG8gY29tcGFyZSBzYW1wbGVzIGluIGRvd25zdHJlYW0gUUMgcGxvdHMgaW4gYW4gdW5iaWFzZWQgbWFubmVyLgoKYGBge3IgQ291bnROb3JtfQpybGQgPSBybG9nKGRkcywgYmxpbmQgPSBUUlVFKQpgYGAKCk5leHQsIHdlJ2xsIGxvb2sgYXQgdGhlIHJlc3VsdHMgb2YgdGhlIHRyYW5zZm9ybWF0aW9uIGJ5IGV4dHJhY3RpbmcgdGhlIHZhbHVlcyB3aXRoIHRoZSBgYXNzYXkoKWAgZnVuY3Rpb24uCgpgYGB7ciBDb3VudE5vcm1DaGVja30KaGVhZChhc3NheShybGQpLCAyKQpgYGAKCkxvb2tpbmcgYXQgdGhlIHJsZCB2YWx1ZXMsIHdlIGNhbiBzZWUgdGhhdCB0aGV5IGFyZSBub3cgaW4gbG9nIHNjYWxlLiBTaW5jZSB3ZSBzZXQgYGJsaW5kPVRSVUVgLCB0aGUgdHJhbnNmb3JtYXRpb24gaXMgYmxpbmQgdG8gdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiB3ZSBzcGVjaWZpZWQgaW4gdGhlIGRlc2lnbiBmb3JtdWxhLiBUaGUgbm9ybWFsaXplZCBjb3VudHMgYXJlIGhlbHBmdWwgZm9yIHZpc3VhbGl6YXRpb24gbWV0aG9kcyBkdXJpbmcgZXhwcmVzc2lvbi1sZXZlbCBxdWFsaXR5IGFzc2Vzc21lbnQgYnV0ICoqYXJlbid0IHVzZWQgaW4gdGhlIG1vZGVsIGZpdHRpbmcqKi4KCldlJ2xsIGNvbWUgYmFjayB0byB0aGVzZSBub3JtYWxpemVkIGRhdGEsIGJ1dCBmaXJzdCBsZXQncyB3cml0ZSBvdXQgYm90aCB0aGUgcmF3IGFuZCBub3JtYWxpemVkIGNvdW50IHRhYmxlcyB0byBmaWxlLgoKIyMjIE91dHB1dCBjb3VudCB0YWJsZXMKClRvIG91dHB1dCB0aGUgcmF3IGNvdW50cywgd2Ugd2lsbCBuZWVkIHRvIHVzZSB0aGUgYGNvdW50c2AgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBjb3VudCB0YWJsZSBmcm9tIHdpdGhpbiBpdHMgbGFyZ2VyIGBERVNlcURhdGFTZXRgIG9iamVjdC4KCmBgYHtyIE91dHB1dENvdW50c1JhdywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoY291bnRzKGRkcywgbm9ybWFsaXplZCA9IEZBTFNFKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jhd19jb3VudHMuY3N2IikKYGBgCgpUaGVuIHdlJ2xsIG91dHB1dCB0aGUgcmxvZyBjb3VudCB0YWJsZSwgdXNpbmcgdGhlIGBhc3NheWAgZnVuY3Rpb24gdG8gYWNjZXNzIHRoZSBub3JtYWxpemVkIGNvdW50IHRhYmxlIGZyb20gd2l0aGluIGl0cyBsYXJnZXIgYERFU2VxRGF0YVNldGAgb2JqZWN0LgoKYGBge3IgT3V0cHV0Q291bnRzUmxvZywgZXZhbCA9IEZBTFNFfQp3cml0ZS5jc3YoYXNzYXkocmxkKSwgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVTZXEyX3Jsb2dfbm9ybWFsaXplZF9jb3VudHMuY3N2IikKYGBgCgoKIyBERVNlcTIgTW9kZWwgRml0dGluZwoKTmV4dCwgd2UnbGwgZml0IG91ciBzdGFuZGFyZCBtb2RlbCB1c2luZyB0aGUgYERFU2VxYCBmdW5jdGlvbiBhbmQgdGFrZSBhIGxvb2sgYXQgdGhlIG9iamVjdHMgd2UgZ2VuZXJhdGUuIFRoaXMgY29tbWFuZCBhcHBsaWVzIHRoZSBtb2RlbCB0byBvdXIgZGF0YSwgdXNpbmcgdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiBzdXBwbGllZCB3aGVuIGdlbmVyYXRpbmcgdGhlIGluaXRpYWwgYGRkc2Agb2JqZWN0IHNvIGNhbiB0YWtlIHNvbWUgdGltZSB0byBydW4uCgpgYGB7ciBGaXRNb2RlbFN0YW5kYXJkLCBtZXNzYWdlPUZBTFNFfQpkZHNfZml0dGVkID0gREVTZXEoZGRzX2ZpbHRlcmVkKQpkZHNfZml0dGVkCmBgYAoKPGRldGFpbHM+CiAgICA8c3VtbWFyeT4qRGlzcGVyc2lvbiBtb2RlbHMgYW5kIHBvc3NpYmxlIHdhcm5pbmcgbWVzc2FnZXMqPC9zdW1tYXJ5PgoKICAgIERlcGVuZGluZyBvbiB0aGUgZGF0YSBzZXQgeW91IGFyZSBhbmFseXppbmcsIHlvdSBtYXkgc2VlIGEgd2FybmluZyB0aGF0IHRoZSBkZWZhdWx0ICdwYXJhbWV0cmljJyBkaXNwZXJzaW9uIG1vZGVsIHNvIGEgbG9jYWwgcmVncmVzc2lvbiB3YXMgc3Vic3RpdHV0ZWQuIFdoZW4gc2VlaW5nIHRoaXMgd2FybmluZywgd2UgcmVjb21tZW5kIGxvb2tpbmcgYXQgYSBkaXNwZXJzaW9uIHBsb3Qgd2l0aCB0aGUgYHBsb3REaXNwRXN0cyhkZHMpYCBmdW5jdGlvbiwgYnV0IGFzIFt0aGlzIGJpb2NvbmR1Y3RvciB0aHJlYWRdKGh0dHBzOi8vc3VwcG9ydC5iaW9jb25kdWN0b3Iub3JnL3AvMTA3OTM3LykgZGlzY3Vzc2VzLCBvdGhlciB2aXN1YWxpemF0aW9ucyBvZiBvdXIgZGF0YSBtaWdodCBiZSBtb3JlIGhlbHBmdWwgYW5kL29yIGVhc2llciB0byBpbnRlcnByZXQgd2h5IHRoZSBkYXRhIGRvZXNuJ3QgZml0IHRoZSBkZWZhdWx0IHBhcmFtZXRyaWMgbW9kZWwuCgo8L2RldGFpbHM+Cjxicj4KCgpOb3RpY2UgdGhhdCB0aGVyZSBpcyBub3cgbW9yZSBpbmZvcm1hdGlvbiBpbiB0aGUgYERFU2VxRGF0YVNldGAgb2JqZWN0IHRoYW4gdGhlcmUgd2FzIHByaW9yIHRvIG91ciBub3JtYWxpemF0aW9uLiBUaGVyZSBpcyBpbmZvcm1hdGlvbiBhYm91dCB0aGUgbW9kZWwgZml0IGFuZCBhYm91dCB0aGUgbGlicmFyeSBzaXplIG5vcm1hbGl6YXRpb24uIERFU2VxMiB3aWxsIHVzZSB0aGlzIGluZm9ybWF0aW9uIHdoZW4gd2UgcGVyZm9ybSB0aGUgdGVzdCBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24uCgpUaGUgYERFU2VxKClgIGZ1bmN0aW9uIGlzIGFjdHVhbGx5IGRvaW5nIHRocmVlIHRoaW5ncyBhdXRvbWF0aWNhbGx5IGZvciB1cy4gSXQgY2FsY3VsYXRlczoKCjEuIFRoZSBzaXplIGZhY3RvcnMgdG8gbm9ybWFsaXplIGZvciBsaWJyYXJ5IHNpemUgd2l0aCBgZXN0aW1hdGVTaXplRmFjdG9ycyhkZHNfZmlsdGVyZWQpYCwKMi4gRGlzcGVyc2lvbiBlc3RpbWF0ZXMgdG8gc2hyaW5rIHRoZSBkaXNwZXJzaW9ucyB3aXRoIGBlc3RpbWF0ZURpc3BlcnNpb25zKGRkc19maWx0ZXJlZClgLCBhbmQKMy4gVGhlIFdhbGQgdGVzdCBzdGF0aXN0aWNzIHdpdGggYG5iaW5vbVdhbGRUZXN0KGRkc19maWx0ZXJlZClgLgoKVGhlIGByZXN1bHRzTmFtZXMoKWAgZnVuY3Rpb24gcmV0dXJucyB0aGUgbmFtZXMgb2YgdGhlIGVzdGltYXRlZCBlZmZlY3RzIG9mIHRoZSBtb2RlbC4KCmBgYHtyIEZpdE1vZGVsU3RhbmRhcmRDaGVjazEsIGV2YWw9VFJVRX0KcmVzdWx0c05hbWVzKGRkc19maXR0ZWQpCmBgYAoKVGhlIHJlc3VsdHMgaW5jbHVkZSB0aGUgc2luZ2xlIGNvbXBhcmlzb24gcmVwcmVzZW50aW5nIHRoZSB0d28gbGV2ZWxzIG9mIGBjb25kaXRpb25gLiBJZiB0aGVyZSB3ZXJlIG1vcmUgbGV2ZWxzIGluIHRoZSBgY29uZGl0aW9uYCBjb2x1bW4sIHRoZXJlIHdvdWxkIGJlIG1vcmUgcmVzdWx0cyBsaXN0ZWQgaGVyZSBiZWNhdXNlIERFU2VxMiB3b3VsZCBpbXBsaWNpdGx5IGNvbXBhcmUgYWxsIG90aGVyIGxldmVscyB0byB0aGUgcmVmZXJlbmNlIGxldmVsLgoKCioqQ2hlY2twb2ludCoqOiAqSWYgeW91IHNlZSB0aGUgc2FtZSByZXN1bHRzIHdoZW4geW91IGV4ZWN1dGUgYHJlc3VsdHNOYW1lcyhkZHNfZmlsdGVyZWQpYCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICd5ZXMnIGJ1dHRvbi4gT3RoZXJ3aXNlLCBwbGVhc2UgdXNlIHRoZSByZWQgJ3gnIGJ1dHRvbiB0byBnZXQgaGVscCBiZWZvcmUgdGhlIGJyZWFrKgoKIyMgU2F2ZSBmaXR0ZWQgbW9kZWwgYW5kIGRhdGEgaW4gUm9iagoKSXQgY2FuIGJlIHVzZWZ1bCB0byBzYXZlIGtleSBSIG9iamVjdHMgdG8gZmlsZSBhcyB3ZSBwcm9jZWVkIHRocm91Z2ggb3VyIGFuYWx5c2lzIC0gYmVmb3JlIHdlIGRvIHRoYXQsIGxldCdzIGxvb2sgYXQgdGhlIGRvY3VtZW50YXRpb24gZm9yIHRoZSBgc2F2ZWAgZnVuY3Rpb24gdG8gc2VlIGlmIGl0IGRvZXMgd2hhdCB3ZSB3YW50LgoKYGBge3IgU2F2ZURvY3VtZW50YXRpb259Cj9zYXZlCmBgYAoKQXMgd2UgY2FuIHNlZSBmcm9tIHRoZSBkb2N1bWVudGF0aW9uIHRoZSBmdW5jdGlvbiBgc2F2ZWAgd3JpdGVzIGFuICJleHRlcm5hbCByZXByZXNlbnRhdGlvbiIgb2YgUiBvYmplY3RzIHRoYXQgY2FuIGJlIHJlYWQgYmFjayBmcm9tIHRoZSBmaWxlIGF0IGEgbGF0ZXIgZGF0ZSBieSB1c2luZyB0aGUgZnVuY3Rpb24gYGxvYWRgIG9yIGBhdHRhY2hgIGluIG1vc3QgY2FzZXMuIFdlJ2xsIHByb2NlZWQgd2l0aCBzYXZpbmcgb3VyIGBkZHNfZml0dGVkYCBvYmplY3QsIGNyZWF0aW5nIGEgc3ViZGlyZWN0b3J5IGZpcnN0LgoKYGBge3IgV3JpdGVGaXRNb2RlbE9iamVjdH0KZGlyLmNyZWF0ZSgib3V0cHV0cy9Sb2JqcyIsIHJlY3Vyc2l2ZT1UUlVFKQpzYXZlKGRkc19maXR0ZWQsCiAgICAgICAgICBmaWxlPSJvdXRwdXRzL1JvYmpzL2Rkc19maXR0ZWQuUm9iaiIpCmBgYAoKCiMgT3B0aW9uYWwgY29udGVudAoKPGRldGFpbHM+CjxzdW1tYXJ5PipDbGljayB0byBmaXQgYSBtb2RlbCB0aGF0IGluY2x1ZGVzIGEgY292YXJpYXRlKjwvc3VtbWFyeT4KCklmIHlvdSBleGVjdXRlZCB0aGUgY29tbWFuZHMgaW4gdGhlIGFkZGl0aW9uYWwgY29udGVudCBzZWN0aW9uIGZyb20gTW9kdWxlIDA3LCB5b3UgY2FuIGZpdCBhIHNlcGFyYXRlIERFU2VxMiBtb2RlbCBmb3IgdGhlIGJhdGNoIGV4YW1wbGUuCgpgYGB7ciBGaXRNb2RlbENvdmFyaWF0ZX0KZGRzX2JhdGNoX2ZpdHRlZCA9IERFU2VxKGRkc19iYXRjaCkKcmVzdWx0c05hbWVzKGRkc19iYXRjaF9maXR0ZWQpCmBgYAoKPC9kZXRhaWxzPgoKPGJyPgoKIyBTdW1tYXJ5CgpJbiB0aGlzIHNlY3Rpb24sIHdlOgoKKiBMZWFybmVkIGFib3V0IGNvdW50IG5vcm1hbGl6YXRpb25zIGFuZCB1c2VzCiogR2VuZXJhdGVkIGEgbm9ybWFsaXplZCBjb3VudCB0YWJsZQoqIEZpdCB0d28gREVTZXEyIG1vZGVscyBmb3Igb3VyIGRhdGEKKiAoT3B0aW9uYWwpIC0gc2F3IHRoZSBpbXBhY3Qgb2YgaW5jbHVkaW5nIGEgY292YXJpYXRlIGluIG91ciBtb2RlbAoqIFdyb3RlIGludGVybWVkaWF0ZSBkYXRhIHRvIGZpbGUKCi0tLQoKIyBTb3VyY2VzCgpUcmFpbmluZyByZXNvdXJjZXMgdXNlZCB0byBkZXZlbG9wIG1hdGVyaWFscwoKKiBIQkMgREdFIHNldHVwOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wMV9ER0Vfc2V0dXBfYW5kX292ZXJ2aWV3Lmh0bWwKKiBIQkMgQ291bnQgTm9ybWFsaXphdGlvbjogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbAoqIERFU2VxMiBzdGFuZGFyZCB2aWduZXR0ZTogaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sCiogREVTZXEyIGJlZ2lubmVycyB2aWduZXR0ZTogaHR0cHM6Ly9iaW9jLmlzbS5hYy5qcC9wYWNrYWdlcy8yLjE0L2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9iZWdpbm5lci5wZGYKKiBCaW9jb25kdWN0b3IgUk5BLXNlcSBXb3JrZmxvd3M6IGh0dHBzOi8vd3d3LmJpb2NvbmR1Y3Rvci5vcmcvaGVscC9jb3Vyc2UtbWF0ZXJpYWxzLzIwMTUvTGVhcm5CaW9jb25kdWN0b3JGZWIyMDE1L0IwMi4xX1JOQVNlcS5odG1sCgpgYGB7ciBXcml0ZU91dC5SRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQojIEhpZGRlbiBjb2RlIGJsb2NrIHRvIHdyaXRlIG91dCBkYXRhIGZvciBrbml0dGluZwojIHNhdmUuaW1hZ2UoZmlsZSA9ICJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKLS0tCgpUaGVzZSBtYXRlcmlhbHMgaGF2ZSBiZWVuIGFkYXB0ZWQgYW5kIGV4dGVuZGVkIGZyb20gbWF0ZXJpYWxzIGxpc3RlZCBhYm92ZS4gVGhlc2UgYXJlIG9wZW4gYWNjZXNzIG1hdGVyaWFscyBkaXN0cmlidXRlZCB1bmRlciB0aGUgdGVybXMgb2YgdGhlIFtDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uIGxpY2Vuc2UgKENDIEJZIDQuMCldKGh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LzQuMC8pLCB3aGljaCBwZXJtaXRzIHVucmVzdHJpY3RlZCB1c2UsIGRpc3RyaWJ1dGlvbiwgYW5kIHJlcHJvZHVjdGlvbiBpbiBhbnkgbWVkaXVtLCBwcm92aWRlZCB0aGUgb3JpZ2luYWwgYXV0aG9yIGFuZCBzb3VyY2UgYXJlIGNyZWRpdGVkLgo=