Objectives

  • Understand advantages of using gene ids when analyzing RNA-seq data.
  • Given a list of ENSEMBL gene ids, add gene symbols and Entrez accessions.
  • Generate common visualizations for differential expression comparisons
  • Understand choosing thresholds for calling differentially expressed genes
  • Discuss options for functional enrichments

Differential Expression Workflow

Here we will generate summary figures for our results and annotate our DE tables.


Summarizing DE results

Part of differential expression analysis is generating visualizations and summary tables to share our results. While the DESeq2 vignette provides examples of other visualizations, a common visualization to summarize DE comparisons are volcano plots.

Tabular DE summary

To generate a general summary of the DE results, we can use the summary function to generate a basic summary by DESeq2.

summary(results_minus_vs_plus)

out of 16249 with nonzero total read count
adjusted p-value < 0.1
LFC > 0 (up)       : 53, 0.33%
LFC < 0 (down)     : 38, 0.23%
outliers [1]       : 72, 0.44%
low counts [2]     : 3781, 23%
(mean count < 7)
[1] see 'cooksCutoff' argument of ?results
[2] see 'independentFiltering' argument of ?results

However, this summary is simply a text output that we are unable to manipulate. Moreover, notice that the thresholds are not quite as we would like them.

To create our own summary, we first need to choose thresholds. A standard threshold for the adjusted p-value is less than 0.05. A reasonable threshold for linear fold-change is less than -1.5 or greater than 1.5 (which corresponds to log2 fold-change -0.585 and 0.585, respectively). Including a fold-change threshold ensures that the DE genes have a reasonable effect size as well as statistical significance.

Let’s set these thresholds as variables to reuse. This is generally good practice because if you want to change those thresholds later then you only have to change them in one location of your script, which is faster and can reduce the risk of errors from missing some instances in your code.

fc = 1.5
fdr = 0.05

Note: Choosing thresholds

Thresholding on adjusted p-values < 0.05 is a standard threshold, but depending on the research question and/or how the results will be used, other thresholds might be reasonable.

There is a nice Biostar post that discusses choosing adjusted p-value thresholds, including cases where a more relaxed threshold might be appropriate and (some heated) discussion of the dangers of adjusting the choosen threshold after running an analysis. Additionally, there is a Dalmon et al 2012 paper about p-value and fold-change thresholds for microarray data that may help provide some context.

If we think back to Computational Foundations, conditional statements could allow us to determine the number of genes that pass our thresholds, which would be useful for annotating our results tables and plots.

Exercise

How would we identify the number of genes with adjusted p-values < 0.05 and a fold-change above 1.5 (or below -1.5)?

Solution

Here is one possible answer:

sum(results_minus_vs_plus$padj < fdr & abs(results_minus_vs_plus$log2FoldChange) >= log2(fc), na.rm = TRUE)
[1] 56

Checkpoint: If you see the same number of DE ganes with our choosen thresholds, please indicate with a green check. Otherwise use a red x.


Let’s now create a new column in results_minus_vs_plus to record the significance “call” based on these thresholds. And let’s separate the call by “Up” or “Down”, noting that these are relative to our “Case” condition of “plus”, or iron replete mice. There are many ways to accomplish this, but the following will work:

First define all values as “NS” or “not significant”:

results_minus_vs_plus$call = 'NS'
head(results_minus_vs_plus)
log2 fold change (MLE): condition minus vs plus 
Wald test p-value: condition minus vs plus 
DataFrame with 6 rows and 7 columns
                      baseMean log2FoldChange     lfcSE      stat    pvalue      padj        call
                     <numeric>      <numeric> <numeric> <numeric> <numeric> <numeric> <character>
ENSMUSG00000000001  1489.83039       0.297760  0.210310  1.415815  0.156830  0.868573          NS
ENSMUSG00000000028  1748.93544       0.226421  0.176795  1.280695  0.200301  0.902900          NS
ENSMUSG00000000031  2151.87715       0.457635  0.764579  0.598545  0.549476  0.995391          NS
ENSMUSG00000000037    24.91672       0.579130  0.561259  1.031840  0.302147  0.950613          NS
ENSMUSG00000000049     7.78377      -0.899483  1.553063 -0.579167  0.562476  0.998043          NS
ENSMUSG00000000056 19653.54030      -0.174048  0.203529 -0.855151  0.392468  0.982479          NS

Next, determine the “Up” and “Down” indices:

up_idx = results_minus_vs_plus$padj < fdr & results_minus_vs_plus$log2FoldChange > log2(fc)
down_idx = results_minus_vs_plus$padj < fdr & results_minus_vs_plus$log2FoldChange < -log2(fc)

Last, use those indices to assign the correct “Up” or “Down” values to the correct indices, and look at the head of the result:

results_minus_vs_plus$call[up_idx] = 'Up'
results_minus_vs_plus$call[down_idx] = 'Down'
head(results_minus_vs_plus)
log2 fold change (MLE): condition minus vs plus 
Wald test p-value: condition minus vs plus 
DataFrame with 6 rows and 7 columns
                      baseMean log2FoldChange     lfcSE      stat    pvalue      padj        call
                     <numeric>      <numeric> <numeric> <numeric> <numeric> <numeric> <character>
ENSMUSG00000000001  1489.83039       0.297760  0.210310  1.415815  0.156830  0.868573          NS
ENSMUSG00000000028  1748.93544       0.226421  0.176795  1.280695  0.200301  0.902900          NS
ENSMUSG00000000031  2151.87715       0.457635  0.764579  0.598545  0.549476  0.995391          NS
ENSMUSG00000000037    24.91672       0.579130  0.561259  1.031840  0.302147  0.950613          NS
ENSMUSG00000000049     7.78377      -0.899483  1.553063 -0.579167  0.562476  0.998043          NS
ENSMUSG00000000056 19653.54030      -0.174048  0.203529 -0.855151  0.392468  0.982479          NS

Finally, since plotting functions often require categorical groupings, let’s make this call column a factor and specify the level ordering:

results_minus_vs_plus$call = factor(results_minus_vs_plus$call, levels = c('Up', 'Down', 'NS'))
head(results_minus_vs_plus)
log2 fold change (MLE): condition minus vs plus 
Wald test p-value: condition minus vs plus 
DataFrame with 6 rows and 7 columns
                      baseMean log2FoldChange     lfcSE      stat    pvalue      padj     call
                     <numeric>      <numeric> <numeric> <numeric> <numeric> <numeric> <factor>
ENSMUSG00000000001  1489.83039       0.297760  0.210310  1.415815  0.156830  0.868573       NS
ENSMUSG00000000028  1748.93544       0.226421  0.176795  1.280695  0.200301  0.902900       NS
ENSMUSG00000000031  2151.87715       0.457635  0.764579  0.598545  0.549476  0.995391       NS
ENSMUSG00000000037    24.91672       0.579130  0.561259  1.031840  0.302147  0.950613       NS
ENSMUSG00000000049     7.78377      -0.899483  1.553063 -0.579167  0.562476  0.998043       NS
ENSMUSG00000000056 19653.54030      -0.174048  0.203529 -0.855151  0.392468  0.982479       NS

Tip

It is often helpful to include code like this in differential expression analyses so there is a clearly labelled column that makes subsetting and summarizing the results easier.

Now we are in a position to quickly summarize our differential expression results:

table(results_minus_vs_plus$call)

   Up  Down    NS 
   41    15 16193 

We see quickly how many genes were “Up” in iron replete, how many were “Down” in iron replete, and how many were not significant.

Checkpoint: If you successfully added the call column and got the same table result as above, please indicate with a green check. Otherwise use a red x.

Visual DE summary

As described by the Galaxy project, a volcano plot is a type of scatterplot that shows statistical significance (adjusted p-value) versus effect size (fold change). In a volcano plot, the most upregulated genes are towards the right, the most downregulated genes are towards the left, and the most statistically significant genes are towards the top.

Let’s coerce the DataFrame which was returned by DESeq2::results() into a tibble in anticipation of using the ggplot2 library to plot. We’re also going to modify our results table so that the row names become a separate column, and so that it’s ordered by adjusted p-value.

# Use the rownames argument to create a new column of gene IDs
# Also arrange by adjusted p-value
results_forPlot = as_tibble(results_minus_vs_plus, rownames = 'id') %>% arrange(padj)

Let’s start with a very simple volcano plot that plots the log2FoldChange on the x-axis, and -log10(padj) on the y-axis.

# Initialize the plot, saving as object 'p' and specifying the plot type as 'geom_point'
p = ggplot(results_forPlot, aes(x = log2FoldChange, y = -log10(padj))) +
    geom_point()
p
Warning: Removed 3853 rows containing missing values (`geom_point()`).

This is a good start, but, as usual it’s good practice to add better labels to the plot with the labs() function:

# Add plot labels and change the theme
p = ggplot(results_forPlot, aes(x = log2FoldChange, y = -log10(padj))) +
    geom_point() +
    theme_bw() +
    labs(
        title = 'Volcano Plot',
        subtitle = 'Plus vs Minus',
        x = 'log2 fold-change',
        y = '-log10 FDR'
    )
p
Warning: Removed 3853 rows containing missing values (`geom_point()`).

What if we now added some visual guides to indicate where the significant genes are? We can use the geom_vline() and geom_hline() functions to accomplish this:

# Add threshold lines
p = p +
    geom_vline(
        xintercept = c(0, -log2(fc), log2(fc)),
        linetype = c(1, 2, 2)) +
    geom_hline(
        yintercept = -log10(fdr),
        linetype = 2)
p
Warning: Removed 3853 rows containing missing values (`geom_point()`).

Finally, why not color the points by their significance status? We already created the call column that has the correct values. In this case we can get away with adding geom_point() to our existing plot and specifying the correct aesthetic:

p = p + geom_point(aes(color = call))
p
Warning: Removed 3853 rows containing missing values (`geom_point()`).
Removed 3853 rows containing missing values (`geom_point()`).

For additional visualizations for our DE results, we included some example code in the Bonus Content module and this HBC tutorial also includes some nice examples.

Subsetting significant genes

You may be interested in creating a table of only the genes that pass your significance thresholds. A useful way to do this is to conditionally subset your results. Again, we already created the call column, which makes this relatively simple to do.

Note: The tidyverse functions you learned in Software Carpentry could also be alternatively used here.

# tidyr (requires table reformatting)
res_sig <- as_tibble(results_minus_vs_plus, rownames = "gene_ids") %>% filter(call != 'NS')

# base
res_sig <- results_minus_vs_plus[results_minus_vs_plus$call != 'NS', ]

head(res_sig)
log2 fold change (MLE): condition minus vs plus 
Wald test p-value: condition minus vs plus 
DataFrame with 6 rows and 7 columns
                    baseMean log2FoldChange     lfcSE      stat      pvalue        padj     call
                   <numeric>      <numeric> <numeric> <numeric>   <numeric>   <numeric> <factor>
ENSMUSG00000001281   532.398       1.110493  0.282999   3.92402 8.70820e-05 2.45334e-02     Up  
ENSMUSG00000004562   499.728       0.750326  0.197826   3.79286 1.48922e-04 3.76743e-02     Up  
ENSMUSG00000019916   352.536       1.305994  0.302853   4.31231 1.61561e-05 7.41746e-03     Up  
ENSMUSG00000020142   624.633       1.720608  0.287145   5.99212 2.07127e-09 4.27924e-06     Up  
ENSMUSG00000020176  3086.303       1.276265  0.262113   4.86914 1.12087e-06 9.26290e-04     Up  
ENSMUSG00000020272  1749.268      -0.640126  0.168531  -3.79827 1.45712e-04 3.76300e-02     Down
dim(res_sig)
[1] 56  7

Summary

In this section, we:

  • Generated a volcano plot for our differential expression results
  • Summarized our differential expression results
  • Discussed choosing thresholds
  • Annotated our tables of results to map gene IDs to gene symbols
  • Saved our results to file

Sources



These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

LS0tCnRpdGxlOiAiTW9kdWxlIDExOiBERSBWaXN1YWxpemF0aW9uIgphdXRob3I6ICJVTSBCaW9pbmZvcm1hdGljcyBDb3JlIgpkYXRlOiAiYHIgU3lzLkRhdGUoKWAiCm91dHB1dDoKICAgICAgICBodG1sX2RvY3VtZW50OgogICAgICAgICAgICBpbmNsdWRlczoKICAgICAgICAgICAgICAgIGluX2hlYWRlcjogaGVhZGVyLmh0bWwKICAgICAgICAgICAgdGhlbWU6IHBhcGVyCiAgICAgICAgICAgIHRvYzogdHJ1ZQogICAgICAgICAgICB0b2NfZGVwdGg6IDQKICAgICAgICAgICAgdG9jX2Zsb2F0OiB0cnVlCiAgICAgICAgICAgIG51bWJlcl9zZWN0aW9uczogZmFsc2UKICAgICAgICAgICAgZmlnX2NhcHRpb246IHRydWUKICAgICAgICAgICAgbWFya2Rvd246IEdGTQogICAgICAgICAgICBjb2RlX2Rvd25sb2FkOiB0cnVlCi0tLQoKPHN0eWxlIHR5cGU9InRleHQvY3NzIj4KYm9keSwgdGQgewogICBmb250LXNpemU6IDE4cHg7Cn0KY29kZS5yewogIGZvbnQtc2l6ZTogMTJweDsKfQpwcmUgewogIGZvbnQtc2l6ZTogMTJweAp9Cjwvc3R5bGU+CgpgYGB7ciwgaW5jbHVkZSA9IEZBTFNFfQpzb3VyY2UoIi4uL2Jpbi9jaHVuay1vcHRpb25zLlIiKQprbml0cl9maWdfcGF0aCgiMTEtIikKYGBgCgo+ICMgT2JqZWN0aXZlcyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQo+ICogVW5kZXJzdGFuZCBhZHZhbnRhZ2VzIG9mIHVzaW5nIGdlbmUgaWRzIHdoZW4gYW5hbHl6aW5nIFJOQS1zZXEgZGF0YS4KPiAqIEdpdmVuIGEgbGlzdCBvZiBFTlNFTUJMIGdlbmUgaWRzLCBhZGQgZ2VuZSBzeW1ib2xzIGFuZCBFbnRyZXogYWNjZXNzaW9ucy4KPiAqIEdlbmVyYXRlIGNvbW1vbiB2aXN1YWxpemF0aW9ucyBmb3IgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gY29tcGFyaXNvbnMKPiAqIFVuZGVyc3RhbmQgY2hvb3NpbmcgdGhyZXNob2xkcyBmb3IgY2FsbGluZyBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMKPiAqIERpc2N1c3Mgb3B0aW9ucyBmb3IgZnVuY3Rpb25hbCBlbnJpY2htZW50cwoKYGBge3IgTW9kdWxlcywgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KHRpZHlyKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KG1hdHJpeFN0YXRzKQpsaWJyYXJ5KGdncmVwZWwpCmxpYnJhcnkocGhlYXRtYXApCmxpYnJhcnkoUkNvbG9yQnJld2VyKQojIGxvYWQoInJkYXRhL1J1bm5pbmdEYXRhLlJEYXRhIikKYGBgCgojIERpZmZlcmVudGlhbCBFeHByZXNzaW9uIFdvcmtmbG93IHsudW5saXN0ZWQgLnVubnVtYmVyZWR9CgpIZXJlIHdlIHdpbGwgZ2VuZXJhdGUgc3VtbWFyeSBmaWd1cmVzIGZvciBvdXIgcmVzdWx0cyBhbmQgYW5ub3RhdGUgb3VyIERFIHRhYmxlcy4KCiFbXSguL2ltYWdlcy93YXlmaW5kZXIvd2F5ZmluZGVyLURFVmlzdWFsaXphdGlvbnMucG5nKXt3aWR0aD03NSV9CgotLS0KCiMgU3VtbWFyaXppbmcgREUgcmVzdWx0cwoKUGFydCBvZiBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyBpcyBnZW5lcmF0aW5nIHZpc3VhbGl6YXRpb25zIGFuZCBzdW1tYXJ5IHRhYmxlcyB0byBzaGFyZSBvdXIgcmVzdWx0cy4gV2hpbGUgdGhlIERFU2VxMiB2aWduZXR0ZSBwcm92aWRlcyBbZXhhbXBsZXMgb2Ygb3RoZXIgdmlzdWFsaXphdGlvbnNdKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNleHBsb3JpbmctYW5kLWV4cG9ydGluZy1yZXN1bHRzKSwgYSBjb21tb24gdmlzdWFsaXphdGlvbiB0byBzdW1tYXJpemUgREUgY29tcGFyaXNvbnMgYXJlIFt2b2xjYW5vIHBsb3RzXShodHRwOi8vcmVzb3VyY2VzLnFpYWdlbmJpb2luZm9ybWF0aWNzLmNvbS9tYW51YWxzL2NsY2dlbm9taWNzd29ya2JlbmNoLzc1Mi9pbmRleC5waHA/bWFudWFsPVZvbGNhbm9fcGxvdHNfaW5zcGVjdGluZ19yZXN1bHRfc3RhdGlzdGljYWxfYW5hbHlzaXMuaHRtbCkuCgojIyBUYWJ1bGFyIERFIHN1bW1hcnkKClRvIGdlbmVyYXRlIGEgZ2VuZXJhbCBzdW1tYXJ5IG9mIHRoZSBERSByZXN1bHRzLCB3ZSBjYW4gdXNlIHRoZSBgc3VtbWFyeWAgZnVuY3Rpb24gdG8gZ2VuZXJhdGUgYSBiYXNpYyBzdW1tYXJ5IGJ5IERFU2VxMi4KCmBgYHtyIERFU2VxMlN1bW1hcnl9CnN1bW1hcnkocmVzdWx0c19taW51c192c19wbHVzKQpgYGAKCkhvd2V2ZXIsIHRoaXMgc3VtbWFyeSBpcyBzaW1wbHkgYSB0ZXh0IG91dHB1dCB0aGF0IHdlIGFyZSB1bmFibGUgdG8gbWFuaXB1bGF0ZS4gTW9yZW92ZXIsIG5vdGljZSB0aGF0IHRoZSB0aHJlc2hvbGRzIGFyZSBub3QgcXVpdGUgYXMgd2Ugd291bGQgbGlrZSB0aGVtLgoKVG8gY3JlYXRlIG91ciBvd24gc3VtbWFyeSwgd2UgZmlyc3QgbmVlZCB0byBjaG9vc2UgdGhyZXNob2xkcy4gQSBzdGFuZGFyZCB0aHJlc2hvbGQgZm9yIHRoZSBhZGp1c3RlZCBwLXZhbHVlIGlzIGxlc3MgdGhhbiAwLjA1LiBBIHJlYXNvbmFibGUgdGhyZXNob2xkIGZvciBsaW5lYXIgZm9sZC1jaGFuZ2UgaXMgbGVzcyB0aGFuIC0xLjUgb3IgZ3JlYXRlciB0aGFuIDEuNSAod2hpY2ggY29ycmVzcG9uZHMgdG8gbG9nMiBmb2xkLWNoYW5nZSAtMC41ODUgYW5kIDAuNTg1LCByZXNwZWN0aXZlbHkpLiBJbmNsdWRpbmcgYSBmb2xkLWNoYW5nZSB0aHJlc2hvbGQgZW5zdXJlcyB0aGF0IHRoZSBERSBnZW5lcyBoYXZlIGEgcmVhc29uYWJsZSBlZmZlY3Qgc2l6ZSBhcyB3ZWxsIGFzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZS4KCkxldCdzIHNldCB0aGVzZSB0aHJlc2hvbGRzIGFzIHZhcmlhYmxlcyB0byByZXVzZS4gVGhpcyBpcyBnZW5lcmFsbHkgZ29vZCBwcmFjdGljZSBiZWNhdXNlIGlmIHlvdSB3YW50IHRvIGNoYW5nZSB0aG9zZSB0aHJlc2hvbGRzIGxhdGVyIHRoZW4geW91IG9ubHkgaGF2ZSB0byBjaGFuZ2UgdGhlbSBpbiBvbmUgbG9jYXRpb24gb2YgeW91ciBzY3JpcHQsIHdoaWNoIGlzIGZhc3RlciBhbmQgY2FuIHJlZHVjZSB0aGUgcmlzayBvZiBlcnJvcnMgZnJvbSBtaXNzaW5nIHNvbWUgaW5zdGFuY2VzIGluIHlvdXIgY29kZS4KCmBgYHtyIFBsb3RTZXR1cDF9CmZjID0gMS41CmZkciA9IDAuMDUKYGBgCgo+ICMgTm90ZTogQ2hvb3NpbmcgdGhyZXNob2xkcyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQo+Cj4gVGhyZXNob2xkaW5nIG9uIGFkanVzdGVkIHAtdmFsdWVzIDwgMC4wNSBpcyBhIHN0YW5kYXJkIHRocmVzaG9sZCwgYnV0IGRlcGVuZGluZyBvbiB0aGUgcmVzZWFyY2ggcXVlc3Rpb24gYW5kL29yIGhvdyB0aGUgcmVzdWx0cyB3aWxsIGJlIHVzZWQsIG90aGVyIHRocmVzaG9sZHMgbWlnaHQgYmUgcmVhc29uYWJsZS4KPgo+IFRoZXJlIGlzIGEgbmljZSBbQmlvc3RhciBwb3N0IHRoYXQgZGlzY3Vzc2VzIGNob29zaW5nIGFkanVzdGVkIHAtdmFsdWUgdGhyZXNob2xkc10oaHR0cHM6Ly93d3cuYmlvc3RhcnMub3JnL3AvMjA5MTE4LyksIGluY2x1ZGluZyBjYXNlcyB3aGVyZSBhIG1vcmUgcmVsYXhlZCB0aHJlc2hvbGQgbWlnaHQgYmUgYXBwcm9wcmlhdGUgYW5kIChzb21lIGhlYXRlZCkgZGlzY3Vzc2lvbiBvZiB0aGUgZGFuZ2VycyBvZiBhZGp1c3RpbmcgdGhlIGNob29zZW4gdGhyZXNob2xkIGFmdGVyIHJ1bm5pbmcgYW4gYW5hbHlzaXMuIEFkZGl0aW9uYWxseSwgdGhlcmUgaXMgYSBbRGFsbW9uIGV0IGFsIDIwMTJdKGh0dHBzOi8vYm1jYmlvaW5mb3JtYXRpY3MuYmlvbWVkY2VudHJhbC5jb20vYXJ0aWNsZXMvMTAuMTE4Ni8xNDcxLTIxMDUtMTMtUzItUzExKSBwYXBlciBhYm91dCBwLXZhbHVlIGFuZCBmb2xkLWNoYW5nZSB0aHJlc2hvbGRzIGZvciBtaWNyb2FycmF5IGRhdGEgdGhhdCBtYXkgaGVscCBwcm92aWRlIHNvbWUgY29udGV4dC4KCklmIHdlIHRoaW5rIGJhY2sgdG8gQ29tcHV0YXRpb25hbCBGb3VuZGF0aW9ucywgY29uZGl0aW9uYWwgc3RhdGVtZW50cyBjb3VsZCBhbGxvdyB1cyB0byBkZXRlcm1pbmUgdGhlIG51bWJlciBvZiBnZW5lcyB0aGF0IHBhc3Mgb3VyIHRocmVzaG9sZHMsIHdoaWNoIHdvdWxkIGJlIHVzZWZ1bCBmb3IgYW5ub3RhdGluZyBvdXIgcmVzdWx0cyB0YWJsZXMgYW5kIHBsb3RzLgoKPiAjIEV4ZXJjaXNlIHsudW5saXN0ZWQgLnVubnVtYmVyZWR9Cj4gSG93IHdvdWxkIHdlIGlkZW50aWZ5IHRoZSBudW1iZXIgb2YgZ2VuZXMgd2l0aCBhZGp1c3RlZCBwLXZhbHVlcyA8IDAuMDUgYW5kIGEgZm9sZC1jaGFuZ2UgYWJvdmUgMS41IChvciBiZWxvdyAtMS41KT8KCgo8ZGV0YWlscz4KPHN1bW1hcnk+U29sdXRpb248L3N1bW1hcnk+CgpIZXJlIGlzIG9uZSBwb3NzaWJsZSBhbnN3ZXI6CgpgYGB7ciBTdGF0U2lnR2VuZXMyfQpzdW0ocmVzdWx0c19taW51c192c19wbHVzJHBhZGogPCBmZHIgJiBhYnMocmVzdWx0c19taW51c192c19wbHVzJGxvZzJGb2xkQ2hhbmdlKSA+PSBsb2cyKGZjKSwgbmEucm0gPSBUUlVFKQpgYGAKCioqQ2hlY2twb2ludCoqOiAqSWYgeW91IHNlZSB0aGUgc2FtZSBudW1iZXIgb2YgREUgZ2FuZXMgd2l0aCBvdXIgY2hvb3NlbiB0aHJlc2hvbGRzLCBwbGVhc2UgaW5kaWNhdGUgd2l0aCBhIGdyZWVuIGNoZWNrLiBPdGhlcndpc2UgdXNlIGEgcmVkIHguKgoKPC9kZXRhaWxzPgo8YnI+CgpMZXQncyBub3cgY3JlYXRlIGEgbmV3IGNvbHVtbiBpbiBgcmVzdWx0c19taW51c192c19wbHVzYCB0byByZWNvcmQgdGhlIHNpZ25pZmljYW5jZSAiY2FsbCIgYmFzZWQgb24gdGhlc2UgdGhyZXNob2xkcy4gQW5kIGxldCdzIHNlcGFyYXRlIHRoZSBjYWxsIGJ5ICJVcCIgb3IgIkRvd24iLCBub3RpbmcgdGhhdCB0aGVzZSBhcmUgcmVsYXRpdmUgdG8gb3VyICJDYXNlIiBjb25kaXRpb24gb2YgInBsdXMiLCBvciBpcm9uIHJlcGxldGUgbWljZS4gVGhlcmUgYXJlIG1hbnkgd2F5cyB0byBhY2NvbXBsaXNoIHRoaXMsIGJ1dCB0aGUgZm9sbG93aW5nIHdpbGwgd29yazoKCkZpcnN0IGRlZmluZSBhbGwgdmFsdWVzIGFzICJOUyIgb3IgIm5vdCBzaWduaWZpY2FudCI6CgpgYGB7ciBOU0NvbHVtbn0KcmVzdWx0c19taW51c192c19wbHVzJGNhbGwgPSAnTlMnCmhlYWQocmVzdWx0c19taW51c192c19wbHVzKQpgYGAKCk5leHQsIGRldGVybWluZSB0aGUgIlVwIiBhbmQgIkRvd24iIGluZGljZXM6CgpgYGB7ciBVcERvd25JbmRpY2VzfQp1cF9pZHggPSByZXN1bHRzX21pbnVzX3ZzX3BsdXMkcGFkaiA8IGZkciAmIHJlc3VsdHNfbWludXNfdnNfcGx1cyRsb2cyRm9sZENoYW5nZSA+IGxvZzIoZmMpCmRvd25faWR4ID0gcmVzdWx0c19taW51c192c19wbHVzJHBhZGogPCBmZHIgJiByZXN1bHRzX21pbnVzX3ZzX3BsdXMkbG9nMkZvbGRDaGFuZ2UgPCAtbG9nMihmYykKYGBgCgpMYXN0LCB1c2UgdGhvc2UgaW5kaWNlcyB0byBhc3NpZ24gdGhlIGNvcnJlY3QgIlVwIiBvciAiRG93biIgdmFsdWVzIHRvIHRoZSBjb3JyZWN0IGluZGljZXMsIGFuZCBsb29rIGF0IHRoZSBoZWFkIG9mIHRoZSByZXN1bHQ6CgpgYGB7ciBDYWxsQ29sdW1ufQpyZXN1bHRzX21pbnVzX3ZzX3BsdXMkY2FsbFt1cF9pZHhdID0gJ1VwJwpyZXN1bHRzX21pbnVzX3ZzX3BsdXMkY2FsbFtkb3duX2lkeF0gPSAnRG93bicKaGVhZChyZXN1bHRzX21pbnVzX3ZzX3BsdXMpCmBgYAoKRmluYWxseSwgc2luY2UgcGxvdHRpbmcgZnVuY3Rpb25zIG9mdGVuIHJlcXVpcmUgY2F0ZWdvcmljYWwgZ3JvdXBpbmdzLCBsZXQncyBtYWtlIHRoaXMgYGNhbGxgIGNvbHVtbiBhIGZhY3RvciBhbmQgc3BlY2lmeSB0aGUgbGV2ZWwgb3JkZXJpbmc6CgpgYGB7ciBGYWN0b3JDYWxsfQpyZXN1bHRzX21pbnVzX3ZzX3BsdXMkY2FsbCA9IGZhY3RvcihyZXN1bHRzX21pbnVzX3ZzX3BsdXMkY2FsbCwgbGV2ZWxzID0gYygnVXAnLCAnRG93bicsICdOUycpKQpoZWFkKHJlc3VsdHNfbWludXNfdnNfcGx1cykKYGBgCgo+ICMgVGlwIHsudW5saXN0ZWQgLnVubnVtYmVyZWR9Cj4gSXQgaXMgb2Z0ZW4gaGVscGZ1bCB0byBpbmNsdWRlIGNvZGUgbGlrZSB0aGlzIGluIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2VzIHNvIHRoZXJlIGlzIGEgY2xlYXJseSBsYWJlbGxlZCBjb2x1bW4gdGhhdCBtYWtlcyBzdWJzZXR0aW5nIGFuZCBzdW1tYXJpemluZyB0aGUgcmVzdWx0cyBlYXNpZXIuCgpOb3cgd2UgYXJlIGluIGEgcG9zaXRpb24gdG8gcXVpY2tseSBzdW1tYXJpemUgb3VyIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIHJlc3VsdHM6CgpgYGB7ciBUYWJsZUNhbGx9CnRhYmxlKHJlc3VsdHNfbWludXNfdnNfcGx1cyRjYWxsKQpgYGAKCldlIHNlZSBxdWlja2x5IGhvdyBtYW55IGdlbmVzIHdlcmUgIlVwIiBpbiBpcm9uIHJlcGxldGUsIGhvdyBtYW55IHdlcmUgIkRvd24iIGluIGlyb24gcmVwbGV0ZSwgYW5kIGhvdyBtYW55IHdlcmUgbm90IHNpZ25pZmljYW50LgoKKipDaGVja3BvaW50Kio6ICpJZiB5b3Ugc3VjY2Vzc2Z1bGx5IGFkZGVkIHRoZSBgY2FsbGAgY29sdW1uIGFuZCBnb3QgdGhlIHNhbWUgdGFibGUgcmVzdWx0IGFzIGFib3ZlLCBwbGVhc2UgaW5kaWNhdGUgd2l0aCBhIGdyZWVuIGNoZWNrLiBPdGhlcndpc2UgdXNlIGEgcmVkIHguKgoKIyMgVmlzdWFsIERFIHN1bW1hcnkKCkFzIGRlc2NyaWJlZCBieSB0aGUgW0dhbGF4eSBwcm9qZWN0XShodHRwczovL2dhbGF4eXByb2plY3QuZ2l0aHViLmlvL3RyYWluaW5nLW1hdGVyaWFsL3RvcGljcy90cmFuc2NyaXB0b21pY3MvdHV0b3JpYWxzL3JuYS1zZXEtdml6LXdpdGgtdm9sY2Fub3Bsb3QvdHV0b3JpYWwuaHRtbCksIGEgdm9sY2FubyBwbG90IGlzIGEgdHlwZSBvZiBzY2F0dGVycGxvdCB0aGF0IHNob3dzIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSAoYWRqdXN0ZWQgcC12YWx1ZSkgdmVyc3VzIGVmZmVjdCBzaXplIChmb2xkIGNoYW5nZSkuIEluIGEgdm9sY2FubyBwbG90LCB0aGUgbW9zdCB1cHJlZ3VsYXRlZCBnZW5lcyBhcmUgdG93YXJkcyB0aGUgcmlnaHQsIHRoZSBtb3N0IGRvd25yZWd1bGF0ZWQgZ2VuZXMgYXJlIHRvd2FyZHMgdGhlIGxlZnQsIGFuZCB0aGUgbW9zdCBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IGdlbmVzIGFyZSB0b3dhcmRzIHRoZSB0b3AuCgpMZXQncyBjb2VyY2UgdGhlIGBEYXRhRnJhbWVgIHdoaWNoIHdhcyByZXR1cm5lZCBieSBgREVTZXEyOjpyZXN1bHRzKClgIGludG8gYSBgdGliYmxlYCBpbiBhbnRpY2lwYXRpb24gb2YgdXNpbmcgdGhlIGBnZ3Bsb3QyYCBsaWJyYXJ5IHRvIHBsb3QuIFdlJ3JlIGFsc28gZ29pbmcgdG8gbW9kaWZ5IG91ciByZXN1bHRzIHRhYmxlIHNvIHRoYXQgdGhlIHJvdyBuYW1lcyBiZWNvbWUgYSBzZXBhcmF0ZSBjb2x1bW4sIGFuZCBzbyB0aGF0IGl0J3Mgb3JkZXJlZCBieSBhZGp1c3RlZCBwLXZhbHVlLgoKYGBge3IgUGxvdFNldHVwMn0KIyBVc2UgdGhlIHJvd25hbWVzIGFyZ3VtZW50IHRvIGNyZWF0ZSBhIG5ldyBjb2x1bW4gb2YgZ2VuZSBJRHMKIyBBbHNvIGFycmFuZ2UgYnkgYWRqdXN0ZWQgcC12YWx1ZQpyZXN1bHRzX2ZvclBsb3QgPSBhc190aWJibGUocmVzdWx0c19taW51c192c19wbHVzLCByb3duYW1lcyA9ICdpZCcpICU+JSBhcnJhbmdlKHBhZGopCmBgYAoKTGV0J3Mgc3RhcnQgd2l0aCBhIHZlcnkgc2ltcGxlIHZvbGNhbm8gcGxvdCB0aGF0IHBsb3RzIHRoZSBgbG9nMkZvbGRDaGFuZ2VgIG9uIHRoZSB4LWF4aXMsIGFuZCBgLWxvZzEwKHBhZGopYCBvbiB0aGUgeS1heGlzLgoKYGBge3IgVm9sY2Fub1Bsb3R9CiMgSW5pdGlhbGl6ZSB0aGUgcGxvdCwgc2F2aW5nIGFzIG9iamVjdCAncCcgYW5kIHNwZWNpZnlpbmcgdGhlIHBsb3QgdHlwZSBhcyAnZ2VvbV9wb2ludCcKcCA9IGdncGxvdChyZXN1bHRzX2ZvclBsb3QsIGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaikpKSArCiAgICBnZW9tX3BvaW50KCkKcApgYGAKClRoaXMgaXMgYSBnb29kIHN0YXJ0LCBidXQsIGFzIHVzdWFsIGl0J3MgZ29vZCBwcmFjdGljZSB0byBhZGQgYmV0dGVyIGxhYmVscyB0byB0aGUgcGxvdCB3aXRoIHRoZSBgbGFicygpYCBmdW5jdGlvbjoKCmBgYHtyIFZvbGNhbm9QbG90Mn0KIyBBZGQgcGxvdCBsYWJlbHMgYW5kIGNoYW5nZSB0aGUgdGhlbWUKcCA9IGdncGxvdChyZXN1bHRzX2ZvclBsb3QsIGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaikpKSArCiAgICBnZW9tX3BvaW50KCkgKwogICAgdGhlbWVfYncoKSArCiAgICBsYWJzKAogICAgICAgIHRpdGxlID0gJ1ZvbGNhbm8gUGxvdCcsCiAgICAgICAgc3VidGl0bGUgPSAnUGx1cyB2cyBNaW51cycsCiAgICAgICAgeCA9ICdsb2cyIGZvbGQtY2hhbmdlJywKICAgICAgICB5ID0gJy1sb2cxMCBGRFInCiAgICApCnAKYGBgCgpXaGF0IGlmIHdlIG5vdyBhZGRlZCBzb21lIHZpc3VhbCBndWlkZXMgdG8gaW5kaWNhdGUgd2hlcmUgdGhlIHNpZ25pZmljYW50IGdlbmVzIGFyZT8gV2UgY2FuIHVzZSB0aGUgYGdlb21fdmxpbmUoKWAgYW5kIGBnZW9tX2hsaW5lKClgIGZ1bmN0aW9ucyB0byBhY2NvbXBsaXNoIHRoaXM6CgpgYGB7ciBWb2xjYW5vUGxvdDN9CiMgQWRkIHRocmVzaG9sZCBsaW5lcwpwID0gcCArCiAgICBnZW9tX3ZsaW5lKAogICAgICAgIHhpbnRlcmNlcHQgPSBjKDAsIC1sb2cyKGZjKSwgbG9nMihmYykpLAogICAgICAgIGxpbmV0eXBlID0gYygxLCAyLCAyKSkgKwogICAgZ2VvbV9obGluZSgKICAgICAgICB5aW50ZXJjZXB0ID0gLWxvZzEwKGZkciksCiAgICAgICAgbGluZXR5cGUgPSAyKQpwCmBgYAoKRmluYWxseSwgd2h5IG5vdCBjb2xvciB0aGUgcG9pbnRzIGJ5IHRoZWlyIHNpZ25pZmljYW5jZSBzdGF0dXM/IFdlIGFscmVhZHkgY3JlYXRlZCB0aGUgYGNhbGxgIGNvbHVtbiB0aGF0IGhhcyB0aGUgY29ycmVjdCB2YWx1ZXMuIEluIHRoaXMgY2FzZSB3ZSBjYW4gZ2V0IGF3YXkgd2l0aCBhZGRpbmcgYGdlb21fcG9pbnQoKWAgdG8gb3VyIGV4aXN0aW5nIHBsb3QgYW5kIHNwZWNpZnlpbmcgdGhlIGNvcnJlY3QgYWVzdGhldGljOgoKYGBge3IgVm9sY2Fub1Bsb3Q0fQpwID0gcCArIGdlb21fcG9pbnQoYWVzKGNvbG9yID0gY2FsbCkpCnAKYGBgCgpGb3IgYWRkaXRpb25hbCB2aXN1YWxpemF0aW9ucyBmb3Igb3VyIERFIHJlc3VsdHMsIHdlIGluY2x1ZGVkIHNvbWUgZXhhbXBsZSBjb2RlIGluIHRoZSBbQm9udXMgQ29udGVudCBtb2R1bGVdKGh0dHBzOi8vdW1pY2gtYnJjZi1iaW9pbmYuZ2l0aHViLmlvLzIwMjMtMDMtMTMtdW1pY2gtcm5hc2VxLWRlbXlzdGlmaWVkL2h0bWwvUl9ib251c19jb250ZW50Lmh0bWwpIGFuZCB0aGlzIFtIQkMgdHV0b3JpYWxdKGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzA2X0RHRV92aXN1YWxpemluZ19yZXN1bHRzLmh0bWwpIGFsc28gaW5jbHVkZXMgc29tZSBuaWNlIGV4YW1wbGVzLgoKIyMjIFN1YnNldHRpbmcgc2lnbmlmaWNhbnQgZ2VuZXMKCllvdSBtYXkgYmUgaW50ZXJlc3RlZCBpbiBjcmVhdGluZyBhIHRhYmxlIG9mIG9ubHkgdGhlIGdlbmVzIHRoYXQgcGFzcyB5b3VyIHNpZ25pZmljYW5jZSB0aHJlc2hvbGRzLiBBIHVzZWZ1bCB3YXkgdG8gZG8gdGhpcyBpcyB0byBjb25kaXRpb25hbGx5IHN1YnNldCB5b3VyIHJlc3VsdHMuIEFnYWluLCB3ZSBhbHJlYWR5IGNyZWF0ZWQgdGhlIGBjYWxsYCBjb2x1bW4sIHdoaWNoIG1ha2VzIHRoaXMgcmVsYXRpdmVseSBzaW1wbGUgdG8gZG8uCgoqTm90ZTogVGhlIHRpZHl2ZXJzZSBmdW5jdGlvbnMgeW91IGxlYXJuZWQgaW4gU29mdHdhcmUgQ2FycGVudHJ5IGNvdWxkIGFsc28gYmUgYWx0ZXJuYXRpdmVseSB1c2VkIGhlcmUuKgoKYGBge3IgQ29uZGl0aW9uYWxTdWJzZXR9CiMgdGlkeXIgKHJlcXVpcmVzIHRhYmxlIHJlZm9ybWF0dGluZykKcmVzX3NpZyA8LSBhc190aWJibGUocmVzdWx0c19taW51c192c19wbHVzLCByb3duYW1lcyA9ICJnZW5lX2lkcyIpICU+JSBmaWx0ZXIoY2FsbCAhPSAnTlMnKQoKIyBiYXNlCnJlc19zaWcgPC0gcmVzdWx0c19taW51c192c19wbHVzW3Jlc3VsdHNfbWludXNfdnNfcGx1cyRjYWxsICE9ICdOUycsIF0KCmhlYWQocmVzX3NpZykKZGltKHJlc19zaWcpCmBgYAoKCiMgU3VtbWFyeQoKSW4gdGhpcyBzZWN0aW9uLCB3ZToKCiogR2VuZXJhdGVkIGEgdm9sY2FubyBwbG90IGZvciBvdXIgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gcmVzdWx0cwoqIFN1bW1hcml6ZWQgb3VyIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIHJlc3VsdHMKKiBEaXNjdXNzZWQgY2hvb3NpbmcgdGhyZXNob2xkcwoqIEFubm90YXRlZCBvdXIgdGFibGVzIG9mIHJlc3VsdHMgdG8gbWFwIGdlbmUgSURzIHRvIGdlbmUgc3ltYm9scwoqIFNhdmVkIG91ciByZXN1bHRzIHRvIGZpbGUKCgotLS0KCiMgU291cmNlcwoKKiBIQkMgREdFIHRyYWluaW5nIG1vZHVsZSwgcGFydCAxOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wNF9ER0VfREVTZXEyX2FuYWx5c2lzLmh0bWwKKiBIQkMgREdFIHRyYWluaW5nIG1vZHVsZSwgcGFydCAyOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wNV9ER0VfREVTZXEyX2FuYWx5c2lzMi5odG1sCiogREVTZXEyIHZpZ25ldHRlOiBodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjZGlmZmVyZW50aWFsLWV4cHJlc3Npb24tYW5hbHlzaXMKCgotLS0KCmBgYHtyIFdyaXRlT3V0LlJEYXRhLCBldmFsPVRSVUUsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CiMgSGlkZGVuIGNvZGUgYmxvY2sgdG8gd3JpdGUgb3V0IGRhdGEgZm9yIGtuaXR0aW5nCiMgc2F2ZS5pbWFnZShmaWxlID0gInJkYXRhL1J1bm5pbmdEYXRhX0Z1bGwuUkRhdGEiKQpgYGAKCgotLS0KClRoZXNlIG1hdGVyaWFscyBoYXZlIGJlZW4gYWRhcHRlZCBhbmQgZXh0ZW5kZWQgZnJvbSBtYXRlcmlhbHMgbGlzdGVkIGFib3ZlLiBUaGVzZSBhcmUgb3BlbiBhY2Nlc3MgbWF0ZXJpYWxzIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgW0NyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gbGljZW5zZSAoQ0MgQlkgNC4wKV0oaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyksIHdoaWNoIHBlcm1pdHMgdW5yZXN0cmljdGVkIHVzZSwgZGlzdHJpYnV0aW9uLCBhbmQgcmVwcm9kdWN0aW9uIGluIGFueSBtZWRpdW0sIHByb3ZpZGVkIHRoZSBvcmlnaW5hbCBhdXRob3IgYW5kIHNvdXJjZSBhcmUgY3JlZGl0ZWQuCg==