Objectives

  • ‘Unblind’ our sample IDs
  • Understand possible confounding factors
  • Understand the impact of batches or additional covariates
  • Filter count table

Differential Expression Workflow

Here we will proceed setting up the inputs needed to initialize DESeq2 before testing for differential expression.


DESeq2 objects

Bioconductor software packages often define and use custom classes within R to store data in a way that better fits expectations around biological data, such as illustrated below from Huber et al. 2015.

A breakdown of the SummarizedExperiment class.

These custom data structures have pre-specified data slots, which hold specific types/classes of data and therefore can be more easily accessed by functions from the same package.

To create the DESeqDataSet we need two tables:

  1. A count matrix (which we already loaded and saved as the object count_table) and
  2. A table that assigns the condition labels for each sample (which we will generate below).

Sample Information

As introduced at the beginning of the workshop, we have downloaded and prepared data from an existing publication (Zhang et al., 2019), wherein one goal is to understand the gene expression differences in wild-type mice that are “iron replete” (we label these “plus”) and “iron deficient” (we label these “minus”).

Read Sample Table

Our next step will be to describe the samples within our R session, so that we make the proper comparisons with DESeq2. Let’s check the sample names from the count table.

colnames(count_table)
[1] "sample_A" "sample_B" "sample_C" "sample_D" "sample_E" "sample_F"

Based on the sample names that are the columns of count_table, our samples are blinded, e.g. the sample names don’t clearly correspond to treatment groups. We will need to specify which sample IDs connect to which experimental conditions.

Typically sample phenotype data, including experimental conditions, are stored as Excel or CSV files that we can read into R, and then use when creating a DESeq2 object. If you are unfamilar with CSV files or how to generate them, there are tutorials available to guide you through the process.

Tip: Sample naming conventions

Use only alpha-numeric characters (A-Z, a-z, 0-9), and separate parts of the name with underscores (_) or dots (.). Do not begin sample names with numbers.

We’ll load our ‘pre-made’ sample information sheet, samplesheet.csv, to unblind our samples.

samplesheet = read.table("data/samplesheet.csv",
                       sep = ",",
                       header = TRUE,
                       row.names = 1)

We can look at the object by typing its name and hitting Enter. Note, for larger experiments, you may want to use head().

samplesheet
         genotype condition
sample_A       wt      plus
sample_B       wt      plus
sample_C       wt      plus
sample_D       wt     minus
sample_E       wt     minus
sample_F       wt     minus

Checkpoint: If you have loaded samplesheet, please indicate with the green ‘check’ button. Otherwise, please use the red ‘x’ button to have the command repeated

Again, all mice are wild-type in this example data, and the iron replete mice are labeled as “plus” and the iron deficient mice are labeled as “minus”. Again, for larger experiments, you may want to examine the coding of the samples by looking at the unique() values of the relevant columns.

unique(samplesheet$condition)
[1] "plus"  "minus"

Next, we’ll format our table so that we have the appropriate data type (an ordered factor) for DESeq2 to recognize our treatment groups and appropriately compare samples.

samplesheet$condition = factor(samplesheet$condition,
                            levels = c('plus', 'minus'))

unique(samplesheet$condition)
[1] plus  minus
Levels: plus minus

Notice that we set the levels in a particular order. This is important for setting the “Control” (or “Reference”) group as the denominator in our comparisons when we setup our DESeq2 model.

Before we proceed, we need to make sure that the sample labels (column names) in the count table match the sample information table (row names), including the order. If the sample labels don’t match, then we will see an error and need to correct the labels prior to proceeding. Checking the sample information table is extremely important to ensure that the correct samples are grouped together for comparisons.

all(colnames(count_table) == rownames(samplesheet))
[1] TRUE

This line of code checks if both the identity and order match between our count_table and our samplesheet. If, in the course of using your own data, this returns FALSE, try using the match() function to rearrange the columns of count_table (or the rows of samplesheet) to get them to match.

Checkpoint: If you your sample info check returns TRUE, please indicate with the green ‘yes’ button. Otherwise, please use the red ‘x’ button to have the command repeated

Creating DESeq2 object

To create the DESeqDataSet we will need the count_table and the samplesheet. We will also need a design formula to specify our model.

Making model choices

The design formula specified informs many of the DESeq2 functions how to treat the samples in the analysis, specifically which column(s) in the sample metadata table are relevant to the experimental design.

In this case, we aren’t aware of any covariates that should be considered in our comparisons. However, if there are additional attributes of the samples that may impact the DE comparisons, like sex, date of collection, or patient of origin, these should be added as additional columns in the sample information table and added to a design formula.

More complex questions, including adding “interaction terms” are helpfully described in this support thread as well as in the DESeq2 vignette, which is an excellent document.

The design formula specifies the column(s) in the metadata table and how they should be used in the analysis. For our dataset we only have one column we are interested in, that is condition. This column has two factor levels, which tells DESeq2 that for each gene we want to evaluate gene expression change with respect to these different levels.

## Create DESeq object, line by line
dds = DESeqDataSetFromMatrix(countData = count_table,
                              colData = samplesheet,
                              design = ~ condition)
converting counts to integer mode
dds
class: DESeqDataSet 
dim: 55492 6 
metadata(1): version
assays(1): counts
rownames(55492): ENSMUSG00000000001 ENSMUSG00000000003 ... mcherry
  tdtomato
rowData names(0):
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(2): genotype condition

Notice that printing the dds object helpfully shows us some helpful information:

  • The dimension (number of genes by number of samples),
  • The gene identifiers,
  • The sample identifiers,
  • The additional column names giving information about the samples

Checkpoint: If you see dds in your environment panel, please indicate with the green ‘check’ button. Otherwise, please use use the red ‘x’ button in your zoom reaction panel to have this step repeated. You can use the red ‘x’ to be put in a breakout room for help


Pre-filtering

While not necessary, pre-filtering the count table helps to not only reduce the size of the DESeq2 object, but also gives you a sense of how many genes were reasonably measured at the sequencing depth generated for your samples.

Here we will filter out any genes that have less than 10 counts across any of the samples. This is a fairly standard level of filtering, but can filter data less/more depending on quality control metrics from alignments and sequencing depth or total number of samples.

keep = rowSums(counts(dds)) >= 10
dds = dds[keep,]
dds
class: DESeqDataSet 
dim: 16249 6 
metadata(1): version
assays(1): counts
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ...
  ENSMUSG00000118651 ENSMUSG00000118653
rowData names(0):
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(2): genotype condition

Notice the dds object has less elements after filtering. There were quite a number of genes that were not measured in our experiment.

Checkpoint: Questions?

Optional content

Click for how to model batch effects with DESeq2

Differences between samples can also be due to technical reasons, such as collection on different days or different sequencing runs. Differences between samples that are not due to biological factors as called batch effects. We can include batch effects in our design model in the same way as covariates, as long as the technical groups do not overlap, or confound, the biological treatment groups.

Let’s try add some additional meta-data information where we have counfounding batch effects and create another DESeq2 object.

samplesheet_batch = samplesheet
samplesheet_batch$batch = factor(c(rep(c("Day1"), 2),
                       rep(c("Day2"), 2),
                       rep(c("Day3"), 2)),
                     levels = c("Day1", "Day2", "Day3"))

dds_batch = DESeqDataSetFromMatrix(countData = count_table,
                      colData = samplesheet_batch,
                      design = ~ batch + condition)
converting counts to integer mode


Summary

In this section, we:

  • Loaded the necessary input files into our R session
  • Discussed model design for DESeq2
  • Initialized a DESeq2 data set
  • Filtered our count data

Now that we’ve created our DESeq2 objects, including specifying what model is appropriate for our data, and filtered our data, we can proceed with assessing the impact of the experimental conditions on gene expression across our samples.


Sources

Training resources used to develop materials:


These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

LS0tCnRpdGxlOiAiTW9kdWxlIDA3OiBERSBJbml0aWFsaXphdGlvbiIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjA3LSIpCmBgYAoKYGBge3IgTG9hZFJ1bm5pbmdEYXRhLCBldmFsPVRSVUUsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CmxpYnJhcnkoREVTZXEyKQojIGxvYWQoInJkYXRhL1J1bm5pbmdEYXRhLlJEYXRhIikKYGBgCgo+ICMgT2JqZWN0aXZlcyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQo+ICogJ1VuYmxpbmQnIG91ciBzYW1wbGUgSURzCj4gKiBVbmRlcnN0YW5kIHBvc3NpYmxlIGNvbmZvdW5kaW5nIGZhY3RvcnMKPiAqIFVuZGVyc3RhbmQgdGhlIGltcGFjdCBvZiBiYXRjaGVzIG9yIGFkZGl0aW9uYWwgY292YXJpYXRlcwo+ICogRmlsdGVyIGNvdW50IHRhYmxlCgojIERpZmZlcmVudGlhbCBFeHByZXNzaW9uIFdvcmtmbG93IHsudW5saXN0ZWQgLnVubnVtYmVyZWR9CgpIZXJlIHdlIHdpbGwgcHJvY2VlZCBzZXR0aW5nIHVwIHRoZSBpbnB1dHMgbmVlZGVkIHRvIGluaXRpYWxpemUgREVTZXEyIGJlZm9yZSB0ZXN0aW5nIGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbi4KCiFbXSguL2ltYWdlcy93YXlmaW5kZXIvd2F5ZmluZGVyLURFU2VxMkluaXQucG5nKXt3aWR0aD03NSV9CgotLS0KCiMgREVTZXEyIG9iamVjdHMKCkJpb2NvbmR1Y3RvciBzb2Z0d2FyZSBwYWNrYWdlcyBvZnRlbiBkZWZpbmUgYW5kIHVzZSBjdXN0b20gY2xhc3NlcyB3aXRoaW4gUiB0byBzdG9yZSBkYXRhIGluIGEgd2F5IHRoYXQgYmV0dGVyIGZpdHMgZXhwZWN0YXRpb25zIGFyb3VuZCBiaW9sb2dpY2FsIGRhdGEsIHN1Y2ggYXMgaWxsdXN0cmF0ZWQgYmVsb3cgZnJvbSBbSHViZXIgZXQgYWwuIDIwMTVdKGh0dHBzOi8vd3d3Lm5hdHVyZS5jb20vYXJ0aWNsZXMvbm1ldGguMzI1MikuCgohW0EgYnJlYWtkb3duIG9mIHRoZSBTdW1tYXJpemVkRXhwZXJpbWVudCBjbGFzcy5dKC4vaW1hZ2VzL1N1bW1hcml6ZWRFeHBlcmltZW50LmpwZykKClRoZXNlIGN1c3RvbSBkYXRhIHN0cnVjdHVyZXMgaGF2ZSBwcmUtc3BlY2lmaWVkIGRhdGEgc2xvdHMsIHdoaWNoIGhvbGQgc3BlY2lmaWMgdHlwZXMvY2xhc3NlcyBvZiBkYXRhIGFuZCB0aGVyZWZvcmUgY2FuIGJlIG1vcmUgZWFzaWx5IGFjY2Vzc2VkIGJ5IGZ1bmN0aW9ucyBmcm9tIHRoZSBzYW1lIHBhY2thZ2UuCgpUbyBjcmVhdGUgdGhlIERFU2VxRGF0YVNldCB3ZSBuZWVkIHR3byB0YWJsZXM6CgoxLiBBIGNvdW50IG1hdHJpeCAod2hpY2ggd2UgYWxyZWFkeSBsb2FkZWQgYW5kIHNhdmVkIGFzIHRoZSBvYmplY3QgYGNvdW50X3RhYmxlYCkgYW5kCjIuIEEgdGFibGUgdGhhdCBhc3NpZ25zIHRoZSBjb25kaXRpb24gbGFiZWxzIGZvciBlYWNoIHNhbXBsZSAod2hpY2ggd2Ugd2lsbCBnZW5lcmF0ZSBiZWxvdykuCgojIFNhbXBsZSBJbmZvcm1hdGlvbgoKQXMgaW50cm9kdWNlZCBhdCB0aGUgYmVnaW5uaW5nIG9mIHRoZSB3b3Jrc2hvcCwgd2UgaGF2ZSBkb3dubG9hZGVkIGFuZCBwcmVwYXJlZCBkYXRhIGZyb20gYW4gZXhpc3RpbmcgcHVibGljYXRpb24gWyhaaGFuZyBldCBhbC4sIDIwMTkpXShodHRwczovL2VsaWZlc2NpZW5jZXMub3JnL2FydGljbGVzLzQ2OTc2KSwgd2hlcmVpbiBvbmUgZ29hbCBpcyB0byB1bmRlcnN0YW5kIHRoZSBnZW5lIGV4cHJlc3Npb24gZGlmZmVyZW5jZXMgaW4gd2lsZC10eXBlIG1pY2UgdGhhdCBhcmUgImlyb24gcmVwbGV0ZSIgKHdlIGxhYmVsIHRoZXNlICJwbHVzIikgYW5kICJpcm9uIGRlZmljaWVudCIgKHdlIGxhYmVsIHRoZXNlICJtaW51cyIpLgoKIVtdKGltYWdlcy9Nb2R1bGUwMF90b3BfZG93bi5qcGcpCgojIyBSZWFkIFNhbXBsZSBUYWJsZQoKT3VyIG5leHQgc3RlcCB3aWxsIGJlIHRvIGRlc2NyaWJlIHRoZSBzYW1wbGVzIHdpdGhpbiBvdXIgUiBzZXNzaW9uLCBzbyB0aGF0IHdlIG1ha2UgdGhlIHByb3BlciBjb21wYXJpc29ucyB3aXRoIERFU2VxMi4gTGV0J3MgY2hlY2sgdGhlIHNhbXBsZSBuYW1lcyBmcm9tIHRoZSBjb3VudCB0YWJsZS4KCmBgYHtyIENvbHVtbk5hbWVzfQpjb2xuYW1lcyhjb3VudF90YWJsZSkKYGBgCgpCYXNlZCBvbiB0aGUgc2FtcGxlIG5hbWVzIHRoYXQgYXJlIHRoZSBjb2x1bW5zIG9mIGBjb3VudF90YWJsZWAsIG91ciBzYW1wbGVzIGFyZSBibGluZGVkLCBlLmcuIHRoZSBzYW1wbGUgbmFtZXMgZG9uJ3QgY2xlYXJseSBjb3JyZXNwb25kIHRvIHRyZWF0bWVudCBncm91cHMuIFdlIHdpbGwgbmVlZCB0byBzcGVjaWZ5IHdoaWNoIHNhbXBsZSBJRHMgY29ubmVjdCB0byB3aGljaCBleHBlcmltZW50YWwgY29uZGl0aW9ucy4KClR5cGljYWxseSBzYW1wbGUgcGhlbm90eXBlIGRhdGEsIGluY2x1ZGluZyBleHBlcmltZW50YWwgY29uZGl0aW9ucywgYXJlIHN0b3JlZCBhcyBFeGNlbCBvciBDU1YgZmlsZXMgdGhhdCB3ZSBjYW4gcmVhZCBpbnRvIFIsIGFuZCB0aGVuIHVzZSB3aGVuIGNyZWF0aW5nIGEgREVTZXEyIG9iamVjdC4gSWYgeW91IGFyZSB1bmZhbWlsYXIgd2l0aCBDU1YgZmlsZXMgb3IgaG93IHRvIGdlbmVyYXRlIHRoZW0sIHRoZXJlIGFyZSBbdHV0b3JpYWxzXShodHRwczovL3d3dy53aWtpaG93LmNvbS9DcmVhdGUtYS1DU1YtRmlsZSkgYXZhaWxhYmxlIHRvIGd1aWRlIHlvdSB0aHJvdWdoIHRoZSBwcm9jZXNzLgoKPiAjIFRpcDogU2FtcGxlIG5hbWluZyBjb252ZW50aW9ucyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQo+IFVzZSBvbmx5IGFscGhhLW51bWVyaWMgY2hhcmFjdGVycyAoQS1aLCBhLXosIDAtOSksIGFuZCBzZXBhcmF0ZSBwYXJ0cyBvZiB0aGUgbmFtZSB3aXRoIHVuZGVyc2NvcmVzIChgX2ApIG9yIGRvdHMgKGAuYCkuIERvIG5vdCBiZWdpbiBzYW1wbGUgbmFtZXMgd2l0aCBudW1iZXJzLgoKV2UnbGwgbG9hZCBvdXIgJ3ByZS1tYWRlJyBzYW1wbGUgaW5mb3JtYXRpb24gc2hlZXQsIGBzYW1wbGVzaGVldC5jc3ZgLCB0byB1bmJsaW5kIG91ciBzYW1wbGVzLgoKYGBge3IgU2FtcGxlc2hlZXQsIGVjaG8gPSBGQUxTRSwgZXZhbCA9IFRSVUV9CnNhbXBsZXNoZWV0ID0gcmVhZC50YWJsZSgiLi4vZGF0YS9zYW1wbGVzaGVldC5jc3YiLAogICAgICAgICAgICAgICAgICAgICAgIHNlcCA9ICIsIiwKICAgICAgICAgICAgICAgICAgICAgICBoZWFkZXIgPSBUUlVFLAogICAgICAgICAgICAgICAgICAgICAgIHJvdy5uYW1lcyA9IDEpCmBgYAoKYGBge3IgU2FtcGxlc2hlZXQyLCBldmFsID0gRkFMU0V9CnNhbXBsZXNoZWV0ID0gcmVhZC50YWJsZSgiZGF0YS9zYW1wbGVzaGVldC5jc3YiLAogICAgICAgICAgICAgICAgICAgICAgIHNlcCA9ICIsIiwKICAgICAgICAgICAgICAgICAgICAgICBoZWFkZXIgPSBUUlVFLAogICAgICAgICAgICAgICAgICAgICAgIHJvdy5uYW1lcyA9IDEpCmBgYAoKV2UgY2FuIGxvb2sgYXQgdGhlIG9iamVjdCBieSB0eXBpbmcgaXRzIG5hbWUgYW5kIGhpdHRpbmcgRW50ZXIuIE5vdGUsIGZvciBsYXJnZXIgZXhwZXJpbWVudHMsIHlvdSBtYXkgd2FudCB0byB1c2UgYGhlYWQoKWAuCgpgYGB7ciBTaG93U2FtcGxlc2hlZXR9CnNhbXBsZXNoZWV0CmBgYAoKKipDaGVja3BvaW50Kio6ICpJZiB5b3UgaGF2ZSBsb2FkZWQgYHNhbXBsZXNoZWV0YCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICdjaGVjaycgYnV0dG9uLiBPdGhlcndpc2UsIHBsZWFzZSB1c2UgdGhlIHJlZCAneCcgYnV0dG9uIHRvIGhhdmUgdGhlIGNvbW1hbmQgcmVwZWF0ZWQqCgpBZ2FpbiwgYWxsIG1pY2UgYXJlIHdpbGQtdHlwZSBpbiB0aGlzIGV4YW1wbGUgZGF0YSwgYW5kIHRoZSBpcm9uIHJlcGxldGUgbWljZSBhcmUgbGFiZWxlZCBhcyAicGx1cyIgYW5kIHRoZSBpcm9uIGRlZmljaWVudCBtaWNlIGFyZSBsYWJlbGVkIGFzICJtaW51cyIuIEFnYWluLCBmb3IgbGFyZ2VyIGV4cGVyaW1lbnRzLCB5b3UgbWF5IHdhbnQgdG8gZXhhbWluZSB0aGUgY29kaW5nIG9mIHRoZSBzYW1wbGVzIGJ5IGxvb2tpbmcgYXQgdGhlIGB1bmlxdWUoKWAgdmFsdWVzIG9mIHRoZSByZWxldmFudCBjb2x1bW5zLgoKYGBge3IgVHJlYXRtZW50R3JvdXBUYWJsZX0KdW5pcXVlKHNhbXBsZXNoZWV0JGNvbmRpdGlvbikKYGBgCgpOZXh0LCB3ZSdsbCBmb3JtYXQgb3VyIHRhYmxlIHNvIHRoYXQgd2UgaGF2ZSB0aGUgYXBwcm9wcmlhdGUgZGF0YSB0eXBlIChhbiBvcmRlcmVkIFtmYWN0b3JdKGh0dHBzOi8vc3djYXJwZW50cnkuZ2l0aHViLmlvL3Itbm92aWNlLWluZmxhbW1hdGlvbi8xMi1zdXBwLWZhY3RvcnMvKSkgZm9yIERFU2VxMiB0byByZWNvZ25pemUgb3VyIHRyZWF0bWVudCBncm91cHMgYW5kIGFwcHJvcHJpYXRlbHkgY29tcGFyZSBzYW1wbGVzLgoKYGBge3IgU2FtcGxlc2hlZXRGYWN0b3J9CgpzYW1wbGVzaGVldCRjb25kaXRpb24gPSBmYWN0b3Ioc2FtcGxlc2hlZXQkY29uZGl0aW9uLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgbGV2ZWxzID0gYygncGx1cycsICdtaW51cycpKQoKdW5pcXVlKHNhbXBsZXNoZWV0JGNvbmRpdGlvbikKYGBgCgpOb3RpY2UgdGhhdCB3ZSBzZXQgdGhlIGxldmVscyBpbiBhIHBhcnRpY3VsYXIgb3JkZXIuIFRoaXMgaXMgaW1wb3J0YW50IGZvciBzZXR0aW5nIHRoZSAiQ29udHJvbCIgKG9yICJSZWZlcmVuY2UiKSBncm91cCBhcyB0aGUgZGVub21pbmF0b3IgaW4gb3VyIGNvbXBhcmlzb25zIHdoZW4gd2Ugc2V0dXAgb3VyIERFU2VxMiBtb2RlbC4KCkJlZm9yZSB3ZSBwcm9jZWVkLCB3ZSBuZWVkIHRvIG1ha2Ugc3VyZSB0aGF0IHRoZSBzYW1wbGUgbGFiZWxzIChjb2x1bW4gbmFtZXMpIGluIHRoZSBjb3VudCB0YWJsZSBtYXRjaCB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIHRhYmxlIChyb3cgbmFtZXMpLCBpbmNsdWRpbmcgdGhlIG9yZGVyLiBJZiB0aGUgc2FtcGxlIGxhYmVscyBkb24ndCBtYXRjaCwgdGhlbiB3ZSB3aWxsIHNlZSBhbiBlcnJvciBhbmQgbmVlZCB0byBjb3JyZWN0IHRoZSBsYWJlbHMgcHJpb3IgdG8gcHJvY2VlZGluZy4gQ2hlY2tpbmcgdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiB0YWJsZSBpcyBleHRyZW1lbHkgaW1wb3J0YW50IHRvIGVuc3VyZSB0aGF0IHRoZSBjb3JyZWN0IHNhbXBsZXMgYXJlIGdyb3VwZWQgdG9nZXRoZXIgZm9yIGNvbXBhcmlzb25zLgoKYGBge3IgQ2hlY2tTYW1wbGVzaGVldH0KYWxsKGNvbG5hbWVzKGNvdW50X3RhYmxlKSA9PSByb3duYW1lcyhzYW1wbGVzaGVldCkpCmBgYApUaGlzIGxpbmUgb2YgY29kZSBjaGVja3MgaWYgYm90aCB0aGUgaWRlbnRpdHkgYW5kIG9yZGVyIG1hdGNoIGJldHdlZW4gb3VyIGBjb3VudF90YWJsZWAgYW5kIG91ciBgc2FtcGxlc2hlZXRgLiBJZiwgaW4gdGhlIGNvdXJzZSBvZiB1c2luZyB5b3VyIG93biBkYXRhLCB0aGlzIHJldHVybnMgYEZBTFNFYCwgdHJ5IHVzaW5nIHRoZSBgbWF0Y2goKWAgZnVuY3Rpb24gdG8gcmVhcnJhbmdlIHRoZSBjb2x1bW5zIG9mIGBjb3VudF90YWJsZWAgKG9yIHRoZSByb3dzIG9mIGBzYW1wbGVzaGVldGApIHRvIGdldCB0aGVtIHRvIG1hdGNoLgoKKipDaGVja3BvaW50Kio6ICpJZiB5b3UgeW91ciBzYW1wbGUgaW5mbyBjaGVjayByZXR1cm5zIGBUUlVFYCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICd5ZXMnIGJ1dHRvbi4gT3RoZXJ3aXNlLCBwbGVhc2UgdXNlIHRoZSByZWQgJ3gnIGJ1dHRvbiB0byBoYXZlIHRoZSBjb21tYW5kIHJlcGVhdGVkKgoKIyBDcmVhdGluZyBERVNlcTIgb2JqZWN0CgpUbyBjcmVhdGUgdGhlIGBERVNlcURhdGFTZXRgIHdlIHdpbGwgbmVlZCB0aGUgYGNvdW50X3RhYmxlYCBhbmQgdGhlIGBzYW1wbGVzaGVldGAuIFdlIHdpbGwgYWxzbyBuZWVkIGEgKipkZXNpZ24gZm9ybXVsYSoqIHRvIHNwZWNpZnkgb3VyIG1vZGVsLgoKIyMgTWFraW5nIG1vZGVsIGNob2ljZXMKClRoZSBkZXNpZ24gZm9ybXVsYSBzcGVjaWZpZWQgaW5mb3JtcyBtYW55IG9mIHRoZSBERVNlcTIgZnVuY3Rpb25zIGhvdyB0byB0cmVhdCB0aGUgc2FtcGxlcyBpbiB0aGUgYW5hbHlzaXMsIHNwZWNpZmljYWxseSB3aGljaCBjb2x1bW4ocykgaW4gdGhlIHNhbXBsZSBtZXRhZGF0YSB0YWJsZSBhcmUgcmVsZXZhbnQgdG8gdGhlIGV4cGVyaW1lbnRhbCBkZXNpZ24uCgpJbiB0aGlzIGNhc2UsIHdlIGFyZW4ndCBhd2FyZSBvZiBhbnkgW2NvdmFyaWF0ZXNdKGh0dHBzOi8vbWV0aG9kcy1zYWdlcHViLWNvbS5wcm94eS5saWIudW1pY2guZWR1L3JlZmVyZW5jZS9lbmN5Yy1vZi1yZXNlYXJjaC1kZXNpZ24vbjg1LnhtbCkgdGhhdCBzaG91bGQgYmUgY29uc2lkZXJlZCBpbiBvdXIgY29tcGFyaXNvbnMuIEhvd2V2ZXIsIGlmIHRoZXJlIGFyZSBhZGRpdGlvbmFsIGF0dHJpYnV0ZXMgb2YgdGhlIHNhbXBsZXMgdGhhdCBtYXkgaW1wYWN0IHRoZSBERSBjb21wYXJpc29ucywgbGlrZSBzZXgsIGRhdGUgb2YgY29sbGVjdGlvbiwgb3IgcGF0aWVudCBvZiBvcmlnaW4sIHRoZXNlIHNob3VsZCBiZSBhZGRlZCBhcyBbYWRkaXRpb25hbCBjb2x1bW5zXShodHRwczovL3N1cHBvcnQuYmlvY29uZHVjdG9yLm9yZy9wLzc1MzA5LykgaW4gdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiB0YWJsZSBhbmQgW2FkZGVkIHRvIGEgZGVzaWduIGZvcm11bGFdKGh0dHBzOi8vc3VwcG9ydC5iaW9jb25kdWN0b3Iub3JnL3AvOTg3MDAvKS4KCk1vcmUgY29tcGxleCBxdWVzdGlvbnMsIGluY2x1ZGluZyBhZGRpbmcgImludGVyYWN0aW9uIHRlcm1zIiBhcmUgaGVscGZ1bGx5IGRlc2NyaWJlZCBpbiBbdGhpcyBzdXBwb3J0IHRocmVhZF0oaHR0cHM6Ly9zdXBwb3J0LmJpb2NvbmR1Y3Rvci5vcmcvcC85ODYyOC8pIGFzIHdlbGwgYXMgaW4gdGhlIFtERVNlcTIgdmlnbmV0dGVdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNpbnRlcmFjdGlvbnMpLCB3aGljaCBpcyBhbiBleGNlbGxlbnQgZG9jdW1lbnQuCgpUaGUgZGVzaWduIGZvcm11bGEgc3BlY2lmaWVzIHRoZSBjb2x1bW4ocykgaW4gdGhlIG1ldGFkYXRhIHRhYmxlIGFuZCBob3cgdGhleSBzaG91bGQgYmUgdXNlZCBpbiB0aGUgYW5hbHlzaXMuIEZvciBvdXIgZGF0YXNldCB3ZSBvbmx5IGhhdmUgb25lIGNvbHVtbiB3ZSBhcmUgaW50ZXJlc3RlZCBpbiwgdGhhdCBpcyBgY29uZGl0aW9uYC4gVGhpcyBjb2x1bW4gaGFzIHR3byBmYWN0b3IgbGV2ZWxzLCB3aGljaCB0ZWxscyBERVNlcTIgdGhhdCBmb3IgZWFjaCBnZW5lIHdlIHdhbnQgdG8gZXZhbHVhdGUgZ2VuZSBleHByZXNzaW9uIGNoYW5nZSB3aXRoIHJlc3BlY3QgdG8gdGhlc2UgZGlmZmVyZW50IGxldmVscy4KCmBgYHtyIERFU2VxMk9iamVjdH0KIyMgQ3JlYXRlIERFU2VxIG9iamVjdCwgbGluZSBieSBsaW5lCmRkcyA9IERFU2VxRGF0YVNldEZyb21NYXRyaXgoY291bnREYXRhID0gY291bnRfdGFibGUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbERhdGEgPSBzYW1wbGVzaGVldCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGVzaWduID0gfiBjb25kaXRpb24pCmRkcwpgYGAKCk5vdGljZSB0aGF0IHByaW50aW5nIHRoZSBgZGRzYCBvYmplY3QgaGVscGZ1bGx5IHNob3dzIHVzIHNvbWUgaGVscGZ1bCBpbmZvcm1hdGlvbjoKCiogVGhlIGRpbWVuc2lvbiAobnVtYmVyIG9mIGdlbmVzIGJ5IG51bWJlciBvZiBzYW1wbGVzKSwKKiBUaGUgZ2VuZSBpZGVudGlmaWVycywKKiBUaGUgc2FtcGxlIGlkZW50aWZpZXJzLAoqIFRoZSBhZGRpdGlvbmFsIGNvbHVtbiBuYW1lcyBnaXZpbmcgaW5mb3JtYXRpb24gYWJvdXQgdGhlIHNhbXBsZXMKCioqQ2hlY2twb2ludCoqOiAqSWYgeW91IHNlZSBgZGRzYCBpbiB5b3VyIGVudmlyb25tZW50IHBhbmVsLCBwbGVhc2UgaW5kaWNhdGUgd2l0aCB0aGUgZ3JlZW4gJ2NoZWNrJyBidXR0b24uIE90aGVyd2lzZSwgcGxlYXNlIHVzZSAgdXNlIHRoZSByZWQgJ3gnIGJ1dHRvbiBpbiB5b3VyIHpvb20gcmVhY3Rpb24gcGFuZWwgdG8gaGF2ZSB0aGlzIHN0ZXAgcmVwZWF0ZWQuIFlvdSBjYW4gdXNlIHRoZSByZWQgJ3gnIHRvIGJlIHB1dCBpbiBhIGJyZWFrb3V0IHJvb20gZm9yIGhlbHAqCgotLS0KCgojIFByZS1maWx0ZXJpbmcKCldoaWxlIG5vdCBuZWNlc3NhcnksIFtwcmUtZmlsdGVyaW5nXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjcHJlLWZpbHRlcmluZykgdGhlIGNvdW50IHRhYmxlIGhlbHBzIHRvIG5vdCBvbmx5IHJlZHVjZSB0aGUgc2l6ZSBvZiB0aGUgREVTZXEyIG9iamVjdCwgYnV0IGFsc28gZ2l2ZXMgeW91IGEgc2Vuc2Ugb2YgaG93IG1hbnkgZ2VuZXMgd2VyZSByZWFzb25hYmx5IG1lYXN1cmVkIGF0IHRoZSBzZXF1ZW5jaW5nIGRlcHRoIGdlbmVyYXRlZCBmb3IgeW91ciBzYW1wbGVzLgoKSGVyZSB3ZSB3aWxsIGZpbHRlciBvdXQgYW55IGdlbmVzIHRoYXQgaGF2ZSBsZXNzIHRoYW4gMTAgY291bnRzIGFjcm9zcyBhbnkgb2YgdGhlIHNhbXBsZXMuIFRoaXMgaXMgYSBmYWlybHkgc3RhbmRhcmQgbGV2ZWwgb2YgZmlsdGVyaW5nLCBidXQgY2FuIGZpbHRlciBkYXRhIGxlc3MvbW9yZSBkZXBlbmRpbmcgb24gcXVhbGl0eSBjb250cm9sIG1ldHJpY3MgZnJvbSBhbGlnbm1lbnRzIGFuZCBzZXF1ZW5jaW5nIGRlcHRoIG9yIHRvdGFsIG51bWJlciBvZiBzYW1wbGVzLgoKYGBge3IgUHJlRmlsdGVyfQprZWVwID0gcm93U3Vtcyhjb3VudHMoZGRzKSkgPj0gMTAKZGRzID0gZGRzW2tlZXAsXQpkZHMKYGBgCgpOb3RpY2UgdGhlIGBkZHNgIG9iamVjdCBoYXMgbGVzcyBlbGVtZW50cyBhZnRlciBmaWx0ZXJpbmcuIFRoZXJlIHdlcmUgcXVpdGUgYSBudW1iZXIgb2YgZ2VuZXMgdGhhdCB3ZXJlIG5vdCBtZWFzdXJlZCBpbiBvdXIgZXhwZXJpbWVudC4KCioqQ2hlY2twb2ludCoqOiAqUXVlc3Rpb25zPyoKCiMgT3B0aW9uYWwgY29udGVudAoKPGRldGFpbHM+CiAgICA8c3VtbWFyeT4qQ2xpY2sgZm9yIGhvdyB0byBtb2RlbCBiYXRjaCBlZmZlY3RzIHdpdGggREVTZXEyKjwvc3VtbWFyeT4KCiAgICBEaWZmZXJlbmNlcyBiZXR3ZWVuIHNhbXBsZXMgY2FuIGFsc28gYmUgZHVlIHRvIHRlY2huaWNhbCByZWFzb25zLCBzdWNoIGFzIGNvbGxlY3Rpb24gb24gZGlmZmVyZW50IGRheXMgb3IgZGlmZmVyZW50IHNlcXVlbmNpbmcgcnVucy4gRGlmZmVyZW5jZXMgYmV0d2VlbiBzYW1wbGVzIHRoYXQgYXJlIG5vdCBkdWUgdG8gYmlvbG9naWNhbCBmYWN0b3JzIGFzIGNhbGxlZCAqKmJhdGNoIGVmZmVjdHMqKi4gV2UgY2FuIGluY2x1ZGUgYmF0Y2ggZWZmZWN0cyBpbiBvdXIgZGVzaWduIG1vZGVsIGluIHRoZSBzYW1lIHdheSBhcyBjb3ZhcmlhdGVzLCBhcyBsb25nIGFzIHRoZSB0ZWNobmljYWwgZ3JvdXBzIGRvIG5vdCBvdmVybGFwLCBvciAqKmNvbmZvdW5kKiosIHRoZSBiaW9sb2dpY2FsIHRyZWF0bWVudCBncm91cHMuCgogICAgTGV0J3MgdHJ5IGFkZCBzb21lIGFkZGl0aW9uYWwgbWV0YS1kYXRhIGluZm9ybWF0aW9uIHdoZXJlIHdlIGhhdmUgY291bmZvdW5kaW5nIGJhdGNoIGVmZmVjdHMgYW5kIGNyZWF0ZSBhbm90aGVyIERFU2VxMiBvYmplY3QuCgpgYGB7ciBDb25mb3VuZGVyc30Kc2FtcGxlc2hlZXRfYmF0Y2ggPSBzYW1wbGVzaGVldApzYW1wbGVzaGVldF9iYXRjaCRiYXRjaCA9IGZhY3RvcihjKHJlcChjKCJEYXkxIiksIDIpLAogICAgICAgICAgICAgICAgICAgICAgIHJlcChjKCJEYXkyIiksIDIpLAogICAgICAgICAgICAgICAgICAgICAgIHJlcChjKCJEYXkzIiksIDIpKSwKICAgICAgICAgICAgICAgICAgICAgbGV2ZWxzID0gYygiRGF5MSIsICJEYXkyIiwgIkRheTMiKSkKCmRkc19iYXRjaCA9IERFU2VxRGF0YVNldEZyb21NYXRyaXgoY291bnREYXRhID0gY291bnRfdGFibGUsCiAgICAgICAgICAgICAgICAgICAgICBjb2xEYXRhID0gc2FtcGxlc2hlZXRfYmF0Y2gsCiAgICAgICAgICAgICAgICAgICAgICBkZXNpZ24gPSB+IGJhdGNoICsgY29uZGl0aW9uKQpgYGAKPC9kZXRhaWxzPgo8YnI+CgojIFN1bW1hcnkKCkluIHRoaXMgc2VjdGlvbiwgd2U6CgoqIExvYWRlZCB0aGUgbmVjZXNzYXJ5IGlucHV0IGZpbGVzIGludG8gb3VyIFIgc2Vzc2lvbgoqIERpc2N1c3NlZCBtb2RlbCBkZXNpZ24gZm9yIERFU2VxMgoqIEluaXRpYWxpemVkIGEgREVTZXEyIGRhdGEgc2V0CiogRmlsdGVyZWQgb3VyIGNvdW50IGRhdGEKCk5vdyB0aGF0IHdlJ3ZlIGNyZWF0ZWQgb3VyIERFU2VxMiBvYmplY3RzLCBpbmNsdWRpbmcgc3BlY2lmeWluZyB3aGF0IG1vZGVsIGlzIGFwcHJvcHJpYXRlIGZvciBvdXIgZGF0YSwgYW5kIGZpbHRlcmVkIG91ciBkYXRhLCB3ZSBjYW4gcHJvY2VlZCB3aXRoIGFzc2Vzc2luZyB0aGUgaW1wYWN0IG9mIHRoZSBleHBlcmltZW50YWwgY29uZGl0aW9ucyBvbiBnZW5lIGV4cHJlc3Npb24gYWNyb3NzIG91ciBzYW1wbGVzLgoKLS0tCgojIFNvdXJjZXMKClRyYWluaW5nIHJlc291cmNlcyB1c2VkIHRvIGRldmVsb3AgbWF0ZXJpYWxzOgoKKiBIQkMgREdFIHNldHVwOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wMV9ER0Vfc2V0dXBfYW5kX292ZXJ2aWV3Lmh0bWwKKiBIQkMgQ291bnQgTm9ybWFsaXphdGlvbjogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDJfREdFX2NvdW50X25vcm1hbGl6YXRpb24uaHRtbAoqIERFU2VxMiBzdGFuZGFyZCB2aWduZXR0ZTogaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sCiogREVTZXEyIGJlZ2lubmVycyB2aWduZXR0ZTogaHR0cHM6Ly9iaW9jLmlzbS5hYy5qcC9wYWNrYWdlcy8yLjE0L2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9iZWdpbm5lci5wZGYKKiBCaW9jb25kdWN0b3IgUk5BLXNlcSBXb3JrZmxvd3M6IGh0dHBzOi8vd3d3LmJpb2NvbmR1Y3Rvci5vcmcvaGVscC9jb3Vyc2UtbWF0ZXJpYWxzLzIwMTUvTGVhcm5CaW9jb25kdWN0b3JGZWIyMDE1L0IwMi4xX1JOQVNlcS5odG1sCiogQ0NETCBHYXN0cmljIGNhbmNlciB0cmFpbmluZyBtYXRlcmlhbHM6IGh0dHBzOi8vYWxleHNsZW1vbmFkZS5naXRodWIuaW8vdHJhaW5pbmctbW9kdWxlcy9STkEtc2VxLzAzLWdhc3RyaWNfY2FuY2VyX2V4cGxvcmF0b3J5Lm5iLmh0bWwKKiBDQ0RMIE5ldXJvYmxhc3RvbWEgdHJhaW5pbmcgbWF0ZXJpYWxzOiBodHRwczovL2FsZXhzbGVtb25hZGUuZ2l0aHViLmlvL3RyYWluaW5nLW1vZHVsZXMvUk5BLXNlcS8wNS1uYl9jZWxsX2xpbmVfREVTZXEyLm5iLmh0bWwKCgoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQpgYGAKCi0tLQoKVGhlc2UgbWF0ZXJpYWxzIGhhdmUgYmVlbiBhZGFwdGVkIGFuZCBleHRlbmRlZCBmcm9tIG1hdGVyaWFscyBsaXN0ZWQgYWJvdmUuIFRoZXNlIGFyZSBvcGVuIGFjY2VzcyBtYXRlcmlhbHMgZGlzdHJpYnV0ZWQgdW5kZXIgdGhlIHRlcm1zIG9mIHRoZSBbQ3JlYXRpdmUgQ29tbW9ucyBBdHRyaWJ1dGlvbiBsaWNlbnNlIChDQyBCWSA0LjApXShodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS80LjAvKSwgd2hpY2ggcGVybWl0cyB1bnJlc3RyaWN0ZWQgdXNlLCBkaXN0cmlidXRpb24sIGFuZCByZXByb2R1Y3Rpb24gaW4gYW55IG1lZGl1bSwgcHJvdmlkZWQgdGhlIG9yaWdpbmFsIGF1dGhvciBhbmQgc291cmNlIGFyZSBjcmVkaXRlZC4K