In this module, we will learn:

  • How to ‘unblind’ our samples
  • How to ensure high quality count data with filtering


Differential Expression Workflow

Here we will setup the inputs needed to initialize DESeq2 before testing for differential expression.


Sample Information

As introduced at the beginning of the workshop, we have downloaded and prepared data from an existing publication (Zhang et al., 2019), wherein one goal is to understand the gene expression differences in wild-type mice that that were “iron replete” and fed a control diet (we label these “control”) and “iron deficient” (we label these “deficient”).

Read Sample Table

Our next step will be to describe the samples within our R session, so that we make the proper comparisons with DESeq2. Let’s check the sample names from the count table.

colnames(count_table)
[1] "sample_A" "sample_B" "sample_C" "sample_D" "sample_E" "sample_F"

Based on the sample names that are the columns of count_table, our samples are blinded, e.g. the sample names don’t clearly correspond to treatment groups. We will need to specify which sample IDs connect to which experimental conditions.

Typically sample phenotype data, including experimental conditions, are stored as Excel or CSV files that we can read into R, and then use when creating a DESeq2 object. If you are unfamilar with CSV files or how to generate them, there are tutorials available to guide you through the process.

Tip: Sample naming conventions

Use only alpha-numeric characters (A-Z, a-z, 0-9), and separate parts of the name with underscores (_) or dots (.). Do not begin sample names with numbers.

We’ll load our ‘pre-made’ sample information sheet, samplesheet.csv, to unblind our samples.

samplesheet = read.table("data/samplesheet.csv",
                       sep = ",",
                       header = TRUE,
                       row.names = 1)

Alternatively, we could manually create a sample sheet, ensuring that the sample IDs match our count table.

We can look at the object by typing its name and hitting Enter. Note, for larger experiments, you may want to use head().

samplesheet
         genotype condition
sample_A       wt   control
sample_B       wt   control
sample_C       wt   control
sample_D       wt deficient
sample_E       wt deficient
sample_F       wt deficient

Checkpoint: If you have loaded samplesheet, please indicate with the green ‘check’ button. Otherwise, please use the red ‘x’ button to have the command repeated

In this example data, all mice are of a wild-type genotype with three samples from control diet mice (labeled as “control”) and three samples from iron deficient diet fed mice (labeled as “deficient”). Again, for larger experiments, you may want to examine the coding of the samples and how many group labels are present by looking at the unique() values of the relevant columns.

unique(samplesheet$condition)
[1] "control"   "deficient"

Replicates and sequencing depth in RNA-seq experiments

Question In this experiment, we only have samples from three mice per each condition. Do we have enough replicates?

The goal of our analysis is to separate the “interesting” biological contributions from the “uninteresting” technical or other contributions that either cannot be or were not controlled in the experimental design. The more sources of variation, such as samples coming from heterogenous tissues or experiments with incomplete knockdowns, the more replicates (>3) are recommended.

Image of technical, biological, and experimental contributors to gene expression, from HBC training materials

For a more in depth discussion of experimental design considerations, particularly for the number of replicates, please read A Beginner’s Guide to Analysis of RNA Sequencing Data or papers like this one by Hart et al that focus on estimating statistical power for RNA-seq experiments.

Sample table formatting

Next, we’ll format our table so that we have the appropriate data type (an ordered factor) for DESeq2 to recognize our treatment groups and appropriately compare samples.

# tidy version
samplesheet_ready <- samplesheet %>% mutate(condition = factor(condition, 
                                          levels = c('control', 'deficient')))

# base version
samplesheet_ready <- samplesheet
samplesheet_ready$condition = factor(samplesheet_ready$condition,
                            levels = c('control', 'deficient'))

unique(samplesheet_ready$condition)
[1] control   deficient
Levels: control deficient

Notice that we set the levels in a particular order. This is important for setting the “Control” (or “Reference”) group as the denominator in the default comparisons when we setup our DESeq2 model.

Before we proceed, we need to make sure that the sample labels (column names) in the count table match the sample information table (row names), including the order. If the sample labels don’t match, then we will see an error and need to correct the labels prior to proceeding. Checking the sample information table is extremely important to ensure that the correct samples are grouped together for comparisons.

all(colnames(count_table) == rownames(samplesheet_ready))
[1] TRUE

This line of code checks if both the identity and order match between our count_table and our samplesheet. If, in the course of using your own data, this returns FALSE, try using the match() function to rearrange the columns of count_table (or the rows of samplesheet) to get them to match.

Checkpoint: If you your sample info check returns TRUE, please indicate with the green ‘yes’ button. Otherwise, please use the red ‘x’ button to have the command repeated

Creating DESeq2 object

Bioconductor software packages often define and use custom structures to store data in a way that suits biological data, such as illustrated below from Huber et al. 2015.

A breakdown of the SummarizedExperiment class.

These custom data structures have pre-defined data slots, which hold specific types/classes of data and therefore can be more easily accessed by functions from the same package.

To create the DESeqDataSet we need two important inputs:

  1. A table that assigns the condition labels for each sample (that we read in and formatted)
  2. A raw count matrix

Is our count matrix from our RSEM outputs considered a “raw” count matrix by DESeq2?

If we think back to the RSEM outputs, the ‘expected_counts’ table may include fractional amounts due to how the alignment tool resolves reads mapping to multiple locuses). However, recent versions of DESeq2 handle this without rounding.

To create the DESeqDataSet we will need the count_table and the samplesheet. We will also need a design formula to specify our model.

Making model choices

We’ll talk about the design formula more later but for now it’s important to understand that the design specifies the relevant column(s) in the metadata table and how they should be used when fitting a model for our data.

For our dataset we only have one column we are interested in, that is condition. This column has two factor levels, which tells DESeq2 that for each gene we want to evaluate gene expression change with respect to these different levels. Although the design formula is required, it is not used at this step.

## Create DESeq object, line by line
dds = DESeqDataSetFromMatrix(countData = count_table,
                              colData = samplesheet_ready,
                              design = ~ condition)
dds
class: DESeqDataSet 
dim: 55492 6 
metadata(1): version
assays(1): counts
rownames(55492): ENSMUSG00000000001 ENSMUSG00000000003 ... mcherry tdtomato
rowData names(0):
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(2): genotype condition

Notice that printing the dds object helpfully shows us some helpful information:

  • The dimension (number of genes by number of samples),
  • The gene identifiers,
  • The sample identifiers,
  • The additional column names giving information about the samples

Checkpoint: If you see dds in your environment panel, please indicate with the green ‘check’ button. Otherwise, please use use the red ‘x’ button in your zoom reaction panel to have this step repeated. You can use the red ‘x’ to be put in a breakout room for help


Pre-filtering

While not necessary, pre-filtering helps to not only reduce the size of the DESeq2 object, but also gives you a sense of how many genes were reasonably measured at the sequencing depth generated for your samples.

Here we will filter out any genes that have less than 10 counts across any of the samples. This is a fairly standard level of filtering, but can filter data less/more depending on quality control metrics from alignments and sequencing depth or total number of samples.

keep = rowSums(counts(dds)) >= 10
dds_filtered = dds[keep,]
dds_filtered
class: DESeqDataSet 
dim: 16249 6 
metadata(1): version
assays(1): counts
rownames(16249): ENSMUSG00000000001 ENSMUSG00000000028 ... ENSMUSG00000118651 ENSMUSG00000118653
rowData names(0):
colnames(6): sample_A sample_B ... sample_E sample_F
colData names(2): genotype condition

Notice the dds_filtered object has less elements than the unfiltered dds object, indicating that a number of genes were not measured in our experiment.

Checkpoint: Questions?

Summary

In this section, we:

  • Loaded the necessary input files into our R session
  • Discussed single variable model design for DESeq2
  • Initialized a DESeq2 data set
  • Filtered our count data

Now that we’ve created our DESeq2 objects, including specifying what model is appropriate for our data, and filtered our data, we can proceed with assessing the impact of the experimental conditions on gene expression for our samples.


Sources

Training resources used to develop materials:


These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.




Previous lesson Top of this lesson Next lesson
LS0tCnRpdGxlOiAiREUgSW5pdGlhbGl6YXRpb24iCmF1dGhvcjogIlVNIEJpb2luZm9ybWF0aWNzIENvcmUiCmRhdGU6ICJgciBTeXMuRGF0ZSgpYCIKb3V0cHV0OgogICAgICAgIGh0bWxfZG9jdW1lbnQ6CiAgICAgICAgICAgIGluY2x1ZGVzOgogICAgICAgICAgICAgICAgaW5faGVhZGVyOiBoZWFkZXIuaHRtbAogICAgICAgICAgICB0aGVtZTogcGFwZXIKICAgICAgICAgICAgdG9jOiB0cnVlCiAgICAgICAgICAgIHRvY19kZXB0aDogNAogICAgICAgICAgICB0b2NfZmxvYXQ6IHRydWUKICAgICAgICAgICAgbnVtYmVyX3NlY3Rpb25zOiBmYWxzZQogICAgICAgICAgICBmaWdfY2FwdGlvbjogdHJ1ZQogICAgICAgICAgICBtYXJrZG93bjogR0ZNCiAgICAgICAgICAgIGNvZGVfZG93bmxvYWQ6IHRydWUKLS0tCgo8c3R5bGUgdHlwZT0idGV4dC9jc3MiPgpib2R5LCB0ZCB7CiAgIGZvbnQtc2l6ZTogMThweDsKfQpjb2RlLnJ7CiAgZm9udC1zaXplOiAxMnB4Owp9CnByZSB7CiAgZm9udC1zaXplOiAxMnB4Cn0KPC9zdHlsZT4KCmBgYHtyLCBpbmNsdWRlID0gRkFMU0V9CnNvdXJjZSgiLi4vYmluL2NodW5rLW9wdGlvbnMuUiIpCmtuaXRyX2ZpZ19wYXRoKCIwNy0iKQpgYGAKCmBgYHtyIExvYWRSdW5uaW5nRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KERFU2VxMikKIyBsb2FkKCJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKSW4gdGhpcyBtb2R1bGUsIHdlIHdpbGwgbGVhcm46CgoqIEhvdyB0byAndW5ibGluZCcgb3VyIHNhbXBsZXMKKiBIb3cgdG8gZW5zdXJlIGhpZ2ggcXVhbGl0eSBjb3VudCBkYXRhIHdpdGggZmlsdGVyaW5nCgo8YnI+CgojIERpZmZlcmVudGlhbCBFeHByZXNzaW9uIFdvcmtmbG93IHsudW5saXN0ZWQgLnVubnVtYmVyZWR9CgpIZXJlIHdlIHdpbGwgc2V0dXAgdGhlIGlucHV0cyBuZWVkZWQgdG8gaW5pdGlhbGl6ZSBERVNlcTIgYmVmb3JlIHRlc3RpbmcgZm9yIGRpZmZlcmVudGlhbCBleHByZXNzaW9uLgoKIVtdKC4vaW1hZ2VzL3dheWZpbmRlci93YXlmaW5kZXItREVTZXEySW5pdC5wbmcpe3dpZHRoPTc1JX0KCi0tLQoKIyBTYW1wbGUgSW5mb3JtYXRpb24KCkFzIGludHJvZHVjZWQgYXQgdGhlIGJlZ2lubmluZyBvZiB0aGUgd29ya3Nob3AsIHdlIGhhdmUgZG93bmxvYWRlZCBhbmQgcHJlcGFyZWQgZGF0YSBmcm9tIGFuIGV4aXN0aW5nIHB1YmxpY2F0aW9uIFsoWmhhbmcgZXQgYWwuLCAyMDE5KV0oaHR0cHM6Ly9lbGlmZXNjaWVuY2VzLm9yZy9hcnRpY2xlcy80Njk3NiksIHdoZXJlaW4gb25lIGdvYWwgaXMgdG8gdW5kZXJzdGFuZCB0aGUgZ2VuZSBleHByZXNzaW9uIGRpZmZlcmVuY2VzIGluIHdpbGQtdHlwZSBtaWNlIHRoYXQgdGhhdCB3ZXJlICJpcm9uIHJlcGxldGUiIGFuZCBmZWQgYSBjb250cm9sIGRpZXQgKHdlIGxhYmVsIHRoZXNlICJjb250cm9sIikgYW5kICJpcm9uIGRlZmljaWVudCIgKHdlIGxhYmVsIHRoZXNlICJkZWZpY2llbnQiKS4KCiFbXShpbWFnZXMvTW9kdWxlMDBfdG9wX2Rvd24uanBnKQoKIyMgUmVhZCBTYW1wbGUgVGFibGUKCk91ciBuZXh0IHN0ZXAgd2lsbCBiZSB0byBkZXNjcmliZSB0aGUgc2FtcGxlcyB3aXRoaW4gb3VyIFIgc2Vzc2lvbiwgc28gdGhhdCB3ZSBtYWtlIHRoZSBwcm9wZXIgY29tcGFyaXNvbnMgd2l0aCBERVNlcTIuIExldCdzIGNoZWNrIHRoZSBzYW1wbGUgbmFtZXMgZnJvbSB0aGUgY291bnQgdGFibGUuCgpgYGB7ciBDb2x1bW5OYW1lc30KY29sbmFtZXMoY291bnRfdGFibGUpCmBgYAoKQmFzZWQgb24gdGhlIHNhbXBsZSBuYW1lcyB0aGF0IGFyZSB0aGUgY29sdW1ucyBvZiBgY291bnRfdGFibGVgLCBvdXIgc2FtcGxlcyBhcmUgYmxpbmRlZCwgZS5nLiB0aGUgc2FtcGxlIG5hbWVzIGRvbid0IGNsZWFybHkgY29ycmVzcG9uZCB0byB0cmVhdG1lbnQgZ3JvdXBzLiBXZSB3aWxsIG5lZWQgdG8gc3BlY2lmeSB3aGljaCBzYW1wbGUgSURzIGNvbm5lY3QgdG8gd2hpY2ggZXhwZXJpbWVudGFsIGNvbmRpdGlvbnMuCgpUeXBpY2FsbHkgc2FtcGxlIHBoZW5vdHlwZSBkYXRhLCBpbmNsdWRpbmcgZXhwZXJpbWVudGFsIGNvbmRpdGlvbnMsIGFyZSBzdG9yZWQgYXMgRXhjZWwgb3IgQ1NWIGZpbGVzIHRoYXQgd2UgY2FuIHJlYWQgaW50byBSLCBhbmQgdGhlbiB1c2Ugd2hlbiBjcmVhdGluZyBhIERFU2VxMiBvYmplY3QuIElmIHlvdSBhcmUgdW5mYW1pbGFyIHdpdGggQ1NWIGZpbGVzIG9yIGhvdyB0byBnZW5lcmF0ZSB0aGVtLCB0aGVyZSBhcmUgW3R1dG9yaWFsc10oaHR0cHM6Ly93d3cud2lraWhvdy5jb20vQ3JlYXRlLWEtQ1NWLUZpbGUpIGF2YWlsYWJsZSB0byBndWlkZSB5b3UgdGhyb3VnaCB0aGUgcHJvY2Vzcy4KCj4gIyBUaXA6IFNhbXBsZSBuYW1pbmcgY29udmVudGlvbnMgey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiBVc2Ugb25seSBhbHBoYS1udW1lcmljIGNoYXJhY3RlcnMgKEEtWiwgYS16LCAwLTkpLCBhbmQgc2VwYXJhdGUgcGFydHMgb2YgdGhlIG5hbWUgd2l0aCB1bmRlcnNjb3JlcyAoYF9gKSBvciBkb3RzIChgLmApLiBEbyBub3QgYmVnaW4gc2FtcGxlIG5hbWVzIHdpdGggbnVtYmVycy4KCldlJ2xsIGxvYWQgb3VyICdwcmUtbWFkZScgc2FtcGxlIGluZm9ybWF0aW9uIHNoZWV0LCBgc2FtcGxlc2hlZXQuY3N2YCwgdG8gdW5ibGluZCBvdXIgc2FtcGxlcy4KCmBgYHtyIFNhbXBsZXNoZWV0LCBlY2hvID0gRkFMU0UsIGV2YWwgPSBUUlVFfQpzYW1wbGVzaGVldCA9IHJlYWQudGFibGUoIi4uL2RhdGEvUl9kYXRhL3NhbXBsZXNoZWV0LmNzdiIsCiAgICAgICAgICAgICAgICAgICAgICAgc2VwID0gIiwiLAogICAgICAgICAgICAgICAgICAgICAgIGhlYWRlciA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgcm93Lm5hbWVzID0gMSkKYGBgCgpgYGB7ciBTYW1wbGVzaGVldDIsIGV2YWwgPSBGQUxTRX0Kc2FtcGxlc2hlZXQgPSByZWFkLnRhYmxlKCJkYXRhL3NhbXBsZXNoZWV0LmNzdiIsCiAgICAgICAgICAgICAgICAgICAgICAgc2VwID0gIiwiLAogICAgICAgICAgICAgICAgICAgICAgIGhlYWRlciA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgcm93Lm5hbWVzID0gMSkKYGBgCgpBbHRlcm5hdGl2ZWx5LCB3ZSBjb3VsZCBtYW51YWxseSBjcmVhdGUgYSBzYW1wbGUgc2hlZXQsIGVuc3VyaW5nIHRoYXQgdGhlIHNhbXBsZSBJRHMgbWF0Y2ggb3VyIGNvdW50IHRhYmxlLgoKV2UgY2FuIGxvb2sgYXQgdGhlIG9iamVjdCBieSB0eXBpbmcgaXRzIG5hbWUgYW5kIGhpdHRpbmcgRW50ZXIuIE5vdGUsIGZvciBsYXJnZXIgZXhwZXJpbWVudHMsIHlvdSBtYXkgd2FudCB0byB1c2UgYGhlYWQoKWAuCgpgYGB7ciBTaG93U2FtcGxlc2hlZXR9CnNhbXBsZXNoZWV0CmBgYAoKKipDaGVja3BvaW50Kio6ICpJZiB5b3UgaGF2ZSBsb2FkZWQgYHNhbXBsZXNoZWV0YCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICdjaGVjaycgYnV0dG9uLiBPdGhlcndpc2UsIHBsZWFzZSB1c2UgdGhlIHJlZCAneCcgYnV0dG9uIHRvIGhhdmUgdGhlIGNvbW1hbmQgcmVwZWF0ZWQqCgoKSW4gdGhpcyBleGFtcGxlIGRhdGEsIGFsbCBtaWNlIGFyZSBvZiBhIHdpbGQtdHlwZSBnZW5vdHlwZSB3aXRoIHRocmVlIHNhbXBsZXMgZnJvbSBjb250cm9sIGRpZXQgbWljZSAobGFiZWxlZCBhcyAiY29udHJvbCIpIGFuZCB0aHJlZSBzYW1wbGVzIGZyb20gaXJvbiBkZWZpY2llbnQgZGlldCBmZWQgbWljZSAobGFiZWxlZCBhcyAiZGVmaWNpZW50IikuIEFnYWluLCBmb3IgbGFyZ2VyIGV4cGVyaW1lbnRzLCB5b3UgbWF5IHdhbnQgdG8gZXhhbWluZSB0aGUgY29kaW5nIG9mIHRoZSBzYW1wbGVzIGFuZCBob3cgbWFueSBncm91cCBsYWJlbHMgYXJlIHByZXNlbnQgYnkgbG9va2luZyBhdCB0aGUgYHVuaXF1ZSgpYCB2YWx1ZXMgb2YgdGhlIHJlbGV2YW50IGNvbHVtbnMuCgpgYGB7ciBUcmVhdG1lbnRHcm91cFRhYmxlfQp1bmlxdWUoc2FtcGxlc2hlZXQkY29uZGl0aW9uKQpgYGAKCgojIyMgUmVwbGljYXRlcyBhbmQgc2VxdWVuY2luZyBkZXB0aCBpbiBSTkEtc2VxIGV4cGVyaW1lbnRzCgo+ICoqUXVlc3Rpb24qKiBJbiB0aGlzIGV4cGVyaW1lbnQsIHdlIG9ubHkgaGF2ZSBzYW1wbGVzIGZyb20gdGhyZWUgbWljZSBwZXIgZWFjaCBjb25kaXRpb24uIERvIHdlIGhhdmUgZW5vdWdoIHJlcGxpY2F0ZXM/CgpUaGUgZ29hbCBvZiBvdXIgYW5hbHlzaXMgaXMgdG8gc2VwYXJhdGUgdGhlIOKAnGludGVyZXN0aW5n4oCdIGJpb2xvZ2ljYWwgY29udHJpYnV0aW9ucyBmcm9tIHRoZSDigJx1bmludGVyZXN0aW5n4oCdIHRlY2huaWNhbCBvciBvdGhlciBjb250cmlidXRpb25zIHRoYXQgZWl0aGVyIGNhbm5vdCBiZSBvciB3ZXJlIG5vdCBjb250cm9sbGVkIGluIHRoZSBleHBlcmltZW50YWwgZGVzaWduLiBUaGUgbW9yZSBzb3VyY2VzIG9mIHZhcmlhdGlvbiwgc3VjaCBhcyBzYW1wbGVzIGNvbWluZyBmcm9tIGhldGVyb2dlbm91cyB0aXNzdWVzIG9yIGV4cGVyaW1lbnRzIHdpdGggaW5jb21wbGV0ZSBrbm9ja2Rvd25zLCB0aGUgbW9yZSByZXBsaWNhdGVzICg+MykgYXJlIHJlY29tbWVuZGVkLgoKIVtJbWFnZSBvZiB0ZWNobmljYWwsIGJpb2xvZ2ljYWwsIGFuZCBleHBlcmltZW50YWwgY29udHJpYnV0b3JzIHRvIGdlbmUgZXhwcmVzc2lvbiwgZnJvbSBIQkMgdHJhaW5pbmcgbWF0ZXJpYWxzXShpbWFnZXMvZGVfdmFyaWF0aW9uLnBuZyl7d2lkdGg9NzUlfQoKRm9yIGEgbW9yZSBpbiBkZXB0aCBkaXNjdXNzaW9uIG9mIGV4cGVyaW1lbnRhbCBkZXNpZ24gY29uc2lkZXJhdGlvbnMsIHBhcnRpY3VsYXJseSBmb3IgdGhlIG51bWJlciBvZiByZXBsaWNhdGVzLCBwbGVhc2UgcmVhZCBbQSBCZWdpbm5lcuKAmXMgR3VpZGUgdG8gQW5hbHlzaXMgb2YgUk5BIFNlcXVlbmNpbmcgRGF0YV0oaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wbWMvYXJ0aWNsZXMvUE1DNjA5NjM0Ni8pIG9yIHBhcGVycyBsaWtlIHRoaXMgb25lIGJ5IFtIYXJ0IGV0IGFsXShodHRwczovL3d3dy5uY2JpLm5sbS5uaWguZ292L3BtYy9hcnRpY2xlcy9QTUMzODQyODg0LykgdGhhdCBmb2N1cyBvbiBlc3RpbWF0aW5nIHN0YXRpc3RpY2FsIHBvd2VyIGZvciBSTkEtc2VxIGV4cGVyaW1lbnRzLgoKCiMjIyBTYW1wbGUgdGFibGUgZm9ybWF0dGluZwoKTmV4dCwgd2UnbGwgZm9ybWF0IG91ciB0YWJsZSBzbyB0aGF0IHdlIGhhdmUgdGhlIGFwcHJvcHJpYXRlIGRhdGEgdHlwZSAoYW4gb3JkZXJlZCBbZmFjdG9yXShodHRwczovL3N3Y2FycGVudHJ5LmdpdGh1Yi5pby9yLW5vdmljZS1pbmZsYW1tYXRpb24vMTItc3VwcC1mYWN0b3JzLykpIGZvciBERVNlcTIgdG8gcmVjb2duaXplIG91ciB0cmVhdG1lbnQgZ3JvdXBzIGFuZCBhcHByb3ByaWF0ZWx5IGNvbXBhcmUgc2FtcGxlcy4KCmBgYHtyIGVjaG8gPSBGQUxTRSwgZXZhbCA9IEZBTFNFfQojIENvbnNpZGVyIGNvcHlpbmcgc2FtcGxlIHNoZWV0ICYgdXBkYXRpbmcvbXV0YXRpbmcgY29uZGl0aW9uIGNvbHVtbiB0byBiZSBhIGZhY3RvcgpgYGAKCmBgYHtyIFNhbXBsZXNoZWV0RmFjdG9yfQojIHRpZHkgdmVyc2lvbgpzYW1wbGVzaGVldF9yZWFkeSA8LSBzYW1wbGVzaGVldCAlPiUgbXV0YXRlKGNvbmRpdGlvbiA9IGZhY3Rvcihjb25kaXRpb24sIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKCdjb250cm9sJywgJ2RlZmljaWVudCcpKSkKCiMgYmFzZSB2ZXJzaW9uCnNhbXBsZXNoZWV0X3JlYWR5IDwtIHNhbXBsZXNoZWV0CnNhbXBsZXNoZWV0X3JlYWR5JGNvbmRpdGlvbiA9IGZhY3RvcihzYW1wbGVzaGVldF9yZWFkeSRjb25kaXRpb24sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKCdjb250cm9sJywgJ2RlZmljaWVudCcpKQoKdW5pcXVlKHNhbXBsZXNoZWV0X3JlYWR5JGNvbmRpdGlvbikKYGBgCgpOb3RpY2UgdGhhdCB3ZSBzZXQgdGhlIGxldmVscyBpbiBhIHBhcnRpY3VsYXIgb3JkZXIuIFRoaXMgaXMgaW1wb3J0YW50IGZvciBzZXR0aW5nIHRoZSAiQ29udHJvbCIgKG9yICJSZWZlcmVuY2UiKSBncm91cCBhcyB0aGUgZGVub21pbmF0b3IgaW4gdGhlIGRlZmF1bHQgY29tcGFyaXNvbnMgd2hlbiB3ZSBzZXR1cCBvdXIgREVTZXEyIG1vZGVsLgoKQmVmb3JlIHdlIHByb2NlZWQsIHdlIG5lZWQgdG8gbWFrZSBzdXJlIHRoYXQgdGhlIHNhbXBsZSBsYWJlbHMgKGNvbHVtbiBuYW1lcykgaW4gdGhlIGNvdW50IHRhYmxlIG1hdGNoIHRoZSBzYW1wbGUgaW5mb3JtYXRpb24gdGFibGUgKHJvdyBuYW1lcyksIGluY2x1ZGluZyB0aGUgb3JkZXIuIElmIHRoZSBzYW1wbGUgbGFiZWxzIGRvbid0IG1hdGNoLCB0aGVuIHdlIHdpbGwgc2VlIGFuIGVycm9yIGFuZCBuZWVkIHRvIGNvcnJlY3QgdGhlIGxhYmVscyBwcmlvciB0byBwcm9jZWVkaW5nLiBDaGVja2luZyB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIHRhYmxlIGlzIGV4dHJlbWVseSBpbXBvcnRhbnQgdG8gZW5zdXJlIHRoYXQgdGhlIGNvcnJlY3Qgc2FtcGxlcyBhcmUgZ3JvdXBlZCB0b2dldGhlciBmb3IgY29tcGFyaXNvbnMuCgpgYGB7ciBDaGVja1NhbXBsZXNoZWV0fQphbGwoY29sbmFtZXMoY291bnRfdGFibGUpID09IHJvd25hbWVzKHNhbXBsZXNoZWV0X3JlYWR5KSkKYGBgClRoaXMgbGluZSBvZiBjb2RlIGNoZWNrcyBpZiBib3RoIHRoZSBpZGVudGl0eSBhbmQgb3JkZXIgbWF0Y2ggYmV0d2VlbiBvdXIgYGNvdW50X3RhYmxlYCBhbmQgb3VyIGBzYW1wbGVzaGVldGAuIElmLCBpbiB0aGUgY291cnNlIG9mIHVzaW5nIHlvdXIgb3duIGRhdGEsIHRoaXMgcmV0dXJucyBgRkFMU0VgLCB0cnkgdXNpbmcgdGhlIGBtYXRjaCgpYCBmdW5jdGlvbiB0byByZWFycmFuZ2UgdGhlIGNvbHVtbnMgb2YgYGNvdW50X3RhYmxlYCAob3IgdGhlIHJvd3Mgb2YgYHNhbXBsZXNoZWV0YCkgdG8gZ2V0IHRoZW0gdG8gbWF0Y2guCgoqKkNoZWNrcG9pbnQqKjogKklmIHlvdSB5b3VyIHNhbXBsZSBpbmZvIGNoZWNrIHJldHVybnMgYFRSVUVgLCBwbGVhc2UgaW5kaWNhdGUgd2l0aCB0aGUgZ3JlZW4gJ3llcycgYnV0dG9uLiBPdGhlcndpc2UsIHBsZWFzZSB1c2UgdGhlIHJlZCAneCcgYnV0dG9uIHRvIGhhdmUgdGhlIGNvbW1hbmQgcmVwZWF0ZWQqCgojIENyZWF0aW5nIERFU2VxMiBvYmplY3QKCkJpb2NvbmR1Y3RvciBzb2Z0d2FyZSBwYWNrYWdlcyBvZnRlbiBkZWZpbmUgYW5kIHVzZSBjdXN0b20gc3RydWN0dXJlcyB0byBzdG9yZSBkYXRhIGluIGEgd2F5IHRoYXQgc3VpdHMgYmlvbG9naWNhbCBkYXRhLCBzdWNoIGFzIGlsbHVzdHJhdGVkIGJlbG93IGZyb20gW0h1YmVyIGV0IGFsLiAyMDE1XShodHRwczovL3d3dy5uYXR1cmUuY29tL2FydGljbGVzL25tZXRoLjMyNTIpLgoKIVtBIGJyZWFrZG93biBvZiB0aGUgU3VtbWFyaXplZEV4cGVyaW1lbnQgY2xhc3MuXSguL2ltYWdlcy9TdW1tYXJpemVkRXhwZXJpbWVudC5qcGcpCgpUaGVzZSBjdXN0b20gZGF0YSBzdHJ1Y3R1cmVzIGhhdmUgcHJlLWRlZmluZWQgZGF0YSBzbG90cywgd2hpY2ggaG9sZCBzcGVjaWZpYyB0eXBlcy9jbGFzc2VzIG9mIGRhdGEgYW5kIHRoZXJlZm9yZSBjYW4gYmUgbW9yZSBlYXNpbHkgYWNjZXNzZWQgYnkgZnVuY3Rpb25zIGZyb20gdGhlIHNhbWUgcGFja2FnZS4KClRvIGNyZWF0ZSB0aGUgREVTZXFEYXRhU2V0IHdlIG5lZWQgdHdvIGltcG9ydGFudCBpbnB1dHM6CgoxLiBBIHRhYmxlIHRoYXQgYXNzaWducyB0aGUgY29uZGl0aW9uIGxhYmVscyBmb3IgZWFjaCBzYW1wbGUgKHRoYXQgd2UgcmVhZCBpbiBhbmQgZm9ybWF0dGVkKQoxLiBBICpyYXcqIGNvdW50IG1hdHJpeAoKSXMgb3VyIGNvdW50IG1hdHJpeCBmcm9tIG91ciBSU0VNIG91dHB1dHMgY29uc2lkZXJlZCBhICJyYXciIGNvdW50IG1hdHJpeCBieSBERVNlcTI/Cgo+IElmIHdlIHRoaW5rIGJhY2sgdG8gdGhlIFJTRU0gb3V0cHV0cywgdGhlICdleHBlY3RlZF9jb3VudHMnIHRhYmxlIG1heSBpbmNsdWRlIGZyYWN0aW9uYWwgYW1vdW50cyBkdWUgdG8gaG93IHRoZSBhbGlnbm1lbnQgdG9vbCByZXNvbHZlcyByZWFkcyBtYXBwaW5nIHRvIG11bHRpcGxlIGxvY3VzZXMpLiBIb3dldmVyLCByZWNlbnQgdmVyc2lvbnMgb2YgREVTZXEyIGhhbmRsZSB0aGlzIHdpdGhvdXQgcm91bmRpbmcuCgoKVG8gY3JlYXRlIHRoZSBgREVTZXFEYXRhU2V0YCB3ZSB3aWxsIG5lZWQgdGhlIGBjb3VudF90YWJsZWAgYW5kIHRoZSBgc2FtcGxlc2hlZXRgLiBXZSB3aWxsIGFsc28gbmVlZCBhICoqZGVzaWduIGZvcm11bGEqKiB0byBzcGVjaWZ5IG91ciBtb2RlbC4KCiMjIE1ha2luZyBtb2RlbCBjaG9pY2VzCgpXZSdsbCB0YWxrIGFib3V0IHRoZSBkZXNpZ24gZm9ybXVsYSBtb3JlIGxhdGVyIGJ1dCBmb3Igbm93IGl0J3MgaW1wb3J0YW50IHRvIHVuZGVyc3RhbmQgdGhhdCB0aGUgZGVzaWduIHNwZWNpZmllcyB0aGUgcmVsZXZhbnQgY29sdW1uKHMpIGluIHRoZSBtZXRhZGF0YSB0YWJsZSBhbmQgaG93IHRoZXkgc2hvdWxkIGJlIHVzZWQgd2hlbiBmaXR0aW5nIGEgbW9kZWwgZm9yIG91ciBkYXRhLiAKCkZvciBvdXIgZGF0YXNldCB3ZSBvbmx5IGhhdmUgb25lIGNvbHVtbiB3ZSBhcmUgaW50ZXJlc3RlZCBpbiwgdGhhdCBpcyBgY29uZGl0aW9uYC4gVGhpcyBjb2x1bW4gaGFzIHR3byBmYWN0b3IgbGV2ZWxzLCB3aGljaCB0ZWxscyBERVNlcTIgdGhhdCBmb3IgZWFjaCBnZW5lIHdlIHdhbnQgdG8gZXZhbHVhdGUgZ2VuZSBleHByZXNzaW9uIGNoYW5nZSB3aXRoIHJlc3BlY3QgdG8gdGhlc2UgZGlmZmVyZW50IGxldmVscy4gQWx0aG91Z2ggdGhlIGRlc2lnbiBmb3JtdWxhIGlzIHJlcXVpcmVkLCBpdCBpcyBub3QgdXNlZCBhdCB0aGlzIHN0ZXAuCgpgYGB7ciBERVNlcTJPYmplY3R9CiMjIENyZWF0ZSBERVNlcSBvYmplY3QsIGxpbmUgYnkgbGluZQpkZHMgPSBERVNlcURhdGFTZXRGcm9tTWF0cml4KGNvdW50RGF0YSA9IGNvdW50X3RhYmxlLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xEYXRhID0gc2FtcGxlc2hlZXRfcmVhZHksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRlc2lnbiA9IH4gY29uZGl0aW9uKQpkZHMKYGBgCgpOb3RpY2UgdGhhdCBwcmludGluZyB0aGUgYGRkc2Agb2JqZWN0IGhlbHBmdWxseSBzaG93cyB1cyBzb21lIGhlbHBmdWwgaW5mb3JtYXRpb246CgoqIFRoZSBkaW1lbnNpb24gKG51bWJlciBvZiBnZW5lcyBieSBudW1iZXIgb2Ygc2FtcGxlcyksCiogVGhlIGdlbmUgaWRlbnRpZmllcnMsCiogVGhlIHNhbXBsZSBpZGVudGlmaWVycywKKiBUaGUgYWRkaXRpb25hbCBjb2x1bW4gbmFtZXMgZ2l2aW5nIGluZm9ybWF0aW9uIGFib3V0IHRoZSBzYW1wbGVzCgoqKkNoZWNrcG9pbnQqKjogKklmIHlvdSBzZWUgYGRkc2AgaW4geW91ciBlbnZpcm9ubWVudCBwYW5lbCwgcGxlYXNlIGluZGljYXRlIHdpdGggdGhlIGdyZWVuICdjaGVjaycgYnV0dG9uLiBPdGhlcndpc2UsIHBsZWFzZSB1c2UgIHVzZSB0aGUgcmVkICd4JyBidXR0b24gaW4geW91ciB6b29tIHJlYWN0aW9uIHBhbmVsIHRvIGhhdmUgdGhpcyBzdGVwIHJlcGVhdGVkLiBZb3UgY2FuIHVzZSB0aGUgcmVkICd4JyB0byBiZSBwdXQgaW4gYSBicmVha291dCByb29tIGZvciBoZWxwKgoKLS0tCgoKIyBQcmUtZmlsdGVyaW5nCgpXaGlsZSBub3QgbmVjZXNzYXJ5LCBbcHJlLWZpbHRlcmluZ10oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI3ByZS1maWx0ZXJpbmcpIGhlbHBzIHRvIG5vdCBvbmx5IHJlZHVjZSB0aGUgc2l6ZSBvZiB0aGUgREVTZXEyIG9iamVjdCwgYnV0IGFsc28gZ2l2ZXMgeW91IGEgc2Vuc2Ugb2YgaG93IG1hbnkgZ2VuZXMgd2VyZSByZWFzb25hYmx5IG1lYXN1cmVkIGF0IHRoZSBzZXF1ZW5jaW5nIGRlcHRoIGdlbmVyYXRlZCBmb3IgeW91ciBzYW1wbGVzLgoKSGVyZSB3ZSB3aWxsIGZpbHRlciBvdXQgYW55IGdlbmVzIHRoYXQgaGF2ZSBsZXNzIHRoYW4gMTAgY291bnRzIGFjcm9zcyBhbnkgb2YgdGhlIHNhbXBsZXMuIFRoaXMgaXMgYSBmYWlybHkgc3RhbmRhcmQgbGV2ZWwgb2YgZmlsdGVyaW5nLCBidXQgY2FuIGZpbHRlciBkYXRhIGxlc3MvbW9yZSBkZXBlbmRpbmcgb24gcXVhbGl0eSBjb250cm9sIG1ldHJpY3MgZnJvbSBhbGlnbm1lbnRzIGFuZCBzZXF1ZW5jaW5nIGRlcHRoIG9yIHRvdGFsIG51bWJlciBvZiBzYW1wbGVzLgoKYGBge3IgUHJlRmlsdGVyfQprZWVwID0gcm93U3Vtcyhjb3VudHMoZGRzKSkgPj0gMTAKZGRzX2ZpbHRlcmVkID0gZGRzW2tlZXAsXQpkZHNfZmlsdGVyZWQKYGBgCgpOb3RpY2UgdGhlIGBkZHNfZmlsdGVyZWRgIG9iamVjdCBoYXMgbGVzcyBlbGVtZW50cyB0aGFuIHRoZSB1bmZpbHRlcmVkIGBkZHNgIG9iamVjdCwgaW5kaWNhdGluZyB0aGF0IGEgbnVtYmVyIG9mIGdlbmVzIHdlcmUgbm90IG1lYXN1cmVkIGluIG91ciBleHBlcmltZW50LgoKKipDaGVja3BvaW50Kio6ICpRdWVzdGlvbnM/KgoKCgojIFN1bW1hcnkKCkluIHRoaXMgc2VjdGlvbiwgd2U6CgoqIExvYWRlZCB0aGUgbmVjZXNzYXJ5IGlucHV0IGZpbGVzIGludG8gb3VyIFIgc2Vzc2lvbgoqIERpc2N1c3NlZCBzaW5nbGUgdmFyaWFibGUgbW9kZWwgZGVzaWduIGZvciBERVNlcTIKKiBJbml0aWFsaXplZCBhIERFU2VxMiBkYXRhIHNldAoqIEZpbHRlcmVkIG91ciBjb3VudCBkYXRhCgpOb3cgdGhhdCB3ZSd2ZSBjcmVhdGVkIG91ciBERVNlcTIgb2JqZWN0cywgaW5jbHVkaW5nIHNwZWNpZnlpbmcgd2hhdCBtb2RlbCBpcyBhcHByb3ByaWF0ZSBmb3Igb3VyIGRhdGEsIGFuZCBmaWx0ZXJlZCBvdXIgZGF0YSwgd2UgY2FuIHByb2NlZWQgd2l0aCBhc3Nlc3NpbmcgdGhlIGltcGFjdCBvZiB0aGUgZXhwZXJpbWVudGFsIGNvbmRpdGlvbnMgb24gZ2VuZSBleHByZXNzaW9uIGZvciBvdXIgc2FtcGxlcy4KCgotLS0KCiMgU291cmNlcwoKVHJhaW5pbmcgcmVzb3VyY2VzIHVzZWQgdG8gZGV2ZWxvcCBtYXRlcmlhbHM6CgoqIEhCQyBER0Ugc2V0dXA6IGh0dHBzOi8vaGJjdHJhaW5pbmcuZ2l0aHViLmlvL0RHRV93b3Jrc2hvcC9sZXNzb25zLzAxX0RHRV9zZXR1cF9hbmRfb3ZlcnZpZXcuaHRtbAoqIEhCQyBDb3VudCBOb3JtYWxpemF0aW9uOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wMl9ER0VfY291bnRfbm9ybWFsaXphdGlvbi5odG1sCiogREVTZXEyIHN0YW5kYXJkIHZpZ25ldHRlOiBodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwKKiBERVNlcTIgYmVnaW5uZXJzIHZpZ25ldHRlOiBodHRwczovL2Jpb2MuaXNtLmFjLmpwL3BhY2thZ2VzLzIuMTQvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL2JlZ2lubmVyLnBkZgoqIEJpb2NvbmR1Y3RvciBSTkEtc2VxIFdvcmtmbG93czogaHR0cHM6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9oZWxwL2NvdXJzZS1tYXRlcmlhbHMvMjAxNS9MZWFybkJpb2NvbmR1Y3RvckZlYjIwMTUvQjAyLjFfUk5BU2VxLmh0bWwKKiBDQ0RMIEdhc3RyaWMgY2FuY2VyIHRyYWluaW5nIG1hdGVyaWFsczogaHR0cHM6Ly9hbGV4c2xlbW9uYWRlLmdpdGh1Yi5pby90cmFpbmluZy1tb2R1bGVzL1JOQS1zZXEvMDMtZ2FzdHJpY19jYW5jZXJfZXhwbG9yYXRvcnkubmIuaHRtbAoqIENDREwgTmV1cm9ibGFzdG9tYSB0cmFpbmluZyBtYXRlcmlhbHM6IGh0dHBzOi8vYWxleHNsZW1vbmFkZS5naXRodWIuaW8vdHJhaW5pbmctbW9kdWxlcy9STkEtc2VxLzA1LW5iX2NlbGxfbGluZV9ERVNlcTIubmIuaHRtbAoKCgpgYGB7ciBXcml0ZU91dC5SRGF0YSwgZXZhbD1UUlVFLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQojIEhpZGRlbiBjb2RlIGJsb2NrIHRvIHdyaXRlIG91dCBkYXRhIGZvciBrbml0dGluZwojIHNhdmUuaW1hZ2UoZmlsZSA9ICJyZGF0YS9SdW5uaW5nRGF0YS5SRGF0YSIpCmBgYAoKLS0tCgpUaGVzZSBtYXRlcmlhbHMgaGF2ZSBiZWVuIGFkYXB0ZWQgYW5kIGV4dGVuZGVkIGZyb20gbWF0ZXJpYWxzIGxpc3RlZCBhYm92ZS4gVGhlc2UgYXJlIG9wZW4gYWNjZXNzIG1hdGVyaWFscyBkaXN0cmlidXRlZCB1bmRlciB0aGUgdGVybXMgb2YgdGhlIFtDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uIGxpY2Vuc2UgKENDIEJZIDQuMCldKGh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LzQuMC8pLCB3aGljaCBwZXJtaXRzIHVucmVzdHJpY3RlZCB1c2UsIGRpc3RyaWJ1dGlvbiwgYW5kIHJlcHJvZHVjdGlvbiBpbiBhbnkgbWVkaXVtLCBwcm92aWRlZCB0aGUgb3JpZ2luYWwgYXV0aG9yIGFuZCBzb3VyY2UgYXJlIGNyZWRpdGVkLgoKPGJyLz4KPGJyLz4KPGhyLz4KfCBbUHJldmlvdXMgbGVzc29uXShNb2R1bGUwNl9ERUFuYWx5c2lzU2V0dXAuaHRtbCkgfCBbVG9wIG9mIHRoaXMgbGVzc29uXSgjdG9wKSB8IFtOZXh0IGxlc3Nvbl0oTW9kdWxlMDhfREVTZXEyREUuaHRtbCkgfAp8IDotLS0gfCA6LS0tLTogfCAtLS06IHwK