In this module, we will learn:
- The advantages of using gene ids when analyzing RNA-seq data.
- How to find gene symbols and other annotations, using ENSEMBL gene
ids
- How to output our results to file
- General options for functional enrichments and other follow-ups
Differential Expression Workflow
Here we will generate summary figures for our results and annotate
our DE tables.
Generating gene annotations
Since, gene symbols can change over time or be ambiguous we use, and
recommend, using the EMSEMBL reference genome and ENSEMBL IDs for
alignments and we’ve been working with tables and data where all genes
are labeled only by their long ENSEMBL ID. However, this can make it
difficult to quickly look for genes of interest.
Luckily, Bioconductor provides many tools and resources to facilitate
access to genomic
annotation resources.
To start, we will first load the biomaRt
library and choose what reference we want to access. For a more
detailed walk through of using biomaRt, this
training module might be useful, including what to do when
annotations are not 1:1 mappings.
We’ll start by loading the biomaRt
library and calling
the useEnsembl()
function to select the database we’ll use
to extract the information we need. This will download the mapping of
ENSEMBL IDs to gene symbols, enabling us to eventually add the gene
symbol column we want.
library('biomaRt')
ensembl = useEnsembl(dataset = 'mmusculus_gene_ensembl', biomart='ensembl')
Note - this process takes some time and will
take up a larger amount of working memory so proceed with caution if you
try to run these commands on a laptop with less than 4G of
memory
To identify possible filters to restrict our data,
we can use the listFilters
function. To identify the
attributes we want to retrive, we can use the
listAttributes
function. The best approach is to use list
or search functions to help narrow down the available options.
head(listFilters(mart = ensembl), n = 20)
head(listAttributes(ensembl), n = 30)
We can access additional genomic annotations using the bioMart
package. To identify we’ll structure our ‘query’ or search of the
bioMart resources to use the ENSEMBL
id from our alignment to add the gene symbols and gene description
for each gene.
id_mapping = getBM(attributes=c('ensembl_gene_id', 'external_gene_name'),
filters = 'ensembl_gene_id',
values = row.names(assay(dds_batch_fitted)),
mart = ensembl)
Batch submitting query [=======>------------------------] 25% eta: 34sBatch
submitting query [===============>----------------] 50% eta: 16sBatch submitting
query [=======================>--------] 75% eta: 7s
# will take some time for the query to run
# Preview the result
head(id_mapping)
ensembl_gene_id external_gene_name
1 ENSMUSG00000000001 Gnai3
2 ENSMUSG00000000028 Cdc45
3 ENSMUSG00000000031 H19
4 ENSMUSG00000000037 Scml2
5 ENSMUSG00000000049 Apoh
6 ENSMUSG00000000056 Narf
The id_mapping
table now includes the ENSEMBL
information and a gene symbol only for the genes included in our
results. This table should look familiar as it’s the same table we used
to annotate our results table in the last module.
Note: For additional information regarding bioMart,
please consult the ENSEMBL
bioMart vignette or the broader Bioconductor
Annotation Resources vignette.
Outputting results to file
A key aspect of our analysis is preserving the relevant datasets for
both our records and for downstream applications, such as functional
enrichments.
DE results table
We’ll write out our DE results, now that we’ve added information to
the table to help us or our collaborators interpret the results.
write.csv(results_deficient_vs_control,
row.names = FALSE,
na = ".",
file="outputs/tables/DE_results_deficient_vs_control.csv")
write.csv(results_deficient_vs_control_annotated,
row.names = FALSE,
file="outputs/tables/DE_results_deficient_vs_control_annotated.csv")
Subsetting significant genes
You may be interested in creating a table of only the genes that pass
your significance thresholds. A useful way to do this is to
conditionally subset your results. Again, we already created the
call
column, which makes this relatively simple to do:
# tidyr (requires table reformatting)
res_sig <- as_tibble(results_deficient_vs_control, rownames = "gene_ids") %>% filter(call != 'NS')
head(res_sig)
# A tibble: 6 × 8
gene_ids baseMean log2FoldChange lfcSE stat pvalue padj call
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <fct>
1 ENSMUSG00000000275 1662. -0.674 0.200 -3.38 0.000734 3.12e-2 Down
2 ENSMUSG00000000861 1679. 0.685 0.210 3.27 0.00109 3.98e-2 Up
3 ENSMUSG00000001281 532. 1.15 0.242 4.75 0.00000203 5.54e-4 Up
4 ENSMUSG00000002109 125. 1.06 0.300 3.54 0.000398 2.07e-2 Up
5 ENSMUSG00000002985 157. -1.16 0.345 -3.35 0.000814 3.29e-2 Down
6 ENSMUSG00000003865 89.3 2.26 0.627 3.61 0.000310 1.80e-2 Up
dim(res_sig)
[1] 189 8
Once we’ve created this table, we can also write it out to file:
write.csv(res_sig,
row.names = FALSE,
na = ".",
file="outputs/tables/DEGs-only_deficient_vs_control.csv")
R session data
In addition to the individual RObj(s) we saved earlier, we can
capture a snapshot our entire session using the save.image
function. This can be loaded in the same manner as an individual
Robj.
First, we’ll save our session info so we can reference the packages
and versions used to generate these data.
session_summary <- sessionInfo()
save.image(file = "outputs/Robjs/DE_iron.RData")
Overall takeaways
We’ve run through most of the building blocks needed to run a
differential expression analysis and hopefully built up a better
understanding of how differential expression comparisons work,
particularly how experimental design can impact our results.
What to consider moving forward:
- How can I control for technical variation in my experimental
design?
- How much variation is expected with a treatment group?
- What is my RNA quality, and how can that be optimized?
- Are there quality concerns for my sequencing data?
- What comparisons are relevant to my biological question?
- Are there covariates that should be considered?
- What will a differential expression analysis tell me?
Let’s pause here for general questions
Next steps - How do we make sense of large numbers of DE genes?
A way to determine possible broader
biological interpretations from the observed DE results, is
functional enrichments.
There are many options, such as some included in this discussion
thread. Other common functional enrichments approaches are gene set
enrichment analysis, aka GSEA,
Database for Annotation, Visualization and Integrated Discovery, aka DAVID, Ingenity, and iPathway Guide
The University of Michigan has license and support for additional
tools, such as Cytoscape, so we recommend reaching out to staff with Taubman
Library to learn more about resources that might be application
toyour research.
Session Info
sessionInfo()
R version 4.4.0 (2024-04-24)
Platform: x86_64-apple-darwin20
Running under: macOS Sonoma 14.4.1
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: America/Detroit
tzcode source: internal
attached base packages:
[1] stats4 stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] biomaRt_2.60.1 data.table_1.15.4
[3] RColorBrewer_1.1-3 pheatmap_1.0.12
[5] ggrepel_0.9.5 lubridate_1.9.3
[7] forcats_1.0.0 stringr_1.5.1
[9] dplyr_1.1.4 purrr_1.0.2
[11] readr_2.1.5 tidyr_1.3.1
[13] tibble_3.2.1 ggplot2_3.5.1
[15] tidyverse_2.0.0 DESeq2_1.44.0
[17] SummarizedExperiment_1.34.0 Biobase_2.64.0
[19] MatrixGenerics_1.16.0 matrixStats_1.3.0
[21] GenomicRanges_1.56.1 GenomeInfoDb_1.40.1
[23] IRanges_2.38.1 S4Vectors_0.42.1
[25] BiocGenerics_0.50.0 knitr_1.47
[27] rmarkdown_2.27
loaded via a namespace (and not attached):
[1] DBI_1.2.3 httr2_1.0.2 rlang_1.1.4
[4] magrittr_2.0.3 compiler_4.4.0 RSQLite_2.3.7
[7] png_0.1-8 vctrs_0.6.5 pkgconfig_2.0.3
[10] crayon_1.5.3 fastmap_1.2.0 dbplyr_2.5.0
[13] XVector_0.44.0 labeling_0.4.3 utf8_1.2.4
[16] tzdb_0.4.0 UCSC.utils_1.0.0 bit_4.0.5
[19] xfun_0.44 zlibbioc_1.50.0 cachem_1.1.0
[22] jsonlite_1.8.8 progress_1.2.3 blob_1.2.4
[25] highr_0.11 DelayedArray_0.30.1 BiocParallel_1.38.0
[28] parallel_4.4.0 prettyunits_1.2.0 R6_2.5.1
[31] bslib_0.7.0 stringi_1.8.4 jquerylib_0.1.4
[34] Rcpp_1.0.13 Matrix_1.7-0 timechange_0.3.0
[37] tidyselect_1.2.1 rstudioapi_0.16.0 abind_1.4-5
[40] yaml_2.3.8 codetools_0.2-20 curl_5.2.1
[43] lattice_0.22-6 withr_3.0.1 KEGGREST_1.44.1
[46] evaluate_0.23 BiocFileCache_2.12.0 xml2_1.3.6
[49] Biostrings_2.72.1 filelock_1.0.3 pillar_1.9.0
[52] BiocManager_1.30.23 generics_0.1.3 hms_1.1.3
[55] munsell_0.5.1 scales_1.3.0 glue_1.7.0
[58] tools_4.4.0 locfit_1.5-9.10 grid_4.4.0
[61] AnnotationDbi_1.66.0 colorspace_2.1-1 GenomeInfoDbData_1.2.12
[64] cli_3.6.2 rappdirs_0.3.3 fansi_1.0.6
[67] S4Arrays_1.4.1 gtable_0.3.5 sass_0.4.9
[70] digest_0.6.35 SparseArray_1.4.8 farver_2.1.2
[73] memoise_2.0.1 htmltools_0.5.8.1 lifecycle_1.0.4
[76] httr_1.4.7 bit64_4.0.5
These materials have been adapted and extended from materials listed
above. These are open access materials distributed under the terms of
the Creative
Commons Attribution license (CC BY 4.0), which permits unrestricted
use, distribution, and reproduction in any medium, provided the original
author and source are credited.
LS0tCnRpdGxlOiAiREUgQW5ub3RhdGlvbnMgJiBEb3duc3RyZWFtIGFwcGxpY2F0aW9ucyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjExLSIpCmBgYAoKSW4gdGhpcyBtb2R1bGUsIHdlIHdpbGwgbGVhcm46CgoqIFRoZSBhZHZhbnRhZ2VzIG9mIHVzaW5nIGdlbmUgaWRzIHdoZW4gYW5hbHl6aW5nIFJOQS1zZXEgZGF0YS4KKiBIb3cgdG8gZmluZCBnZW5lIHN5bWJvbHMgYW5kIG90aGVyIGFubm90YXRpb25zLCB1c2luZyBFTlNFTUJMIGdlbmUgaWRzCiogSG93IHRvIG91dHB1dCBvdXIgcmVzdWx0cyB0byBmaWxlCiogR2VuZXJhbCBvcHRpb25zIGZvciBmdW5jdGlvbmFsIGVucmljaG1lbnRzIGFuZCBvdGhlciBmb2xsb3ctdXBzCgo8YnI+CgpgYGB7ciBNb2R1bGVzLCBldmFsPVRSVUUsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CmxpYnJhcnkoREVTZXEyKQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkodGlkeXIpCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkobWF0cml4U3RhdHMpCmxpYnJhcnkoZ2dyZXBlbCkKbGlicmFyeShwaGVhdG1hcCkKbGlicmFyeShSQ29sb3JCcmV3ZXIpCiMgbG9hZCgicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQoKIyMgZW5zZW1ibCBTU0wgY2VydCBpc3N1ZSBmbGFnZ2VkIGluIGtuaXR0aW5nIHdpdGggc2VsZWN0ZWQgbWlycm9yCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIGdlbmVyYXRlIHN1bW1hcnkgZmlndXJlcyBmb3Igb3VyIHJlc3VsdHMgYW5kIGFubm90YXRlIG91ciBERSB0YWJsZXMuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERUFubm90YXRpb24ucG5nKXt3aWR0aD03NSV9CgotLS0KCiMgR2VuZXJhdGluZyBnZW5lIGFubm90YXRpb25zCgpTaW5jZSwgZ2VuZSBzeW1ib2xzIGNhbiBjaGFuZ2Ugb3ZlciB0aW1lIG9yIGJlIGFtYmlndW91cyB3ZSB1c2UsIGFuZCByZWNvbW1lbmQsIHVzaW5nIHRoZSBFTVNFTUJMIHJlZmVyZW5jZSBnZW5vbWUgYW5kIEVOU0VNQkwgSURzIGZvciBhbGlnbm1lbnRzIGFuZCB3ZSd2ZSBiZWVuIHdvcmtpbmcgd2l0aCB0YWJsZXMgYW5kIGRhdGEgd2hlcmUgYWxsIGdlbmVzIGFyZSBsYWJlbGVkIG9ubHkgYnkgdGhlaXIgbG9uZyBFTlNFTUJMIElELiBIb3dldmVyLCB0aGlzIGNhbiBtYWtlIGl0IGRpZmZpY3VsdCB0byBxdWlja2x5IGxvb2sgZm9yIGdlbmVzIG9mIGludGVyZXN0LgoKTHVja2lseSwgQmlvY29uZHVjdG9yIHByb3ZpZGVzIG1hbnkgdG9vbHMgYW5kIHJlc291cmNlcyB0byBmYWNpbGl0YXRlIGFjY2VzcyB0byBbZ2Vub21pYyBhbm5vdGF0aW9uIHJlc291cmNlc10oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvd29ya2Zsb3dzL3ZpZ25ldHRlcy9hbm5vdGF0aW9uL2luc3QvZG9jL0Fubm90YXRpb25fUmVzb3VyY2VzLmh0bWwpLgoKVG8gc3RhcnQsIHdlIHdpbGwgZmlyc3QgbG9hZCB0aGUgW2Jpb21hUnQgbGlicmFyeV0oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzLzMuMTQvYmlvYy9odG1sL2Jpb21hUnQuaHRtbCkgYW5kIGNob29zZSB3aGF0IHJlZmVyZW5jZSB3ZSB3YW50IHRvIGFjY2Vzcy4gRm9yIGEgbW9yZSBkZXRhaWxlZCB3YWxrIHRocm91Z2ggb2YgdXNpbmcgYmlvbWFSdCwgW3RoaXMgdHJhaW5pbmcgbW9kdWxlXShodHRwczovL2Jpb2luZm9ybWF0aWNzLWNvcmUtc2hhcmVkLXRyYWluaW5nLmdpdGh1Yi5pby9jcnVrLXN1bW1lci1zY2hvb2wtMjAxOS9STkFzZXEvaHRtbC8wNV9Bbm5vdGF0aW9uX2FuZF9WaXN1YWxpc2F0aW9uLmh0bWwpIG1pZ2h0IGJlIHVzZWZ1bCwgaW5jbHVkaW5nIHdoYXQgdG8gZG8gd2hlbiBhbm5vdGF0aW9ucyBhcmUgbm90IDE6MSBtYXBwaW5ncy4KCldlJ2xsIHN0YXJ0IGJ5IGxvYWRpbmcgdGhlIGBiaW9tYVJ0YCBsaWJyYXJ5IGFuZCBjYWxsaW5nIHRoZSBgdXNlRW5zZW1ibCgpYCBmdW5jdGlvbiB0byBzZWxlY3QgdGhlIGRhdGFiYXNlIHdlJ2xsIHVzZSB0byBleHRyYWN0IHRoZSBpbmZvcm1hdGlvbiB3ZSBuZWVkLiBUaGlzIHdpbGwgZG93bmxvYWQgdGhlIG1hcHBpbmcgb2YgRU5TRU1CTCBJRHMgdG8gZ2VuZSBzeW1ib2xzLCBlbmFibGluZyB1cyB0byBldmVudHVhbGx5IGFkZCB0aGUgZ2VuZSBzeW1ib2wgY29sdW1uIHdlIHdhbnQuCgpgYGB7ciBQdWxsbWFydCwgd2FybmluZz1GQUxTRX0KbGlicmFyeSgnYmlvbWFSdCcpCmVuc2VtYmwgPSB1c2VFbnNlbWJsKGRhdGFzZXQgPSAnbW11c2N1bHVzX2dlbmVfZW5zZW1ibCcsIGJpb21hcnQ9J2Vuc2VtYmwnKQpgYGAKCioqTm90ZSoqICotIHRoaXMgcHJvY2VzcyB0YWtlcyBzb21lIHRpbWUgYW5kIHdpbGwgdGFrZSB1cCBhIGxhcmdlciBhbW91bnQgb2Ygd29ya2luZyBtZW1vcnkgc28gcHJvY2VlZCB3aXRoIGNhdXRpb24gaWYgeW91IHRyeSB0byBydW4gdGhlc2UgY29tbWFuZHMgb24gYSBsYXB0b3Agd2l0aCBsZXNzIHRoYW4gNEcgb2YgbWVtb3J5KgoKVG8gaWRlbnRpZnkgcG9zc2libGUgKipmaWx0ZXJzKiogdG8gcmVzdHJpY3Qgb3VyIGRhdGEsIHdlIGNhbiB1c2UgdGhlIGBsaXN0RmlsdGVyc2AgZnVuY3Rpb24uIFRvIGlkZW50aWZ5IHRoZSAqKmF0dHJpYnV0ZXMqKiB3ZSB3YW50IHRvIHJldHJpdmUsIHdlIGNhbiB1c2UgdGhlIGBsaXN0QXR0cmlidXRlc2AgZnVuY3Rpb24uIFRoZSBiZXN0IGFwcHJvYWNoIGlzIHRvIHVzZSBbbGlzdCBvciBzZWFyY2ggZnVuY3Rpb25zXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy9iaW9tYVJ0L2luc3QvZG9jL2FjY2Vzc2luZ19lbnNlbWJsLmh0bWwjaG93LXRvLWJ1aWxkLWEtYmlvbWFydC1xdWVyeSkgdG8gaGVscCBuYXJyb3cgZG93biB0aGUgYXZhaWxhYmxlIG9wdGlvbnMuCgpgYGB7ciBBZGRBbm5vdGF0aW9uczIsIHdhcm5pbmc9RkFMU0UsIGV2YWw9RkFMU0V9CmhlYWQobGlzdEZpbHRlcnMobWFydCA9IGVuc2VtYmwpLCBuID0gMjApCmhlYWQobGlzdEF0dHJpYnV0ZXMoZW5zZW1ibCksIG4gPSAzMCkKYGBgCgpXZSBjYW4gYWNjZXNzIGFkZGl0aW9uYWwgZ2Vub21pYyBhbm5vdGF0aW9ucyB1c2luZyB0aGUgW2BiaW9NYXJ0YCBwYWNrYWdlXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL2h0bWwvYmlvbWFSdC5odG1sKS4gVG8gaWRlbnRpZnkgIHdlJ2xsIHN0cnVjdHVyZSBvdXIgJ3F1ZXJ5JyBvciBzZWFyY2ggb2YgdGhlIGJpb01hcnQgcmVzb3VyY2VzIHRvIHVzZSB0aGUgW0VOU0VNQkwgaWRdKGh0dHBzOi8vbS5lbnNlbWJsLm9yZy9pbmZvL2dlbm9tZS9nZW5lYnVpbGQvZ2VuZV9uYW1lcy5odG1sKSBmcm9tIG91ciBhbGlnbm1lbnQgdG8gYWRkIHRoZSBnZW5lIHN5bWJvbHMgYW5kIGdlbmUgZGVzY3JpcHRpb24gZm9yIGVhY2ggZ2VuZS4KCmBgYHtyIEFkZEFub3RhdGlvbjMsIHdhcm5pbmc9RkFMU0V9CmlkX21hcHBpbmcgPSBnZXRCTShhdHRyaWJ1dGVzPWMoJ2Vuc2VtYmxfZ2VuZV9pZCcsICdleHRlcm5hbF9nZW5lX25hbWUnKSwKICAgICAgZmlsdGVycyA9ICdlbnNlbWJsX2dlbmVfaWQnLAogICAgICB2YWx1ZXMgPSByb3cubmFtZXMoYXNzYXkoZGRzX2JhdGNoX2ZpdHRlZCkpLAogICAgICBtYXJ0ID0gZW5zZW1ibCkKIyB3aWxsIHRha2Ugc29tZSB0aW1lIGZvciB0aGUgcXVlcnkgdG8gcnVuCgojIFByZXZpZXcgdGhlIHJlc3VsdApoZWFkKGlkX21hcHBpbmcpCmBgYAoKVGhlIGBpZF9tYXBwaW5nYCB0YWJsZSBub3cgaW5jbHVkZXMgdGhlIEVOU0VNQkwgaW5mb3JtYXRpb24gYW5kIGEgZ2VuZSBzeW1ib2wgb25seSBmb3IgdGhlIGdlbmVzIGluY2x1ZGVkIGluIG91ciByZXN1bHRzLiBUaGlzIHRhYmxlIHNob3VsZCBsb29rIGZhbWlsaWFyIGFzIGl0J3MgdGhlIHNhbWUgdGFibGUgd2UgdXNlZCB0byBhbm5vdGF0ZSBvdXIgcmVzdWx0cyB0YWJsZSBpbiB0aGUgbGFzdCBtb2R1bGUuCgoqKk5vdGUqKjogRm9yIGFkZGl0aW9uYWwgaW5mb3JtYXRpb24gcmVnYXJkaW5nIGJpb01hcnQsIHBsZWFzZSBjb25zdWx0IHRoZSBbRU5TRU1CTCBiaW9NYXJ0IHZpZ25ldHRlXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy9iaW9tYVJ0L2luc3QvZG9jL2FjY2Vzc2luZ19lbnNlbWJsLmh0bWwpIG9yIHRoZSBicm9hZGVyIFtCaW9jb25kdWN0b3IgQW5ub3RhdGlvbiBSZXNvdXJjZXMgdmlnbmV0dGUgXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC93b3JrZmxvd3MvdmlnbmV0dGVzL2Fubm90YXRpb24vaW5zdC9kb2MvQW5ub3RhdGlvbl9SZXNvdXJjZXMuaHRtbCkuCgotLS0KCiMgT3V0cHV0dGluZyByZXN1bHRzIHRvIGZpbGUKCkEga2V5IGFzcGVjdCBvZiBvdXIgYW5hbHlzaXMgaXMgcHJlc2VydmluZyB0aGUgcmVsZXZhbnQgZGF0YXNldHMgZm9yIGJvdGggb3VyIHJlY29yZHMgYW5kIGZvciBkb3duc3RyZWFtIGFwcGxpY2F0aW9ucywgc3VjaCBhcyBmdW5jdGlvbmFsIGVucmljaG1lbnRzLgoKIyMgREUgcmVzdWx0cyB0YWJsZQoKV2UnbGwgd3JpdGUgb3V0IG91ciBERSByZXN1bHRzLCBub3cgdGhhdCB3ZSd2ZSBhZGRlZCBpbmZvcm1hdGlvbiB0byB0aGUgdGFibGUgdG8gaGVscCB1cyBvciBvdXIgY29sbGFib3JhdG9ycyBpbnRlcnByZXQgdGhlIHJlc3VsdHMuCgpgYGB7ciBERVJlc3VsdHNPdXRwdXQsIGV2YWwgPSBGQUxTRX0Kd3JpdGUuY3N2KHJlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2wsCiAgICAgICAgICByb3cubmFtZXMgPSBGQUxTRSwKICAgICAgICAgIG5hID0gIi4iLAogICAgICAgICAgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVfcmVzdWx0c19kZWZpY2llbnRfdnNfY29udHJvbC5jc3YiKQoKd3JpdGUuY3N2KHJlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2xfYW5ub3RhdGVkLAogICAgICAgICAgcm93Lm5hbWVzID0gRkFMU0UsCiAgICAgICAgICBmaWxlPSJvdXRwdXRzL3RhYmxlcy9ERV9yZXN1bHRzX2RlZmljaWVudF92c19jb250cm9sX2Fubm90YXRlZC5jc3YiKQpgYGAKCgojIyBTdWJzZXR0aW5nIHNpZ25pZmljYW50IGdlbmVzCgpZb3UgbWF5IGJlIGludGVyZXN0ZWQgaW4gY3JlYXRpbmcgYSB0YWJsZSBvZiBvbmx5IHRoZSBnZW5lcyB0aGF0IHBhc3MgeW91ciBzaWduaWZpY2FuY2UgdGhyZXNob2xkcy4gQSB1c2VmdWwgd2F5IHRvIGRvIHRoaXMgaXMgdG8gY29uZGl0aW9uYWxseSBzdWJzZXQgeW91ciByZXN1bHRzLiBBZ2Fpbiwgd2UgYWxyZWFkeSBjcmVhdGVkIHRoZSBgY2FsbGAgY29sdW1uLCB3aGljaCBtYWtlcyB0aGlzIHJlbGF0aXZlbHkgc2ltcGxlIHRvIGRvOgoKYGBge3IgQ29uZGl0aW9uYWxTdWJzZXR9CiMgdGlkeXIgKHJlcXVpcmVzIHRhYmxlIHJlZm9ybWF0dGluZykKcmVzX3NpZyA8LSBhc190aWJibGUocmVzdWx0c19kZWZpY2llbnRfdnNfY29udHJvbCwgcm93bmFtZXMgPSAiZ2VuZV9pZHMiKSAlPiUgZmlsdGVyKGNhbGwgIT0gJ05TJykKCmhlYWQocmVzX3NpZykKZGltKHJlc19zaWcpCmBgYAoKPCEtLSBhZGQgYmFzZSB2ZXJzaW9uIGFzIGRyb3Bkb3duOiAjIGJhc2UKcmVzX3NpZyA8LSByZXN1bHRzX2RlZmljaWVudF92c19jb250cm9sW3Jlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2wkY2FsbCAhPSAnTlMnLCBdIC0tPgoKT25jZSB3ZSd2ZSBjcmVhdGVkIHRoaXMgdGFibGUsIHdlIGNhbiBhbHNvIHdyaXRlIGl0IG91dCB0byBmaWxlOgpgYGB7ciBERU9ubHlPdXRwdXQsIGV2YWwgPSBGQUxTRX0Kd3JpdGUuY3N2KHJlc19zaWcsCiAgICAgICAgICByb3cubmFtZXMgPSBGQUxTRSwKICAgICAgICAgIG5hID0gIi4iLAogICAgICAgICAgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVHcy1vbmx5X2RlZmljaWVudF92c19jb250cm9sLmNzdiIpCmBgYAoKCiMjIFIgc2Vzc2lvbiBkYXRhCgpJbiBhZGRpdGlvbiB0byB0aGUgaW5kaXZpZHVhbCBST2JqKHMpIHdlIHNhdmVkIGVhcmxpZXIsIHdlIGNhbiBjYXB0dXJlIGEgc25hcHNob3Qgb3VyIGVudGlyZSBzZXNzaW9uIHVzaW5nIHRoZSBgc2F2ZS5pbWFnZWAgZnVuY3Rpb24uIFRoaXMgY2FuIGJlIGxvYWRlZCBpbiB0aGUgc2FtZSBtYW5uZXIgYXMgYW4gaW5kaXZpZHVhbCBSb2JqLgoKRmlyc3QsIHdlJ2xsIHNhdmUgb3VyIHNlc3Npb24gaW5mbyBzbyB3ZSBjYW4gcmVmZXJlbmNlIHRoZSBwYWNrYWdlcyBhbmQgdmVyc2lvbnMgdXNlZCB0byBnZW5lcmF0ZSB0aGVzZSBkYXRhLgoKYGBge3J9CnNlc3Npb25fc3VtbWFyeSA8LSBzZXNzaW9uSW5mbygpCmBgYAoKYGBge3IgfQpzYXZlLmltYWdlKGZpbGUgPSAib3V0cHV0cy9Sb2Jqcy9ERV9pcm9uLlJEYXRhIikKYGBgCgoKCiMgT3ZlcmFsbCB0YWtlYXdheXMKCldlJ3ZlIHJ1biB0aHJvdWdoIG1vc3Qgb2YgdGhlIGJ1aWxkaW5nIGJsb2NrcyBuZWVkZWQgdG8gcnVuIGEgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgYW5kIGhvcGVmdWxseSBidWlsdCB1cCBhIGJldHRlciB1bmRlcnN0YW5kaW5nIG9mIGhvdyBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBjb21wYXJpc29ucyB3b3JrLCBwYXJ0aWN1bGFybHkgaG93IGV4cGVyaW1lbnRhbCBkZXNpZ24gY2FuIGltcGFjdCBvdXIgcmVzdWx0cy4KCldoYXQgdG8gY29uc2lkZXIgbW92aW5nIGZvcndhcmQ6CgoqIEhvdyBjYW4gSSBjb250cm9sIGZvciB0ZWNobmljYWwgdmFyaWF0aW9uIGluIG15IGV4cGVyaW1lbnRhbCBkZXNpZ24/CiogSG93IG11Y2ggdmFyaWF0aW9uIGlzIGV4cGVjdGVkIHdpdGggYSB0cmVhdG1lbnQgZ3JvdXA/CiogV2hhdCBpcyBteSBSTkEgcXVhbGl0eSwgYW5kIGhvdyBjYW4gdGhhdCBiZSBvcHRpbWl6ZWQ/CiogQXJlIHRoZXJlIHF1YWxpdHkgY29uY2VybnMgZm9yIG15IHNlcXVlbmNpbmcgZGF0YT8KKiBXaGF0IGNvbXBhcmlzb25zIGFyZSByZWxldmFudCB0byBteSBiaW9sb2dpY2FsIHF1ZXN0aW9uPwoqIEFyZSB0aGVyZSBjb3ZhcmlhdGVzIHRoYXQgc2hvdWxkIGJlIGNvbnNpZGVyZWQ/CiogV2hhdCB3aWxsIGEgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgdGVsbCBtZT8KCgoqKkxldCdzIHBhdXNlIGhlcmUgZm9yIGdlbmVyYWwgcXVlc3Rpb25zKioKCi0tLQoKIyBOZXh0IHN0ZXBzIC0gSG93IGRvIHdlIG1ha2Ugc2Vuc2Ugb2YgbGFyZ2UgbnVtYmVycyBvZiBERSBnZW5lcz8KCkEgd2F5IHRvIGRldGVybWluZSBwb3NzaWJsZSBbYnJvYWRlciBiaW9sb2dpY2FsIGludGVycHJldGF0aW9uc10oaHR0cHM6Ly93d3cuZWJpLmFjLnVrL3RyYWluaW5nLWJldGEvb25saW5lL2NvdXJzZXMvZnVuY3Rpb25hbC1nZW5vbWljcy1paS1jb21tb24tdGVjaG5vbG9naWVzLWFuZC1kYXRhLWFuYWx5c2lzLW1ldGhvZHMvYmlvbG9naWNhbC1pbnRlcnByZXRhdGlvbi1vZi1nZW5lLWV4cHJlc3Npb24tZGF0YS0yLykgZnJvbSB0aGUgb2JzZXJ2ZWQgREUgcmVzdWx0cywgaXMgZnVuY3Rpb25hbCBlbnJpY2htZW50cy4gCgpUaGVyZSBhcmUgbWFueSBvcHRpb25zLCBzdWNoIGFzIHNvbWUgaW5jbHVkZWQgaW4gdGhpcyBbZGlzY3Vzc2lvbiB0aHJlYWRdKGh0dHBzOi8vd3d3LnJlc2VhcmNoZ2F0ZS5uZXQvcG9zdC9Ib3dfY2FuX0lfYW5hbHl6ZV9hX3NldF9vZl9ERUdzX2RpZmZlcmVudGlhbGx5X2V4cHJlc3NlZF9nZW5lc190b19vYnRhaW5faW5mb3JtYXRpb25fZnJvbV90aGVtKS4gT3RoZXIgY29tbW9uIGZ1bmN0aW9uYWwgZW5yaWNobWVudHMgYXBwcm9hY2hlcyBhcmUgZ2VuZSBzZXQgZW5yaWNobWVudCBhbmFseXNpcywgYWthIFtHU0VBXShodHRwOi8vc29mdHdhcmUuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEvaW5kZXguanNwKSwgRGF0YWJhc2UgZm9yIEFubm90YXRpb24sIFZpc3VhbGl6YXRpb24gYW5kIEludGVncmF0ZWQgRGlzY292ZXJ5LCBha2EgW0RBVklEXShodHRwczovL2RhdmlkLm5jaWZjcmYuZ292LyksIFtJbmdlbml0eV0oaHR0cHM6Ly9kaWdpdGFsaW5zaWdodHMucWlhZ2VuLmNvbS8pLCBhbmQgW2lQYXRod2F5IEd1aWRlXShodHRwczovL2FkdmFpdGFiaW8uY29tL2lwYXRod2F5Z3VpZGUvKQoKVGhlIFVuaXZlcnNpdHkgb2YgTWljaGlnYW4gaGFzIGxpY2Vuc2UgYW5kIHN1cHBvcnQgZm9yIGFkZGl0aW9uYWwgdG9vbHMsIHN1Y2ggYXMgQ3l0b3NjYXBlLCBzbyB3ZSByZWNvbW1lbmQgcmVhY2hpbmcgb3V0IHRvIHN0YWZmIHdpdGggW1RhdWJtYW4gTGlicmFyeV0oaHR0cHM6Ly93d3cubGliLnVtaWNoLmVkdS9sb2NhdGlvbnMtYW5kLWhvdXJzL3RhdWJtYW4taGVhbHRoLXNjaWVuY2VzLWxpYnJhcnkvcmVzZWFyY2gtYW5kLWNsaW5pY2FsLXN1cHBvcnQpIHRvIGxlYXJuIG1vcmUgYWJvdXQgcmVzb3VyY2VzIHRoYXQgbWlnaHQgYmUgYXBwbGljYXRpb24gdG95b3VyIHJlc2VhcmNoLgoKLS0tCgojIFNvdXJjZXMKCiogSEJDIERHRSB0cmFpbmluZyBtb2R1bGUsIHBhcnQgMTogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDRfREdFX0RFU2VxMl9hbmFseXNpcy5odG1sCiogSEJDIERHRSB0cmFpbmluZyBtb2R1bGUsIHBhcnQgMjogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDVfREdFX0RFU2VxMl9hbmFseXNpczIuaHRtbAoqIERFU2VxMiB2aWduZXR0ZTogaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2RpZmZlcmVudGlhbC1leHByZXNzaW9uLWFuYWx5c2lzCiogQmlvY29uZHVjdG9yIEdlbm9taWMgQW5ub3RhdGlvbiByZXNvdXJjZXM6IGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL3dvcmtmbG93cy92aWduZXR0ZXMvYW5ub3RhdGlvbi9pbnN0L2RvYy9Bbm5vdGF0aW9uX1Jlc291cmNlcy5odG1sCiogQmlvTWFydCB2aWduZXR0ZTogaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvYmlvbWFSdC9pbnN0L2RvYy9hY2Nlc3NpbmdfZW5zZW1ibC5odG1sCgojIEFkZGl0aW9uYWwgUmVzb3VyY2VzCiogTUlEQVMgUmVwcm9kdWNpYmxpdHkgSHViOiBodHRwczovL21pZGFzLnVtaWNoLmVkdS9yZXByb2R1Y2liaWxpdHktb3ZlcnZpZXcvCiogQVJDIHJlc291cmNlczogaHR0cHM6Ly9hcmMtdHMudW1pY2guZWR1LwoqIEdlbmUgU2V0IEVucmljaG1lbnQgUmVzb3VyY2VzIGZyb20gQmlvY29uZHVjdG9yOiBodHRwczovL2Jpb2luZm9ybWF0aWNzLWNvcmUtc2hhcmVkLXRyYWluaW5nLmdpdGh1Yi5pby9jcnVrLXN1bW1lci1zY2hvb2wtMjAxOC9STkFTZXEyMDE4L2h0bWwvMDZfR2VuZV9zZXRfdGVzdGluZy5uYi5odG1sCiogVXNpbmcgSFRTZXEgZGF0YSB3aXRoIERFU2VxMjogaHR0cHM6Ly9hbmd1cy5yZWFkdGhlZG9jcy5pby9lbi8yMDE5L2RpZmYtZXgtYW5kLXZpei5odG1sCiogRGV0YWlsZWQgUk5BLXNlcSBhbmFseXNpcyBwYXBlcjogaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wbWMvYXJ0aWNsZXMvUE1DNjA5NjM0Ni8KKiBPdmVydmlldyBvZiBSTkEtc2VxIGFuYWx5c2lzIGNvbnNpZGVyYXRpb25zOiBodHRwczovL2FjYWRlbWljLW91cC1jb20ucHJveHkubGliLnVtaWNoLmVkdS9iZmcvYXJ0aWNsZS8xNC8yLzEzMC8yNTczNzAKKiBBbHRlcm5hdGl2ZSBvdmVydmlldyBvZiBERVNlcTIsIGluY2x1ZGluZyB2aXN1YWxpemF0aW9ucyBhbmQgZnVuY3Rpb25hbCBlbnJpY2htZW50czogaHR0cDovL2RwdXRoaWVyLmdpdGh1Yi5pby9qZ2I3MWUtcG9seXRlY2gtYmlvaW5mby1hcHAvcHJhY3RpY2FsL3JuYS1zZXFfUi9ybmFzZXFfZGlmZl9TbmYyLmh0bWwKCi0tLQoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGFfRnVsbC5SRGF0YSIpCmBgYAoKIyBTZXNzaW9uIEluZm8KYGBge3IgU2Vzc2lvbkluZm99CnNlc3Npb25JbmZvKCkKYGBgCgotLS0KClRoZXNlIG1hdGVyaWFscyBoYXZlIGJlZW4gYWRhcHRlZCBhbmQgZXh0ZW5kZWQgZnJvbSBtYXRlcmlhbHMgbGlzdGVkIGFib3ZlLiBUaGVzZSBhcmUgb3BlbiBhY2Nlc3MgbWF0ZXJpYWxzIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgW0NyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gbGljZW5zZSAoQ0MgQlkgNC4wKV0oaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyksIHdoaWNoIHBlcm1pdHMgdW5yZXN0cmljdGVkIHVzZSwgZGlzdHJpYnV0aW9uLCBhbmQgcmVwcm9kdWN0aW9uIGluIGFueSBtZWRpdW0sIHByb3ZpZGVkIHRoZSBvcmlnaW5hbCBhdXRob3IgYW5kIHNvdXJjZSBhcmUgY3JlZGl0ZWQuCgo8YnIvPgo8YnIvPgo8aHIvPgp8IFtQcmV2aW91cyBsZXNzb25dKE1vZHVsZTExX0RFVmlzdWFsaXphdGlvbnMuaHRtbCkgfCBbVG9wIG9mIHRoaXMgbGVzc29uXSgjdG9wKSB8IFtOZXh0IGxlc3Nvbl0oUl9ib251c19jb250ZW50Lmh0bWwpIHwKfCA6LS0tIHwgOi0tLS06IHwgLS0tOiB8