In this module, we will learn:

  • The advantages of using gene ids when analyzing RNA-seq data.
  • How to find gene symbols and other annotations, using ENSEMBL gene ids
  • How to output our results to file
  • General options for functional enrichments and other follow-ups


Differential Expression Workflow

Here we will generate summary figures for our results and annotate our DE tables.


Generating gene annotations

Since, gene symbols can change over time or be ambiguous we use, and recommend, using the EMSEMBL reference genome and ENSEMBL IDs for alignments and we’ve been working with tables and data where all genes are labeled only by their long ENSEMBL ID. However, this can make it difficult to quickly look for genes of interest.

Luckily, Bioconductor provides many tools and resources to facilitate access to genomic annotation resources.

To start, we will first load the biomaRt library and choose what reference we want to access. For a more detailed walk through of using biomaRt, this training module might be useful, including what to do when annotations are not 1:1 mappings.

We’ll start by loading the biomaRt library and calling the useEnsembl() function to select the database we’ll use to extract the information we need. This will download the mapping of ENSEMBL IDs to gene symbols, enabling us to eventually add the gene symbol column we want.

library('biomaRt')
ensembl = useEnsembl(dataset = 'mmusculus_gene_ensembl', biomart='ensembl')

Note - this process takes some time and will take up a larger amount of working memory so proceed with caution if you try to run these commands on a laptop with less than 4G of memory

To identify possible filters to restrict our data, we can use the listFilters function. To identify the attributes we want to retrive, we can use the listAttributes function. The best approach is to use list or search functions to help narrow down the available options.

head(listFilters(mart = ensembl), n = 20)
head(listAttributes(ensembl), n = 30)

We can access additional genomic annotations using the bioMart package. To identify we’ll structure our ‘query’ or search of the bioMart resources to use the ENSEMBL id from our alignment to add the gene symbols and gene description for each gene.

id_mapping = getBM(attributes=c('ensembl_gene_id', 'external_gene_name'),
      filters = 'ensembl_gene_id',
      values = row.names(assay(dds_batch_fitted)),
      mart = ensembl)
Batch submitting query [=======>------------------------] 25% eta: 34sBatch
submitting query [===============>----------------] 50% eta: 16sBatch submitting
query [=======================>--------] 75% eta: 7s
# will take some time for the query to run

# Preview the result
head(id_mapping)
     ensembl_gene_id external_gene_name
1 ENSMUSG00000000001              Gnai3
2 ENSMUSG00000000028              Cdc45
3 ENSMUSG00000000031                H19
4 ENSMUSG00000000037              Scml2
5 ENSMUSG00000000049               Apoh
6 ENSMUSG00000000056               Narf

The id_mapping table now includes the ENSEMBL information and a gene symbol only for the genes included in our results. This table should look familiar as it’s the same table we used to annotate our results table in the last module.

Note: For additional information regarding bioMart, please consult the ENSEMBL bioMart vignette or the broader Bioconductor Annotation Resources vignette.


Outputting results to file

A key aspect of our analysis is preserving the relevant datasets for both our records and for downstream applications, such as functional enrichments.

DE results table

We’ll write out our DE results, now that we’ve added information to the table to help us or our collaborators interpret the results.

write.csv(results_deficient_vs_control,
          row.names = FALSE,
          na = ".",
          file="outputs/tables/DE_results_deficient_vs_control.csv")

write.csv(results_deficient_vs_control_annotated,
          row.names = FALSE,
          file="outputs/tables/DE_results_deficient_vs_control_annotated.csv")

Subsetting significant genes

You may be interested in creating a table of only the genes that pass your significance thresholds. A useful way to do this is to conditionally subset your results. Again, we already created the call column, which makes this relatively simple to do:

# tidyr (requires table reformatting)
res_sig <- as_tibble(results_deficient_vs_control, rownames = "gene_ids") %>% filter(call != 'NS')

head(res_sig)
# A tibble: 6 × 8
  gene_ids           baseMean log2FoldChange lfcSE  stat     pvalue    padj call 
  <chr>                 <dbl>          <dbl> <dbl> <dbl>      <dbl>   <dbl> <fct>
1 ENSMUSG00000000275   1662.          -0.674 0.200 -3.38 0.000734   3.12e-2 Down 
2 ENSMUSG00000000861   1679.           0.685 0.210  3.27 0.00109    3.98e-2 Up   
3 ENSMUSG00000001281    532.           1.15  0.242  4.75 0.00000203 5.54e-4 Up   
4 ENSMUSG00000002109    125.           1.06  0.300  3.54 0.000398   2.07e-2 Up   
5 ENSMUSG00000002985    157.          -1.16  0.345 -3.35 0.000814   3.29e-2 Down 
6 ENSMUSG00000003865     89.3          2.26  0.627  3.61 0.000310   1.80e-2 Up   
dim(res_sig)
[1] 189   8

Once we’ve created this table, we can also write it out to file:

write.csv(res_sig,
          row.names = FALSE,
          na = ".",
          file="outputs/tables/DEGs-only_deficient_vs_control.csv")

R session data

In addition to the individual RObj(s) we saved earlier, we can capture a snapshot our entire session using the save.image function. This can be loaded in the same manner as an individual Robj.

First, we’ll save our session info so we can reference the packages and versions used to generate these data.

session_summary <- sessionInfo()
save.image(file = "outputs/Robjs/DE_iron.RData")

Overall takeaways

We’ve run through most of the building blocks needed to run a differential expression analysis and hopefully built up a better understanding of how differential expression comparisons work, particularly how experimental design can impact our results.

What to consider moving forward:

  • How can I control for technical variation in my experimental design?
  • How much variation is expected with a treatment group?
  • What is my RNA quality, and how can that be optimized?
  • Are there quality concerns for my sequencing data?
  • What comparisons are relevant to my biological question?
  • Are there covariates that should be considered?
  • What will a differential expression analysis tell me?

Let’s pause here for general questions


Next steps - How do we make sense of large numbers of DE genes?

A way to determine possible broader biological interpretations from the observed DE results, is functional enrichments.

There are many options, such as some included in this discussion thread. Other common functional enrichments approaches are gene set enrichment analysis, aka GSEA, Database for Annotation, Visualization and Integrated Discovery, aka DAVID, Ingenity, and iPathway Guide

The University of Michigan has license and support for additional tools, such as Cytoscape, so we recommend reaching out to staff with Taubman Library to learn more about resources that might be application toyour research.


Session Info

sessionInfo()
R version 4.4.0 (2024-04-24)
Platform: x86_64-apple-darwin20
Running under: macOS Sonoma 14.4.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Detroit
tzcode source: internal

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] biomaRt_2.60.1              data.table_1.15.4          
 [3] RColorBrewer_1.1-3          pheatmap_1.0.12            
 [5] ggrepel_0.9.5               lubridate_1.9.3            
 [7] forcats_1.0.0               stringr_1.5.1              
 [9] dplyr_1.1.4                 purrr_1.0.2                
[11] readr_2.1.5                 tidyr_1.3.1                
[13] tibble_3.2.1                ggplot2_3.5.1              
[15] tidyverse_2.0.0             DESeq2_1.44.0              
[17] SummarizedExperiment_1.34.0 Biobase_2.64.0             
[19] MatrixGenerics_1.16.0       matrixStats_1.3.0          
[21] GenomicRanges_1.56.1        GenomeInfoDb_1.40.1        
[23] IRanges_2.38.1              S4Vectors_0.42.1           
[25] BiocGenerics_0.50.0         knitr_1.47                 
[27] rmarkdown_2.27             

loaded via a namespace (and not attached):
 [1] DBI_1.2.3               httr2_1.0.2             rlang_1.1.4            
 [4] magrittr_2.0.3          compiler_4.4.0          RSQLite_2.3.7          
 [7] png_0.1-8               vctrs_0.6.5             pkgconfig_2.0.3        
[10] crayon_1.5.3            fastmap_1.2.0           dbplyr_2.5.0           
[13] XVector_0.44.0          labeling_0.4.3          utf8_1.2.4             
[16] tzdb_0.4.0              UCSC.utils_1.0.0        bit_4.0.5              
[19] xfun_0.44               zlibbioc_1.50.0         cachem_1.1.0           
[22] jsonlite_1.8.8          progress_1.2.3          blob_1.2.4             
[25] highr_0.11              DelayedArray_0.30.1     BiocParallel_1.38.0    
[28] parallel_4.4.0          prettyunits_1.2.0       R6_2.5.1               
[31] bslib_0.7.0             stringi_1.8.4           jquerylib_0.1.4        
[34] Rcpp_1.0.13             Matrix_1.7-0            timechange_0.3.0       
[37] tidyselect_1.2.1        rstudioapi_0.16.0       abind_1.4-5            
[40] yaml_2.3.8              codetools_0.2-20        curl_5.2.1             
[43] lattice_0.22-6          withr_3.0.1             KEGGREST_1.44.1        
[46] evaluate_0.23           BiocFileCache_2.12.0    xml2_1.3.6             
[49] Biostrings_2.72.1       filelock_1.0.3          pillar_1.9.0           
[52] BiocManager_1.30.23     generics_0.1.3          hms_1.1.3              
[55] munsell_0.5.1           scales_1.3.0            glue_1.7.0             
[58] tools_4.4.0             locfit_1.5-9.10         grid_4.4.0             
[61] AnnotationDbi_1.66.0    colorspace_2.1-1        GenomeInfoDbData_1.2.12
[64] cli_3.6.2               rappdirs_0.3.3          fansi_1.0.6            
[67] S4Arrays_1.4.1          gtable_0.3.5            sass_0.4.9             
[70] digest_0.6.35           SparseArray_1.4.8       farver_2.1.2           
[73] memoise_2.0.1           htmltools_0.5.8.1       lifecycle_1.0.4        
[76] httr_1.4.7              bit64_4.0.5            

These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.




Previous lesson Top of this lesson Next lesson
LS0tCnRpdGxlOiAiREUgQW5ub3RhdGlvbnMgJiBEb3duc3RyZWFtIGFwcGxpY2F0aW9ucyIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjExLSIpCmBgYAoKSW4gdGhpcyBtb2R1bGUsIHdlIHdpbGwgbGVhcm46CgoqIFRoZSBhZHZhbnRhZ2VzIG9mIHVzaW5nIGdlbmUgaWRzIHdoZW4gYW5hbHl6aW5nIFJOQS1zZXEgZGF0YS4KKiBIb3cgdG8gZmluZCBnZW5lIHN5bWJvbHMgYW5kIG90aGVyIGFubm90YXRpb25zLCB1c2luZyBFTlNFTUJMIGdlbmUgaWRzCiogSG93IHRvIG91dHB1dCBvdXIgcmVzdWx0cyB0byBmaWxlCiogR2VuZXJhbCBvcHRpb25zIGZvciBmdW5jdGlvbmFsIGVucmljaG1lbnRzIGFuZCBvdGhlciBmb2xsb3ctdXBzCgo8YnI+CgpgYGB7ciBNb2R1bGVzLCBldmFsPVRSVUUsIGVjaG89RkFMU0UsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CmxpYnJhcnkoREVTZXEyKQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkodGlkeXIpCmxpYnJhcnkoZHBseXIpCmxpYnJhcnkobWF0cml4U3RhdHMpCmxpYnJhcnkoZ2dyZXBlbCkKbGlicmFyeShwaGVhdG1hcCkKbGlicmFyeShSQ29sb3JCcmV3ZXIpCiMgbG9hZCgicmRhdGEvUnVubmluZ0RhdGEuUkRhdGEiKQoKIyMgZW5zZW1ibCBTU0wgY2VydCBpc3N1ZSBmbGFnZ2VkIGluIGtuaXR0aW5nIHdpdGggc2VsZWN0ZWQgbWlycm9yCmBgYAoKIyBEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiBXb3JrZmxvdyB7LnVubGlzdGVkIC51bm51bWJlcmVkfQoKSGVyZSB3ZSB3aWxsIGdlbmVyYXRlIHN1bW1hcnkgZmlndXJlcyBmb3Igb3VyIHJlc3VsdHMgYW5kIGFubm90YXRlIG91ciBERSB0YWJsZXMuCgohW10oLi9pbWFnZXMvd2F5ZmluZGVyL3dheWZpbmRlci1ERUFubm90YXRpb24ucG5nKXt3aWR0aD03NSV9CgotLS0KCiMgR2VuZXJhdGluZyBnZW5lIGFubm90YXRpb25zCgpTaW5jZSwgZ2VuZSBzeW1ib2xzIGNhbiBjaGFuZ2Ugb3ZlciB0aW1lIG9yIGJlIGFtYmlndW91cyB3ZSB1c2UsIGFuZCByZWNvbW1lbmQsIHVzaW5nIHRoZSBFTVNFTUJMIHJlZmVyZW5jZSBnZW5vbWUgYW5kIEVOU0VNQkwgSURzIGZvciBhbGlnbm1lbnRzIGFuZCB3ZSd2ZSBiZWVuIHdvcmtpbmcgd2l0aCB0YWJsZXMgYW5kIGRhdGEgd2hlcmUgYWxsIGdlbmVzIGFyZSBsYWJlbGVkIG9ubHkgYnkgdGhlaXIgbG9uZyBFTlNFTUJMIElELiBIb3dldmVyLCB0aGlzIGNhbiBtYWtlIGl0IGRpZmZpY3VsdCB0byBxdWlja2x5IGxvb2sgZm9yIGdlbmVzIG9mIGludGVyZXN0LgoKTHVja2lseSwgQmlvY29uZHVjdG9yIHByb3ZpZGVzIG1hbnkgdG9vbHMgYW5kIHJlc291cmNlcyB0byBmYWNpbGl0YXRlIGFjY2VzcyB0byBbZ2Vub21pYyBhbm5vdGF0aW9uIHJlc291cmNlc10oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvd29ya2Zsb3dzL3ZpZ25ldHRlcy9hbm5vdGF0aW9uL2luc3QvZG9jL0Fubm90YXRpb25fUmVzb3VyY2VzLmh0bWwpLgoKVG8gc3RhcnQsIHdlIHdpbGwgZmlyc3QgbG9hZCB0aGUgW2Jpb21hUnQgbGlicmFyeV0oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzLzMuMTQvYmlvYy9odG1sL2Jpb21hUnQuaHRtbCkgYW5kIGNob29zZSB3aGF0IHJlZmVyZW5jZSB3ZSB3YW50IHRvIGFjY2Vzcy4gRm9yIGEgbW9yZSBkZXRhaWxlZCB3YWxrIHRocm91Z2ggb2YgdXNpbmcgYmlvbWFSdCwgW3RoaXMgdHJhaW5pbmcgbW9kdWxlXShodHRwczovL2Jpb2luZm9ybWF0aWNzLWNvcmUtc2hhcmVkLXRyYWluaW5nLmdpdGh1Yi5pby9jcnVrLXN1bW1lci1zY2hvb2wtMjAxOS9STkFzZXEvaHRtbC8wNV9Bbm5vdGF0aW9uX2FuZF9WaXN1YWxpc2F0aW9uLmh0bWwpIG1pZ2h0IGJlIHVzZWZ1bCwgaW5jbHVkaW5nIHdoYXQgdG8gZG8gd2hlbiBhbm5vdGF0aW9ucyBhcmUgbm90IDE6MSBtYXBwaW5ncy4KCldlJ2xsIHN0YXJ0IGJ5IGxvYWRpbmcgdGhlIGBiaW9tYVJ0YCBsaWJyYXJ5IGFuZCBjYWxsaW5nIHRoZSBgdXNlRW5zZW1ibCgpYCBmdW5jdGlvbiB0byBzZWxlY3QgdGhlIGRhdGFiYXNlIHdlJ2xsIHVzZSB0byBleHRyYWN0IHRoZSBpbmZvcm1hdGlvbiB3ZSBuZWVkLiBUaGlzIHdpbGwgZG93bmxvYWQgdGhlIG1hcHBpbmcgb2YgRU5TRU1CTCBJRHMgdG8gZ2VuZSBzeW1ib2xzLCBlbmFibGluZyB1cyB0byBldmVudHVhbGx5IGFkZCB0aGUgZ2VuZSBzeW1ib2wgY29sdW1uIHdlIHdhbnQuCgpgYGB7ciBQdWxsbWFydCwgd2FybmluZz1GQUxTRX0KbGlicmFyeSgnYmlvbWFSdCcpCmVuc2VtYmwgPSB1c2VFbnNlbWJsKGRhdGFzZXQgPSAnbW11c2N1bHVzX2dlbmVfZW5zZW1ibCcsIGJpb21hcnQ9J2Vuc2VtYmwnKQpgYGAKCioqTm90ZSoqICotIHRoaXMgcHJvY2VzcyB0YWtlcyBzb21lIHRpbWUgYW5kIHdpbGwgdGFrZSB1cCBhIGxhcmdlciBhbW91bnQgb2Ygd29ya2luZyBtZW1vcnkgc28gcHJvY2VlZCB3aXRoIGNhdXRpb24gaWYgeW91IHRyeSB0byBydW4gdGhlc2UgY29tbWFuZHMgb24gYSBsYXB0b3Agd2l0aCBsZXNzIHRoYW4gNEcgb2YgbWVtb3J5KgoKVG8gaWRlbnRpZnkgcG9zc2libGUgKipmaWx0ZXJzKiogdG8gcmVzdHJpY3Qgb3VyIGRhdGEsIHdlIGNhbiB1c2UgdGhlIGBsaXN0RmlsdGVyc2AgZnVuY3Rpb24uIFRvIGlkZW50aWZ5IHRoZSAqKmF0dHJpYnV0ZXMqKiB3ZSB3YW50IHRvIHJldHJpdmUsIHdlIGNhbiB1c2UgdGhlIGBsaXN0QXR0cmlidXRlc2AgZnVuY3Rpb24uIFRoZSBiZXN0IGFwcHJvYWNoIGlzIHRvIHVzZSBbbGlzdCBvciBzZWFyY2ggZnVuY3Rpb25zXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy9iaW9tYVJ0L2luc3QvZG9jL2FjY2Vzc2luZ19lbnNlbWJsLmh0bWwjaG93LXRvLWJ1aWxkLWEtYmlvbWFydC1xdWVyeSkgdG8gaGVscCBuYXJyb3cgZG93biB0aGUgYXZhaWxhYmxlIG9wdGlvbnMuCgpgYGB7ciBBZGRBbm5vdGF0aW9uczIsIHdhcm5pbmc9RkFMU0UsIGV2YWw9RkFMU0V9CmhlYWQobGlzdEZpbHRlcnMobWFydCA9IGVuc2VtYmwpLCBuID0gMjApCmhlYWQobGlzdEF0dHJpYnV0ZXMoZW5zZW1ibCksIG4gPSAzMCkKYGBgCgpXZSBjYW4gYWNjZXNzIGFkZGl0aW9uYWwgZ2Vub21pYyBhbm5vdGF0aW9ucyB1c2luZyB0aGUgW2BiaW9NYXJ0YCBwYWNrYWdlXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL2h0bWwvYmlvbWFSdC5odG1sKS4gVG8gaWRlbnRpZnkgIHdlJ2xsIHN0cnVjdHVyZSBvdXIgJ3F1ZXJ5JyBvciBzZWFyY2ggb2YgdGhlIGJpb01hcnQgcmVzb3VyY2VzIHRvIHVzZSB0aGUgW0VOU0VNQkwgaWRdKGh0dHBzOi8vbS5lbnNlbWJsLm9yZy9pbmZvL2dlbm9tZS9nZW5lYnVpbGQvZ2VuZV9uYW1lcy5odG1sKSBmcm9tIG91ciBhbGlnbm1lbnQgdG8gYWRkIHRoZSBnZW5lIHN5bWJvbHMgYW5kIGdlbmUgZGVzY3JpcHRpb24gZm9yIGVhY2ggZ2VuZS4KCmBgYHtyIEFkZEFub3RhdGlvbjMsIHdhcm5pbmc9RkFMU0V9CmlkX21hcHBpbmcgPSBnZXRCTShhdHRyaWJ1dGVzPWMoJ2Vuc2VtYmxfZ2VuZV9pZCcsICdleHRlcm5hbF9nZW5lX25hbWUnKSwKICAgICAgZmlsdGVycyA9ICdlbnNlbWJsX2dlbmVfaWQnLAogICAgICB2YWx1ZXMgPSByb3cubmFtZXMoYXNzYXkoZGRzX2JhdGNoX2ZpdHRlZCkpLAogICAgICBtYXJ0ID0gZW5zZW1ibCkKIyB3aWxsIHRha2Ugc29tZSB0aW1lIGZvciB0aGUgcXVlcnkgdG8gcnVuCgojIFByZXZpZXcgdGhlIHJlc3VsdApoZWFkKGlkX21hcHBpbmcpCmBgYAoKVGhlIGBpZF9tYXBwaW5nYCB0YWJsZSBub3cgaW5jbHVkZXMgdGhlIEVOU0VNQkwgaW5mb3JtYXRpb24gYW5kIGEgZ2VuZSBzeW1ib2wgb25seSBmb3IgdGhlIGdlbmVzIGluY2x1ZGVkIGluIG91ciByZXN1bHRzLiBUaGlzIHRhYmxlIHNob3VsZCBsb29rIGZhbWlsaWFyIGFzIGl0J3MgdGhlIHNhbWUgdGFibGUgd2UgdXNlZCB0byBhbm5vdGF0ZSBvdXIgcmVzdWx0cyB0YWJsZSBpbiB0aGUgbGFzdCBtb2R1bGUuCgoqKk5vdGUqKjogRm9yIGFkZGl0aW9uYWwgaW5mb3JtYXRpb24gcmVnYXJkaW5nIGJpb01hcnQsIHBsZWFzZSBjb25zdWx0IHRoZSBbRU5TRU1CTCBiaW9NYXJ0IHZpZ25ldHRlXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy9iaW9tYVJ0L2luc3QvZG9jL2FjY2Vzc2luZ19lbnNlbWJsLmh0bWwpIG9yIHRoZSBicm9hZGVyIFtCaW9jb25kdWN0b3IgQW5ub3RhdGlvbiBSZXNvdXJjZXMgdmlnbmV0dGUgXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC93b3JrZmxvd3MvdmlnbmV0dGVzL2Fubm90YXRpb24vaW5zdC9kb2MvQW5ub3RhdGlvbl9SZXNvdXJjZXMuaHRtbCkuCgotLS0KCiMgT3V0cHV0dGluZyByZXN1bHRzIHRvIGZpbGUKCkEga2V5IGFzcGVjdCBvZiBvdXIgYW5hbHlzaXMgaXMgcHJlc2VydmluZyB0aGUgcmVsZXZhbnQgZGF0YXNldHMgZm9yIGJvdGggb3VyIHJlY29yZHMgYW5kIGZvciBkb3duc3RyZWFtIGFwcGxpY2F0aW9ucywgc3VjaCBhcyBmdW5jdGlvbmFsIGVucmljaG1lbnRzLgoKIyMgREUgcmVzdWx0cyB0YWJsZQoKV2UnbGwgd3JpdGUgb3V0IG91ciBERSByZXN1bHRzLCBub3cgdGhhdCB3ZSd2ZSBhZGRlZCBpbmZvcm1hdGlvbiB0byB0aGUgdGFibGUgdG8gaGVscCB1cyBvciBvdXIgY29sbGFib3JhdG9ycyBpbnRlcnByZXQgdGhlIHJlc3VsdHMuCgpgYGB7ciBERVJlc3VsdHNPdXRwdXQsIGV2YWwgPSBGQUxTRX0Kd3JpdGUuY3N2KHJlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2wsCiAgICAgICAgICByb3cubmFtZXMgPSBGQUxTRSwKICAgICAgICAgIG5hID0gIi4iLAogICAgICAgICAgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVfcmVzdWx0c19kZWZpY2llbnRfdnNfY29udHJvbC5jc3YiKQoKd3JpdGUuY3N2KHJlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2xfYW5ub3RhdGVkLAogICAgICAgICAgcm93Lm5hbWVzID0gRkFMU0UsCiAgICAgICAgICBmaWxlPSJvdXRwdXRzL3RhYmxlcy9ERV9yZXN1bHRzX2RlZmljaWVudF92c19jb250cm9sX2Fubm90YXRlZC5jc3YiKQpgYGAKCgojIyBTdWJzZXR0aW5nIHNpZ25pZmljYW50IGdlbmVzCgpZb3UgbWF5IGJlIGludGVyZXN0ZWQgaW4gY3JlYXRpbmcgYSB0YWJsZSBvZiBvbmx5IHRoZSBnZW5lcyB0aGF0IHBhc3MgeW91ciBzaWduaWZpY2FuY2UgdGhyZXNob2xkcy4gQSB1c2VmdWwgd2F5IHRvIGRvIHRoaXMgaXMgdG8gY29uZGl0aW9uYWxseSBzdWJzZXQgeW91ciByZXN1bHRzLiBBZ2Fpbiwgd2UgYWxyZWFkeSBjcmVhdGVkIHRoZSBgY2FsbGAgY29sdW1uLCB3aGljaCBtYWtlcyB0aGlzIHJlbGF0aXZlbHkgc2ltcGxlIHRvIGRvOgoKYGBge3IgQ29uZGl0aW9uYWxTdWJzZXR9CiMgdGlkeXIgKHJlcXVpcmVzIHRhYmxlIHJlZm9ybWF0dGluZykKcmVzX3NpZyA8LSBhc190aWJibGUocmVzdWx0c19kZWZpY2llbnRfdnNfY29udHJvbCwgcm93bmFtZXMgPSAiZ2VuZV9pZHMiKSAlPiUgZmlsdGVyKGNhbGwgIT0gJ05TJykKCmhlYWQocmVzX3NpZykKZGltKHJlc19zaWcpCmBgYAoKPCEtLSBhZGQgYmFzZSB2ZXJzaW9uIGFzIGRyb3Bkb3duOiAjIGJhc2UKcmVzX3NpZyA8LSByZXN1bHRzX2RlZmljaWVudF92c19jb250cm9sW3Jlc3VsdHNfZGVmaWNpZW50X3ZzX2NvbnRyb2wkY2FsbCAhPSAnTlMnLCBdIC0tPgoKT25jZSB3ZSd2ZSBjcmVhdGVkIHRoaXMgdGFibGUsIHdlIGNhbiBhbHNvIHdyaXRlIGl0IG91dCB0byBmaWxlOgpgYGB7ciBERU9ubHlPdXRwdXQsIGV2YWwgPSBGQUxTRX0Kd3JpdGUuY3N2KHJlc19zaWcsCiAgICAgICAgICByb3cubmFtZXMgPSBGQUxTRSwKICAgICAgICAgIG5hID0gIi4iLAogICAgICAgICAgZmlsZT0ib3V0cHV0cy90YWJsZXMvREVHcy1vbmx5X2RlZmljaWVudF92c19jb250cm9sLmNzdiIpCmBgYAoKCiMjIFIgc2Vzc2lvbiBkYXRhCgpJbiBhZGRpdGlvbiB0byB0aGUgaW5kaXZpZHVhbCBST2JqKHMpIHdlIHNhdmVkIGVhcmxpZXIsIHdlIGNhbiBjYXB0dXJlIGEgc25hcHNob3Qgb3VyIGVudGlyZSBzZXNzaW9uIHVzaW5nIHRoZSBgc2F2ZS5pbWFnZWAgZnVuY3Rpb24uIFRoaXMgY2FuIGJlIGxvYWRlZCBpbiB0aGUgc2FtZSBtYW5uZXIgYXMgYW4gaW5kaXZpZHVhbCBSb2JqLgoKRmlyc3QsIHdlJ2xsIHNhdmUgb3VyIHNlc3Npb24gaW5mbyBzbyB3ZSBjYW4gcmVmZXJlbmNlIHRoZSBwYWNrYWdlcyBhbmQgdmVyc2lvbnMgdXNlZCB0byBnZW5lcmF0ZSB0aGVzZSBkYXRhLgoKYGBge3J9CnNlc3Npb25fc3VtbWFyeSA8LSBzZXNzaW9uSW5mbygpCmBgYAoKYGBge3IgfQpzYXZlLmltYWdlKGZpbGUgPSAib3V0cHV0cy9Sb2Jqcy9ERV9pcm9uLlJEYXRhIikKYGBgCgoKCiMgT3ZlcmFsbCB0YWtlYXdheXMKCldlJ3ZlIHJ1biB0aHJvdWdoIG1vc3Qgb2YgdGhlIGJ1aWxkaW5nIGJsb2NrcyBuZWVkZWQgdG8gcnVuIGEgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgYW5kIGhvcGVmdWxseSBidWlsdCB1cCBhIGJldHRlciB1bmRlcnN0YW5kaW5nIG9mIGhvdyBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBjb21wYXJpc29ucyB3b3JrLCBwYXJ0aWN1bGFybHkgaG93IGV4cGVyaW1lbnRhbCBkZXNpZ24gY2FuIGltcGFjdCBvdXIgcmVzdWx0cy4KCldoYXQgdG8gY29uc2lkZXIgbW92aW5nIGZvcndhcmQ6CgoqIEhvdyBjYW4gSSBjb250cm9sIGZvciB0ZWNobmljYWwgdmFyaWF0aW9uIGluIG15IGV4cGVyaW1lbnRhbCBkZXNpZ24/CiogSG93IG11Y2ggdmFyaWF0aW9uIGlzIGV4cGVjdGVkIHdpdGggYSB0cmVhdG1lbnQgZ3JvdXA/CiogV2hhdCBpcyBteSBSTkEgcXVhbGl0eSwgYW5kIGhvdyBjYW4gdGhhdCBiZSBvcHRpbWl6ZWQ/CiogQXJlIHRoZXJlIHF1YWxpdHkgY29uY2VybnMgZm9yIG15IHNlcXVlbmNpbmcgZGF0YT8KKiBXaGF0IGNvbXBhcmlzb25zIGFyZSByZWxldmFudCB0byBteSBiaW9sb2dpY2FsIHF1ZXN0aW9uPwoqIEFyZSB0aGVyZSBjb3ZhcmlhdGVzIHRoYXQgc2hvdWxkIGJlIGNvbnNpZGVyZWQ/CiogV2hhdCB3aWxsIGEgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgdGVsbCBtZT8KCgoqKkxldCdzIHBhdXNlIGhlcmUgZm9yIGdlbmVyYWwgcXVlc3Rpb25zKioKCi0tLQoKIyBOZXh0IHN0ZXBzIC0gSG93IGRvIHdlIG1ha2Ugc2Vuc2Ugb2YgbGFyZ2UgbnVtYmVycyBvZiBERSBnZW5lcz8KCkEgd2F5IHRvIGRldGVybWluZSBwb3NzaWJsZSBbYnJvYWRlciBiaW9sb2dpY2FsIGludGVycHJldGF0aW9uc10oaHR0cHM6Ly93d3cuZWJpLmFjLnVrL3RyYWluaW5nLWJldGEvb25saW5lL2NvdXJzZXMvZnVuY3Rpb25hbC1nZW5vbWljcy1paS1jb21tb24tdGVjaG5vbG9naWVzLWFuZC1kYXRhLWFuYWx5c2lzLW1ldGhvZHMvYmlvbG9naWNhbC1pbnRlcnByZXRhdGlvbi1vZi1nZW5lLWV4cHJlc3Npb24tZGF0YS0yLykgZnJvbSB0aGUgb2JzZXJ2ZWQgREUgcmVzdWx0cywgaXMgZnVuY3Rpb25hbCBlbnJpY2htZW50cy4gCgpUaGVyZSBhcmUgbWFueSBvcHRpb25zLCBzdWNoIGFzIHNvbWUgaW5jbHVkZWQgaW4gdGhpcyBbZGlzY3Vzc2lvbiB0aHJlYWRdKGh0dHBzOi8vd3d3LnJlc2VhcmNoZ2F0ZS5uZXQvcG9zdC9Ib3dfY2FuX0lfYW5hbHl6ZV9hX3NldF9vZl9ERUdzX2RpZmZlcmVudGlhbGx5X2V4cHJlc3NlZF9nZW5lc190b19vYnRhaW5faW5mb3JtYXRpb25fZnJvbV90aGVtKS4gT3RoZXIgY29tbW9uIGZ1bmN0aW9uYWwgZW5yaWNobWVudHMgYXBwcm9hY2hlcyBhcmUgZ2VuZSBzZXQgZW5yaWNobWVudCBhbmFseXNpcywgYWthIFtHU0VBXShodHRwOi8vc29mdHdhcmUuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEvaW5kZXguanNwKSwgRGF0YWJhc2UgZm9yIEFubm90YXRpb24sIFZpc3VhbGl6YXRpb24gYW5kIEludGVncmF0ZWQgRGlzY292ZXJ5LCBha2EgW0RBVklEXShodHRwczovL2RhdmlkLm5jaWZjcmYuZ292LyksIFtJbmdlbml0eV0oaHR0cHM6Ly9kaWdpdGFsaW5zaWdodHMucWlhZ2VuLmNvbS8pLCBhbmQgW2lQYXRod2F5IEd1aWRlXShodHRwczovL2FkdmFpdGFiaW8uY29tL2lwYXRod2F5Z3VpZGUvKQoKVGhlIFVuaXZlcnNpdHkgb2YgTWljaGlnYW4gaGFzIGxpY2Vuc2UgYW5kIHN1cHBvcnQgZm9yIGFkZGl0aW9uYWwgdG9vbHMsIHN1Y2ggYXMgQ3l0b3NjYXBlLCBzbyB3ZSByZWNvbW1lbmQgcmVhY2hpbmcgb3V0IHRvIHN0YWZmIHdpdGggW1RhdWJtYW4gTGlicmFyeV0oaHR0cHM6Ly93d3cubGliLnVtaWNoLmVkdS9sb2NhdGlvbnMtYW5kLWhvdXJzL3RhdWJtYW4taGVhbHRoLXNjaWVuY2VzLWxpYnJhcnkvcmVzZWFyY2gtYW5kLWNsaW5pY2FsLXN1cHBvcnQpIHRvIGxlYXJuIG1vcmUgYWJvdXQgcmVzb3VyY2VzIHRoYXQgbWlnaHQgYmUgYXBwbGljYXRpb24gdG95b3VyIHJlc2VhcmNoLgoKLS0tCgojIFNvdXJjZXMKCiogSEJDIERHRSB0cmFpbmluZyBtb2R1bGUsIHBhcnQgMTogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDRfREdFX0RFU2VxMl9hbmFseXNpcy5odG1sCiogSEJDIERHRSB0cmFpbmluZyBtb2R1bGUsIHBhcnQgMjogaHR0cHM6Ly9oYmN0cmFpbmluZy5naXRodWIuaW8vREdFX3dvcmtzaG9wL2xlc3NvbnMvMDVfREdFX0RFU2VxMl9hbmFseXNpczIuaHRtbAoqIERFU2VxMiB2aWduZXR0ZTogaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2RpZmZlcmVudGlhbC1leHByZXNzaW9uLWFuYWx5c2lzCiogQmlvY29uZHVjdG9yIEdlbm9taWMgQW5ub3RhdGlvbiByZXNvdXJjZXM6IGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL3dvcmtmbG93cy92aWduZXR0ZXMvYW5ub3RhdGlvbi9pbnN0L2RvYy9Bbm5vdGF0aW9uX1Jlc291cmNlcy5odG1sCiogQmlvTWFydCB2aWduZXR0ZTogaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvYmlvbWFSdC9pbnN0L2RvYy9hY2Nlc3NpbmdfZW5zZW1ibC5odG1sCgojIEFkZGl0aW9uYWwgUmVzb3VyY2VzCiogTUlEQVMgUmVwcm9kdWNpYmxpdHkgSHViOiBodHRwczovL21pZGFzLnVtaWNoLmVkdS9yZXByb2R1Y2liaWxpdHktb3ZlcnZpZXcvCiogQVJDIHJlc291cmNlczogaHR0cHM6Ly9hcmMtdHMudW1pY2guZWR1LwoqIEdlbmUgU2V0IEVucmljaG1lbnQgUmVzb3VyY2VzIGZyb20gQmlvY29uZHVjdG9yOiBodHRwczovL2Jpb2luZm9ybWF0aWNzLWNvcmUtc2hhcmVkLXRyYWluaW5nLmdpdGh1Yi5pby9jcnVrLXN1bW1lci1zY2hvb2wtMjAxOC9STkFTZXEyMDE4L2h0bWwvMDZfR2VuZV9zZXRfdGVzdGluZy5uYi5odG1sCiogVXNpbmcgSFRTZXEgZGF0YSB3aXRoIERFU2VxMjogaHR0cHM6Ly9hbmd1cy5yZWFkdGhlZG9jcy5pby9lbi8yMDE5L2RpZmYtZXgtYW5kLXZpei5odG1sCiogRGV0YWlsZWQgUk5BLXNlcSBhbmFseXNpcyBwYXBlcjogaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wbWMvYXJ0aWNsZXMvUE1DNjA5NjM0Ni8KKiBPdmVydmlldyBvZiBSTkEtc2VxIGFuYWx5c2lzIGNvbnNpZGVyYXRpb25zOiBodHRwczovL2FjYWRlbWljLW91cC1jb20ucHJveHkubGliLnVtaWNoLmVkdS9iZmcvYXJ0aWNsZS8xNC8yLzEzMC8yNTczNzAKKiBBbHRlcm5hdGl2ZSBvdmVydmlldyBvZiBERVNlcTIsIGluY2x1ZGluZyB2aXN1YWxpemF0aW9ucyBhbmQgZnVuY3Rpb25hbCBlbnJpY2htZW50czogaHR0cDovL2RwdXRoaWVyLmdpdGh1Yi5pby9qZ2I3MWUtcG9seXRlY2gtYmlvaW5mby1hcHAvcHJhY3RpY2FsL3JuYS1zZXFfUi9ybmFzZXFfZGlmZl9TbmYyLmh0bWwKCi0tLQoKYGBge3IgV3JpdGVPdXQuUkRhdGEsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBIaWRkZW4gY29kZSBibG9jayB0byB3cml0ZSBvdXQgZGF0YSBmb3Iga25pdHRpbmcKIyBzYXZlLmltYWdlKGZpbGUgPSAicmRhdGEvUnVubmluZ0RhdGFfRnVsbC5SRGF0YSIpCmBgYAoKIyBTZXNzaW9uIEluZm8KYGBge3IgU2Vzc2lvbkluZm99CnNlc3Npb25JbmZvKCkKYGBgCgotLS0KClRoZXNlIG1hdGVyaWFscyBoYXZlIGJlZW4gYWRhcHRlZCBhbmQgZXh0ZW5kZWQgZnJvbSBtYXRlcmlhbHMgbGlzdGVkIGFib3ZlLiBUaGVzZSBhcmUgb3BlbiBhY2Nlc3MgbWF0ZXJpYWxzIGRpc3RyaWJ1dGVkIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgW0NyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gbGljZW5zZSAoQ0MgQlkgNC4wKV0oaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyksIHdoaWNoIHBlcm1pdHMgdW5yZXN0cmljdGVkIHVzZSwgZGlzdHJpYnV0aW9uLCBhbmQgcmVwcm9kdWN0aW9uIGluIGFueSBtZWRpdW0sIHByb3ZpZGVkIHRoZSBvcmlnaW5hbCBhdXRob3IgYW5kIHNvdXJjZSBhcmUgY3JlZGl0ZWQuCgo8YnIvPgo8YnIvPgo8aHIvPgp8IFtQcmV2aW91cyBsZXNzb25dKE1vZHVsZTExX0RFVmlzdWFsaXphdGlvbnMuaHRtbCkgfCBbVG9wIG9mIHRoaXMgbGVzc29uXSgjdG9wKSB8IFtOZXh0IGxlc3Nvbl0oUl9ib251c19jb250ZW50Lmh0bWwpIHwKfCA6LS0tIHwgOi0tLS06IHwgLS0tOiB8