Objectives

  • Additional visualizations for gene/sample level QC assessment

Count boxplots

To evaluate the difference between the count distributions before and after normalization, let’s extract the raw counts and the rlog normalized counts, and coerce them to tibbles.

raw_counts = as_tibble(counts(dds), rownames = 'id')
norm_counts = as_tibble(assay(rld), rownames = 'id')

raw_counts
# A tibble: 16,249 × 7
   id                 sample_A sample_B sample_C sample_D sample_E sample_F
   <chr>                 <int>    <int>    <int>    <int>    <int>    <int>
 1 ENSMUSG00000000001     1041      905     1296     3481     1283     1921
 2 ENSMUSG00000000028     1043     1232     1664     2690     1825     2720
 3 ENSMUSG00000000031     1819      914     1618     8618     1350     1222
 4 ENSMUSG00000000037       19       11       18       48       37       29
 5 ENSMUSG00000000049       18        1        4       24        1        1
 6 ENSMUSG00000000056    14972    14768    21026    22962    22263    23622
 7 ENSMUSG00000000078      888      607      911     1689      738     1180
 8 ENSMUSG00000000085      402      483      744      898      811     1261
 9 ENSMUSG00000000088      904     1212     1733     2434     1387     1267
10 ENSMUSG00000000093        7        8       16       23       17       29
# … with 16,239 more rows
norm_counts
# A tibble: 16,249 × 7
   id                 sample_A sample_B sample_C sample_D sample_E sample_F
   <chr>                 <dbl>    <dbl>    <dbl>    <dbl>    <dbl>    <dbl>
 1 ENSMUSG00000000001    10.5     10.4     10.4     10.8     10.4     10.6 
 2 ENSMUSG00000000028    10.6     10.7     10.7     10.7     10.8     11.0 
 3 ENSMUSG00000000031    11.2     10.5     10.7     11.9     10.6     10.3 
 4 ENSMUSG00000000037     4.64     4.56     4.58     4.66     4.72     4.62
 5 ENSMUSG00000000049     3.02     2.82     2.84     2.94     2.81     2.81
 6 ENSMUSG00000000056    14.3     14.3     14.3     13.9     14.4     14.2 
 7 ENSMUSG00000000078    10.1      9.77     9.86     9.93     9.68     9.90
 8 ENSMUSG00000000085     9.28     9.41     9.53     9.23     9.60     9.80
 9 ENSMUSG00000000088    10.4     10.6     10.7     10.5     10.5     10.2 
10 ENSMUSG00000000093     3.88     3.89     3.93     3.91     3.94     3.99
# … with 16,239 more rows

Next, this is the perfect opportunity to use tidyr::pivot_longer() because we want to use ggplot() for the bar plots, but the data is currently in the wide form, not the tidy form!

tidy_raw = tidyr::pivot_longer(raw_counts, -id, names_to = 'sample', values_to = 'counts')
tidy_norm = tidyr::pivot_longer(norm_counts, -id, names_to = 'sample', values_to = 'counts')

We should also join in the sample metadata so that we can color on the sample groups.

samplesheet_tbl = as_tibble(colData(dds), rownames = 'sample')

tidy_raw = tidy_raw %>% left_join(samplesheet_tbl, by = 'sample')
tidy_norm = tidy_norm %>% left_join(samplesheet_tbl, by = 'sample')
raw_boxplot = ggplot(tidy_raw, aes(x = sample, y = log2(counts), fill = condition)) +
    geom_boxplot(notch = TRUE) +
    labs(
        title = 'Raw Counts',
        x = 'Sample',
        y = 'log2 Counts') +
    theme_bw() + theme(axis.text.x = element_text(angle = 90))
raw_boxplot
Warning: Removed 6656 rows containing non-finite values (`stat_boxplot()`).

Question

Why did we get that warning about non-finite values?

norm_boxplot = ggplot(tidy_norm, aes(x = sample, y = counts, fill = condition)) +
    geom_boxplot(notch = TRUE) +
    labs(
        title = 'rlog Normalized Counts',
        x = 'Sample',
        y = 'rlog Counts') +
    theme_bw() + theme(axis.text.x = element_text(angle = 90))
norm_boxplot

Observe that the normalized plots truly do appear normalized; their means are more uniform. Let’s go ahead and save these plots in our outputs/figures folder.

ggsave(filename = 'outputs/figures/Boxplot_raw_condition.pdf', plot = raw_boxplot, height = 6, width = 6)
ggsave(filename = 'outputs/figures/Boxplot_rlog_condition.pdf', plot = norm_boxplot, height = 6, width = 6)

Heatmaps

Let’s create a heatmap based on the distance between pair-wise sample expression profiles. This is another vantage point of how similar and dissimilar the samples are from one another. To get started, we actually want a plain matrix, with out a column for the gene IDs, as we did for the boxplot.

norm_mat = assay(rld)
head(norm_mat)
                    sample_A  sample_B  sample_C  sample_D  sample_E  sample_F
ENSMUSG00000000001 10.514813 10.366709 10.419463 10.840373 10.410449 10.578771
ENSMUSG00000000028 10.604461 10.734506 10.735026 10.682714 10.820938 10.990999
ENSMUSG00000000031 11.160276 10.498747 10.742763 11.861617 10.578156 10.298022
ENSMUSG00000000037  4.642433  4.555500  4.578934  4.656950  4.719012  4.620204
ENSMUSG00000000049  3.017478  2.820198  2.843894  2.936455  2.814833  2.811619
ENSMUSG00000000056 14.321672 14.284652 14.337197 13.904238 14.393912 14.235971

Next, we’ll use the dist() function on the transpose of norm_mat to compute the distance. We will use the default Euclidean distance.

dist_mat = dist(t(norm_mat), upper = TRUE)

Next, we’ll plot a very simple pheatmap() using this matrix. But let’s check the class of dist_mat first, because the input to pheatmap() needs to be a matrix.

class(dist_mat)
[1] "dist"
# Have to coerce it
dist_mat = as.matrix(dist_mat)

dist_heatmap = pheatmap(
    mat = dist_mat,
    cluster_rows = TRUE,
    cluster_cols = TRUE,
    show_rownames = TRUE,
    show_colnames = TRUE
)
dist_heatmap

This is nice, but there are a few tweaks we might consider:

  1. The color scale used by default is divergent, but our distances are values in [0, Inf), so a linear color scale would be more appropriate here.
  2. We might like for the sample metadata to be included so that we can easily tell if the samples cluster by their condition.

To accomplish the first change, we’ll use the colorRampPalette() function from the RColorBrewer package. This is the package to use for creating color scales of all varieties. We highly recommend exploring the package website and documentation.

colors = colorRampPalette(brewer.pal(9, 'Blues'))(50)

Now let’s use that in the pheatmap() call using the color parameter:

dist_heatmap = pheatmap(
    mat = dist_mat,
    color = colors,
    cluster_rows = TRUE,
    cluster_cols = TRUE,
    show_rownames = TRUE,
    show_colnames = TRUE
)
dist_heatmap

This is a lot more reasonable. More distant samples are a deeper blue, while more similar samples are closer to white. Next, let’s add some annotation data.

annotation_tbl = samplesheet_tbl %>%
    dplyr::select(sample, condition) %>%
    tibble::column_to_rownames(var = 'sample')

dist_heatmap = pheatmap(
    mat = dist_mat,
    color = colors,
    cluster_rows = TRUE,
    cluster_cols = TRUE,
    show_rownames = TRUE,
    show_colnames = TRUE,
    annotation_col = annotation_tbl
)
dist_heatmap

And now we have our conditions as a colored annotation bar along the columns.

Session Info

sessionInfo()
R version 4.2.1 (2022-06-23)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur ... 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] forcats_0.5.2               stringr_1.4.1              
 [3] purrr_0.3.5                 readr_2.1.3                
 [5] tibble_3.1.8                tidyverse_1.3.2            
 [7] biomaRt_2.52.0              data.table_1.14.4          
 [9] RColorBrewer_1.1-3          pheatmap_1.0.12            
[11] ggrepel_0.9.2               dplyr_1.0.10               
[13] tidyr_1.2.1                 ggplot2_3.4.0              
[15] DESeq2_1.36.0               SummarizedExperiment_1.26.1
[17] Biobase_2.56.0              MatrixGenerics_1.8.1       
[19] matrixStats_0.62.0          GenomicRanges_1.48.0       
[21] GenomeInfoDb_1.32.4         IRanges_2.30.1             
[23] S4Vectors_0.34.0            BiocGenerics_0.42.0        
[25] knitr_1.40                  rmarkdown_2.17             

loaded via a namespace (and not attached):
 [1] googledrive_2.0.0      colorspace_2.0-3       ellipsis_0.3.2        
 [4] XVector_0.36.0         fs_1.5.2               farver_2.1.1          
 [7] bit64_4.0.5            AnnotationDbi_1.58.0   fansi_1.0.3           
[10] lubridate_1.9.0        xml2_1.3.3             codetools_0.2-18      
[13] splines_4.2.1          cachem_1.0.6           geneplotter_1.74.0    
[16] jsonlite_1.8.3         broom_1.0.1            annotate_1.74.0       
[19] dbplyr_2.2.1           png_0.1-7              compiler_4.2.1        
[22] httr_1.4.4             backports_1.4.1        assertthat_0.2.1      
[25] Matrix_1.5-1           fastmap_1.1.0          gargle_1.2.1          
[28] cli_3.4.1              htmltools_0.5.3        prettyunits_1.1.1     
[31] tools_4.2.1            gtable_0.3.1           glue_1.6.2            
[34] GenomeInfoDbData_1.2.8 rappdirs_0.3.3         Rcpp_1.0.9            
[37] cellranger_1.1.0       jquerylib_0.1.4        vctrs_0.5.0           
[40] Biostrings_2.64.1      xfun_0.34              rvest_1.0.3           
[43] timechange_0.1.1       lifecycle_1.0.3        XML_3.99-0.12         
[46] googlesheets4_1.0.1    zlibbioc_1.42.0        scales_1.2.1          
[49] hms_1.1.2              parallel_4.2.1         yaml_2.3.6            
[52] curl_4.3.3             memoise_2.0.1          sass_0.4.2            
[55] stringi_1.7.8          RSQLite_2.2.18         highr_0.9             
[58] genefilter_1.78.0      filelock_1.0.2         BiocParallel_1.30.4   
[61] rlang_1.0.6            pkgconfig_2.0.3        bitops_1.0-7          
[64] evaluate_0.18          lattice_0.20-45        labeling_0.4.2        
[67] bit_4.0.4              tidyselect_1.2.0       magrittr_2.0.3        
[70] R6_2.5.1               generics_0.1.3         DelayedArray_0.22.0   
[73] DBI_1.1.3              pillar_1.8.1           haven_2.5.1           
[76] withr_2.5.0            survival_3.4-0         KEGGREST_1.36.3       
[79] RCurl_1.98-1.9         modelr_0.1.9           crayon_1.5.2          
[82] utf8_1.2.2             BiocFileCache_2.4.0    tzdb_0.3.0            
[85] progress_1.2.2         readxl_1.4.1           locfit_1.5-9.6        
[88] grid_4.2.1             blob_1.2.3             reprex_2.0.2          
[91] digest_0.6.30          xtable_1.8-4           munsell_0.5.0         
[94] bslib_0.4.1           

These materials have been adapted and extended from materials listed above. These are open access materials distributed under the terms of the Creative Commons Attribution license (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

LS0tCnRpdGxlOiAiTW9kdWxlIDExWDogQm9udXMgQ29udGVudCIKYXV0aG9yOiAiVU0gQmlvaW5mb3JtYXRpY3MgQ29yZSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiAgICAgICAgaHRtbF9kb2N1bWVudDoKICAgICAgICAgICAgaW5jbHVkZXM6CiAgICAgICAgICAgICAgICBpbl9oZWFkZXI6IGhlYWRlci5odG1sCiAgICAgICAgICAgIHRoZW1lOiBwYXBlcgogICAgICAgICAgICB0b2M6IHRydWUKICAgICAgICAgICAgdG9jX2RlcHRoOiA0CiAgICAgICAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICAgICAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgICAgICAgICAgIGZpZ19jYXB0aW9uOiB0cnVlCiAgICAgICAgICAgIG1hcmtkb3duOiBHRk0KICAgICAgICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+CmJvZHksIHRkIHsKICAgZm9udC1zaXplOiAxOHB4Owp9CmNvZGUucnsKICBmb250LXNpemU6IDEycHg7Cn0KcHJlIHsKICBmb250LXNpemU6IDEycHgKfQo8L3N0eWxlPgoKYGBge3IsIGluY2x1ZGUgPSBGQUxTRX0Kc291cmNlKCIuLi9iaW4vY2h1bmstb3B0aW9ucy5SIikKa25pdHJfZmlnX3BhdGgoIjExWC0iKQpgYGAKCmBgYHtyIE1vZHVsZXMsIGV2YWw9VFJVRSwgZWNobz1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KbGlicmFyeShERVNlcTIpCmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KGdncmVwZWwpCmxpYnJhcnkocGhlYXRtYXApCmxpYnJhcnkoUkNvbG9yQnJld2VyKQpgYGAKCj4gIyBPYmplY3RpdmVzIHsudW5saXN0ZWQgLnVubnVtYmVyZWR9Cj4gKiBBZGRpdGlvbmFsIHZpc3VhbGl6YXRpb25zIGZvciBnZW5lL3NhbXBsZSBsZXZlbCBRQyBhc3Nlc3NtZW50CgojIENvdW50IGJveHBsb3RzCgpUbyBldmFsdWF0ZSB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBjb3VudCBkaXN0cmlidXRpb25zIGJlZm9yZSBhbmQgYWZ0ZXIgbm9ybWFsaXphdGlvbiwgbGV0J3MgZXh0cmFjdCB0aGUgcmF3IGNvdW50cyBhbmQgdGhlIHJsb2cgbm9ybWFsaXplZCBjb3VudHMsIGFuZCBjb2VyY2UgdGhlbSB0byBgdGliYmxlYHMuCgpgYGB7ciBSYXdDb3VudHNTZXR1cH0KcmF3X2NvdW50cyA9IGFzX3RpYmJsZShjb3VudHMoZGRzKSwgcm93bmFtZXMgPSAnaWQnKQpub3JtX2NvdW50cyA9IGFzX3RpYmJsZShhc3NheShybGQpLCByb3duYW1lcyA9ICdpZCcpCgpyYXdfY291bnRzCm5vcm1fY291bnRzCmBgYAoKTmV4dCwgdGhpcyBpcyB0aGUgKipwZXJmZWN0Kiogb3Bwb3J0dW5pdHkgdG8gdXNlIGB0aWR5cjo6cGl2b3RfbG9uZ2VyKClgIGJlY2F1c2Ugd2Ugd2FudCB0byB1c2UgYGdncGxvdCgpYCBmb3IgdGhlIGJhciBwbG90cywgYnV0IHRoZSBkYXRhIGlzIGN1cnJlbnRseSBpbiB0aGUgd2lkZSBmb3JtLCBub3QgdGhlIHRpZHkgZm9ybSEKCmBgYHtyIFBpdm90TG9uZ2VyfQp0aWR5X3JhdyA9IHRpZHlyOjpwaXZvdF9sb25nZXIocmF3X2NvdW50cywgLWlkLCBuYW1lc190byA9ICdzYW1wbGUnLCB2YWx1ZXNfdG8gPSAnY291bnRzJykKdGlkeV9ub3JtID0gdGlkeXI6OnBpdm90X2xvbmdlcihub3JtX2NvdW50cywgLWlkLCBuYW1lc190byA9ICdzYW1wbGUnLCB2YWx1ZXNfdG8gPSAnY291bnRzJykKYGBgCgpXZSBzaG91bGQgYWxzbyBqb2luIGluIHRoZSBzYW1wbGUgbWV0YWRhdGEgc28gdGhhdCB3ZSBjYW4gY29sb3Igb24gdGhlIHNhbXBsZSBncm91cHMuCgpgYGB7ciBKb2luTWV0YWRhdGF9CnNhbXBsZXNoZWV0X3RibCA9IGFzX3RpYmJsZShjb2xEYXRhKGRkcyksIHJvd25hbWVzID0gJ3NhbXBsZScpCgp0aWR5X3JhdyA9IHRpZHlfcmF3ICU+JSBsZWZ0X2pvaW4oc2FtcGxlc2hlZXRfdGJsLCBieSA9ICdzYW1wbGUnKQp0aWR5X25vcm0gPSB0aWR5X25vcm0gJT4lIGxlZnRfam9pbihzYW1wbGVzaGVldF90YmwsIGJ5ID0gJ3NhbXBsZScpCmBgYAoKYGBge3IgUmF3Qm94cGxvdH0KcmF3X2JveHBsb3QgPSBnZ3Bsb3QodGlkeV9yYXcsIGFlcyh4ID0gc2FtcGxlLCB5ID0gbG9nMihjb3VudHMpLCBmaWxsID0gY29uZGl0aW9uKSkgKwogICAgZ2VvbV9ib3hwbG90KG5vdGNoID0gVFJVRSkgKwogICAgbGFicygKICAgICAgICB0aXRsZSA9ICdSYXcgQ291bnRzJywKICAgICAgICB4ID0gJ1NhbXBsZScsCiAgICAgICAgeSA9ICdsb2cyIENvdW50cycpICsKICAgIHRoZW1lX2J3KCkgKyB0aGVtZShheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDkwKSkKcmF3X2JveHBsb3QKYGBgCgo+ICMgUXVlc3Rpb24gey51bmxpc3RlZCAudW5udW1iZXJlZH0KPiBXaHkgZGlkIHdlIGdldCB0aGF0IHdhcm5pbmcgYWJvdXQgbm9uLWZpbml0ZSB2YWx1ZXM/CgpgYGB7ciBOb3JtQm94cGxvdH0Kbm9ybV9ib3hwbG90ID0gZ2dwbG90KHRpZHlfbm9ybSwgYWVzKHggPSBzYW1wbGUsIHkgPSBjb3VudHMsIGZpbGwgPSBjb25kaXRpb24pKSArCiAgICBnZW9tX2JveHBsb3Qobm90Y2ggPSBUUlVFKSArCiAgICBsYWJzKAogICAgICAgIHRpdGxlID0gJ3Jsb2cgTm9ybWFsaXplZCBDb3VudHMnLAogICAgICAgIHggPSAnU2FtcGxlJywKICAgICAgICB5ID0gJ3Jsb2cgQ291bnRzJykgKwogICAgdGhlbWVfYncoKSArIHRoZW1lKGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gOTApKQpub3JtX2JveHBsb3QKYGBgCgpPYnNlcnZlIHRoYXQgdGhlIG5vcm1hbGl6ZWQgcGxvdHMgdHJ1bHkgZG8gYXBwZWFyIG5vcm1hbGl6ZWQ7IHRoZWlyIG1lYW5zIGFyZSBtb3JlIHVuaWZvcm0uIExldCdzIGdvIGFoZWFkIGFuZCBzYXZlIHRoZXNlIHBsb3RzIGluIG91ciBgb3V0cHV0cy9maWd1cmVzYCBmb2xkZXIuCgpgYGB7ciBCb3hwbG90c1Jhd1NhdmUsIGV2YWwgPSBGQUxTRX0KZ2dzYXZlKGZpbGVuYW1lID0gJ291dHB1dHMvZmlndXJlcy9Cb3hwbG90X3Jhd19jb25kaXRpb24ucGRmJywgcGxvdCA9IHJhd19ib3hwbG90LCBoZWlnaHQgPSA2LCB3aWR0aCA9IDYpCmdnc2F2ZShmaWxlbmFtZSA9ICdvdXRwdXRzL2ZpZ3VyZXMvQm94cGxvdF9ybG9nX2NvbmRpdGlvbi5wZGYnLCBwbG90ID0gbm9ybV9ib3hwbG90LCBoZWlnaHQgPSA2LCB3aWR0aCA9IDYpCmBgYAoKIyBIZWF0bWFwcwoKTGV0J3MgY3JlYXRlIGEgaGVhdG1hcCBiYXNlZCBvbiB0aGUgZGlzdGFuY2UgYmV0d2VlbiBwYWlyLXdpc2Ugc2FtcGxlIGV4cHJlc3Npb24gcHJvZmlsZXMuIFRoaXMgaXMgYW5vdGhlciB2YW50YWdlIHBvaW50IG9mIGhvdyBzaW1pbGFyIGFuZCBkaXNzaW1pbGFyIHRoZSBzYW1wbGVzIGFyZSBmcm9tIG9uZSBhbm90aGVyLiBUbyBnZXQgc3RhcnRlZCwgd2UgYWN0dWFsbHkgd2FudCBhIHBsYWluIG1hdHJpeCwgd2l0aCBvdXQgYSBjb2x1bW4gZm9yIHRoZSBnZW5lIElEcywgYXMgd2UgZGlkIGZvciB0aGUgYm94cGxvdC4KCmBgYHtyIE5vcm1Db3VudHN9Cm5vcm1fbWF0ID0gYXNzYXkocmxkKQpoZWFkKG5vcm1fbWF0KQpgYGAKCk5leHQsIHdlJ2xsIHVzZSB0aGUgYGRpc3QoKWAgZnVuY3Rpb24gb24gdGhlIHRyYW5zcG9zZSBvZiBgbm9ybV9tYXRgIHRvIGNvbXB1dGUgdGhlIGRpc3RhbmNlLiBXZSB3aWxsIHVzZSB0aGUgZGVmYXVsdCBFdWNsaWRlYW4gZGlzdGFuY2UuCgpgYGB7ciBEaXN0YW5jZU1hdH0KZGlzdF9tYXQgPSBkaXN0KHQobm9ybV9tYXQpLCB1cHBlciA9IFRSVUUpCmBgYAoKTmV4dCwgd2UnbGwgcGxvdCBhIHZlcnkgc2ltcGxlIGBwaGVhdG1hcCgpYCB1c2luZyB0aGlzIG1hdHJpeC4gQnV0IGxldCdzIGNoZWNrIHRoZSBjbGFzcyBvZiBgZGlzdF9tYXRgIGZpcnN0LCBiZWNhdXNlIHRoZSBpbnB1dCB0byBgcGhlYXRtYXAoKWAgbmVlZHMgdG8gYmUgYSBgbWF0cml4YC4KCmBgYHtyIERpc3RhbmNlSGVhdG1hcH0KY2xhc3MoZGlzdF9tYXQpCgojIEhhdmUgdG8gY29lcmNlIGl0CmRpc3RfbWF0ID0gYXMubWF0cml4KGRpc3RfbWF0KQoKZGlzdF9oZWF0bWFwID0gcGhlYXRtYXAoCiAgICBtYXQgPSBkaXN0X21hdCwKICAgIGNsdXN0ZXJfcm93cyA9IFRSVUUsCiAgICBjbHVzdGVyX2NvbHMgPSBUUlVFLAogICAgc2hvd19yb3duYW1lcyA9IFRSVUUsCiAgICBzaG93X2NvbG5hbWVzID0gVFJVRQopCmRpc3RfaGVhdG1hcApgYGAKClRoaXMgaXMgbmljZSwgYnV0IHRoZXJlIGFyZSBhIGZldyB0d2Vha3Mgd2UgbWlnaHQgY29uc2lkZXI6CgoxLiBUaGUgY29sb3Igc2NhbGUgdXNlZCBieSBkZWZhdWx0IGlzIGRpdmVyZ2VudCwgYnV0IG91ciBkaXN0YW5jZXMgYXJlIHZhbHVlcyBpbiBbMCwgSW5mKSwgc28gYSBsaW5lYXIgY29sb3Igc2NhbGUgd291bGQgYmUgbW9yZSBhcHByb3ByaWF0ZSBoZXJlLgoyLiBXZSBtaWdodCBsaWtlIGZvciB0aGUgc2FtcGxlIG1ldGFkYXRhIHRvIGJlIGluY2x1ZGVkIHNvIHRoYXQgd2UgY2FuIGVhc2lseSB0ZWxsIGlmIHRoZSBzYW1wbGVzIGNsdXN0ZXIgYnkgdGhlaXIgYGNvbmRpdGlvbmAuCgpUbyBhY2NvbXBsaXNoIHRoZSBmaXJzdCBjaGFuZ2UsIHdlJ2xsIHVzZSB0aGUgYGNvbG9yUmFtcFBhbGV0dGUoKWAgZnVuY3Rpb24gZnJvbSB0aGUgYFJDb2xvckJyZXdlcmAgcGFja2FnZS4gVGhpcyBpcyAqKnRoZSoqIHBhY2thZ2UgdG8gdXNlIGZvciBjcmVhdGluZyBjb2xvciBzY2FsZXMgb2YgYWxsIHZhcmlldGllcy4gV2UgaGlnaGx5IHJlY29tbWVuZCBleHBsb3JpbmcgdGhlIHBhY2thZ2Ugd2Vic2l0ZSBhbmQgZG9jdW1lbnRhdGlvbi4KCmBgYHtyIENob29zZUNvbG9yc30KY29sb3JzID0gY29sb3JSYW1wUGFsZXR0ZShicmV3ZXIucGFsKDksICdCbHVlcycpKSg1MCkKYGBgCgpOb3cgbGV0J3MgdXNlIHRoYXQgaW4gdGhlIGBwaGVhdG1hcCgpYCBjYWxsIHVzaW5nIHRoZSBgY29sb3JgIHBhcmFtZXRlcjoKCmBgYHtyIERpc3RhbmNlSGVhdG1hQ29sb3JzfQpkaXN0X2hlYXRtYXAgPSBwaGVhdG1hcCgKICAgIG1hdCA9IGRpc3RfbWF0LAogICAgY29sb3IgPSBjb2xvcnMsCiAgICBjbHVzdGVyX3Jvd3MgPSBUUlVFLAogICAgY2x1c3Rlcl9jb2xzID0gVFJVRSwKICAgIHNob3dfcm93bmFtZXMgPSBUUlVFLAogICAgc2hvd19jb2xuYW1lcyA9IFRSVUUKKQpkaXN0X2hlYXRtYXAKYGBgCgpUaGlzIGlzIGEgbG90IG1vcmUgcmVhc29uYWJsZS4gTW9yZSBkaXN0YW50IHNhbXBsZXMgYXJlIGEgZGVlcGVyIGJsdWUsIHdoaWxlIG1vcmUgc2ltaWxhciBzYW1wbGVzIGFyZSBjbG9zZXIgdG8gd2hpdGUuIE5leHQsIGxldCdzIGFkZCBzb21lIGFubm90YXRpb24gZGF0YS4KCmBgYHtyIERpc3RhbmNlSGVhdG1hcEFubm90YXRlZH0KYW5ub3RhdGlvbl90YmwgPSBzYW1wbGVzaGVldF90YmwgJT4lCiAgICBkcGx5cjo6c2VsZWN0KHNhbXBsZSwgY29uZGl0aW9uKSAlPiUKICAgIHRpYmJsZTo6Y29sdW1uX3RvX3Jvd25hbWVzKHZhciA9ICdzYW1wbGUnKQoKZGlzdF9oZWF0bWFwID0gcGhlYXRtYXAoCiAgICBtYXQgPSBkaXN0X21hdCwKICAgIGNvbG9yID0gY29sb3JzLAogICAgY2x1c3Rlcl9yb3dzID0gVFJVRSwKICAgIGNsdXN0ZXJfY29scyA9IFRSVUUsCiAgICBzaG93X3Jvd25hbWVzID0gVFJVRSwKICAgIHNob3dfY29sbmFtZXMgPSBUUlVFLAogICAgYW5ub3RhdGlvbl9jb2wgPSBhbm5vdGF0aW9uX3RibAopCmRpc3RfaGVhdG1hcApgYGAKCkFuZCBub3cgd2UgaGF2ZSBvdXIgY29uZGl0aW9ucyBhcyBhIGNvbG9yZWQgYW5ub3RhdGlvbiBiYXIgYWxvbmcgdGhlIGNvbHVtbnMuCgojIFNvdXJjZXMKCiogSEJDIFFDIHR1dG9yaWFsOiBodHRwczovL2hiY3RyYWluaW5nLmdpdGh1Yi5pby9ER0Vfd29ya3Nob3AvbGVzc29ucy8wM19ER0VfUUNfYW5hbHlzaXMuaHRtbAoqIERldGFpbGVkIEhlYXRtYXAgdHV0b3JpYWwgZnJvbSBHYWxheHk6IGh0dHBzOi8vdHJhaW5pbmcuZ2FsYXh5cHJvamVjdC5vcmcvdHJhaW5pbmctbWF0ZXJpYWwvdG9waWNzL3RyYW5zY3JpcHRvbWljcy90dXRvcmlhbHMvcm5hLXNlcS12aXotd2l0aC1oZWF0bWFwMi90dXRvcmlhbC5odG1sCgotLS0KCiMgU2Vzc2lvbiBJbmZvCmBgYHtyIFNlc3Npb25JbmZvfQpzZXNzaW9uSW5mbygpCmBgYAoKLS0tCgpUaGVzZSBtYXRlcmlhbHMgaGF2ZSBiZWVuIGFkYXB0ZWQgYW5kIGV4dGVuZGVkIGZyb20gbWF0ZXJpYWxzIGxpc3RlZCBhYm92ZS4gVGhlc2UgYXJlIG9wZW4gYWNjZXNzIG1hdGVyaWFscyBkaXN0cmlidXRlZCB1bmRlciB0aGUgdGVybXMgb2YgdGhlIFtDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uIGxpY2Vuc2UgKENDIEJZIDQuMCldKGh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LzQuMC8pLCB3aGljaCBwZXJtaXRzIHVucmVzdHJpY3RlZCB1c2UsIGRpc3RyaWJ1dGlvbiwgYW5kIHJlcHJvZHVjdGlvbiBpbiBhbnkgbWVkaXVtLCBwcm92aWRlZCB0aGUgb3JpZ2luYWwgYXV0aG9yIGFuZCBzb3VyY2UgYXJlIGNyZWRpdGVkLgo=