#+title: Personal Genome Project: Hackathon 1.0 example projects
#+author: Brad Chapman \newline Bioinformatics Core, Harvard Chan School \newline http://bit.ly/pgp-resources-1 \newline http://bit.ly/pgp-analysis
#+date: 21 July 2018

#+OPTIONS: toc:nil H:2

#+startup: beamer
#+LaTeX_CLASS: beamer
#+latex_header: \usepackage{url}
#+latex_header: \usepackage{hyperref}
#+latex_header: \hypersetup{colorlinks=true}
#+BEAMER_THEME: default
#+BEAMER_COLOR_THEME: seahorse
#+BEAMER_INNER_THEME: rectangles

* Introduction

** What we'll do

\LARGE
- Understand the types of data in the personal genome project
- Learn how to query and find a genome to analyze
- Analyses you can do with a single genome

** Types of analyses

\LARGE
- Looking at small variants for traits
- HLA typing: the adaptive immune system
- Structural variants: larger events

** Resources
\LARGE
- PGP data
- Compute on AWS http://bit.ly/pgp-resources-1
- Example analyses  http://bit.ly/pgp-analysis
- Mentors and participants
- Your ideas

** Outline
\Large
- \textbf{Overview of the Personal Genome Project and Data}
- Identify participants of interest
- Overview of human variant data analysis
- Example of looking at small variant data: ApoE
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

* PGP overview

** Personal Genome Project

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/pgp.png]]

http://www.personalgenomes.org/us

** Whole genome sequencing data plus metadata

#+ATTR_LATEX: :width 0.6\textwidth
[[./images12/pgp_huD57BBF.png]]

https://my.pgp-hms.org/profile/huD57BBF

** Rich set of associated data

#+ATTR_LATEX: :width 0.7\textwidth
[[./images12/openhumans.png]]

https://www.openhumans.org/member/jameslvick/

** Collections of data in PGP

\Large
- Processed data: variants
   - \Large Per participant portable ready to use VCFs
- Raw data: reads
   - \Large Per participant BAM files of reads

http://bit.ly/pgp-analysis

\vspace{1cm}

\scriptsize
https://github.com/bcbio/bcbio_validation_workflows/tree/master/pgp#find-bam-and-vcf-files-in-arvados-collections

* Finding a participant of interest

** Outline
\Large
- Overview of the Personal Genome Project and Data
- \textbf{Identify participants of interest}
- Overview of human variant data analysis
- Example of looking at small variant data: ApoE
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

** Find a participant of interest

- \Large Untap SQL database: \newline \small https://github.com/abeconnelly/untap
- \Large Participants plus associated metadata
- \Large Regularly updated with new participants

\scriptsize
https://collections.su92l.arvadosapi.com/c=2210f7ee07fc1c8b926e5db28eff9635-3284/_/html/index.html?disposition=inline

** Query for participant

\Large
- Example query and selection of participant \newline
http://bit.ly/pgp-analysis

\vspace{0.1cm}

- huD57BBF \normalsize \newline https://my.pgp-hms.org/profile/huD57BBF

** Command line

\Large
#+BEGIN_SRC sh
$ cd /mnt/work/pgp/examples
$ bcbiovm_python \
    scripts/extract_veritas_pgp.py \
    untap.db

huD57BBF 53Gb No demographics
 [u'Family Tree DNA', u'Veritas Genetics',
  u'23andMe']
#+END_SRC

** Query, SQL to python pandas dataframe

#+BEGIN_SRC python
query = ("SELECT uploaded_data.human_id, date, name "
         "FROM uploaded_data WHERE "
         "data_type == 'Veritas Genetics' AND "
         "uploaded_data.name GLOB '*VCF'")
conn = sqlite3.connect(sys.argv[1])
df = pd.read_sql_query(query, conn)
#+END_SRC

http://bit.ly/pgp-analysis

** Other example queries

\Large
- Jupyter notebooks
- Summarize Age, Bloodtype, Ethnicity, Gender

\normalsize
https://github.com/swzCuroverse/PGPGraphics

** Outline
\Large
- Overview of the Personal Genome Project and Data
- Identify participants of interest
- \textbf{Overview of human variant data analysis}
- Example of looking at small variant data: ApoE
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

* Variant calling overview
** Human whole genome sequencing

[[./images5/human_genome.png]]

\footnotesize
http://ensembl.org/Homo_sapiens/Location/Genome
\normalsize

** High throughput sequencing

[[./images5/reads.png]]

** Variant calling

[[./images5/SNV_calling.png]]

\footnotesize
http://en.wikipedia.org/wiki/SNV_calling_from_NGS_data
\normalsize

* Calling methods

** SNPs and Indels

[[./images8/mutations.png]]

http://carolguze.com/text/442-2-mutations.shtml

** Genome Analysis Toolkit (GATK)

[[./images8/gatk.png]]

\vspace{0.5cm}

https://www.broadinstitute.org/gatk/

** GATK Best Practices

#+BEGIN_CENTER
#+ATTR_LATEX: :width 1.0\textwidth
[[./images11/gatk_bp.png]]
#+END_CENTER

\scriptsize
https://software.broadinstitute.org/gatk/best-practices/

** HaplotypeCaller

[[./images8/gatk-hc.png]]

\tiny
http://gatkforums.broadinstitute.org/discussion/5464/workshop-presentations-2015-uk-4-20-24

* Post-calling annotation and analysis
** Effects prediction

#+ATTR_LATEX: :width 1.1\textwidth
[[./images8/vep-consequences.jpg]]

\scriptsize
http://www.ensembl.org/info/genome/variation/predicted_data.html

** Annotation and analysis -- GEMINI

[[./images9/gemini_overview.png]]

\small
https://github.com/arq5x/gemini
\normalsize

* Understanding outputs

** VCF -- overview

#+ATTR_LATEX: :width 1.05\textwidth
[[./images8/vcf-overview.png]]

http://vcftools.sourceforge.net/VCF-poster.pdf

** VCF -- representations

[[./images8/vcf-representation.png]]

http://vcftools.sourceforge.net/VCF-poster.pdf

** Learning to read VCFs

\Large
- Step by step guide from Broad
\small
https://www.broadinstitute.org/gatk/guide/article?id=1268

\vspace{0.5cm}
\Large
- Specification
\small
http://samtools.github.io/hts-specs/


** Outline
\Large
- Overview of the Personal Genome Project and Data
- Identify participants of interest
- Overview of human variant data analysis
- \textbf{Example of looking at small variant data: ApoE}
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

** Examine existing variation files

\Large
- Portable VCFs with small variant data
- Hosted as data collection with standard wget retrieval
- Also downloaded on work machines for PGP event: /mnt/work/pgp/vcf

\vspace{0.5cm}
\tiny
https://workbench.su92l.arvadosapi.com/collections/su92l-4zz18-2rwb81xy8f1eh42

* ApoE status

** 

\Large
- ApoE \small https://www.snpedia.com/index.php/APOE
- \Large Two variants, on chromosome 19, that impact risk of Alzheimer's disease and
  cholesterol metabolism

#+ATTR_LATEX: :width 0.7\textwidth
[[./images12/apoe.png]]


** ApoE analysis

#+BEGIN_SRC sh
$ tabix huD57BBF-gatk-haplotype.vcf.gz
    chr19:44908684-44908684
chr19   44908684        rs429358        T       C
1116.80    PASS
ANN=C|missense_variant|MODERATE|APOE|c.388T>C|p.Cys130Arg
GT:AD:DP:GQ:MMQ:PL      1/1:0,26:26:78:60:1145,78,0
$ tabix huD57BBF-gatk-haplotype.vcf.gz
     chr19:44908822-44908822
#+END_SRC

http://bit.ly/pgp-analysis

* Read analysis

** Outline
\Large
- Overview of the Personal Genome Project and Data
- Identify participants of interest
- Overview of human variant data analysis
- Example of looking at small variant data: ApoE
- \textbf{Additional analyses with BAM reads}:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

** Performing additional analyses

\Large
- Raw files of reads in BAM format
- Also hosted as data collection by participant
- Demonstrate using open platforms for performing additional data analyses

\vspace{0.5cm}
\tiny
https://workbench.su92l.arvadosapi.com/collections/su92l-4zz18-1rqqi0kpkfmfite

* HLA

** Major histocompatibility complex (MHC) -- HLA

\Large
- human leukocyte antigen (HLA)
- Apative immune system
- Cell surface display and recognition
- Organ transplants, Cancer immunotherapy

\vspace{0.5cm}

\normalsize
https://en.wikipedia.org/wiki/Human_leukocyte_antigen
** HLA -- complex and repetitive

[[./images8/MHC.png]]

\small
\vspace{1cm}
http://www.ebi.ac.uk/ipd/imgt/hla/ \\
\scriptsize
http://sciscogenetics.com/technology/human-leukocyte-antigen-complex/

** HLA typing

\Large
- 1000 genomes: build 38 + IMGT/HLA-3.18.0
- bwa mem extracts HLA reads
- Map reads only to HLA sequences
- OptiType: Call HLA types

\vspace{0.5cm}
\footnotesize
https://github.com/lh3/bwa/blob/master/README-alt.md\#hla-typing \newline
https://github.com/FRED-2/OptiType


** HLA outputs

\Large
#+BEGIN_SRC sh
HLA-A*11:01;HLA-A*24:02
HLA-B*27:05;HLA-B*55:01
HLA-C*07:02;HLA-C*07:02
#+END_SRC

* Structural variation
** Structural variations

#+BEGIN_CENTER
#+ATTR_LATEX: :width .55\textwidth
[[./images8/svs.jpg]]
#+END_CENTER

\footnotesize
http://www.nature.com/nmeth/journal/v9/n2/full/nmeth.1858.html

** Tools used

- \Large Manta: \footnotesize https://github.com/Illumina/manta \\
   \Large Split and paired end reads
- \Large Lumpy: \footnotesize https://github.com/arq5x/lumpy-sv \\
   \Large Split and paired ends reads
- \Large CNVkit: \footnotesize https://github.com/etal/cnvkit \\
   \Large Read depth based

** Example deletion call -- 3 callers

#+BEGIN_SRC sh
chr19   50827242        MantaDEL:67020:0:1:0:0:0
T   <DEL>   658.0 PASS
END=50830636;SVTYPE=DEL;SVLEN=-3394;
ANN=<DEL>|bidirectional_gene_fusion|HIGH|AC011523.2&KLK15|
ENSG00000267968&ENSG00000174562|gene_variant|
GT:FT:GQ:PL:PR:SR       0/1:PASS:504:708,0,501:18,16:23,12
#+END_SRC

** Viewing deletion -- SV-plaudit

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/huD57BBF-KLK15-DEL-plaudit.png]]

https://github.com/jbelyeu/SV-plaudit

** Viewing deletion -- svviz

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/hud57bbf-klk15-del.png]]

http://svviz.readthedocs.io

** Genomic region with deletion -- KLK15

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/chr19_klk15.png]]

http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38

** KLK15 known function

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/klk15_wikipedia.png]]

https://en.wikipedia.org/wiki/KLK15

** Tissue specific gene expression

#+ATTR_LATEX: :width 0.7\textwidth
[[./images12/klk15_expression.png]]

\small
https://www.gtexportal.org/home/gene/ENSG00000174562.9

** Self reported conditions

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/pgp_huD57BBF_conditions.png]]

https://my.pgp-hms.org/profile/huD57BBF


* bcbio motivation

** Outline
\Large
- Overview of the Personal Genome Project and Data
- Identify participants of interest
- Overview of human variant data analysis
- Example of looking at small variant data: ApoE
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- \textbf{Platform for data analysis: CWL, Arvados, bcbio}

** Open source community analysis

#+ATTR_LATEX: :width 1.0\textwidth
[[./images3/bcbio_nextgen_highlevel.png]]

\vspace{1cm}
https://github.com/bcbio/bcbio-nextgen

** Supported analysis types

#+BEGIN_CENTER
#+ATTR_LATEX: :width 0.4\textwidth
[[./images9/bcbio_pipelines.png]]
#+END_CENTER

\scriptsize
https://bcbio-nextgen.readthedocs.org/en/latest/contents/pipelines.html

** Community: sustainability and support

#+ATTR_LATEX: :width 0.9\textwidth
[[./images12/bcbio_contrib_apr2018.png]]

\vspace{0.5cm}

#+ATTR_LATEX: :width 0.9\textwidth
[[./images12/bcbio_issues_apr2018.png]]

\vspace{0.5cm}

[[https://github.com/bcbio/bcbio-nextgen]]


#+ATTR_LATEX: :width 0.9\textwidth
[[./images11/bcbio_commits_apr2017.png]]

\vspace{0.5cm}

#+ATTR_LATEX: :width 0.9\textwidth
[[./images11/bcbio_issues_apr2017.png]]

\vspace{0.5cm}

[[https://github.com/bcbio/bcbio-nextgen]]

* Interoperable infrastructure

** Infrastructure Goals
\Large
- Local machines
- Clusters: SLURM, SGE, Torque, PBS, LSF
- Clouds: Amazon, Google, Azure
- Clinical environments
- User interface for researchers
- Integrate with LIMS
- Accessible to the general public

** Better abstractions = more interoperability

[[./images10/abstractions.png]]

\scriptsize
https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html

** Common Workflow Language (CWL)

#+ATTR_LATEX: :width 1.0\textwidth
[[./images10/cwl_pipeline_example.png]]

#+BEGIN_CENTER
http://www.commonwl.org/ \\
\vspace{0.5cm}
\scriptsize
https://f1000research.com/slides/5-1617
#+END_CENTER

** Arvados + Curoverse + Veritas

#+ATTR_LATEX: :width 1.0\textwidth
[[./images12/arvados.png]]

https://arvados.org/

** Why use a workflow abstraction?

\Large
- Integrate with multiple platforms
   - \Large Arvados -- AWS, Azure
   - \Large Cromwell -- HPC, local, GCP
   - \Large Toil -- HPC, local
   - \Large DNAnexus -- AWS, Azure
   - \Large Seven Bridges + Cancer Genomics Cloud
- Stop maintaining bcbio specific infrastructure
- Focus on hard biological problems

** CWL in bcbio

\Large
- Start with high level configuration file
- Generate CWL
- Run, on any infrastructure that supports CWL
   - \Large Generated CWL
   - \Large Docker or local bcbio installation
   - \Large Genome data

\scriptsize
https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html

** bcbio-vm: CWL wrapper

\Large
- bcbio-like interface integrating with external tools
- Install wrapper plus supported runners

\vspace{0.2cm}
\normalsize
#+BEGIN_SRC sh
conda install -c conda-forge -c bioconda bcbio-nextgen-vm
#+END_SRC

\vspace{0.4cm}
https://github.com/bcbio/bcbio-nextgen-vm \\
https://bioconda.github.io/

** Describe your analysis

#+BEGIN_SRC yaml
  - files: huD57BBF.bam
    description: huD57BBF
    analysis: variant
    genome_build: hg38
    algorithm:
      aligner: bwa
      variantcaller: gatk-haplotype
      svcaller: [manta, lumpy, cnvkit]
      hlacaller: optitype
#+END_SRC

https://github.com/bcbio/bcbio_validation_workflows

** Describe the platform resources

#+BEGIN_SRC yaml
arvados:
  reference: su92l-4zz18-3p00f79y4p535ia
  input: [su92l-4zz18-ihm3wrgyuwcmsx1]
resources:
  default: {cores: 16, memory: 3500M,
            jvm_opts: [-Xms1g, -Xmx3500m]}
#+END_SRC

** Build Common Workflow Language description

#+BEGIN_SRC sh
bcbio_vm.py cwl --systemconfig bcbio_system-arvados.yaml \
  pgp_sv_hla.yaml
#+END_SRC

** Launch analysis

#+BEGIN_SRC sh
bcbio_vm.py cwlrun arvados pgp_sv_hla-workflow -- \
  --project-uuid su92l-j7d0g-eoibug3nrwg8ysj
#+END_SRC

\scriptsize
https://workbench.su92l.arvadosapi.com/projects/su92l-j7d0g-eoibug3nrwg8ysj

** Arvados pipeline run

#+ATTR_LATEX: :width 0.7\textwidth
[[./images12/arvados_pgp.png]]

\scriptsize
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-iprauko4kegv1kz

* Summary

** Summary

\Large
- Overview of the Personal Genome Project and Data
- Identify participants of interest
- Overview of human variant data analysis
- Example of looking at small variant data: ApoE
- Additional analyses with BAM reads:
   - \Large HLA typing
   - \Large Structural variant analysis
- Platform for data analysis: CWL, Arvados, bcbio

** Next steps

\Large
- Work through examples to get started
- Propose your own projects building off these ideas
- Brainstorm new research ideas from PGP data
- Help us improve data access and organization
- Improve documentation and resources

** Resources
\LARGE
- PGP data
- Compute on AWS http://bit.ly/pgp-resources-1
- Example analyses  http://bit.ly/pgp-analysis
- Mentors and participants
- Your ideas
