#+DATE: [2014-12-19 Fri 10:17]
#+BLOG: bcbio
#+POSTID: 702
#+TITLE: Benchmarking variation and RNA-seq analyses on Amazon Web Services with Docker
#+CATEGORY: benchmarking
#+TAGS: bioinformatics, variant, rna-seq, ngs, validation, benchmarking, collectl, docker, aws
#+OPTIONS: toc:nil num:nil

* Overview

We developed a freely available, easy to run implementation of [[bcbio][bcbio-nextgen]] on
[[aws][Amazon Web Services (AWS)]] using [[docker][Docker]]. bcbio is a community developed tool
providing validated and scalable variant calling and RNA-seq analysis. The AWS
implementation automates all of the steps of building a cluster, attaching high
performance shared filesystems, and running an analysis. This makes bcbio
readily available to the research community without the need to install and
configure a local installation.

The entire installation bootstraps from standard Linux AMIs, enabling
adjustment of the tools, genome data and installation without needing to
prepare custom AMIs. The implementation uses [[elasticluster][Elasticluster]] to provision and
configure the cluster. We automate the process with
[[boto][the boto Python interface to AWS]] and [[ansible][Ansible scripts]]. [[bcbiovm][bcbio-vm]] isolates
code and tools inside a [[docker][Docker]] container allowing runs on any remote machine
with a download of the Docker image and access to the shared filesystem.
Analyses run directly from S3 buckets, with automatic streaming download
of input data and upload of final processed data.

We provide timing benchmarks for running a full variant calling analysis using
[[bcbio][bcbio]] on AWS. The benchmark dataset was [[dream_about][a cancer tumor/normal evaluation]], from
[[dream][the ICGC-TCGA DREAM challenge]], with 100x coverage in exome regions. We compared
the results of running this dataset on 2 different networked filesystems:
Lustre and NFS. We also show benchmarks for an RNA-seq dataset
using inputs from [[seqc_paper][the Sequencing Quality Control (SEQC) project]].

#+LINK: bcbio http://github.com/chapmanb/bcbio-nextgen
#+LINK: aws http://aws.amazon.com/
#+LINK: docker https://docker.com/
#+LINK: dream https://www.synapse.org/#!Synapse:syn312572
#+LINK: dream_about https://bcbio-nextgen.readthedocs.org/en/latest/contents/testing.html#cancer-tumor-normal
#+LINK: boto http://boto.readthedocs.org/en/latest/
#+LINK: bcbiovm https://github.com/chapmanb/bcbio-nextgen-vm

We developed bcbio on AWS and ran these timing benchmarks thanks to work with
great partners. A collaboration with [[biogen][Biogen]] and [[http://www.massgeneral.org/research/researchlab.aspx?id=1402][Rudy Tanzi's group at MGH]] funded
the development of bcbio on AWS. A second collaboration with
[[http://www.intel.com/content/www/us/en/healthcare-it/healthcare-overview.html][Intel Health and Life Sciences]]
and [[az][AstraZenenca]] funded the somatic variant calling
benchmarking work. We're thankful for all the relationships that make this work
possible:

- John Morrissey automated the process of starting a bcbio cluster on AWS and
  attaching a Lustre filesystem. He also automated the approach to generating
  graphs of resource usage from collectl stats and provided critical front line
  testing and improvements to all the components of the bcbio AWS
  interaction.

- Kristina Kermanshahche and Robert Read at [[http://www.intel.com/content/www/us/en/healthcare-it/healthcare-overview.html][Intel]] provided great support
  helping us get the [[icel][Lustre ICEL CloudFormation]] templates running.

- Ronen Artzi, Michael Heimlich, and Justin Johnson at [[az][AstraZenenca]] setup
  Lustre, Gluster and NFS benchmarks using a bcbio [[starcluster][StarCluster]] instance. This
  initial validation was essential for convincing us of the value of moving to a
  shared filesystem on AWS.

- Jason Tetrault, Karl Gutwin and Hank Wu at [[biogen][Biogen]] provided valuable feedback,
  suggestions and resources for developing bcbio on AWS.

- Glen Otero parsed the collectl data and provided graphs, which gave us a
  detailed look into the potential causes of bottlenecks we found in the
  timings.

- James Cuff, Paul Edmon and the team at [[fas][Harvard FAS research computing]]
  built and administered the Regal Lustre setup used for local testing.

- John Kern and other members of the bcbio community tested, debugged and helped
  identify issues with the implementation. Community feedback and contributions are
  essential to bcbio development.

#+LINK: icel https://wiki.hpdd.intel.com/display/PUB/Intel+Cloud+Edition+for+Lustre*+Software
#+LINK: fas https://rc.fas.harvard.edu/
#+LINK: az http://www.astrazeneca.com
#+LINK: biogen http://www.biogenidec.com/

* Architecture

The implementation provides both a practical way to run large scale variant
calling and RNA-seq analysis, as well as a flexible backend architecture
suitable for production quality runs. This
writeup might feel a bit like a [[blacktriangle][black triangle moment]] since I also wrote about
[[bcbiocloudman][running bcbio on AWS three years ago]]. That implementation was a demonstration
for small scale usage rather than a production ready system. We now have a setup we can
support and run on [[bcbioscaling][large scale projects]] thanks to numerous changes in
the backend architecture:

- Amazon, and cloud based providers in general, now provide high end filesystems
  and networking. Our AWS runs are fast because they use SSD backend storage, fast
  networking connectivity and high end processors that would be difficult to
  invest in for a local cluster. Renting these is economically feasible now
  that we have an approach to provision resources, run the analysis, and tear
  everything down. The dichotomy between local cluster hardware and cloud
  hardware will continue to expand with upcoming improvements in
  [[awsc4][compute (Haswell processors)]] and [[aws16tb][storage (16Tb EBS SSD volumes]]).

- Isolating all of the software and code inside [[docker][Docker]] containers enables rapid
  deployment of fixes and improvements. From an open source support perspective,
  Amazon provides a consistent cluster environment we have full control
  over, limiting the space of potential system specific issues. From a
  researcher's perspective, this will allow use of bcbio without needing to
  spend time installing and testing locally.

- The setup runs from standard Linux base images using [[ansible][Ansible scripts]] and
  [[elasticluster][Elasticluster]]. This means we no longer need to build and update
  AMIs for changes in the architecture or code. This simplifies testing and
  pushing fixes, letting us spend less time on support and more on
  development. Since we avoid having a pre-built AMI, the process of
  building and running bcbio on AWS is fully auditable for both security and
  scientific accuracy. Finally, it provides a path to support bcbio on container
  specific management services like [[awsecs][Amazon's EC2 container service]].

- All long term data storage happens in [[awss3][Amazon's S3 object store]], including both
  analysis specific data as well as general reference genome data. Downloading
  reference data for an analysis on demand removes the requirement to maintain
  large shared EBS volumes. On the analysis side, you maintain only the input
  files and high value output files in S3, removing the intermediates upon
  completion of the analysis. Removing the need to manage storage of
  EBS volumes also provides a cost savings ([[http://aws.amazon.com/s3/pricing/][$0.03/Gb/month for S3]] versus
  [[http://aws.amazon.com/ebs/pricing/][$0.10+/Gb/month for EBS]]) and allows the option of archiving in [[https://aws.amazon.com/glacier/][Glacier]] for
  long term storage.

All of these architectural changes provide a setup that is easier to maintain
and scale over time. Our goal moving ahead is to provide a researcher friendly
interface to setting up and running analyses. We hope to achieve that through
the in-development [[cwl][Common Workflow Language]] from [[http://galaxyproject.org/][Galaxy]], [[arvados][Arvados]], [[sevenbridges][Seven Bridges]],
[[taverna][Taverna]] and the [[openbio][open bioinformatics community]].

#+LINK: blacktriangle https://web.archive.org/web/20131122230658/http://rampantgames.com/blog/2004/10/black-triangle.html
#+LINK: slurm http://slurm.schedmd.com
#+LINK: bcbiocloudman https://bcbio.wordpress.com/2011/11/29/making-next-generation-sequencing-analysis-pipelines-easier-with-biocloudcentral-and-galaxy-integration/
#+LINK: bcbioscaling https://bcbio.wordpress.com/2013/05/22/scaling-variant-detection-pipelines-for-whole-genome-sequencing-analysis/
#+LINK: awsc4 http://aws.amazon.com/blogs/aws/new-c4-instances/
#+LINK: aws16tb http://www.infoq.com/news/2014/11/new-features-ec2-ebs-s3
#+LINK: awsecs http://aws.amazon.com/ecs/
#+LINK: awss3 http://aws.amazon.com/s3/
#+LINK: cwl https://github.com/rabix/common-workflow-language
#+LINK: arvados https://arvados.org/
#+LINK: sevenbridges https://www.sbgenomics.com/
#+LINK: openbio http://www.open-bio.org/wiki/Main_Page
#+LINK: taverna http://www.taverna.org.uk/

* Variant calling -- benchmarking AWS versus local

We benchmarked somatic variant calling in two environments: on the
elasticluster Docker AWS implementation and on local [[fas][Harvard FAS]] machines.

- AWS processing was twice as fast as a local run. The gains occur in disk IO intensive
  steps like alignment post-processing. AWS offers the opportunity to rent SSD backed
  storage and obtain a 10GigE connected cluster without contention for network
  resources. Our local test machines have an in-production Lustre filesystem attached to
  a large highly utilized cluster provided by [[fas][Harvard FAS research computing]].

- At this scale Lustre and NFS have similar throughput, with Lustre
  outperforming NFS during IO intensive steps like alignment, post-processing
  and large BAM file merging. From [[bcbioscaling][previous benchmarking work]] we'll need to
  process additional samples in parallel to fully stress the shared filesystem and
  differentiate Lustre versus NFS performance. However, the resource plots at this
  scale show potential future bottlenecks during alignment,
  post-processing and other IO intensive steps. Generally, having Lustre scaled
  across 4 [[lun][LUNs]] per object storage server (OSS) enables better
  distribution of disk and network resources.

AWS runs use two c3.8xlarge instances clustered in a single [[awspg][placement group]],
providing 32 cores and 60Gb of memory per machine. Our local run was comparable
with two compute machines, each with 32 cores and 128Gb of memory, connected to
a Lustre filesystem. The benchmark is [[dream_about][a cancer tumor/normal evaluation]] consisting of alignment, recalibration,
realignment and variant detection with four different callers. The input is
a tumor/normal pair from the [[dream][the ICGC-TCGA DREAM challenge]] with 100x
coverage in exome regions. Here are the times, in hours, for each benchmark:

#+LINK: awspg http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html
#+LINK: lun http://en.wikipedia.org/wiki/Logical_unit_number

|                           | AWS (Lustre) | AWS (NFS) | Local (Lustre) |
|---------------------------+--------------+-----------+----------------|
| Total                     |         4:42 |      5:05 |          10:30 |
|---------------------------+--------------+-----------+----------------|
| genome data preparation   |         0:04 |      0:10 |                |
| alignment preparation     |         0:12 |      0:15 |                |
| alignment                 |         0:29 |      0:52 |           0:53 |
| callable regions          |         0:44 |      0:44 |           1:25 |
| alignment post-processing |         0:13 |      0:21 |           4:36 |
| variant calling           |         2:35 |      2:05 |           2:36 |
| variant post-processing   |         0:05 |      0:03 |           0:22 |
| prepped BAM merging       |         0:03 |      0:18 |           0:06 |
| validation                |         0:05 |      0:05 |           0:09 |
| population database       |         0:06 |      0:07 |           0:09 |
#+begin_html
 <br />
#+end_html

To provide more insight into the timing differences between these benchmarks, we
automated collection and plotting of resource usage on AWS runs.

* Variant calling -- resource usage plots

bcbio retrieves [[collectl][collectl]] usage statistics from the server [[bcbiodoc_graph][and prepares graphs]] of
CPU, memory, disk and network usage. These plots allow in-depth insight into
limiting factors during individual steps in the workflow.

We'll highlight some interesting comparisons between NFS and Lustre during the
variant calling benchmarking. During this benchmark, the two critical resources
were CPU usage and disk IO on the shared filesystems.  We also measure memory
usage but that was not a limiting factor with these analyses. In addition to the
comparisons highlighted below, we have the full set of resource usage graphs
available for each run:

- [[nfsstats][Variant calling with NFS on AWS]]
- [[lustre_stats][Variant calling with Lustre on AWS]]
- [[rnaseq_stats][RNA-seq on a single machine on AWS]]

#+LINK: bcbiodoc_graph https://bcbio-nextgen.readthedocs.org/en/latest/contents/cloud.html#graphing-resource-usage
#+LINK: collectl http://collectl.sourceforge.net/
#+LINK: nfsstats http://imgur.com/a/AZjuC
#+LINK: lustre_stats http://imgur.com/a/HfrqY
#+LINK: rnaseq_stats http://imgur.com/a/LSDFz

** CPU

These plots compare CPU usage during processing steps for Lustre and NFS. The largest
differences between the two runs are in the alignment, alignment post-processing
and variant calling steps:

*** NFS

#+BEGIN_HTML
<a href="http://i.imgur.com/iUpvyHx.png">
  <img src="http://i.imgur.com/iUpvyHx.png" width="700"
       alt="CPU resource usage for NFS during variant calling">
</a>
#+END_HTML

*** Lustre

#+BEGIN_HTML
<a href="http://i.imgur.com/59W8YvL.png">
  <img src="http://i.imgur.com/59W8YvL.png" width="700"
       alt="CPU resource usage for Lustre during variant calling">
</a>
#+END_HTML


For alignment and alignment post-processing the Lustre runs show more stable CPU
usage. NFS specifically spends more time in the CPU wait state (red line) during
IO intensive steps. On larger scale projects this may become a limiting factor
in processing throughput. The variant calling step was slower on Lustre than
NFS, with inconsistent CPU usage. We'll have to investigate this
slowdown further, since no other metrics point to an obvious bottleneck.

** Shared filesystem network usage and IO

These plots compare network usage during processing for Lustre and NFS. We use
this as a consistent proxy for the performance of the shared filesystem and disk
IO (the [[nfsstats][NFS plots]] do have directly measured disk IO for comparison purposes).

*** NFS

#+BEGIN_HTML
<a href="http://i.imgur.com/vvru0sv.png">
  <img src="http://i.imgur.com/vvru0sv.png" width="700"
       alt="Network resource usage NFS">
</a>
#+END_HTML

*** Lustre

#+BEGIN_HTML
<a href="http://i.imgur.com/onut3GI.png">
  <img src="http://i.imgur.com/onut3GI.png" width="700"
       alt="Network resource usage Lustre">
</a>
#+END_HTML

The biggest difference in the IO intensive steps is that Lustre network usage is smoother
compared to the spiky NFS input/output, due to spreading out read/writes over
multiple disks. Including more processes with additional read/writes will
help determine how these differences translate to scaling on larger numbers of
simultaneous samples.

* RNA-seq benchmarking

We also ran an RNA-seq analysis
using 4 samples from [[seqc_paper][the Sequencing Quality Control (SEQC) project]]. Each sample
has 15 million 100bp paired reads. bcbio handled trimming, alignment with [[star][STAR]],
and quantitation with [[dexseq][DEXSeq]] and [[cufflinks][Cufflinks]]. We ran on a single AWS c3.8xlarge
machines with 32 cores, 60Gb of memory, and attached SSD storage.

RNA-seq optimization in bcbio is at an earlier stage than variant calling.
We've done work to speed up trimming and aligning, but haven't yet optimized the
expression and count steps. The analysis runs quickly in 6 1/2 hours, but there
is still room for further optimization, and this is a nice example of how we can
use benchmarking plots to identify targets for additional work:

| Total                   | 6:25 |
|-------------------------+------|
| genome data preparation | 0:32 |
| adapter trimming        | 0:32 |
| alignment               | 0:24 |
| estimate expression     | 3:41 |
| quality control         | 1:16 |
#+begin_html
 <br />
#+end_html

The [[rnaseq_stats][RNA-seq collectl plots]] show the cause of the slower steps during expression
estimation and quality control. Here is CPU usage over the run:

#+BEGIN_HTML
<a href="http://i.imgur.com/D43c94L.png">
  <img src="http://i.imgur.com/D43c94L.png" width="700"
       alt="RNA-seq CPU usage">
</a>
#+END_HTML

The low CPU usage during the first 2 hours of expression estimation corresponds
to DEXSeq running serially over the 4 samples. In contrast with Cufflinks, which
parallelizes over all 32 cores, DEXSeq runs in a single core. We
could run these steps in parallel by using multiprocessing to launch the jobs, split
by sample. Similarly, the QC steps could benefit from parallel
processing. Alternatively, we're looking at validating other approaches for
doing quantification like [[express][eXpress]]. These are the type of benchmarking and
validation steps that are continually ongoing in the development of bcbio
pipelines.

#+LINK: dexseq http://www.bioconductor.org/packages/release/bioc/html/DEXSeq.html
#+LINK: express http://bio.math.berkeley.edu/eXpress/

* Reproducing the analysis

The process to launch the cluster and an NFS or optional Lustre shared
filesystem is [[awsdocs][fully automated and documented]]. It sets up permissions, VPCs,
clusters and shared filesystems from a basic AWS account, so requires minimal
manual work. ~bcbio_vm.py~ has commands to:

- Add an IAM user, a VPC and prepare the Elasticluster config.
- Launch a cluster and bootstrap with the latest bcbio code and data.
- Create and mount a Lustre filesystem attached to the cluster.
- Terminate the cluster and Lustre stack upon completion.

The processing handles download of input data from S3 and upload back to S3 on
finalization. We store data encrypted on S3 and manage access using
[[instance_profile][IAM instance profiles]]. We store in progress analysis data on an
[[aws_en][encrypted EBS volume]] that is NFS-mounted across the cluster.
The examples below show how to run both a somatic variant
calling evaluation and an RNA-seq evaluation.

#+LINK: starcluster http://star.mit.edu/cluster/index.html
#+LINK: elasticluster https://github.com/gc3-uzh-ch/elasticluster
#+LINK: ansible http://www.ansible.com/home
#+LINK: awsdocs https://bcbio-nextgen.readthedocs.org/en/latest/contents/cloud.html
#+LINK: instance_profile http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
#+LINK: aws_en http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

** Running the somatic variant calling evaluation

This analysis performs evaluation of variant calling using
[[dream_about][tumor/normal somatic sample from the DREAM challenge]].
To run, prepare an S3 bucket to run the analysis from. Copy the [[evalconfig][configuration file]]
to your own personal bucket and add a [[gatk][GATK]] jar. You can use the AWS console or
any available S3 client to do this. For example, using the [[awscli][AWS command line client]]:

#+BEGIN_SRC
aws s3 mb s3://YOURBUCKET-syn3-eval/
aws s3 cp s3://bcbio-syn3-eval/cancer-dream-syn3-aws.yaml s3://YOURBUCKET-syn3-eval/
aws s3 cp GenomeAnalysisTK.jar s3://YOURBUCKET-syn3-eval/jars/
#+END_SRC
#+BEGIN_HTML
<br/>
#+END_HTML

Now ssh to the cluster head node, create the work directory and use bcbio_vm to
create a batch script that we submit to SLURM. This example uses an attached
Lustre filesystem:

#+BEGIN_SRC
bcbio_vm.py elasticluster ssh bcbio
sudo mkdir -p /scratch/cancer-dream-syn3-exome
sudo chown ubuntu !$
cd !$ && mkdir work && cd work
bcbio_vm.py ipythonprep s3://YOURBUCKET-syn3-eval/cancer-dream-syn3-aws.yaml \
                        slurm cloud -n 60
sbatch bcbio_submit.sh
#+END_SRC
#+BEGIN_HTML
<br/>
#+END_HTML

This runs alignment and variant calling with multiple callers (MuTect,
FreeBayes, VarDict and VarScan), validates against the
[[dream][DREAM validation dataset truth calls]] and uploads the results back
to S3 in YOURBUCKET-syn3-eval/final.

#+LINK: evalconfig https://s3.amazonaws.com/bcbio-syn3-eval/cancer-dream-syn3-aws.yaml
#+LINK: awscli https://aws.amazon.com/cli/
#+LINK: gatk https://www.broadinstitute.org/gatk/

** Running the RNA-seq evaluation

This example runs an RNA-seq analysis using inputs from
[[seqc_paper][the Sequencing Quality Control (SEQC) project]]. Full details on the analysis are
available in the [[seqc_details][bcbio example run documentation]]. To setup the run, we copy the
input configuration from a publicly available S3 bucket into your own personal bucket:

#+BEGIN_SRC
aws s3 mb s3://YOURBUCKET-eval-rna-seqc/
aws s3 cp s3://bcbio-eval-rna-seqc/eval-rna-seqc.yaml s3://YOURBUCKET-eval-rnaseqc/
#+END_SRC
#+BEGIN_HTML
<br/>
#+END_HTML

Now ssh to the cluster head node, create the work directory and use bcbio_vm to
run on 32 cores using the attached EBS SSD filesystem:

#+BEGIN_SRC
bcbio_vm.py elasticluster ssh bcbio
mkdir -p /encrypted/eval-rna-seqc/work
cd /encrypted/eval-rna-seqc/work
bcbio_vm.py run s3://YOURBUCKET-eval-rna-seqc/eval-rna-seqc.yaml -n 32
#+END_SRC
#+BEGIN_HTML
<br/>
#+END_HTML

This will process three replicates from two different SEQC panels, performing
adapter trimming, alignment with [[star][STAR]] and produce counts, [[cufflinks][Cufflinks quantitation]]
and quality control metrics. The results upload back into your initial S3 bucket as
YOURBUCKET-eval-rna-seqc/final, and you can shut down the cluster used for processing.

#+LINK: seqc_paper http://www.nature.com/nbt/journal/v32/n9/full/nbt.2957.html
#+LINK: seqc_details https://bcbio-nextgen.readthedocs.org/en/latest/contents/testing.html#rnaseq-example
#+LINK: star https://github.com/alexdobin/STAR
#+LINK: cufflinks http://cufflinks.cbcb.umd.edu/

* Costs per hour

These are the instance costs, per hour, for running a 2 node 64 core cluster and
associated Lustre filesystem on AWS. For NFS runs, the compute costs are identical but
there will not be any additional instances for the shared filesystem.
Other run costs will include EBS volumes, but these are small ($0.13/Gb/month)
compared to the instance costs over these time periods. We use S3
for long term storage rather than the Lustre or NFS filesystems.

|                         | AWS type   | n | each  | total    |
|-------------------------+------------+---+-------+----------|
| compute entry node      | c3.large   | 1 | $0.11 |          |
| compute worker nodes    | c3.8xlarge | 2 | $1.68 |          |
|                         |            |   |       | $3.47/hr |
| ost (object data store) | c3.2xlarge | 4 | $0.42 |          |
| mdt (metadata target)   | c3.4xlarge | 1 | $0.84 |          |
| mgt (management target) | c3.xlarge  | 1 | $0.21 |          |
| NATDevice               | m3.medium  | 1 | $0.07 |          |
| Lustre licensing        |            | 1 | $0.48 |          |
|                         |            |   |       | $3.28/hr |
|-------------------------+------------+---+-------+----------|
|                         |            |   |       | $6.75/hr |

* Work to do

The bcbio AWS implementation is [[awsdocs][freely available and documented]] and we'll
continue to develop and support it. Some of the areas of immediate improvement
we hope to tackle in the future include:

- Supporting encryption at rest for the Lustre filesystem in continuing work to
  meet the [[aws_security][security requirements for storing genomic data on AWS]]. We currently
  encrypt data stored in S3 and at rest on the NFS-mounted ~/encrypted~ drive.

- Run directly on container-based parallel frameworks like Amazon's
  [[ec2_cont][EC2 container service]], which is also supported by the [[coreos_cont][CoreOS framework]].

- Add spot instance support using [[clusterk][Clusterk]] or Elasticluster.

We welcome feedback on the implementation, usage and benchmarking results.

#+LINK: aws_security http://d0.awsstatic.com/whitepapers/compliance/AWS_dBGaP_Genomics_on_AWS_Best_Practices.pdf
#+LINK: clusterk http://clusterk.com/
#+LINK: ec2_cont http://aws.amazon.com/blogs/aws/ec2-container-service-in-action/
#+LINK: coreos_cont https://coreos.com/docs/running-coreos/cloud-providers/ecs/
