cryoem-cloud-tools: A software platform to deploy and manage cryo-EM jobs in the cloud

Abstract

Access to streamlined computational resources remains a significant bottleneck for new users of cryo-electron microscopy (cryo-EM). To address this, we have developed tools that will submit cryo-EM analysis routines and atomic model building jobs directly to Amazon Web Services (AWS) from a local computer or laptop. These new software tools ("cryoem-cloud-tools") have incorporated optimal data movement, security, and cost-saving strategies, giving novice users access to complex cryo-EM data processing pipelines. Integrating these tools into the RELION processing pipeline and graphical user interface we determined a 2.2 Å structure of ß-galactosidase in ∼55 h on AWS. We implemented a similar strategy to submit Rosetta atomic model building and refinement to AWS. These software tools dramatically reduce the barrier for entry of new users to cloud computing for cryo-EM and are freely available at cryoem-tools.cloud.

Keywords: Amazon Web Services; Cloud computing; Cryo-electron microscopy.

Figure 1 -. AWS architecture for ‘advanced’ cryo-EM data processing with RELION.

Shown is a schematic of AWS resources deployed by cryoem-cloud-tools through the program ‘qsub_aws’. For all job types shown, the software places VMs within security groups that restrict access to the IP address of the end-user. Within a security group, the software determines the appropriate VM and storage choices, using S3 as a distribution point between local and AWS resources.

Figure 2 -. Performance of AWS vs. local GPU workstation.

Processing times (A) and FSC curve (B) for the determination of a 2.2 Å ß-galactosidase structure on AWS. (C) Processing times from the determination of 2.2 Å ß-galactosidase structure on GPU workstation (Kimanius et al. 2016). (D) Comparison of percent speed-up increases between AWS and a GPU workstation.

Figure 3 -. AWS architecture for ‘common’ cryo-EM data processing with RELION.

Shown is a schematic of AWS resources deployed by cryoem-cloud-tools through the program ‘qsub_aws’. For this common pipeline, there is no data storage on S3 and RELION jobs are run directly on p2 instances.

Figure 4 -. Rosetta atomic model refinement in the cloud.

(A) WS architecture for running Rosetta model refinement across multiple VMs. (B) Run time comparisons between a local workstation (16 cores) and AWS (252 vCPUs). (C) Representative region of the cryo-EM map with the top five atomic models built by Rosetta FastRelax (D) FSC curves between the best atomic model from FastRelax and the cryo-EM map of ß- galactosidase. The resolution corresponding to the FSC value of 0.5 for the full map is shown.