Streamlining large-scale genomic data management: Insights from the UK Biobank whole-genome sequencing data

Xihao Li; Andrew R Wood; Yuxin Yuan; Manrui Zhang; Yushu Huang; Gareth Hawkes; Robin N Beaumont; Michael N Weedon; Wenyuan Li; Xiaoyu Li; Xihong Lin; Zilin Li

doi:10.1016/j.xgen.2025.101009

Streamlining large-scale genomic data management: Insights from the UK Biobank whole-genome sequencing data

Cell Genom. 2025 Dec 10;5(12):101009. doi: 10.1016/j.xgen.2025.101009. Epub 2025 Sep 18.

Authors

Xihao Li¹, Andrew R Wood², Yuxin Yuan³, Manrui Zhang⁴, Yushu Huang⁵, Gareth Hawkes⁶, Robin N Beaumont⁶, Michael N Weedon⁶, Wenyuan Li⁵, Xiaoyu Li⁴, Xihong Lin⁷, Zilin Li⁸

Affiliations

¹ Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA; Department of Genetics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. Electronic address: xihaoli@unc.edu.
² Department of Biomedical and Clinical Sciences, Faculty of Health and Life Sciences, University of Exeter, Exeter, UK. Electronic address: a.r.wood@exeter.ac.uk.
³ School of Mathematics and Statistics and KLAS, Northeast Normal University, Changchun, Jilin, China.
⁴ Department of Sociology, Tsinghua University, Beijing, China.
⁵ Department of Big Data in Health Science, School of Public Health and Center of Clinical Big Data and Analytics of the Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China.
⁶ Department of Biomedical and Clinical Sciences, Faculty of Health and Life Sciences, University of Exeter, Exeter, UK.
⁷ Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA; Department of Statistics, Harvard University, Cambridge, MA, USA. Electronic address: xlin@hsph.harvard.edu.
⁸ School of Mathematics and Statistics and KLAS, Northeast Normal University, Changchun, Jilin, China. Electronic address: lizl@nenu.edu.cn.

Abstract

Biobank-scale whole-genome sequencing (WGS) studies are increasingly pivotal in unraveling the genetic bases of diverse health outcomes. However, managing and analyzing these datasets' sheer volume and complexity presents significant challenges. We highlight the annotated genomic data structure (aGDS) format, substantially reducing the WGS data file size while enabling seamless integration of genomic and functional information for comprehensive WGS analyses. The aGDS format yielded 23 chromosome-specific files for the UK Biobank 500k WGS dataset, occupying only 1.10 tebibytes of storage. We develop the vcf2agds toolkit that streamlines the conversion of WGS data from VCF to aGDS format. Additionally, the STAARpipeline equipped with the aGDS files enabled scalable, comprehensive, and functionally informed WGS analysis, facilitating the detection of common and rare coding and noncoding phenotype-genotype associations. Overall, the vcf2agds toolkit and STAARpipeline provide a streamlined solution that facilitates efficient data management and analysis of biobank-scale WGS data across hundreds of thousands of samples.

Keywords: annotated genomic data structure; big data management; cloud computing; functional annotations; functionally informed association analyses; vcf2agds toolkit; whole-genome sequencing.

MeSH terms

Biological Specimen Banks*
Data Management* / methods
Databases, Genetic
Genome, Human*
Genomics* / methods
Humans
Software
UK Biobank
United Kingdom
Whole Genome Sequencing* / methods