Indexing and searching petabase-scale nucleotide resources

Sergey A Shiryev; Richa Agarwala

doi:10.1038/s41592-024-02280-z

Indexing and searching petabase-scale nucleotide resources

Nat Methods. 2024 May 16. doi: 10.1038/s41592-024-02280-z. Online ahead of print.

Authors

Sergey A Shiryev¹, Richa Agarwala²

Affiliations

¹ Department of Health and Human Services, National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
² Department of Health and Human Services, National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA. agarwala@ncbi.nlm.nih.gov.

PMID: 38755321
DOI: 10.1038/s41592-024-02280-z

Abstract

Searching vast and rapidly growing nucleotide content in resources, such as runs in the Sequence Read Archive and assemblies for whole-genome shotgun sequencing projects in GenBank, is currently impractical for most researchers. Here we present Pebblescout, a tool that navigates such content by providing indexing and search capabilities. Indexing uses dense sampling of the sequences in the resource. Search finds subjects (runs or assemblies) that have short sequence matches to a user query, with well-defined guarantees and ranks them using informativeness of the matches. We illustrate the functionality of Pebblescout by creating eight databases that index over 3.7 petabases. The web service of Pebblescout can be reached at https://pebblescout.ncbi.nlm.nih.gov . We show that for a wide range of query lengths, Pebblescout provides a data-driven way for finding relevant subsets of large nucleotide resources, reducing the effort for downstream analysis substantially. We also show that Pebblescout results compare favorably to MetaGraph and Sourmash.