Toward scalable reuse of vEM data: OME-Zarr to the rescue

Abstract

The growing size of EM volumes is a significant barrier to findable, accessible, interoperable, and reusable (FAIR) sharing. Storage, sharing, visualization and processing are challenging for large datasets. Here we discuss a recent development toward the standardized storage of volume electron microscopy (vEM) data which addresses many of the issues that researchers face. The OME-Zarr format splits data into more manageable, performant chunks enabling streaming-based access, and unifies important metadata such as multiresolution pyramid descriptions. The file format is designed for centralized and remote storage (e.g., cloud storage or file system) and is therefore ideal for sharing large data. By coalescing on a common, community-wide format, these benefits will expand as ever more data is made available to the scientific community.

Keywords: Big data handling; Chunked data storage; FAIR data sharing; OME-Zarr.