Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 19 (2), 107-115

A Link Is Not Enough - Reproducibility of Data

Affiliations

A Link Is Not Enough - Reproducibility of Data

Mateusz Pawlik et al. Datenbank Spektrum.

Abstract

Although many works in the database community use open data in their experimental evaluation, repeating the empirical results of previous works remains a challenge. This holds true even if the source code or binaries of the tested algorithms are available. In this paper, we argue that providing access to the raw, original datasets is not enough. Real-world datasets are rarely processed without modification. Instead, the data is adapted to the needs of the experimental evaluation in the data preparation process. We showcase that the details of the data preparation process matter and subtle differences during data conversion can have a large impact on the outcome of runtime results. We introduce a data reproducibility model, identify three levels of data reproducibility, report about our own experience, and exemplify our best practices.

Keywords: Data preparation process; Reproducibility.

Figures

Fig. 1
Fig. 1
XML tree using Conversion A
Fig. 2
Fig. 2
XML tree using Conversion B
Fig. 3
Fig. 3
RPI data reproducibility model
Fig. 4
Fig. 4
Data reproducibility levels

Similar articles

See all similar articles

References

    1. Artifact review and badging. https://www.acm.org/publications/policies/artifact-review-badging. Accessed: 03 June 2019
    1. CERN. https://home.cern/. Accessed: 03 June 2019
    1. Creative Commons. https://creativecommons.org/. Accessed: 03 June 2019
    1. Data in brief. https://www.journals.elsevier.com/data-in-brief. Accessed: 03 June 2019.
    1. Data Mendeley. https://data.mendeley.com/. Accessed: 03 June 2019.

LinkOut - more resources

Feedback