Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 1, e148
eCollection

On the Reproducibility of Science: Unique Identification of Research Resources in the Biomedical Literature

Affiliations

On the Reproducibility of Science: Unique Identification of Research Resources in the Biomedical Literature

Nicole A Vasilevsky et al. PeerJ.

Abstract

Scientific reproducibility has been at the forefront of many news stories and there exist numerous initiatives to help address this problem. We posit that a contributor is simply a lack of specificity that is required to enable adequate research reproducibility. In particular, the inability to uniquely identify research resources, such as antibodies and model organisms, makes it difficult or impossible to reproduce experiments even where the science is otherwise sound. In order to better understand the magnitude of this problem, we designed an experiment to ascertain the "identifiability" of research resources in the biomedical literature. We evaluated recent journal articles in the fields of Neuroscience, Developmental Biology, Immunology, Cell and Molecular Biology and General Biology, selected randomly based on a diversity of impact factors for the journals, publishers, and experimental method reporting guidelines. We attempted to uniquely identify model organisms (mouse, rat, zebrafish, worm, fly and yeast), antibodies, knockdown reagents (morpholinos or RNAi), constructs, and cell lines. Specific criteria were developed to determine if a resource was uniquely identifiable, and included examining relevant repositories (such as model organism databases, and the Antibody Registry), as well as vendor sites. The results of this experiment show that 54% of resources are not uniquely identifiable in publications, regardless of domain, journal impact factor, or reporting requirements. For example, in many cases the organism strain in which the experiment was performed or antibody that was used could not be identified. Our results show that identifiability is a serious problem for reproducibility. Based on these results, we provide recommendations to authors, reviewers, journal editors, vendors, and publishers. Scientific efficiency and reproducibility depend upon a research-wide improvement of this substantial problem in science today.

Keywords: Antibodies; Cell lines; Constructs; Knockdown reagents; Materials and Methods; Model organisms; Scientific reproducibility.

Figures

Figure 1
Figure 1. Resource identifiability across disciplines.
(A) Summary of average fraction identified for each resource type. (B–F) Identifiability of each resource type by discipline. The total number of resources for each type is: (B) antibodies, n = 703; (C) cell lines, n = 104; (D) constructs, n = 258; (E) knockdown reagents, n = 210; (F) organisms, n = 428. The y-axis is the average for each resource type across each domain. Variation from this average is shown by the bars, error bars indicate upper and lower 95% confidence intervals.
Figure 2
Figure 2. Resource identification rates across journals of varying impact factors.
(A) An overview of fraction identified by impact factor for all resource types. (B–F) Fraction identified by impact factor for each individual resource type. Increasing height on the x-axis corresponds with a higher impact factor for each journal.
Figure 3
Figure 3. Identification of resource varies across journals with varying resource-reporting requirements.
The classifications of reporting requirements are summarized in the methods. A total of 53 out of 118 resources were identifiable in the stringent reporting guidelines category (17 papers were analyzed), 201 resources were identifiable out of 329 resources for the satisfactory category (48 papers were analyzed) and 662 out of 1,217 resources were identifiable in the loose category (182 papers were analyzed). Variation from this average is shown by the bars, error bars indicate upper and lower 95% confidence intervals.

Similar articles

See all similar articles

Cited by 77 PubMed Central articles

See all "Cited by" articles

References

    1. Bandrowski AE, Cachat J, Li Y, Müller HM, Sternberg PW, Ciccarese P, Clark T, Marenco L, Wang R, Astakhov V, Grethe JS, Martone ME. A hybrid human and machine resource curation pipeline for the neuroscience information framework. Database. 2012;bas005 doi: 10.1093/database/bas005. - DOI - PMC - PubMed
    1. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature. 2012;483(7391):531–533. doi: 10.1038/483531a. - DOI - PubMed
    1. Bourbeillon J, Orchard S, Benhar I, Borrebaeck C, de Daruvar A, Dübel S, Frank R, Gibson F, Gloriam D, Haslam N, Hiltker T, Humphrey-Smith I, Hust M, Juncker D, Koegl M, Konthur Z, Korn B, Krobitsch S, Muyldermans S, Nygren P-Å, Palcy S, Polic B, Rodriguez H, Sawyer A, Schlapshy M, Snyder M, Stoevesandt O, Taussig MJ, Templin M, Uhlen M, van der Maarel S, Wingren C, Hermjakob H, Sherman D. Minimum information about a protein affinity reagent (MIAPAR) Nature Biotechnology. 2010;28(7):650–653. doi: 10.1038/nbt0710-650. - DOI - PubMed
    1. Bowes JB, Snyder KA, Segerdell E, Jarabek CJ, Azam K, Zorn AM, Vize PD. Xenbase: gene expression and improved integration. Nucleic Acids Research. 2010;38(Suppl 1):D607–D612. doi: 10.1093/nar/gkp953. - DOI - PMC - PubMed
    1. Bradford Y, Conlin T, Dunn N, Fashena D, Frazer K, Howe DG, Knight J, Mani P, Martin R, Moxon SAT, Paddock H, Pich C, Ramachandran S, Ruef BJ, Ruzicka L, Schaper HB, Schaper K, Shao X, Singer A, Sprague J, Sprunger B, Slyke CV, Westerfield M. ZFIN: enhancements and updates to the zebrafish model organism database. Nucleic Acids Research. 2011;39(Database issue):D822–D829. doi: 10.1093/nar/gkq1077. - DOI - PMC - PubMed

LinkOut - more resources

Feedback