Unknown unknowns in biomedical research: does an inability to deal with ambiguity contribute to issues of irreproducibility?

Abstract

The credibility and consequent sustainability of the biomedical research "ecosystem" is in jeopardy, in part due to an inability to reproduce data from the peer-reviewed literature. Despite obvious and relatively inexpensive solutions to improve reproducibility-ensuring that experimental reagents, specifically cancer cell lines and antibodies, are authenticated/validated before use and that best practices in statistical usage are incorporated into the design, analysis, and reporting of experiments-these are routinely ignored, a reflection of hubris and a comfort with the status quo on the part of many investigators. New guidelines for the peer review of publications and grant applications introduced in the past year, while well-intended, lack the necessary consequences, e.g., denial of funding, that would result in sustained improvements when scientific rigor is lacking and/or transparency is, at best, opaque. An additional factor contributing to irreproducibility is a reductionist mindset that prioritizes certainty in research outcomes over the ambiguity intrinsic to biological systems that is often reflected in "unknown unknowns". This has resulted in a tendency towards codifying "rules" that can provide "yes-no" outcomes that represent a poor substitute for the intellectual challenge and skepticism that leads to an awareness and consideration of "unknown unknowns". When acknowledged as potential causes of unexpected experimental outcomes, these can often transition into the "knowns" that facilitate positive, disruptive innovation in biomedical research like the human microbiome. Changes in investigator mindset, both in terms of validating reagents and embracing ambiguity, are necessary to aid in reducing issues with reproducibility.

Keywords: Peer review; Reagent validation; Reproducibility, unknown unknowns.