The rigor and reproducibility of science methods depends heavily on the appropriate use of statistical methods to answer research questions and make meaningful and accurate inferences based on data. The increasing analytic complexity and valuation of novel statistical and methodological approaches to data place greater emphasis on statistical review. We will outline the controversies within statistical sciences that threaten rigor and reproducibility of research published in the behavioral sciences and discuss ongoing approaches to generate reliable and valid inferences from data. We outline nine major areas to consider for generally evaluating the rigor and reproducibility of published articles and apply this framework to the 116 Behaviour Research and Therapy (BRAT) articles published in 2018. The results of our analysis highlight a pattern of missing rigor and reproducibility elements, especially pre-registration of study hypotheses, links to statistical code/output, and explicit archiving or sharing data used in analyses. We recommend reviewers consider these elements in their peer review and that journals consider publishing results of these rigor and reproducibility ratings with manuscripts to incentivize authors to publish these elements with their manuscript.
Keywords: Big data; P-hacking; Reliability; Reproducibility; Statistics.
Copyright © 2020 Elsevier Ltd. All rights reserved.
Conflict of interest statement
Declaration of competing interest The authors declare no conflict of interest.
Improving transparency and scientific rigor in academic publishing.J Neurosci Res. 2019 Apr;97(4):377-390. doi: 10.1002/jnr.24340. Epub 2018 Dec 2. J Neurosci Res. 2019. PMID: 30506706
Improving transparency and scientific rigor in academic publishing.Brain Behav. 2019 Jan;9(1):e01141. doi: 10.1002/brb3.1141. Epub 2018 Dec 2. Brain Behav. 2019. PMID: 30506879 Free PMC article.
Statistical considerations in reporting cardiovascular research.Am J Physiol Heart Circ Physiol. 2018 Aug 1;315(2):H303-H313. doi: 10.1152/ajpheart.00309.2018. Epub 2018 Jul 20. Am J Physiol Heart Circ Physiol. 2018. PMID: 30028200 Free PMC article.
Translational Metabolomics of Head Injury: Exploring Dysfunctional Cerebral Metabolism with Ex Vivo NMR Spectroscopy-Based Metabolite Quantification.In: Kobeissy FH, editor. Brain Neurotrauma: Molecular, Neuropsychological, and Rehabilitation Aspects. Boca Raton (FL): CRC Press/Taylor & Francis; 2015. Chapter 25. Brain Neurotrauma: Molecular, Neuropsychological, and Rehabilitation Aspects. 2015. PMID: 26269925 Free Books & Documents. Review.
Reviewing a Manuscript: Disparity Amongst Peer Reviewers' Priorities from Basic Health Sciences and Clinicians.J Coll Physicians Surg Pak. 2016 Aug;26(8):677-80. J Coll Physicians Surg Pak. 2016. PMID: 27539762 Review.