No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications

John Protzko; Jonathan W Schooler

doi:10.7717/peerj.8014

No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications

PeerJ. 2020 Mar 24:8:e8014. doi: 10.7717/peerj.8014. eCollection 2020.

Authors

John Protzko¹, Jonathan W Schooler¹

Affiliation

¹ Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, United States of America.

Abstract

What explanation is there when teams of researchers are unable to successfully replicate already established 'canonical' findings? One suggestion that has been put forward, but left largely untested, is that those researchers who fail to replicate prior studies are of low 'expertise and diligence' and lack the skill necessary to successfully replicate the conditions of the original experiment. Here we examine the replication success of 100 scientists of differing 'expertise and diligence' who attempted to replicate five different studies. Using a bibliometric tool (h-index) as our indicator of researcher 'expertise and diligence', we examine whether this was predictive of replication success. Although there was substantial variability in replication success and in the h-factor of the investigators, we find no relationship between these variables. The present results provide no evidence for the hypothesis that systematic replications fail because of low 'expertise and diligence' among replicators.

Keywords: Bibliometrics; Expertise; Hypothesis Testing; Laboratories; Metascience; Registered Replication Reports; Replication Crisis; Reproducibility; Scientometrics; h-index.

Grants and funding

John Protzko was supported by grant #44069-59380 from the Fetzer Franklin Fund. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.