Background and aims: Journal impact factor has historically been taken as a proxy for quality. However, this is open to significant manipulation and bias. There is currently not widely adopted, robust journal and paper ranking metric which is focused solely on risk of bias.
Methods: Risk of bias data was extracted from all Cochrane database systematic reviews in Child Health, Lungs, and Airways for the years 2017-2019. A novel paper quality score, the Clinical Research Bias Index (CRBI), was applied. Individual paper data were pooled for each journal. A comparison was made to journal impact factors, individual paper citations, reads, and altmetric scores.
Results: 927 papers were analyzed for risk of bias. 119 (12·8%) scored a CRBI of 100%, with a mean score of 70%. A journal's overall CRBI risk of bias score was poorly correlated with impact factor (r 0.25). Citations (r 0.02), and reads (r 0.01) of individual papers showed very little association with the paper's risk of bias. Likewise, reads were not correlated with citations (r 0.03). H-index and Altmetric scores were similarly poorly correlated with CRBI.
Conclusion: The novel research quality tool CRBI demonstrates the poor correlation between journal impact factor, citations, and risk of bias. Journal and paper ranking metrics should ensure that they are fit for purpose, and enable the dissemination of high-quality research for the benefit of patients. We propose the CRBI as a potential solution which is resistant to manipulation and will reward the creation and publication of bias-free research.
Keywords: Impact Factor; bias; citations; quality.
© 2022 The Authors. Health Science Reports published by Wiley Periodicals LLC.