Measuring the developmental function of peer review: a multi-dimensional, cross-disciplinary analysis of peer review reports from 740 academic journals

PeerJ. 2022 Jun 7:10:e13539. doi: 10.7717/peerj.13539. eCollection 2022.

Abstract

Reviewers do not only help editors to screen manuscripts for publication in academic journals; they also serve to increase the rigor and value of manuscripts by constructive feedback. However, measuring this developmental function of peer review is difficult as it requires fine-grained data on reports and journals without any optimal benchmark. To fill this gap, we adapted a recently proposed quality assessment tool and tested it on a sample of 1.3 million reports submitted to 740 Elsevier journals in 2018-2020. Results showed that the developmental standards of peer review are shared across areas of research, yet with remarkable differences. Reports submitted to social science and economics journals show the highest developmental standards. Reports from junior reviewers, women and reviewers from Western Europe are generally more developmental than those from senior, men and reviewers working in academic institutions outside Western regions. Our findings suggest that increasing the standards of peer review at journals requires effort to assess interventions and measure practices with context-specific and multi-dimensional frameworks.

Keywords: Academic journals; Natural language processing; Peer review; Reviewers; Standards.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Benchmarking
  • Female
  • Humans
  • Male
  • Organizations
  • Peer Review
  • Periodicals as Topic*
  • Schools

Grants and funding

Daniel Garcia-Costa and Francisco Grimaldo are supported by the Spanish Ministry of Science, Innovation and Universities (MCIU), the Spanish State Research Agency (AEI) and the European Regional Development Fund (ERDF) under project RTI2018-095820-B-I00. Flaminio Squazzoni is supported by a ”Department of Excellence” grant from the Italian Ministry of Education, University and Research to the Department of Social and Political Sciences of the University of Milan, a grant from PRIN-MIUR (Progetti di Rilevante Interesse Nazionale –Italian Ministry of University and Research) (Grant Number: 20178TRM3F001 “14All”) and a grant from the University of Milan (Grant Number: PSR2015-17 Transition Grant). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.