Context-Aware Reviewer Assignment for Trust Enhanced Peer Review

PLoS One. 2015 Jun 19;10(6):e0130493. doi: 10.1371/journal.pone.0130493. eCollection 2015.

Abstract

Reviewer assignment is critical to peer review systems, such as peer-reviewed research conferences or peer-reviewed funding applications, and its effectiveness is a deep concern of all academics. However, there are some problems in existing peer review systems during reviewer assignment. For example, some of the reviewers are much more stringent than others, leading to an unfair final decision, i.e., some submissions (i.e., papers or applications) with better quality are rejected. In this paper, we propose a context-aware reviewer assignment for trust enhanced peer review. More specifically, in our approach, we first consider the research area specific expertise of reviewers, and the institution relevance and co-authorship between reviewers and authors, so that reviewers with the right expertise are assigned to the corresponding submissions without potential conflict of interest. In addition, we propose a novel cross-assignment paradigm, and reviewers are cross-assigned in order to avoid assigning a group of stringent reviewers or a group of lenient reviewers to the same submission. More importantly, on top of them, we propose an academic CONtext-aware expertise relevanCe oriEnted Reviewer cross-assignmenT approach (CONCERT), which aims to effectively estimate the "true" ratings of submissions based on the ratings from all reviewers, even though no prior knowledge exists about the distribution of stringent reviewers and lenient reviewers. The experiments illustrate that compared with existing approaches, our proposed CONCERT approach can less likely assign more than one stringent reviewers or lenient reviewers to a submission simultaneously and significantly reduce the influence of ratings from stringent reviewers and lenient reviewers, leading to trust enhanced peer review and selection, no matter what kind of distributions of stringent reviewers and lenient reviewers are.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Peer Review / standards*
  • Publishing*

Grant support

This work was supported by National 973 Program of China under grant 2013CB329604; Ministry of Science and Technology of China, http://www.most.gov.cn/; LL MW XW: Program for Changjiang Scholars and Innovative Research Team in University (PCSIRT) of the Ministry of Education China under grant IRT13059; Ministry of Education China, http://www.moe.edu.cn/; LL MW XW: Anhui Provincial Natural Science Foundation under grant 1408085QF130; Anhui Province Natural Science Fund Committee, http://220.178.98.52/zrkxjj/; LL: the National Natural Science Foundation of China (NSFC) under grant 61229301, and the Fundamental Research Funds for the Central Universities under grant 2015HGCH0012. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.