Peer review of grant applications: criteria used and qualitative study of reviewer practices

PLoS One. 2012;7(9):e46054. doi: 10.1371/journal.pone.0046054. Epub 2012 Sep 28.

Abstract

Background: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

Methods and findings: We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

Conclusions: Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Data Collection
  • Financing, Organized / methods*
  • Financing, Organized / standards
  • France
  • Humans
  • Intellectual Property
  • Peer Review, Research / methods*
  • Peer Review, Research / standards
  • Qualitative Research
  • Research / economics

Grant support

This work was supported by a research grant from the French Ministry of Heath and sponsored by Département à la Recherche Clinique et au Développement, Assistance Publique-Hôpitaux de Paris (AOM 08 074). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.