Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2012;7(10):e48509.
doi: 10.1371/journal.pone.0048509. Epub 2012 Oct 31.

Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach

Rüdiger Mutz  1 , Lutz Bornmann, Hans-Dieter Daniel
Affiliations

Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach

Rüdiger Mutz et al. PLoS One. 2012.

Abstract

Background: One of the most important weaknesses of the peer review process is that different reviewers' ratings of the same grant proposal typically differ. Studies on the inter-rater reliability of peer reviews mostly report only average values across all submitted proposals. But inter-rater reliabilities can vary depending on the scientific discipline or the requested grant sum, for instance.

Goal: Taking the Austrian Science Fund (FWF) as an example, we aimed to investigate empirically the heterogeneity of inter-rater reliabilities (intraclass correlation) and its determinants.

Methods: The data consisted of N = 8,329 proposals with N = 23,414 overall ratings by reviewers, which were statistically analyzed using the generalized estimating equations approach (GEE).

Results: We found an overall intraclass correlation (ICC) of reviewer? ratings of ρ = .259 with a 95% confidence interval of [.249,.279]. In humanities the ICCs were statistically significantly higher than in all other research areas except technical sciences. The ICC in biosciences deviated statistically significantly from the average ICC. Other factors (besides the research areas), such as the grant sum requested, had negligible influence on the ICC.

Conclusions: Especially in biosciences, the number of reviewers of each proposal should be increased so as to increase the ICC.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Intraclass correlations, overall and for the separate research areas.
Lines are shown as dotted because research area is categorical, so interpolation between research areas is not intended.
Figure 2
Figure 2. Intraclass correlations, overall and for the separate years of the final decision by the board of trustees of the Austrian Science Fund.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Bornmann L (2011) Scientific peer review. Annual Review of Information Science and Technology 45: 199–245.
    1. Hemlin S, Rasmussen S (2006) The shift in academic quality control. Science Technology & Human Values 31: 173–198.
    1. Marsh HW, Jayasinghe UW, Bond NW (2008) Improving the peer-review process for grant applications - reliability, validity, bias, and generalizability. American Psychologist 63: 160–168. - PubMed
    1. Cicchetti DV (1991) The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behavioral and Brain Sciences 14: 119–135.
    1. Jayasinghe UW, Marsh HW, Bond N (2003) A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society - Series A: Statistics in Society 166: 279–300.

Grants and funding

The authors have no funding or support to report.