Bias in Research Grant Evaluation Has Dire Consequences for Small Universities

Abstract

Federal funding for basic scientific research is the cornerstone of societal progress, economy, health and well-being. There is a direct relationship between financial investment in science and a nation's scientific discoveries, making it a priority for governments to distribute public funding appropriately in support of the best science. However, research grant proposal success rate and funding level can be skewed toward certain groups of applicants, and such skew may be driven by systemic bias arising during grant proposal evaluation and scoring. Policies to best redress this problem are not well established. Here, we show that funding success and grant amounts for applications to Canada's Natural Sciences and Engineering Research Council (NSERC) Discovery Grant program (2011-2014) are consistently lower for applicants from small institutions. This pattern persists across applicant experience levels, is consistent among three criteria used to score grant proposals, and therefore is interpreted as representing systemic bias targeting applicants from small institutions. When current funding success rates are projected forward, forecasts reveal that future science funding at small schools in Canada will decline precipitously in the next decade, if skews are left uncorrected. We show that a recently-adopted pilot program to bolster success by lowering standards for select applicants from small institutions will not erase funding skew, nor will several other post-evaluation corrective measures. Rather, to support objective and robust review of grant applications, it is necessary for research councils to address evaluation skew directly, by adopting procedures such as blind review of research proposals and bibliometric assessment of performance. Such measures will be important in restoring confidence in the objectivity and fairness of science funding decisions. Likewise, small institutions can improve their research success by more strongly supporting productive researchers and developing competitive graduate programming opportunities.

Fig 1

(A) Mean (± SD) percent success of NSERC Discovery Grant applications (2011–2014) relative to institution size and applicant status.

Fig 2. Funding level scores for NSERC Discovery Grant applications (2011–2014) by established researchers and early career researchers, according to institution size.

Normally a score of “J” or earlier letter is required for funding.

Fig 3. Evaluation scores for NSERC Discovery Grant applications relative to institution size (2011–2014).

(A) Excellence of the Researcher for established researchers. (B) Excellence of the Researcher for early career researchers. (C) Merit of the Proposal for established researchers. (D) Merit of the Proposal for early career researchers. (E) Contribution to the Training of High Quality Personnel for established researchers. (F) Contribution to the Training of High Quality Personnel for early career researchers.

Fig 4. Discovery Grant success rates 2004–2015.

(A) Established researchers currently seeking Discovery Grant renewal (ER-R). (B) Established researchers currently not holding a Discovery Grant (ER-NHG). (C) Early career researchers (ECR). The dashed line reflects the 2009–2010 adoption of a new grant application evaluation system.

Fig 5

(A) Projected 10-year mean change in NSERC Discovery Grant funding at Canadian universities assuming no corrective measures to address bias related to institution size. (B) Projected mean (± SD) 10-year change in NSERC Discovery Grant funding at universities according to different bias-corrective measures, relative to current funding rates. Values less than 1 represent attrition in funded researchers.