Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute

Circ Res. 2015 Jul 17;117(3):239-43. doi: 10.1161/CIRCRESAHA.115.306830. Epub 2015 Jun 18.

Abstract

There are conflicting data about the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute (NHLBI) between 1980 and 2011. Our outcomes focus on top-10% articles, meaning articles that were cited more often than 90% of other articles on the same topic, of the same type (eg, article, editorial), and published in the same year. The 6873 grants yielded 62 468 articles, of which 13 507 (or 22%) were top-10% articles. There was a modest association between better grant percentile ranking and number of top-10% articles. However, discrimination was poor (area under receiver operating characteristic curve [ROC], 0.52; 95% confidence interval, 0.51-0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% articles per $million spent. Hence, the seemingly conflicting findings on peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best National Institutes of Health (NIH) should use peer review assessments to make complex funding decisions.

Keywords: National Heart, Lung, and Blood Institute (US); National Institutes of Health (US); ROC curve; bibliometrics; peer review.

Publication types

  • Research Support, N.I.H., Extramural

MeSH terms

  • Area Under Curve
  • Bibliometrics
  • Biomedical Research / economics
  • Budgets / statistics & numerical data
  • Decision Making
  • Efficiency, Organizational / statistics & numerical data
  • Efficiency, Organizational / trends
  • Financing, Government / statistics & numerical data*
  • Financing, Organized / economics
  • Forecasting
  • Government Publications as Topic
  • Inflation, Economic
  • Investments / economics*
  • Investments / trends
  • National Heart, Lung, and Blood Institute (U.S.) / economics
  • National Heart, Lung, and Blood Institute (U.S.) / organization & administration*
  • National Heart, Lung, and Blood Institute (U.S.) / trends
  • Peer Review* / trends
  • ROC Curve
  • Research Support as Topic* / economics
  • Research Support as Topic* / statistics & numerical data
  • Research Support as Topic* / trends
  • United States