Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level

Abstract

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.

Fig 1. Properties of co-citation networks.

(A) Schematic of a co-citation network. The reference article (RA) (red, middle row) cites previous papers from the literature (orange, bottom row); subsequent papers cite the RA (blue, top row). The co-citation network is the set of papers that appear alongside the article in the subsequent citing papers (green, middle row). The field citation rate is calculated as the mean of the latter articles’ journal citation rates. (B) Growth of co-citation networks over time. Three RAs published in 2006 (red dots) were cited 5 (top row), 9 (middle row), or 31 times (bottom row) by 2011. Three intervals were chosen to illustrate the growth of the corresponding co-citation networks: 2006–2007, 2006–2009, and 2006–2011 (the first, second, and third columns, respectively). Each article in one of the three co-citation networks is shown as a separate green dot; the edges (connections between dots) indicates their presence together in the same reference list. (C) Cluster algorithm-based content analysis of the 215 papers in the co-citation network of a sample RA (panel B, bottom network series) identified a changing pattern of relevance to different subdisciplines over time. This RA described the identification of new peptides of possible clinical utility due to their similarity to known conotoxins. Papers in the co-citation network of this RA focused on (1) α-conotoxin mechanisms of action, (2) structure and evolution of conotoxins, (3) cyclotide biochemistry, (4) conotoxin phylogenetics, and (5) identification and synthesis of lantibiotics. (D) Growth of an article’s co-citation network is proportional to the number of times it has been cited. Each point is the average network size of 1,000 randomly chosen papers with between 1 and 100 citations (error bars represent the standard error of the mean). Each paper is only counted once, even if it is co-cited with the article of interest multiple times. An average of 17.8 new papers is added to the co-citation network for each additional citation. This suggests substantial duplication of articles within a co-citation network, since on average 32.4 papers (median of 30) are referenced in each citing article.

Fig 2. Text similarity of articles is defined more accurately by their co-citation networks than by the journals in which they appear.

(A, B) The text in each of 1,397 RAs was compared, either with the text in each corresponding co-citation network or separately with the collection of articles appearing in the same journal. Both primary and review articles are included. Cosine similarity scores were then calculated using either the top 100 terms (A) or all terms appearing in at least ten documents (B). Filled circles in green, co-citation network comparison; filled circles in shades of red, multidisciplinary journal comparison; filled circles in shades of blue, disciplinary journal comparison. Curves shifted to the right show more text similarity: RAs are least similar to papers in the same multidisciplinary journals, more similar to papers in the same disciplinary journal, and most similar to papers in their co-citation network.

Fig 3. Algorithm for calculating the Relative Citation Ratio (RCR).

(A) Article citation rate (ACR) is calculated as the total citations divided by the number of years excluding the calendar year of publication (Supporting Equation S1 in S1 Text), when few, if any, citations accrue (S2 Fig). Numbers in the bars correspond to the number of citations in that year. (B) Box-and-whisker plots of 88,835 NIH-funded papers (published between 2003 and 2010), summarizing their ACR, journal impact factor (matched to the article’s year of publication), and field citation rate (FCR). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles. (C) Correlation of FCR as generated in 2012 versus 2 y later in 2014 for the same set of articles, as a function of the number of starting citations in 2012. Data were sliced by the number of initial citations in 2012, to assess stability as a function of the number of citing articles (and thereby the starting size of the network). Each point, correlation coefficient for >1,000 articles. Between 2012 and 2014, articles accrued a median of 5 additional citations. The inclusion of the full span of years ensures a representative spread of ACRs at each value of the independent axis. Furthermore, since papers in this analysis receive a nearly identical number of citations in their ninth year as in their first full year after publication (S2 Fig), the FCRs of articles published later in the chosen time frame (2003 to 2010) do not undergo substantially more change than those published earlier. (D) Generate an expectation for ACRs based on a preselected benchmark group, by regressing the ACR of the benchmark papers onto their FCRs (Supporting Equations S3, S4 in S1 Text), one regression each publication year. The graphed examples were sampled from a random distribution for illustrative purposes. (E) The coefficients from each year’s regression equation transforms the FCRs of papers published in the same year into expected citation rates (ECRs) (Supporting Equation S5 in S1 Text). Each paper’s RCR is its ACR/ECR ratio. A portfolio’s RCR is simply the average of the individual articles’ RCRs (Supporting Equation S6 in S1 Text).

Fig 4. RCRs correspond with expert reviewer scores.

(A–C) Bubble plots of reviewer scores versus RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (A) F1000 scores for 2,193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“good,” “very good,” and “exceptional”, respectively); those scores were summed into a composite F1000 score for each article (S3 Fig). (B) Reviewer scores of 430 HHMI and NIH-funded papers collected by STPI. (C) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression. (D) Box-and-whisker plots illustrating the distribution of journal impact factors (JIFs) citations per year (CPY) and RCRs for two areas of NIH-funded research from 2007–2011. Cell biology, n = 5,936; neurological function, n = 5,417. *** p < 0.001, Kruskal-Wallis with Dunn’s multiple comparison test. n.s., not significant. Mean represented by a “+.” (E) Comparison of RCRs (orange) and Thompson Reuters ratios (blue) [17,28] for the same 544 articles with a low denominator. Data points are partially transparent to allow coordinates with multiple papers (darker) to be more clearly identified.

Fig 5. iCite, a publicly available tool for calculating RCR and accessing related citation information.

(A) Screenshot of a sample iCite result. Four hundred sample PMIDs from papers published over a 4-y window were entered into the iCite tool, which returned the maximum, mean +/− standard error of the mean (SEM), and median values for both CPY and RCR; weighted RCR is equal to the sum of the RCRs for this group. The box-and-whisker plot shows the distribution of article RCRs; bar graphs show the number of publications per year and weighted RCR per year, respectively. (B) Sample data download for an iCite result. iCite returns the total number of citations, number of CPY, expected CPY based on an NIH benchmark, FCR, Relative Citation Ratio, and percentile ranking in a downloadable Excel format for each PMID entered, as well as the corresponding title, author information, and year/journal of publication.

Fig 6. RCR-based evaluation of two NIH-funded research programs.

(A) Bar graph showing the percentage of papers that experience a drop in RCR from 2012 to 2014. Black bars, decrease in RCR; grey bars, decrease in RCR of 0.1 or more. (B) Box-and-whisker plots showing the distribution of RCR values for articles describing the human microbiome, published with support from the Human Microbiome Project (HMP) of the NIH Common Fund or another source (other). Boxes show the 25th–75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles.

Fig 7. Properties of RCRs at the article and investigator level.

(A, B) Frequency distribution of article-level RCRs (A) and JIFs (B), from 88,835 papers (authored by 3,089 R01-funded principal investigators [PIs]) for which co-citation networks were generated. Article RCRs are well fit by a log-normal distribution (R² = 0.99), and JIFs less so (R² = 0.79). (C) Box-and-whisker plots summarizing JIFs for the same papers, binned by impact factor quintile (line, median; box, 25th–75th percentiles; whiskers, 10th–90th percentiles). (D) RCR for the same papers using the same bins by JIF quintile (same scale as C). Although the median RCR for each bin generally corresponds to the impact factor quintile, there is a wide range of article RCRs in each category. (E) Box-and-whisker plots summarizing RCRs of these same papers published in selected journals. In each journal, there are papers with article RCRs surpassing the median RCR of the highest impact factor journals (left three). The impact factor of each journal is shown above. (F, G) Frequency distribution of investigator-level RCRs (F) and JIFs (G), representing the mean values for papers authored by each of 3,089 R01-funded PIs. Dashed line in (F), mode of RCR for PIs.

Fig 8. Scientific mobility of investigators’ influence relative to their field.

Color intensity is proportional to the percentage of PIs in each quintile. (A) The 3,089 investigators who were continuously funded by at least one R01 were ranked by their articles’ average RCR in each time window and split into quintiles. From left to right, investigators starting in different quintiles were tracked to see their rank in the next 4-y period. (B) This panel shows the same analysis, but the number of published articles was multiplied by their average RCR to calculate an influence-weighted article count. PIs were ranked by this aggregate score and split into quintiles. (C) Scatter plot illustrating the relationship between PI RCR at earlier and later time frames. Black points, actual RCR values; black line, linear regression of actual RCR values. Red points, random assignment model (PI RCRs for the second 4-y period are reshuffled and randomly assigned); red line, linear regression of modeled data.