Effectiveness of journal ranking schemes as a tool for locating information

Abstract

Background: The rise of electronic publishing, preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer immediate, reliable information on the value of that research.

Methodology/principal findings: Here, we systematically evaluate the effectiveness of journals, through the work of editors and reviewers, at evaluating unpublished research. We find that the distribution of the number of citations to a paper published in a given journal in a specific year converges to a steady state after a journal-specific transient time, and demonstrate that in the steady state the logarithm of the number of citations has a journal-specific typical value. We then develop a model for the asymptotic number of citations accrued by papers published in a journal that closely matches the data.

Conclusions/significance: Our model enables us to quantify both the typical impact and the range of impacts of papers published in a journal. Finally, we propose a journal-ranking scheme that maximizes the efficiency of locating high impact research.

Figure 1. Time evolution of the distribution of number of citations of the papers published in a given academic journal.

(A) Probability density function , where Y is a year in the period 1998–2004, J is the Journal of Biological Chemistry, and ℓ≡log₁₀(n) where n is the number of citations accrued by a paper between its publication date and December 31, 2006. Because the papers published in those years are still accruing citations by December 2006, the distributions are not stationary, but instead “drift” to higher values of ℓ. (B) for the Journal of Biological Chemistry and for Y in the period 1991–1993. For this period, the distributions are essentially identical, indicating that has converged to its steady-state form . The steady-state distribution is well described by a normal with mean 1.65 and standard deviation 0.35 (black dashed curve). (C) Time dependence of for three journals: Astrophysical Journal, Ecology, and Circulation. As for the Journal of Biological Chemistry, we find that after some transient period, reaches a stationary value (see Methods). The orange region highlights the set of years for which we consider that is stationary. The time scale τ(J) for reaching the steady-state strongly depends on the journal: τ(Astrophysical Journal) = 18 years, τ(Ecology) = 12 years, and τ(Circulation) = 9 years. Significantly, we find no correlations between τ(J) and , whose values are 1.44 for Astrophysical Journal, 1.70 for Ecology, and 1.66 for Circulation. (D) Pairwise comparison of citation distributions for different years for a given journal. We show the matrices of p-values obtained using the Kolmogorov-Smirnov test for the Astrophysical Journal, Ecology, and Circulation. We color the matrix elements following the color code on the right. p-values close to one mean that it is likely that both distributions come from a common underlying distribution; p-values close to zero mean that is it very unlikely that both distributions come from a common underlying distribution. We then use a box-diagonal model to identify contiguous blocks of years for which the p-value is large enough that the null hypothesis cannot be rejected. The white lines in the matrices indicate the best fit of a box-diagonal model. We identify the first box with more than 2 years for which to be the steady-state period (see Methods).

Figure 2. Modeling the steady-state distributions of the number of citations for papers published in a given journal.

(A) Our model assumes that the “quality” of the papers published by a journal obeys a normal distribution with mean μ and standard deviation σ. The number of citations of a paper with quality q∈N(μ,σ) is given by Eq. (3). Because the quality is a continuous variable whereas the number of citations is an integer quantity, the same number of citations will occur for papers with qualities spanning a certain range of q. In particular, all papers for which q<log₁₀(1+γ) will receive no citations. In the panel, the areas of differently shaded regions yield the probability of a paper accruing a given number of citations. (B) Scatter plot of the estimated value of σ versus for all 2,267 journals considered in our analysis (see Methods and Appendices S1 and S4 for details on the fits). Notice that σ is almost independent of . The solid line corresponds to σ = 0.419, the mean of the estimated values of σ for all journals (see Methods). (C) Scatter plot of the estimated value of γ+1 for versus . Notice the strong correlation between the two variables. The solid line corresponds to (see Methods for details on the fit). (D) Fraction of uncited papers as a function of . For this and all subsequent panels, solid lines show the predictions of the model using , σ = 0.419, and a value of μ for each (see Methods). (E) Variance of ℓ as a function of . (F) Skewness of ℓ as a function of . The skewness of the normal distribution is zero. (G) Kurtosis excess of ℓ as a function of . The kurtosis excess of the normal distribution is zero. Note how, for the case of , the moments of the distribution of citations for cited papers deviate significantly from those expected for a normal distribution. In contrast, for , only a small fraction of papers remains uncited, so deviations from the expectations for a normal distribution are small.

Figure 3. Comparison of citation-based journal ranking schemes.

We present results for 13 journals that the ISI classifies primarily in experimental psychology, and 36 journals that the ISI classifies primarily in ecology (see Appendix S3 for other fields). For every pair of journals, J_i and J_j, belonging to the same field, we obtain the probability p_ij that a randomly selected paper published in J_i has received more citations than a randomly selected paper published in J_j. We rank the journals in each field according to three schemes: (A) optimal ranking R^AUC, that is, the ranking that maximizes p_ij for R(i)<R(j); (B) ranking according to decreasing q̅(J); (C) ranking according to decreasing JIF. We plot {p_ij} matrices for each of the fields and ranking schemes using the color scheme on the right. Green indicates adequate ranking, whereas red indicates inadequate ranking. It is visually apparent that the ranking according to decreasing q̅(J) provides nearly optimal ranking, whereas ranking according to decreasing JIF does not. As an example, consider the journals Brain and Cognition and Journal of Experimental Psychology: Learning, Memory, and Cognition. The JIF ranks Brain Cogn. third and J. Exp. Psy. fourth. However, the median number of cumulative citations to the papers published in the latter is 34, and only 3 for papers published in the former. Not surprisingly, the probability of a randomly selected paper published in J. Exp. Psy. to have received more cumulative citations than a randomly selected paper published in Brain Cogn. is 0.88.

Figure 4. Effect of JIF biases on the ranking of journals.

(A) Comparison of the rankings of journals obtained using the JIF and the AUC statistic. Though there are clear correlations between the two rankings, deviations can be extremely large. (B) Probability density function of ΔR(i) = R^JIF(i)−R^AUC(i). Positive values of ΔR indicate under-rating of the journal. (C) Probability density function of change in the median ranking of the journals primarily classified in a given field, for fields with at least two journals. The papers published in journals classified in fields that are over-rated tend to get cited quickly (probably because of faster publication times), whereas papers published in journals in under-rated fields take longer to start accruing citations. Table S1 lists the median change of rank for each field.