Gender bias in scholarly peer review

Abstract

Peer review is the cornerstone of scholarly publishing and it is essential that peer reviewers are appointed on the basis of their expertise alone. However, it is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. Here, using public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals, we show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. We also show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.

Keywords: gender bias; large-scale data analysis; peer review.

Figure 1.. Women review and author even less articles than expected from their numeric underrepresentation.

(a₁) We represent peer-reviewing interactions as directed graphs, in which vertices denote scientists. In the editor-to-reviewer network every edge represents the act of an editor (source vertex) appointing a reviewer (target vertex) to review a manuscript (and the reviewer has accepted the invitation). Analogously, in the reviewer-to-author network edges represent a reviewer reviewing a manuscript of an author. (b) The development of the fraction of contributions for each gender are shown for editors, reviewers and authors. Since the start of the Frontiers journals in 2007 until 2015, women (circles) edit, review and author much less than 50% of manuscripts, as expected from their numeric underrepresentation. However, the actual numbers of reviewing and authoring contributions by women are even smaller than expected by chance, taking into account their numeric underrepresentation. This is revealed by comparison with a null hypothesis in which gender and number of contributions are assumed to be independent. To this end, we generated surrogate ensembles by shuffling the genders of scientists appearing in a given role in the network (a₂). From the surrogate ensembles, we obtained 95% confidence intervals (CIs; shaded areas in b). *, **, *** over (under) the data symbols denote the data lying over (under) the 95%, 99%, 99.9% CIs. Note that for all three subnetworks, there is a noticeable, but extremely slow trend towards equity (dashed line) for the fraction of contributions. (c) The fraction of female contributors, ranked in increasing order of authoring contributions, for the 47 frontier journals, whose published articles were handled by at least 25 distinct editors. Women were underrepresented consistently across all fields and particularly severely in math-intensive disciplines. DOI: http://dx.doi.org/10.7554/eLife.21718.002

Figure 1—figure supplement 1.. Analysis of network topology.

(a) Number of articles (blue), as well as number of people (green) in the reviewer-to-author (left) and editor-to-reviewer (right) network grow approximately exponentially over the years. (b) At the same time, network connectivity also steadily gets denser, giving rise to a growing weak graph giant component (that is, the biggest connected component in the network, irrespective of edge directions) of socially interacting editors, reviewers and authors. The cluster sizes are normalized with the network sizes at each time point. The latest reviewer-to-author (editor-to-reviewer) network sampled at the cutoff date has a giant component reaching 95% (93%) of its global size. (c) The small-world-ness index (see methods) suggests that a small-world topology is robustly established. (d) Global undirected transitivity (see methods), (e) as well as average shortest path lengths (see methods) have stabilized in the 3-5 last considered years, despite the fact that the network keeps growing at an exponential pace. DOI: http://dx.doi.org/10.7554/eLife.21718.003

Figure 1—figure supplement 2.. Gender disparities vary between countries.

Based on scientists' affiliations, we plot the fractions of (a) editors, (b) reviewers and (c) authors in each country. In case of multiple affiliations we counted one contribution for each distinct country. Dark gray color indicates a fraction of female contributions aligned to the overall median value. Countries providing less than seven contributions are excluded from this analysis (white color). DOI: http://dx.doi.org/10.7554/eLife.21718.004

Figure 2.. Women are underrepresented in the fat tail of contributions.

A break-down of the number of individuals contributing a given number of times as editors, reviewers and authors (binned, x-axis is marking the bin edges) shows that the majority of scientists (a) edited, (b) reviewed or (c) authored (corresponding zooms for small contribution numbers are shown in e-f) only a small number of manuscripts. Chance levels (shaded) were derived from an ensemble of reference networks constructed as shown in Figure 1a. The underrepresentation of women in relation to these chance levels tends to increase towards the fat tail of the distribution, associated to the relatively few individuals that made many contributions. In the group of one-time authors or reviewers however, women are overrepresented. Time resolved distributions are shown in Figure 2—figure supplement 1. DOI: http://dx.doi.org/10.7554/eLife.21718.005

Figure 2—figure supplement 1.. Time- and gender-resolved histograms of the number of contributions.

For each year we calculate the histograms of contributions for men (a–c) and (e–f) women (see Figure 2 for the year 2015) and color code the normalized deviations from the expected median (i.e. the median of the shaded area in Figure 2). Bins for which less than 7 contributions were available in a given year are shown in white. In the course of time the network grows (Figure 1—figure supplement 1) and higher contribution numbers become available, but the overall pattern of deviations from the expectation remains significant for reviewers (b,e) and authors (c,f). The tendency towards equity is weaker at the distribution’s right tail. Data for editors (a,d) is more variable. DOI: http://dx.doi.org/10.7554/eLife.21718.006

Figure 3.. Editors have a same-gender preference for appointing reviewers.

(a) Female editors (orange) appoint significantly more female reviewers than expected under the gender-blind assumption (shaded area). At the same time, male editors (green) appoint less women than expected. The development of this trend over time is shown, including articles cumulatively until the indicated year. (b) Likewise, female/male reviewers review significantly more female/male-authored articles than expected. (c) Homophily is widespread across scientific fields, including those with relatively mild underrepresentation of women. We here report four example disciplinary groupings, with large numbers of contributions (from left to right, respectively, 13416, 4721, 4020, 5680) and the propensity of appointing a female reviewer depending on the editor's gender for each of these groupings. Only assignments by female neuroscience editors were not homophilic, otherwise the occurrence of same-gender preferences was general, arguing against heterogeneity between subfields as a cause for homophily in assignments. (d) Plotted here are distributions of a measure of inbreeding homophily. To control for baseline homophily at the level of a narrow local neighborhood, we measure, for each editor node, the actual number of reviewer assignments given to women and subtract the expected number, which would be observed if the considered editor appointed women with the same frequency as in his/her local vicinity. For male editors (green) the distribution is skewed towards an underrepresentation of female assignments (left-leaning), while for female editors the distribution is skewed towards an overrepresentation of female assignments. This highlights that homophily bias is detectable even at the level of the reachable narrow surrounding of each editor. (e) Histogram of the probability that an editor assigns as least as many reviews to people of the same gender as he/she actually does reveals that there's an excess of strongly inbreeding-homophilic editors (small Φ_hom -values) among both men and women compared to expectation (shaded area). Note that below Φ_hom < 0.1 there are only few strongly homophilic female editors. For male editors, significant homophily extends through many more editors until Φ_hom < 0.6. (f) Using all data until 2015, the probability that a women is appointed is above expectation (shaded areas) only for female editors and only when all or all but the most extremely inbreeding-homophilic editors are included in the analysis. DOI: http://dx.doi.org/10.7554/eLife.21718.007