Background: Objectives of this study were to analyze the impact of including librarians and information specialist as methodological peer-reviewers. We sought to determine if and how librarians' comments differed from subject peer-reviewers'; whether there were differences in the implementation of their recommendations; how this impacted editorial decision-making; and the perceived utility of librarian peer-review by librarians and authors.
Methods: We used a mixed method approach, conducting a qualitative analysis of reviewer reports, author replies and editors' decisions of submissions to the International Journal of Health Governance. Our content analysis categorized 16 thematic areas, so that methodological and subject peer-reviewers' comments, decisions and rejection rates could be compared. Categories were based on the standard areas covered in peer-review (e.g., title, originality, etc.) as well as additional in-depth categories relating to the methodology (e.g., search strategy, reporting guidelines, etc.). We developed and used criteria to judge reviewers' perspectives and code their comments. We conducted two online multiple-choice surveys which were qualitatively analyzed: one of methodological peer-reviewers' perceptions of peer-reviewing, the other of published authors' views on the suggested revisions.
Results: Methodological peer-reviewers assessed 13 literature reviews submitted between September 2020 and March 2023. 55 reviewer reports were collected: 25 from methodological peer-reviewers, 30 from subject peer-reviewers (mean: 4.2 reviews per manuscript). Methodological peer-reviewers made more comments on methodologies, with authors more likely to implement their changes (52 of 65 changes, vs. 51 of 82 by subject peer-reviewers); they were also more likely to reject submissions (seven vs. four times, respectively). Where there were differences in recommendations to editors, journal editors were more likely to follow methodological peer-reviewers (nine vs. three times, respectively). The survey of published authors (87.5% response rate) revealed four of seven found comments on methodologies helpful. Librarians' survey responses (66.5% response rate) revealed those who conducted peer-reviews felt they improved quality of publications.
Conclusions: Librarians can enhance evidence synthesis publications by ensuring methodologies have been conducted and reported appropriately. Their recommendations helped authors revise submissions and facilitated editorial decision-making. Further research could determine if sharing reviews with subject peer-reviewers and journal editors could benefit them in better understanding of evidence synthesis methodologies.
Keywords: Evidence synthesis; Health science librarians; Information specialists; Methodological peer-reviewers; Segmented peer-review.
© 2024. The Author(s).