Background: Each year, the International Medical Informatics Association Yearbook recognizes significant scientific papers, labelled as "best papers", published the previous year in the subfields of biomedical informatics that correspond to the different section topics of the journal. For each section, about fifteen pre-selected "candidate" best papers are externally peer-reviewed to select the actual best papers. Although based on the available literature, little is known about the pre-selection process.
Objective: To move toward an explicit formalization of the candidate best papers selection process to reduce variability in the literature search across sections and over years.
Methods: A methodological framework is proposed to build for each section topic specific queries tailored to PubMed and Web of Science citation databases. The two sets of returned papers are merged and reviewed by two independent section editors and citations are tagged as "discarded", "pending", and "kept". A protocolized consolidation step is then jointly conducted to resolve conflicts. A bibliographic software tool, BibReview, was developed to support the whole process.
Results: The proposed search strategy was fully applied to the Decision Support section of the 2013 edition of the Yearbook. For this section, 1124 references were returned (689 PubMed-specific, 254 WoS-specific, 181 common to both databases) among which the 15 candidate best papers were selected.
Conclusions: The search strategy for determining candidate best papers for an IMIA Yearbook's section is now explicitly specified and allows for reproducibility. However, some aspects of the whole process remain reviewer-dependent, mostly because there is no characterization of a "best paper".
Keywords: IMIA Yearbook; Review literature as topic; decision support; medical informatics; publishing.