The use of bibliography enriched features for automatic citation screening

Abstract

Context: Citation screening (also called study selection) is a phase of systematic review process that has attracted a growing interest on the use of text mining (TM) methods to support it to reduce time and effort. Search results are usually imbalanced between the relevant and the irrelevant classes of returned citations. Class imbalance among other factors has been a persistent problem that impairs the performance of TM models, particularly in the context of automatic citation screening for systematic reviews. This has often caused the performance of classification models using the basic title and abstract data to ordinarily fall short of expectations.

Objective: In this study, we explore the effects of using full bibliography data in addition to title and abstract on text classification performance for automatic citation screening.

Methods: We experiment with binary and Word2vec feature representations and SVM models using 4 software engineering (SE) and 15 medical review datasets. We build and compare 3 types of models (binary-non-linear, Word2vec-linear and Word2vec-non-linear kernels) with each dataset using the two feature sets.

Results: The bibliography enriched data exhibited consistent improved performance in terms of recall, work saved over sampling (WSS) and Matthews correlation coefficient (MCC) in 3 of the 4 SE datasets that are fairly large in size. For the medical datasets, the results vary, however in the majority of cases the performance is the same or better.

Conclusion: Inclusion of the bibliography data provides the potential of improving the performance of the models but to date results are inconclusive.

Keywords: Citation screening automation; Computing methodologies; Feature enrichment; Systematic reviews; Text mining.