Using altmetrics for detecting impactful research in quasi-zero-day time-windows: the case of COVID-19

Abstract

On December 31st 2019, the World Health Organization China Country Office was informed of cases of pneumonia of unknown etiology detected in Wuhan City. The cause of the syndrome was a new type of coronavirus isolated on January 7th 2020 and named Severe Acute Respiratory Syndrome CoronaVirus 2 (SARS-CoV-2). SARS-CoV-2 is the cause of the coronavirus disease 2019 (COVID-19). Since January 2020 an ever increasing number of scientific works related to the new pathogen have appeared in literature. Identifying relevant research outcomes at very early stages is challenging. In this work we use COVID-19 as a use-case for investigating: (1) which tools and frameworks are mostly used for early scholarly communication; (2) to what extent altmetrics can be used to identify potential impactful research in tight (i.e. quasi-zero-day) time-windows. A literature review with rigorous eligibility criteria is performed for gathering a sample composed of scientific papers about SARS-CoV-2/COVID-19 appeared in literature in the tight time-window ranging from January 15th 2020 to February 24th 2020. This sample is used for building a knowledge graph that represents the knowledge about papers and indicators formally. This knowledge graph feeds a data analysis process which is applied for experimenting with altmetrics as impact indicators. We find moderate correlation among traditional citation count, citations on social media, and mentions on news and blogs. Additionally, correlation coefficients are not inflated by indicators associated with zero values, which are quite common at very early stages after an article has been published. This suggests there is a common intended meaning of the citational acts associated with aforementioned indicators. Then, we define a method, i.e. the Comprehensive Impact Score (CIS), that harmonises different indicators for providing a multi-dimensional impact indicator. CIS shows promising results as a tool for selecting relevant papers even in a tight time-window. Our results foster the development of automated frameworks aimed at helping the scientific community in identifying relevant work even in case of limited literature and observation time.

Keywords: Altmetrics; Bibliometric indicators; COVID-19; Research evaluation; SARS-CoV-2; nCoV-2019.

Fig. 1

Number of papers per publication date in the time-window ranging from January 15th, 2020 to February 24th, 2020

Fig. 2

The UML activity diagram that graphically represents the workflow re-used and extended from (Nuzzolese et al. 2019) for the data processing activities

Fig. 3

Distributions of the number of indicators per paper

Fig. 4

Diagrams of the kernel density estimations obtained for each category

Fig. 5

Diagrams of the KDEs computed on z-scores for categories (5a) and metrics (5b)

Fig. 6

Correlation coefficients among indicators

Fig. 7

Correlation among mentions, social media, and citations with articles without zero-values as indicators

Fig. 8

Correlation among categories of indicators with articles with zero-value indicators filtered out from the sample. The values of the indicators for the three categories have been normalised in the range $[0-1]$ for comparison

Fig. 9

Geometric spaces

Fig. 10

Selection of papers based on the Comprehensive Impact Score (CIS) computed on citations and altmetrics separately

Fig. 11

Selection of papers based on the Comprehensive Impact Score (CIS) computed on I, $I^{\prime }$, and $A^{\prime }$

Fig. 12

STROBE scores