Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

Vincent Chin; John P A Ioannidis; Martin A Tanner; Sally Cripps

doi:10.1016/j.jclinepi.2021.03.014

Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

J Clin Epidemiol. 2021 Aug;136:96-132. doi: 10.1016/j.jclinepi.2021.03.014. Epub 2021 Mar 26.

Authors

Vincent Chin¹, John P A Ioannidis², Martin A Tanner³, Sally Cripps¹

Affiliations

¹ Australian Research Council Training Centre in Data Analytics for Resources and Environments, Sydney, New South Wales, Australia; School of Mathematics and Statistics, The University of Sydney, Sydney, New South Wales, Australia.
² Stanford Prevention Research Center, Department of Medicine, Stanford University, Stanford, CA, USA; Department of Epidemiology and Population Health, Stanford University, Stanford, CA, USA; Department of Biomedical Data Sciences, Stanford University, Stanford, CA, USA; Department of Statistics, Stanford University, Stanford, CA, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA. Electronic address: jioannid@stanford.edu.
³ Department of Statistics, Northwestern University, Evanston, IL, USA.

Abstract

Objective: To compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models.

Study design and setting: We explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons.

Results: While model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons.

Conclusion: Inferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.

Keywords: Bayesian statistics; COVID-19; Information criteria; Model comparison; Non-pharmaceutical interventions; SIR models.

MeSH terms

COVID-19 / prevention & control*
Communicable Disease Control / methods*
Europe
Humans
Models, Statistical*
Physical Distancing*
Quarantine / methods*
SARS-CoV-2