An evaluation and update of methods for estimating the number of influenza cases averted by vaccination in the United States

Abstract

Introduction: To evaluate the public health benefit of yearly influenza vaccinations, CDC estimates the number of influenza cases and hospitalizations averted by vaccine. Available input data on cases and vaccinations is aggregated by month and the estimation model is intentionally simple, raising concerns about the accuracy of estimates.

Methods: We created a synthetic dataset with daily counts of influenza cases and vaccinations, calculated "true" averted cases using a reference model applied to the daily data, aggregated the data by month to simulate data that would actually be available, and evaluated the month-level data with seven test methods (including the current method). Methods with averted case estimates closest to the reference model were considered most accurate. To examine their performance under varying conditions, we re-evaluated the test methods when synthetic data parameters (timing of vaccination relative to cases, vaccination coverage, infection rate, and vaccine effectiveness) were varied over wide ranges. Finally, we analyzed real (i.e., collected by surveillance) data from 2010 to 2017 comparing the current method used by CDC with the best-performing test methods.

Results: In the synthetic dataset (population 1 million persons, vaccination uptake 55%, seasonal infection risk without vaccination 12%, vaccine effectiveness 48%) the reference model estimated 28,768 averted cases. The current method underestimated averted cases by 9%. The two best test methods estimated averted cases with <1% error. These two methods also worked well when synthetic data parameters were varied over wide ranges (≤6.2% error). With the real data, these two methods estimated numbers of averted cases that are a median 8% higher than the currently-used method.

Conclusions: We identified two methods for estimating numbers of influenza cases averted by vaccine that are more accurate than the currently-used algorithm. These methods will help us to better assess the benefits of influenza vaccination.

Keywords: Compartment model; Influenza epidemiology; Influenza vaccine; Prevented cases; Vaccine effectiveness.

Fig. 1.

Reference model for calculation of numbers of cases averted by influenza vaccination. Solid-line ovals are prevalent counts for a given time period. Rectangles are incident counts during the time period. Persons stay in the dotted-line oval (B) for an immune lag period (generally 14 days), during which they are susceptible to infection and at the end of which they become either susceptible (C) or immune (D). Lower-case “b” indicates those vaccinated on individual days; b_d-14 denotes the number vaccinated 14 days before. Abbreviations: ve, vaccine effectiveness; r, infection risk; v, vaccination rate (see Supplemental Table 1).

Fig. 2.

Simulated data used to test estimation methods. A, incident daily counts of vaccinations and cases simulated using the normal probability density distribution. B, daily data aggregated into months was used for test methods 1–6. C, monthly aggregated data was divided by number of days per month to create a proxy for daily data, forming a step function, and used for test method 7.