Linear regression metamodeling as a tool to summarize and present simulation model results

Abstract

Background / objective: Modelers lack a tool to systematically and clearly present complex model results, including those from sensitivity analyses. The objective was to propose linear regression metamodeling as a tool to increase transparency of decision analytic models and better communicate their results.

Methods: We used a simplified cancer cure model to demonstrate our approach. The model computed the lifetime cost and benefit of 3 treatment options for cancer patients. We simulated 10,000 cohorts in a probabilistic sensitivity analysis (PSA) and regressed the model outcomes on the standardized input parameter values in a set of regression analyses. We used the regression coefficients to describe measures of sensitivity analyses, including threshold and parameter sensitivity analyses. We also compared the results of the PSA to deterministic full-factorial and one-factor-at-a-time designs.

Results: The regression intercept represented the estimated base-case outcome, and the other coefficients described the relative parameter uncertainty in the model. We defined simple relationships that compute the average and incremental net benefit of each intervention. Metamodeling produced outputs similar to traditional deterministic 1-way or 2-way sensitivity analyses but was more reliable since it used all parameter values.

Conclusions: Linear regression metamodeling is a simple, yet powerful, tool that can assist modelers in communicating model characteristics and sensitivity analyses.

Keywords: cost-effectiveness; decision analysis; design of experiments; economic evaluation; metamodel; regression; sensitivity analysis.

Figure 1:. Metamodel as an add-on to simulation models.

Simulation models are approximations of real life complexity. Metamodels simplify and increase model transparency by summarizing the relationship between model inputs and outputs.

Figure 2:. One-way threshold analysis from regression results.

The standardized values of x_j are plotted on the X axis, and the NHB is plotted on the Y axis. The horizontal line (−Δθ₀) is equal to the negative expected value of the incremental NHB. The grayed out region represents the values of the standardized parameter z_j at which the preferred decision is reversed. $z_j^{*}$ is the value at which the decision changes and it is a function of the intercept (θ₀) and (θ_j), the coefficient of z_j, in the incremental NHB (ΔNHB) regression. The region of z_j where T₁ is preferred over T₀ is conditional on the direction of Δθ_j which indicates the direction in which ΔNHB changes relative to z_j. If Δθ_j > 0, meaning that ΔNHB increases as z_j increases, then T₁ becomes preferred over T₀ where $z_j>z_j^{*}$. However, if Δθ_j is a negative value, then T₁ must be preferred over T₀ in regions where $z_j<z_j^{*}$.

Figure 3:. Decision tree outline.

This tree involves treating a hypothetical case of cancer with chemotherapy, radiation, or surgery. Each decision branch is followed by a Markov node that computes the lifetime cost and benefit of the simulated patient.

Figure 4:. Sensitivity analysis design.

This figure shows how two parameter values are sampled in different types of simulation designs. Cancer mortality rate (μCancer) is plotted on the X axis and the probability of failing chemotherapy (pFailChemo) is plotted on the Y axis. In probabilistic sensitivity analysis (PSA), 1000 parameter pairs are sampled randomly from their distributions (Table 1). In deterministic designs, the analyst chooses the input values. In 2^K full-factorial design, the inputs are sampled from the edges, or corners, of the parameter space. In contrast, in one-factor-at-a-time design, only one parameter is varied at a time, the other parameter is held at its mean value.

Figure 5:. Two-way threshold analysis from regression results.

Cancer mortality rate (μCancer) is plotted on the X axis and the probability of failing chemotherapy (pFailChemo) is plotted on the Y axis. The ranges of these axes represent the parameters 95% confidence interval. The graph shows the optimal strategies for each combination of input parameters, while the rest of the parameters in the model are equal to their mean values.