Commercial off-the-shelf consumer health informatics interventions: recommendations for their design, evaluation and redesign

Abstract

Objective: The goal of this paper is to describe the successful application of a use case-based evaluation approach to guide the effective design, evaluation and redesign of inexpensive, commercial, off-the-shelf consumer health informatics (CHI) interventions.

Design: Researchers developed four CHI intervention use cases representing two distinct patient populations (patients with diabetes with high blood pressure, post-bariatric surgery patients), two commercial off-the-shelf CHI applications (Microsoft HealthVault, Google Health), and related devices (blood pressure monitor, pedometer, weight scale). Three patient proxies tested each intervention for 10 days.

Measurements: The patient proxies recorded their challenges while completing use case tasks, rating the severity of each challenge based on how much it hindered their use of the intervention. Two independent evaluators categorized the challenges by human factors domain (physical, cognitive, macroergonomic).

Results: The use case-based approach resulted in the identification of 122 challenges, with 12% physical, 50% cognitive and 38% macroergonomic. Thirty-nine challenges (32%) were at least moderately severe. Nine of 22 use case tasks (41%) accounted for 72% of the challenges.

Limitations: The study used two patient proxies and addressed two specific patient populations and low-cost, off-the-shelf CHI interventions, which may not perfectly generalize to a larger number of proxies, actual patient populations, or other CHI interventions.

Conclusion: CHI designers can employ the use case-based evaluation approach to assess the fit of a CHI intervention with patients' health work, in the context of their daily activities and environment, which would be difficult or impossible to evaluate by laboratory-based studies.

Figure 1

Number of challenges identified by the patient proxies for each use case, by human factors domain.