Objective: To determine the extent of explicit attention to formal issues of 'measurement' in studies examining outcomes of the deployment of clinical information systems.
Methods: A structured literature review identified 27 published studies reporting quantitative outcomes including attitudes, clinician behavior, quality of care, and cost of care. These studies were analyzed for types of outcome reported, evidence of 'reuse' of measurement methodology, and evidence of formal study of the reliability and validity of these measures.
Results: The 27 studies meeting the inclusion criteria were published between 1976 and 2002. Several of the studies addressed multiple outcome types. Nine examined clinician attitudes; 22 examined health care behaviors; 15 examined patient health status/quality of care; and four examined economic indicators. There were eight examples of reuse of measurement methods, five of which represented reuse within a single research group. Reliability indices were reported in three studies. There were no reported validity indices.
Conclusion: Based on this sample of studies, specific attention to issues of measurement is sparse in outcome studies of deployed clinical information systems. As such, medical informatics does not appear to be on a par with more mature sciences in its approaches to measurement of key outcome variables.