When and how to evaluate health information systems?

Int J Med Inform. 2003 Mar;69(2-3):251-9. doi: 10.1016/s1386-5056(02)00108-9.


Aims: Evaluating large scale health information systems (HIS) such as hospital systems can be difficult. This article discusses the reasons we need to evaluate these systems and a range of appropriate methods to carry out evaluations. It is written in non-technical language to assist health policy makers and others commissioning or implementing such systems, with references and a web site containing information for those wishing more detail (http://www.ucl.ac.uk/kmc/evaluation/index.html).

Methods: A variety of questions relevant to HIS and qualitative and quantitative methods ranging from simple before-after to controlled before-after and fully randomised designs, are discussed. A running example--evaluating the impact of an order communications system on lab requests--is used to illustrate the potential problems, and how they can be resolved.

Results: The main types of biases affecting impact studies and methods to reduce them are described. The article then discusses some trade-offs between the low cost, easily conducted before-after study with its unreliable results versus the more complex, expensive but much more rigorous randomised trial.

Conclusions: As would be expected, the correct methods to evaluate depend not on what technology is being evaluated--whether an information system or a drug--but on the questions the study is designed to answer, and how reliable the answers must be. Only those commissioning an evaluation study can decide these.

MeSH terms

  • Hospital Information Systems / standards*
  • Humans
  • Program Evaluation / methods*
  • Research Design
  • Technology Assessment, Biomedical*