The reliability of simulated clients' quality-of-care ratings

Stud Fam Plann. May-Jun 1994;25(3):184-90.


Evaluators of family planning programs have begun to use simulated client ratings to assess the quality of services. However, little is known about the reliability of such ratings when they are used to assess individual provider performance. This study examined the reliability of quality-of-care ratings in a Peruvian community-based distribution program by using pairs of concealed observers--a simulated client and a companion. Average interrater agreement, measured by intraclass correlation, was .50, indicating that ratings are not reliable enough for the evaluation of a single provider by a single rater. The study results suggest that checklist-item scores referring to specific provider behaviors will be more reliable and useful than ratings.

Publication types

  • Research Support, Non-U.S. Gov't
  • Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

  • Adolescent
  • Adult
  • Decision Trees
  • Family Planning Services*
  • Female
  • Health Knowledge, Attitudes, Practice*
  • Humans
  • Middle Aged
  • Observer Variation
  • Patient Satisfaction*
  • Program Evaluation / methods*
  • Quality of Health Care*
  • Reproducibility of Results