Assessing quality of care from hospital case notes: comparison of reliability of two methods

Abstract

Objectives: To determine which of the two methods of case note review provide the most useful and reliable information for reviewing quality of care.

Design: Retrospective, multiple reviews of 692 case notes were undertaken using both holistic (implicit) and criterion-based (explicit) review methods. Quality measures were evidence-based review criteria and a quality of care rating scale.

Setting: Nine randomly selected acute hospitals in England.

Participants: Sixteen doctors, 11 specialist nurses and three clinically trained audit staff, and eight non-clinical audit staff. ANALYSIS METHODS: Intrarater consistency, inter-rater reliability between pairs of staff using intraclass correlation coefficients (ICCs), completeness of criterion data capture and between-staff group comparison.

Results: A total of 1473 holistic reviews and 1389 criterion-based reviews were undertaken. When the three same staff types reviewed the same record, holistic scale score inter-rater reliability was moderate within each group (ICC 0.46 to 0.52). Inter-rater reliability for criterion-based scores was moderate to good (ICC 0.61 to 0.88). Comparison of holistic review score and criterion-based score of case notes reviewed by doctors and by non-clinical audit staff showed a reasonable level of agreement between the two methods.

Conclusions: Using a holistic approach to review case notes, same staff groups can achieve reasonable repeatability within their professional groups. When the same clinical record was reviewed twice by the doctors, and by the non-clinical audit staff, using both holistic and criterion-based methods, there are close similarities between the quality of care scores generated by the two methods. When using retrospective review of case notes to examine quality of care, a clear view is required of the purpose and the expected outputs of the project.