Development and Evaluation of an Electronic Health Record-Generated Clinical Coverage Scoring System Compared to Human Decision-Making in Pediatric Surgical Patients: A Single Center Experience

Abstract

Background: Surgical Patients in tertiary care centers can be healthy or extremely ill with comorbidities, and procedures vary from simple to difficult and complicated. High-acuity cases, or those involving anesthesia and procedural complexities, require specific anesthesia staff arrangements, specific nursing team assignments, and additional support staff. These cases are identified manually, and there is no ready-to-use scoring system to stratify high-acuity, complex pediatric surgical patients.

Aims: We aim to develop an electronic medical record-generated rule-based clinical coverage scoring system to identify high-acuity, complex cases and compare its accuracy with human performance.

Methods: In this quality improvement project, an automated scoring system using rule-based clinical criteria was designed and implemented in a quaternary children's hospital. These rules were based on patient characteristics, procedure and anesthetic complexity, and the patient's acute condition. The cases with clinical coverage scores higher than zero were compared to those manually identified as high-acuity, complex cases by the anesthesia clinical directors and operating room charge nurses. The accuracy was reported using sensitivity, specificity, PPV, NPV, accuracy, and F-1 scores.

Results: There were 10 761 pediatric surgical cases during the study period (April 7-September 8, 2023). 1450 (13.5%) cases were manually identified as high-acuity, complex cases, while the automated system identified 1906 (17.7%) cases. The accuracy of the automated scoring system improved over time. Eventually, it became better than manual identification with 95.86% (94.48%-97.24%) sensitivity, 99.84% (99.71%-99.98%) specificity, 99.35% (98.78%-99.92%) PPV, 98.97% (98.62%-99.32%) NPV, and 99.04% (98.62%-99.47%) accuracy by the end of the study period. The most impactful interventions were removing canceled cases and adding procedure codes to the rules for automated scores.

Conclusion: EHR-generated clinical coverage scores can reliably replace manual reviews of high-acuity, complex pediatric surgical patients. This tool can guide clinical decision-making in real time.

Keywords: accuracy; anesthetic complexity; automated scoring system; patient acuity; pediatric surgical patients; surgical complexity.