Use of Simulation Based on an Electronic Health Records Environment to Evaluate the Structure and Accuracy of Notes Generated by Medical Scribes: Proof-of-Concept Study

Abstract

Background: The increasing adoption of electronic health records (EHRs) has been associated with a number of unintended negative consequences with provider efficiency and job satisfaction. To address this, there has been a dramatic increase in the use of medical scribes to perform many of the required EHR functions. Despite this rapid growth, little has been published on the training or assessment tools to appraise the safety and efficacy of scribe-related EHR activities. Given the number of reports documenting that other professional groups suffer from a number of performance errors in EHR interface and data gathering, scribes likely face similar challenges. This highlights the need for new assessment tools for medical scribes.

Objective: The objective of this study was to develop a virtual video-based simulation to demonstrate and quantify the variability and accuracy of scribes' transcribed notes in the EHR.

Methods: From a pool of 8 scribes in one department, a total of 5 female scribes, intent on pursuing careers in health care, with at least 6 months of experience were recruited for our simulation study. We created three simulated patient-provider scenarios. Each scenario contained a corresponding medical record in our simulation instance of our EHR. For each scenario, we video-recorded a standardized patient-provider encounter. Five scribes with at least 6 months of experience both with our EHR and in the specialty of the simulated cases were recruited. Each scribe watched the simulated encounter and transcribed notes into a simulated EHR environment. Transcribed notes were evaluated for interscribe variability and compared with a gold standard for accuracy.

Results: All scribes completed all simulated cases. There was significant interscribe variability in note structure and content. Overall, only 26% of all data elements were unique to the scribe writing them. The term data element was used to define the individual pieces of data that scribes perceived from the simulation. Note length was determined by counting the number of words varied by 31%, 37%, and 57% between longest and shortest note between the three cases, and word economy ranged between 23% and 71%. Overall, there was a wide inter- and intrascribe variation in accuracy for each section of the notes with ranges from 50% to 76%, resulting in an overall positive predictive value for each note between 38% and 81%.

Conclusions: We created a high-fidelity, video-based EHR simulation, capable of assessing multiple performance indicators in medical scribes. In this cohort, we demonstrate significant variability both in terms of structure and accuracy in clinical documentation. This form of simulation can provide a valuable tool for future development of scribe curriculum and assessment of competency.

Keywords: documentation; electronic health record; simulation training.

Figure 1

Distribution of data elements. Each of the 5 scribes completed 3 separate simulation exercises. The absolute number of data elements for each section of the note was tabulated for each individual scribe. Subjective (Panel A), Physical exam (Panel B), and Assessment and plan (Panel C).

Figure 2

Interscribe commonality in data elements. Each of the 5 scribes completed 3 separate simulation exercises. For each section of the note, Subjective (Panel A), Physical exam (Panel B), and Assessment and plan (Panel C), the fraction of data elements for each scribe in common among the other scribes for all three cases is presented.

Figure 3

Distribution of Word Count. Five scribes each completed 3 separate simulation exercises. The absolute number of words for each section of the note was tabulated for each individual scribe. Subjective (Panel A), Physical Exam (Panel B), and Assessment and Plan (Panel C).