Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Oct 1;31(10):2228-2235.
doi: 10.1093/jamia/ocae171.

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

Seunghwan Kim  1   2 , Benjamin C Warner  3 , Daphne Lew  2 , Sunny S Lou  2   4 , Thomas Kannampallil  2   3   4
Affiliations

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

Seunghwan Kim et al. J Am Med Inform Assoc. .

Abstract

Objectives: To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.

Materials and methods: EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.

Results: Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.

Discussion: We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.

Conclusion: An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.

Keywords: audit log; clinical workflow; cognitive effort; entropy; language model.

PubMed Disclaimer

Conflict of interest statement

None declared.

Similar articles

See all similar articles

References

    1. Jha AK. Meaningful use of electronic health records: the road ahead. JAMA. 2010;304(15):1709-1710. - PubMed
    1. Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health records in US hospitals. N Engl J Med. 2009;360(16):1628-1638. - PubMed
    1. DiAngi YT, Stevens LA, Halpern-Felsher B, et al. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers’ perceived control over their workload in the EHR. JAMIA Open. 2019;2(2):222-230. - PMC - PubMed
    1. Ratanawongsa N, Matta GY, Bohsali FB, et al. Reducing misses and near misses related to multitasking on the electronic health record: observational study and qualitative analysis. JMIR Hum Factors. 2018;5(1):e9371. - PMC - PubMed
    1. Ahmed A, Chandra S, Herasevich V, et al. The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med. 2011;39(7):1626-1634. - PubMed

Publication types

LinkOut - more resources