Utilization of Machine Learning-Based Computer Vision and Voice Analysis to Derive Digital Biomarkers of Cognitive Functioning in Trauma Survivors

Katharina Schultebraucks; Vijay Yadav; Isaac R Galatzer-Levy

doi:10.1159/000512394

Utilization of Machine Learning-Based Computer Vision and Voice Analysis to Derive Digital Biomarkers of Cognitive Functioning in Trauma Survivors

Digit Biomark. 2020 Dec 30;5(1):16-23. doi: 10.1159/000512394. eCollection 2021 Jan-Apr.

Authors

Katharina Schultebraucks^{1

2

3}, Vijay Yadav⁴, Isaac R Galatzer-Levy^{2

4}

Affiliations

¹ Vagelos School of Physicians and Surgeons, Department of Emergency Medicine, Columbia University Medical Center, New York, New York, USA.
² Department of Psychiatry, New York University School of Medicine, New York, New York, USA.
³ Data Science Institute, Columbia University, New York, New York, USA.
⁴ AiCure, New York, New York, USA.

Abstract

Background: Alterations in multiple domains of cognition have been observed in individuals who have experienced a traumatic stressor. These domains may provide important insights in identifying underlying neurobiological dysfunction driving an individual's clinical response to trauma. However, such assessments are burdensome, costly, and time-consuming. To overcome barriers, efforts have emerged to measure multiple domains of cognitive functioning through the application of machine learning (ML) models to passive data sources.

Methods: We utilized automated computer vision and voice analysis methods to extract facial, movement, and speech characteristics from semi-structured clinical interviews in 81 trauma survivors who additionally completed a cognitive assessment battery. A ML-based regression framework was used to identify variance in visual and auditory measures that relate to multiple cognitive domains.

Results: Models derived from visual and auditory measures collectively accounted for a large variance in multiple domains of cognitive functioning, including motor coordination (R² = 0.52), processing speed (R² = 0.42), emotional bias (R² = 0.52), sustained attention (R² = 0.51), controlled attention (R² = 0.44), cognitive flexibility (R² = 0.43), cognitive inhibition (R² = 0.64), and executive functioning (R² = 0.63), consistent with the high test-retest reliability of traditional cognitive assessments. Face, voice, speech content, and movement have all significantly contributed to explaining the variance in predicting functioning in all cognitive domains.

Conclusions: The results demonstrate the feasibility of automated measurement of reliable proxies of cognitive functioning through low-burden passive patient evaluations. This makes it easier to monitor cognitive functions and to intervene earlier and at a lower threshold without requiring a time-consuming neurocognitive assessment by, for instance, a licensed psychologist with specialized training in neuropsychology.

Keywords: Cognitive functioning; Computer vision; Deep learning; Digital biomarkers; Emergency department; Voice analysis.