Machine Learning-Based Audiovisual Phenotyping for Measuring Communication, Shared Decision-Making, and Trust

Shely Khaikin; Vineet Tiruvadi; Jeffrey Brooks; Alice Baird; Anne-Catherine Grela-Mpoko; Lindsey Hoffman; Jadyn Crossley; Menachem Leasy; Jaime Fineman; Margot Savoy; Laura Igarabuza; Anuradha Paranjape; Cheryl Ys Foo; Michael L Birnbaum; Yaara Zisman-Ilani

doi:10.2196/85906

Machine Learning-Based Audiovisual Phenotyping for Measuring Communication, Shared Decision-Making, and Trust

JMIR Form Res. 2026 Mar 3:10:e85906. doi: 10.2196/85906.

Authors

Shely Khaikin¹, Vineet Tiruvadi^{2

3}, Jeffrey Brooks³, Alice Baird³, Anne-Catherine Grela-Mpoko¹, Lindsey Hoffman¹, Jadyn Crossley¹, Menachem Leasy⁴, Jaime Fineman⁵, Margot Savoy⁶, Laura Igarabuza⁴, Anuradha Paranjape⁷, Cheryl Ys Foo⁸, Michael L Birnbaum^{9

10}, Yaara Zisman-Ilani^{11

12

13}

Affiliations

¹ Shared Decision Making Laboratory, Temple University, Philadelphia, PA, United States.
² Harvard University, Cambridge, MA, United States.
³ Hume AI, New York, NY, United States.
⁴ Department of Clinical Family and Community Medicine, Lewis Katz School of Medicine, Temple University, Philadelphia, PA, United States.
⁵ Department of Clinical Medicine, Lewis Katz School of Medicine, Temple University, Philadelphia, PA, United States.
⁶ American Academy of Family Physicians, Washington, DC, United States.
⁷ University of Colorado School of Medicine, Aurora, CO, United States.
⁸ Center of Excellence for Psychosocial and Systemic Research, Department of Psychiatry, Massachusetts General Hospital, Boston, MA, United States.
⁹ New York State Psychiatric Institute, New York, NY, United States.
¹⁰ Department of Psychiatry, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY, United States.
¹¹ Department of Clinical, Educational and Health Psychology, Division of Psychology and Language Sciences, University College London, 1-19 Torrington Place, London, WC1E 7Hb, United Kingdom.
¹² Department of Social and Behavioral Sciences, Barnett College of Public Health, Temple University, Philadelphia, PA, United States.
¹³ Department of Psychiatry and Behavioral Sciences, Lewis Katz School of Medicine, Temple University, Philadelphia, PA, United States.

PMID: 41823725
PMCID: PMC12984015
DOI: 10.2196/85906

Abstract

Machine learning-based audiovisual phenotyping can reveal hidden discrepancies between patients' self-reported experiences and nonverbal expressions, offering a promising tool for objectively assessing communication quality and advancing health equity.

Keywords: AI; artificial intelligence; audiovisual digital phenotyping; depression; natural language processing; primary care; shared decision-making.

© Shely Khaikin, Vineet Tiruvadi, Jeffrey Brooks, Alice Baird, Anne-Catherine Grela-Mpoko, Lindsey Hoffman, Jadyn Crossley, Menachem Leasy, Jaime Fineman, Margot Savoy, Laura Igarabuza, Anuradha Paranjape, Cheryl YS Foo, Michael L Birnbaum, Yaara Zisman-Ilani. Originally published in JMIR Formative Research (https://formative.jmir.org).

MeSH terms

Adult
Communication*
Decision Making, Shared*
Female
Humans
Machine Learning*
Male
Middle Aged
Phenotype
Trust* / psychology