Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI

David B Larson; Florence X Doo; Bibb Allen Jr; John Mongan; Adam E Flanders; Christoph Wald

doi:10.1016/j.jacr.2024.01.024

Proceedings From the 2022 ACR-RSNA Workshop on Safety, Effectiveness, Reliability, and Transparency in AI

J Am Coll Radiol. 2024 Jul;21(7):1119-1129. doi: 10.1016/j.jacr.2024.01.024. Epub 2024 Feb 13.

Authors

David B Larson¹, Florence X Doo², Bibb Allen Jr³, John Mongan⁴, Adam E Flanders⁵, Christoph Wald⁶

Affiliations

¹ Executive Vice Chair, Department of Radiology, Stanford University Medical Center, Stanford, California; Chair, Quality and Safety Commission, ACR; and Member, ACR Board of Chancellors. Electronic address: david.larson@stanford.edu.
² Director of Innovation, University of Maryland Medical Intelligent Imaging (UM2ii) Center, Baltimore, Marlyand. Electronic address: https://twitter.com/flo_doo.
³ Department of Radiology, Grandview Medical Center, Birmingham, Alabama; and Chief Medical Officer, ACR Data Science Institute. Electronic address: https://twitter.com/bibballen.
⁴ Associate Chair for Translational Informatics and Director of the Center for Intelligent Imaging, Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, California. Electronic address: https://twitter.com/MonganMD.
⁵ Vice Chair for Informatics, Department of Radiology, Thomas Jefferson University, Philadelphia, Pennsylvania; and Member of the RSNA Board of Directors. Electronic address: https://twitter.com/BFlanksteak.
⁶ Chair, Department of Radiology, Lahey Hospital and Medical Center, Boston, Massachusetts; Chair, Informatics Commission, ACR; and Member of the ACR Board of Chancellors. Electronic address: https://twitter.com/waldchristoph.

PMID: 38354844
DOI: 10.1016/j.jacr.2024.01.024

Abstract

Despite the surge in artificial intelligence (AI) development for health care applications, particularly for medical imaging applications, there has been limited adoption of such AI tools into clinical practice. During a 1-day workshop in November 2022, co-organized by the ACR and the RSNA, participants outlined experiences and problems with implementing AI in clinical practice, defined the needs of various stakeholders in the AI ecosystem, and elicited potential solutions and strategies related to the safety, effectiveness, reliability, and transparency of AI algorithms. Participants included radiologists from academic and community radiology practices, informatics leaders responsible for AI implementation, regulatory agency employees, and specialty society representatives. The major themes that emerged fell into two categories: (1) AI product development and (2) implementation of AI-based applications in clinical practice. In particular, participants highlighted key aspects of AI product development to include clear clinical task definitions; well-curated data from diverse geographic, economic, and health care settings; standards and mechanisms to monitor model reliability; and transparency regarding model performance, both in controlled and real-world settings. For implementation, participants emphasized the need for strong institutional governance; systematic evaluation, selection, and validation methods conducted by local teams; seamless integration into the clinical workflow; performance monitoring and support by local teams; performance monitoring by external entities; and alignment of incentives through credentialing and reimbursement. Participants predicted that clinical implementation of AI in radiology will continue to be limited until the safety, effectiveness, reliability, and transparency of such tools are more fully addressed.

Keywords: AI implementation; AI performance; artificial intelligence; clinical AI; imaging AI.

MeSH terms

Artificial Intelligence*
Diagnostic Imaging
Humans
Patient Safety
Radiology*
Reproducibility of Results
Societies, Medical
United States