Machine Learning for Automation of Radiology Protocols for Quality and Efficiency Improvement

J Am Coll Radiol. 2020 Sep;17(9):1149-1158. doi: 10.1016/j.jacr.2020.03.012. Epub 2020 Apr 9.


Purpose: The aim of this study was to enhance multispecialty CT and MRI protocol assignment quality and efficiency through development, testing, and proposed workflow design of a natural language processing (NLP)-based machine learning classifier.

Methods: NLP-based machine learning classification models were developed using order entry input data and radiologist-assigned protocols from more than 18,000 unique CT and MRI examinations obtained during routine clinical use. k-Nearest neighbor, random forest, and deep neural network classification models were evaluated at baseline and after applying class frequency and confidence thresholding techniques. To simulate performance in real-world deployment, the model was evaluated in two operating modes in combination: automation (automated assignment of the top result) and clinical decision support (CDS; top-three protocol suggestion for clinical review). Finally, model-radiologist discordance was subjectively reviewed to guide explainability and safe use.

Results: Baseline protocol assignment performance achieved weighted precision of 0.757 to 0.824. Simulating real-world deployment using combined thresholding techniques, the optimized deep neural network model assigned 69% of protocols in automation mode with 95% accuracy. In the remaining 31% of cases, the model achieved 92% accuracy in CDS mode. Analysis of discordance with subspecialty radiologist labels revealed both more and less appropriate model predictions.

Conclusions: A multiclass NLP-based classification algorithm was designed to drive local operational improvement in CT and MR radiology protocol assignment at subspecialist quality. The results demonstrate a simulated workflow deployment enabling automated assignment of protocols in nearly 7 of 10 cases with very few errors combined with top-three CDS for remaining cases supporting a high-quality, efficient radiology workflow.

Keywords: Protocols; automation; machine learning; natural language processing; quality improvement.

MeSH terms

  • Automation*
  • Machine Learning*
  • Natural Language Processing
  • Neural Networks, Computer
  • Radiology*