Objective: Relapse rates are consistently high for stimulant user disorders. In order to obtain prognostic information about individuals in treatment, machine learning models have been applied to neuroimaging and clinical data. Yet few efforts have been made to test these models in independent samples or show that they can outperform linear models. In this exploratory study, we examine whether machine learning models relative to linear models provide greater predictive accuracy and less overfitting.
Method: This longitudinal study included 63 methamphetamine-dependent (training sample) and 29 cocaine-dependent (test sample) individuals who completed an MRI scan during residential treatment. Linear and machine learning models predicting relapse at a one-year follow up that were previously developed in the methamphetamine-dependent sample using neuroimaging and clinical variables were applied to the cocaine-dependent sample. Receiver operating characteristic analysis was used to assess performance using area under the curve (AUC) as the primary outcome.
Results: Twelve individuals in the cocaine-dependent sample remained abstinent, and 17 relapsed. The linear models produced more accurate prediction in the training sample than the machine learning models but showed reduced performance in the testing sample, with AUC decreasing by 0.18. The machine learning models produced similar predictive performance in the training and test samples, with AUC changing by 0.03. In the test sample, neither the linear nor the machine learning model predicted relapse at rates above chance.
Conclusions: Although machine learning algorithms may have advantages, in this study neither model's performance was sufficient to be clinically useful. In order to improve predictive models, stronger predictor variables and larger samples are needed.
Copyright © 2019 The Authors. Published by Elsevier Inc. All rights reserved.