Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2019 Sep 25;14(9):e0222907.
doi: 10.1371/journal.pone.0222907. eCollection 2019.

A Comparison of Machine Learning Algorithms for the Surveillance of Autism Spectrum Disorder

Affiliations
Free PMC article
Comparative Study

A Comparison of Machine Learning Algorithms for the Surveillance of Autism Spectrum Disorder

Scott H Lee et al. PLoS One. .
Free PMC article

Abstract

Objective: The Centers for Disease Control and Prevention (CDC) coordinates a labor-intensive process to measure the prevalence of autism spectrum disorder (ASD) among children in the United States. Random forests methods have shown promise in speeding up this process, but they lag behind human classification accuracy by about 5%. We explore whether more recently available document classification algorithms can close this gap.

Materials and methods: Using data gathered from a single surveillance site, we applied 8 supervised learning algorithms to predict whether children meet the case definition for ASD based solely on the words in their evaluations. We compared the algorithms' performance across 10 random train-test splits of the data, using classification accuracy, F1 score, and number of positive calls to evaluate their potential use for surveillance.

Results: Across the 10 train-test cycles, the random forest and support vector machine with Naive Bayes features (NB-SVM) each achieved slightly more than 87% mean accuracy. The NB-SVM produced significantly more false negatives than false positives (P = 0.027), but the random forest did not, making its prevalence estimates very close to the true prevalence in the data. The best-performing neural network performed similarly to the random forest on both measures.

Discussion: The random forest performed as well as more recently available models like the NB-SVM and the neural network, and it also produced good prevalence estimates. NB-SVM may not be a good candidate for use in a fully-automated surveillance workflow due to increased false negatives. More sophisticated algorithms, like hierarchical convolutional neural networks, may not be feasible to train due to characteristics of the data. Current algorithms might perform better if the data are abstracted and processed differently and if they take into account information about the children in addition to their evaluations.

Conclusion: Deep learning models performed similarly to traditional machine learning methods at predicting the clinician-assigned case status for CDC's autism surveillance system. While deep learning methods had limited benefit in this task, they may have applications in other surveillance systems.

Conflict of interest statement

The authors have declared that no competing interests exist.

Similar articles

See all similar articles

Cited by 1 article

References

    1. Maenner MJ, Yeargin-Allsopp M, Braun KV, Christensen DL, Schieve LA. Development of a machine learning algorithm for the surveillance of autism spectrum disorder. PLOS ONE. 2016. December 21;11(12):e0168224 10.1371/journal.pone.0168224 - DOI - PMC - PubMed
    1. American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders: DSM—5. Washington, DC: American Psychiatric Association.
    1. Breiman L. Random forests. Machine learning. 2001. October 1;45(1):5–32.
    1. Autism and Developmental Disabilities Monitoring Network Surveillance Year 2008 Principal Investigators. Prevalence of autism spectrum disorders—Autism and Developmental Disabilities Monitoring Network, 14 sites, United States, 2008. MMWR Surveill Summ 2012;61(No. SS-3):1–19. - PubMed
    1. Rice CE, Baio J, Van Naarden Braun K, Doernberg N, Meaney FJ, Kriby RS. A public health collaboration for the surveillance of autism spectrum disorders. Paediatric and Perinatal Epidemiology. 2007. March 1;21(2):179–90. 10.1111/j.1365-3016.2007.00801.x - DOI - PubMed

Publication types

Grant support

The authors received no specific funding for this work.
Feedback