Digitally Diagnosing Multiple Developmental Delays Using Crowdsourcing Fused With Machine Learning: Protocol for a Human-in-the-Loop Machine Learning Study

Abstract

Background: A considerable number of minors in the United States are diagnosed with developmental or psychiatric conditions, potentially influenced by underdiagnosis factors such as cost, distance, and clinician availability. Despite the potential of digital phenotyping tools with machine learning (ML) approaches to expedite diagnoses and enhance diagnostic services for pediatric psychiatric conditions, existing methods face limitations because they use a limited set of social features for prediction tasks and focus on a single binary prediction, resulting in uncertain accuracies.

Objective: This study aims to propose the development of a gamified web system for data collection, followed by a fusion of novel crowdsourcing algorithms with ML behavioral feature extraction approaches to simultaneously predict diagnoses of autism spectrum disorder and attention-deficit/hyperactivity disorder in a precise and specific manner.

Methods: The proposed pipeline will consist of (1) gamified web applications to curate videos of social interactions adaptively based on the needs of the diagnostic system, (2) behavioral feature extraction techniques consisting of automated ML methods and novel crowdsourcing algorithms, and (3) the development of ML models that classify several conditions simultaneously and that adaptively request additional information based on uncertainties about the data.

Results: A preliminary version of the web interface has been implemented, and a prior feature selection method has highlighted a core set of behavioral features that can be targeted through the proposed gamified approach.

Conclusions: The prospect for high reward stems from the possibility of creating the first artificial intelligence-powered tool that can identify complex social behaviors well enough to distinguish conditions with nuanced differentiators such as autism spectrum disorder and attention-deficit/hyperactivity disorder.

International registered report identifier (irrid): PRR1-10.2196/52205.

Keywords: ADHD; ASD; attention-deficit/hyperactivity disorder; autism spectrum disorder; crowdsourcing; machine learning; precision health.

Figure 1

Overview of the proposed crowd-powered diagnostic system comprising adaptive gamified data curation, behavioral feature extraction by both crowd workers and computational workflows, and machine learning models for multicondition classification that also output individual symptom estimates and dynamically query participants based on crowd ratings. Each of these 3 major steps is independent yet can be combined to produce a synergistic improvement in remote and accessible diagnostics for pediatric psychiatry. ADHD: attention-deficit/hyperactivity disorder.

Figure 2

Feature extraction and quantification of behaviors relevant to neuropsychiatric diagnostics. LSTM: Long Short-Term Memory network.

Figure 3

Crowd worker assignment to labeling tasks. Each crowd worker will only be asked to label those features for which they agreed with clinicians during a worker profiling step performed before the primary study.

Figure 4

Preliminary interface for the study’s central web platform. (A) Users who are not a part of the core user study where participant matchmaking occurs can select their game play partner. (B) One of the implemented games, Charades.