Interpretable classification of Alzheimer's disease pathologies with a convolutional neural network pipeline

Abstract

Neuropathologists assess vast brain areas to identify diverse and subtly-differentiated morphologies. Standard semi-quantitative scoring approaches, however, are coarse-grained and lack precise neuroanatomic localization. We report a proof-of-concept deep learning pipeline that identifies specific neuropathologies-amyloid plaques and cerebral amyloid angiopathy-in immunohistochemically-stained archival slides. Using automated segmentation of stained objects and a cloud-based interface, we annotate > 70,000 plaque candidates from 43 whole slide images (WSIs) to train and evaluate convolutional neural networks. Networks achieve strong plaque classification on a 10-WSI hold-out set (0.993 and 0.743 areas under the receiver operating characteristic and precision recall curve, respectively). Prediction confidence maps visualize morphology distributions at high resolution. Resulting network-derived amyloid beta (Aβ)-burden scores correlate well with established semi-quantitative scores on a 30-WSI blinded hold-out. Finally, saliency mapping demonstrates that networks learn patterns agreeing with accepted pathologic features. This scalable means to augment a neuropathologist's ability suggests a route to neuropathologic deep phenotyping.

Fig. 1

Ways of assessing neuropathologies in human tissues. a Current protocols for neuropathological assessment of WSIs typically rely on comparatively coarse-grained semi-quantitative scoring such as CERAD. b We report an automated computational approach to process entire digitized immunohistochemical stained archival slides, leveraging convolutional neural networks for amyloid plaque classification and localization

Fig. 2

Creation of an annotated plaque and CAA dataset for machine learning. a Summary of the image processing pipeline, including color normalization, IHC stain segmentation, and extraction of candidate objects (red boxes), followed by rapid expert annotation using a cloud-based web application. b Examples of extracted cored plaques (top row), diffuse plaques (middle), and CAA (bottom) and their surrounding tissue area. Rectangles shown in red bound the candidate object during the labeling process. Scale bar = 25 μm. c A custom web interface allows for the rapid annotation of plaques by mouse or keystroke, with visualization of raw (without color adjustments) and normalized images, showing the object bounding boxes around which the tile is automatically centered and cropped

Fig. 3

CNN models identify three Aβ deposit types in image tiles. a The optimized CNN model architecture contained six convolutional layers and two dense layers, using exclusively 3 × 3 kernels and alternating max-pooling layers. b Examples of correct CNN predictions. The ground truth expert label row indicates the pathologies that had been manually found within the tile image. The predicted row shows corresponding model confidences for cored plaque (yellow arrow), diffuse plaque (red), and CAA (blue) classes (from left to right). Model predictions range from 0.00 to 1.00, where a higher score indicates higher predicted confidence by the CNN for that plaque class (e.g., the 1.00 corresponds to 100% model confidence that a cored plaque is present in the leftmost panel). c Examples of CNN predictions that do not agree with the expert manual annotation. Incorrect model predictions are indicated by light orange backgrounds in the predicted column; green backgrounds correspond to correct predictions. Scale bar = 25 μm for all images

Fig. 4

Predictive performance on the held-out Phase-II test set (n = 10,873). a Receiver operator characteristic (ROC) and b Precision-recall curves (PRC) for cored (magenta lines) and diffuse (blue lines) plaques. The blue star marks the best trade-off point where prediction confidence threshold equals 0.91. c Summarized areas under the ROC and PRC (AUPRC and AUROC) of independently-trained CNNs (n = 5 per point) for the task of cored plaque classification, as a function of training dataset size. Dataset was randomly subsetted at each point independent of the date of tile annotation. Error bars represent s.d. d AUPRC and AUROC of CNNs for the same task of cored-plaque classification, as a function of chronological dataset growth by annotation timestamp, over the course of the project, showing chronology-dependent dataset effects. Source data are provided as a Source Data file

Fig. 5

Prediction confidence heatmaps for cored plaques, diffuse plaques, and CAA. a Whole slide overview visualization, revealing broad amyloid distribution patterns. Scale bar = 3 mm. b Higher magnification (×4) view of the blue-boxed region from panel a. Scale bar = 750 μm. c Higher magnification (×20) view of the blue-boxed region from panel b. Green box marks cored plaque manual annotation. Scale bar = 150 μm. Confidence scales for each panel were bottom-capped to aid visualization, such that only confidence scores ≥ 0.8 are plotted, with yellow being the most confidence, and purple the least. Approximate (hand-drawn) boundaries of gray versus white matter (dotted line) and tissue boundaries (solid lines) are overlaid for reference

Fig. 6

Visualization of a representative example from the cored-plaque classification tests plotted in Fig. 4. a CNN model prediction confidence maps (middle panel, as in Fig. 5c) overlaid onto the original slide tile. Bounding boxes mark cored-plaque expert annotations (left panel, green box). The combined map (right panel) assesses agreement between the model’s predictions and the expert labels, where pixels are colored by a semi-transparent overlay as true positive (blue), false positive (orange), true negative (cyan), and false negative (red) areas. Scale bar = 150 μm. b Prediction-versus-annotation agreement map generated as in a, but with a larger field for greater tissue and plaque clustering context. Scale bar = 750 μm. Confidence scales for middle panel are bottom-capped to aid visualization, such that only confidence scores ≥ 0.8 are plotted, with yellow being the most confident and purple the least

Fig. 7

Model interpretability studies using machine-learning introspection techniques. a A cored plaque example (top row, yellow arrow). For the task of cored-plaque prediction, the activation map (by Guided Grad-CAM; left, second row) and the feature occlusion map (right, second row) identify the amyloid core (yellow arrow) as the defining morphological feature. By contrast, the diffuse stained region (red arrow) only arises as a salient feature during diffuse-plaque and CAA prediction tasks (third and fourth rows, respectively). b Diffuse plaque example where activation and feature occlusion maps focus on ill-defined amorphous amyloid contours for diffuse-plaque classification task (third row). c CAA example, where the CAA task’s activation and feature occlusion maps (fourth row) highlight amyloid ring pixels within the media of the cortical vessel (blue arrow), while for cored and diffuse tasks the small punctate IHC staining is considered salient (red arrow; second and third rows). d Example containing both diffuse (red arrow) and cored (yellow arrow) plaques in the same tile illustrate the difference between activation and feature occlusion maps. Confidence scales for feature occlusion maps represent the CNN’s prediction confidence on the occluded image, with red being the most confident and blue the least. Scale bar = 25 μm

Fig. 8

Comparison of CNN-based Aβ-burden scores versus manual CERAD-like semi-quantitative scores at a whole-slide level for each pathology. a The automatic and manual scores correlate well across the entire dataset of 62 independent WSIs, comprising the original Phase I-III slide set plus 20 additional blinded WSIs. b Correlations assessed on the 20 blinded WSIs not used in any previous step of the study combined with the 10 WSIs from the original hold-out set, for a total of n = 30 individual cases. Box plots show median (center line inside the box), interquartile range (IQR, bounds of box), minimums and maximums within 1.5 times the IQR (whiskers), and outliers (points beyond the whiskers), with a dot per WSI. p ≥ 0.05 was considered not significant (ns); *p < 0.05, **p < 0.01, ***p < 0.001. Matrices in the second row of each panel exhaustively plot p-values of CNN-based score distributions between all pairs of CERAD-like categories for the corresponding box plot, where squares are colored in log scale by p < 1e−4 (blue) to p = 0.05 (gray) to p = 1 (red; insignificant) using two-sided Student’s t-tests. Source data are provided as a Source Data file