Using deep learning to predict anti-PD-1 response in melanoma and lung cancer patients from histopathology images

Abstract

Background: Recent studies showed that immune-checkpoint blockade (ICB) has significantly improved clinical outcomes of melanoma and lung cancer patients. However, only a small subset of patients can benefit from ICB. Deep learning has been successfully implemented in complementary clinical diagnosis. The aim of this study is to demonstrate the potential of deep learning to facilitate the prediction of anti-PD-1 response from H&E images directly.

Methods: In this study, 190 H&E slides of melanoma were segmented into 256 × 256 tiles which were used as the training set for the convolutional neural network (CNN). Additional 54 melanoma and 55 lung cancer H&E slides were collected as independent testing sets.

Findings: An AUC of 0.778(95% CI: 63.8%-90.5%) was achieved for 54 melanoma testing samples with 15(65.2%) responders and 23(74.2%) non-responders correctly classified. We also obtained an AUC of 0.645(95% CI: 49.4%-78.4%) for 55 lung cancer samples.

Interpretation: To our knowledge, this is the first study of using deep learning to determine patients' anti-PD-1 response from H&E slides directly. Our CNN model achieved the state-of-the-art performance and has the potential to screen ICB beneficial patients in routine clinical practice.

Keywords: Deep learning; H&E slides; Immunotherapy.

Graphical abstract

Fig. 1

anti-PD-1 response prediction by H&E histology images. Training phase of the deep learning model. Left, Tumor regions were annotated by two pathologists with green polygon border. Tumor regions were segmented and color normalized for downstream analysis. The multi-scale LBP and AP algorithms were applied on gray-scaled tiles. Right, features were extracted by transfer learning using the Xception model and reduced features were fed into SVM for final classification. Testing phase uses the trained model from the training phase to predict clinical outcomes of unseen samples.

Fig. 2

Prediction performance on the validation datasets (A) Area under the curves (AUC) of melanoma testing set (n = 54). (B) Progression-free survival of patients separated by responders and non-responders in melanoma. (C) A waterfall plot of prediction probability score of melanoma samples. (D) AUC curves of lung cancer data set (n = 55). (E) Difference in progression-free survival of lung patients in responders and non-responders. (F) A waterfall plot of prediction probability score of lung cancer patients.

Fig. 3

Examples of TILs from whole slide images of responder and non-responder. Left, a responder example with TILs labeled as red points and tissue regions colored in blue on the masked figure. Right, a non-responder example with TILs labeled as red points and tissue regions colored in blue on the masked figure. Intermediate, randomly selected regions from each slide. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)