Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks

Abstract

Retinal fundus diseases can lead to irreversible visual impairment without timely diagnoses and appropriate treatments. Single disease-based deep learning algorithms had been developed for the detection of diabetic retinopathy, age-related macular degeneration, and glaucoma. Here, we developed a deep learning platform (DLP) capable of detecting multiple common referable fundus diseases and conditions (39 classes) by using 249,620 fundus images marked with 275,543 labels from heterogenous sources. Our DLP achieved a frequency-weighted average F1 score of 0.923, sensitivity of 0.978, specificity of 0.996 and area under the receiver operating characteristic curve (AUC) of 0.9984 for multi-label classification in the primary test dataset and reached the average level of retina specialists. External multihospital test, public data test and tele-reading application also showed high efficiency for multiple retinal diseases and conditions detection. These results indicate that our DLP can be applied for retinal fundus disease triage, especially in remote areas around the world.

Fig. 1. Examples of preprocessed images and heatmaps.

Typical images with one label, two labels, and three labels were selected and shown for heatmaps. First column: typical preprocessed image of selected images with a resolution of 299 × 299 pixels. Second column: their predictions. Third and fourth columns: the heatmaps (CAM and Deepsharp) indicating important regions with typical features of diseases discovered by DLP for predictions.

Fig. 2. ROC and AUC of DLP for detection of bigclasses in primary test and multihospital test data sets.

a ROC curves and AUC for detecting every bigclass in primary test were calculated and plotted. Different colors of ROC curves corresponding to different AUC of each disease and condition are listed. b ROC curves and AUC for detecting every bigclasses in multihospital test. Source data are provided as a Source Data file.

Fig. 3. Performance of DLP in comparative test data set compare to retina experts.

Examples of ROC curves and AUC for detecting referable DR, rhegmatogenous RD, ERM, optic nerve degeneration, disc swelling and elevation, and cotton-wool spots in comparative test data set were calculated and plotted (blue curves for all images, and green curves for JSIEC images). Performance of individual retina expert (more than 10 years’ clinical experiences in the retina specialty) is indicated by the crosses, and averaged expert performance by dots. Blue and green crosses/dots denote performance without patient information and red crosses/dots denote performance with patient information. Source data are provided as a Source Data file.

Fig. 4. Tele-reading application of the DLP.

a Images were uploaded from seven hospitals for primary care located in different parts of China. b Images were rejected and sent back for repeat photography if merely detected as “Blur fundus” with probabilities equal to or larger than 95%. c–f Referral results were calculated in subject-based. Subjects were regarded as nonreferable if no image was detected as referable, otherwise, they were regarded as referable and sent for retina specialist confirmation. c Images were reviewed by retina specialists at the end of tele-reading application, among which 11 subjects were confirmed to be referable. f 1892 subjects with 3956 images were confirmed as referable, among which 3851 images were classifiable and 105 images could not be categorized as any of the diseases and conditions enlisted in Supplementary Table 1. Subjects were counted as unclassifiable only if all their images were judged as unclassifiable.