Using deep learning for dermatologist-level detection of suspicious pigmented skin lesions from wide-field images

Luis R Soenksen; Timothy Kassis; Susan T Conover; Berta Marti-Fuster; Judith S Birkenfeld; Jason Tucker-Schwartz; Asif Naseem; Robert R Stavert; Caroline C Kim; Maryanne M Senna; José Avilés-Izquierdo; James J Collins; Regina Barzilay; Martha L Gray

doi:10.1126/scitranslmed.abb3652

Using deep learning for dermatologist-level detection of suspicious pigmented skin lesions from wide-field images

Sci Transl Med. 2021 Feb 17;13(581):eabb3652. doi: 10.1126/scitranslmed.abb3652.

Authors

Luis R Soenksen^{1

2

3

4

5}, Timothy Kassis⁶, Susan T Conover², Berta Marti-Fuster^{2

5}, Judith S Birkenfeld^{2

5}, Jason Tucker-Schwartz^{2

5}, Asif Naseem^{2

5}, Robert R Stavert^{7

8

9}, Caroline C Kim^{10

11}, Maryanne M Senna^{9

12}, José Avilés-Izquierdo¹³, James J Collins^{2

3

4

6

14

15}, Regina Barzilay^{16

17}, Martha L Gray^{2

4

5

17}

Affiliations

¹ Department of Mechanical Engineering, Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA. soenksen@mit.edu.
² Institute for Medical Engineering and Science, Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA.
³ Wyss Institute for Biologically Inspired Engineering, Harvard University, 3 Blackfan Cir, Boston, MA 02115, USA.
⁴ Harvard-MIT Program in Health Sciences and Technology, Cambridge, MA 02139, USA.
⁵ MIT linQ, Massachusetts Institute of Technology Cambridge, MA 02148, USA.
⁶ Department of Biological Engineering, Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, 02139, MA, USA.
⁷ Division of Dermatology, Cambridge Health Alliance, 1493 Cambridge Street, Cambridge, MA 02139, USA.
⁸ Department of Dermatology, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Boston, MA 02215, USA.
⁹ Department of Dermatology, Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA.
¹⁰ Pigmented Lesion Program, Newton Wellesley Dermatology Associates, 65 Walnut Street Suite 520 Wellesley Hills, MA 02481, USA.
¹¹ Department of Dermatology, Tufts Medical Center, 260 Tremont Street Biewend Building, Boston, MA 02116, USA.
¹² Department of Dermatology, Massachusetts General Hospital, 55 Fruit St, Boston, MA 02114.
¹³ Department of Dermatology, Hospital General Universitario Gregorio Marañón, Calle del Dr. Esquerdo 46, 28007 Madrid, Spain.
¹⁴ Broad Institute of MIT and Harvard, Cambridge, MA 02142, USA.
¹⁵ School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA.
¹⁶ Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA.
¹⁷ Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology Cambridge, MA 02148, USA.

PMID: 33597262
DOI: 10.1126/scitranslmed.abb3652

Abstract

A reported 96,480 people were diagnosed with melanoma in the United States in 2019, leading to 7230 reported deaths. Early-stage identification of suspicious pigmented lesions (SPLs) in primary care settings can lead to improved melanoma prognosis and a possible 20-fold reduction in treatment cost. Despite this clinical and economic value, efficient tools for SPL detection are mostly absent. To bridge this gap, we developed an SPL analysis system for wide-field images using deep convolutional neural networks (DCNNs) and applied it to a 38,283 dermatological dataset collected from 133 patients and publicly available images. These images were obtained from a variety of consumer-grade cameras (15,244 nondermoscopy) and classified by three board-certified dermatologists. Our system achieved more than 90.3% sensitivity (95% confidence interval, 90 to 90.6) and 89.9% specificity (89.6 to 90.2%) in distinguishing SPLs from nonsuspicious lesions, skin, and complex backgrounds, avoiding the need for cumbersome individual lesion imaging. We also present a new method to extract intrapatient lesion saliency (ugly duckling criteria) on the basis of DCNN features from detected lesions. This saliency ranking was validated against three board-certified dermatologists using a set of 135 individual wide-field images from 68 dermatological patients not included in the DCNN training set, exhibiting 82.96% (67.88 to 88.26%) agreement with at least one of the top three lesions in the dermatological consensus ranking. This method could allow for rapid and accurate assessments of pigmented lesion suspiciousness within a primary care visit and could enable improved patient triaging, utilization of resources, and earlier treatment of melanoma.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Deep Learning*
Dermatologists
Humans
Melanoma* / diagnostic imaging
Sensitivity and Specificity
Skin Neoplasms* / diagnostic imaging