How Convolutional Neural Networks Diagnose Plant Disease

Abstract

Deep learning with convolutional neural networks (CNNs) has achieved great success in the classification of various plant diseases. However, a limited number of studies have elucidated the process of inference, leaving it as an untouchable black box. Revealing the CNN to extract the learned feature as an interpretable form not only ensures its reliability but also enables the validation of the model authenticity and the training dataset by human intervention. In this study, a variety of neuron-wise and layer-wise visualization methods were applied using a CNN, trained with a publicly available plant disease image dataset. We showed that neural networks can capture the colors and textures of lesions specific to respective diseases upon diagnosis, which resembles human decision-making. While several visualization methods were used as they are, others had to be optimized to target a specific layer that fully captures the features to generate consequential outputs. Moreover, by interpreting the generated attention maps, we identified several layers that were not contributing to inference and removed such layers inside the network, decreasing the number of parameters by 75% without affecting the classification accuracy. The results provide an impetus for the CNN black box users in the field of plant science to better understand the diagnosis process and lead to further efficient use of deep learning for plant disease diagnosis.

Figure 1

Image-based disease diagnosis training using convolutional neural networks. (a) The PlantVillage image dataset used in this study. This dataset contains 38 categories of diseased or healthy leaf images. See Figure S2 for the names of species and diseases assigned to each label. (b) InceptionV3-based convolutional neural network (CNN) architecture used in this study. Conv, convolutional layer; Mixed, inception module. (c) Accuracy, precision, recall, and mean F1 scores against the training, validation, and test data using the trained weights. (d) Confusion matrix drawn against the test dataset. See Figure S2 for an enlarged view.

Figure 2

Visualization of intermediate outputs generated by the trained CNN. Image of tomato leaf infected with early blight (label 29) was fed to the network and intermediate output values of representative layers were visualized. Layer or inception module names and their output array sizes are described above each intermediate output.

Figure 3

Feature visualization. Four neurons were randomly selected from the indicated layers and feature visualization was performed to obtain a visual interpretation of what the neurons have learned. Neurons trained with (a) ImageNet or (b) PlantVillage were, respectively, visualized.

Figure 4

Semantic dictionary. Semantic dictionary generated using the intermediate outputs of global average pooling (GAP) layer. (a) Mean GAP output of 200 images of tomato early blight from the test dataset was multiplied by the weights of the CNN and sorted based on its value. (b) Neurons that correspond to the top five output values were selected and feature visualization was, respectively, applied. (c) Representative images of disease symptoms selected from the indicated class. Bottom row is a magnified view of the red inset in the top row.

Figure 5

Evaluation of attention map generating algorithms. (a) Input images from three classes used for evaluation (top row). Numbers in parentheses indicate the class label of the dataset. Lesions in the image were manually annotated (bottom row). (b)-(e) Attention map generating methods applied to each image and displayed as a heatmap over the input. See Materials and Methods for details. (b) Perturbation-based visualization. (c) Gradient-based visualization. (d) Grad-CAM visualization. (e) Reference-based visualization. For Grad-CAM and explanation map, the layers of which the gradient and the intermediate output were used are indicated.

Figure 6

Application of attention map generating algorithms on images misclassified by the CNN. From left to right column: (1), images randomly selected from the dataset which were misclassified by the CNN; correct labels are displayed on top of the image; (2), the top three inferences by the CNN; (3), Grad-CAM-based visualization targeted to Mixed0 layer; (4), guided back-propagation-based visualization.

Figure 7

Effect of feature extraction layer shaving. (a) The accuracy and loss value against the test dataset of CNN whose layers posterior to the indicated layers were removed. We performed a transfer learning with the newly prepared global average pooling and output layers. Since the Mixed5 CNN showed a classification performance equivalent to the original model, further analysis was not performed. (b) Network parameters required to run the CNN.