Strategies to Reduce the Expert Supervision Required for Deep Learning-Based Segmentation of Histopathological Images

Abstract

The emergence of computational pathology comes with a demand to extract more and more information from each tissue sample. Such information extraction often requires the segmentation of numerous histological objects (e.g., cell nuclei, glands, etc.) in histological slide images, a task for which deep learning algorithms have demonstrated their effectiveness. However, these algorithms require many training examples to be efficient and robust. For this purpose, pathologists must manually segment hundreds or even thousands of objects in histological images, i.e., a long, tedious and potentially biased task. The present paper aims to review strategies that could help provide the very large number of annotated images needed to automate the segmentation of histological images using deep learning. This review identifies and describes four different approaches: the use of immunohistochemical markers as labels, realistic data augmentation, Generative Adversarial Networks (GAN), and transfer learning. In addition, we describe alternative learning strategies that can use imperfect annotations. Adding real data with high-quality annotations to the training set is a safe way to improve the performance of a well configured deep neural network. However, the present review provides new perspectives through the use of artificially generated data and/or imperfect annotations, in addition to transfer learning opportunities.

Keywords: data augmentation; deep learning; generative adversarial networks; histopathology; image annotation; image segmentation; transfer learning; weak supervision.

Figure 1

The different steps implemented in computational pathology. These steps aim to extract the most accurate information possible from all available data to improve complex diagnosis and therapeutic decisions (2).

Figure 2

Images generated by a data augmentation strategy. (A) The original image and (B–E) various images which are generated from (A) using a data augmentation strategy described in Van Eycke et al. (20). This strategy combines image alterations targeting color, intensity, geometry, and image quality features, such as sharpness.

Figure 3

Generative adversarial networks (GAN) principles. Cylinders represent data while black rectangles represent neural networks. The main path appears in blue while the feedback loops appear in orange. The generator receives input data that allows it to synthesize an image. The discriminator receives either a real image or a synthesized image as an input. It must then determine whether it is a real or generated example. The generator is rewarded if it succeeds in deceiving the discriminator while the discriminator is rewarded if it succeeds in distinguishing the true images from the generated images.

Figure 4

Use of a GAN to generate examples for histological image segmentation (same graphic conventions as in Figure 3). Computer-generated binary images are provided as inputs to the generator to generate images (of the same size) that mimic the targeted tissue structure. The discriminator receives as input a binary image and a tissue image, artificial or real, of which it must determine the origin. The binary images associated with the real images are the segmentation masks of the structures of interest. Therefore, in the generated images the structures of interest must appear at the locations indicated by the white masks in the generator inputs in order to be able to deceive the discriminator. In this way, after system optimization, the binary images provided to the generators correspond to the segmentation masks of the generated images.

Figure 5

Examples of imperfect annotations generated from high quality ones. (A) Original annotations from the GLaS challenge (12), (B) noisy annotation where some labels are removed, (C) weak annotations based on bounding boxes.