High-Resolution Encoder-Decoder Networks for Low-Contrast Medical Image Segmentation

Abstract

Automatic image segmentation is an essential step for many medical image analysis applications, include computer-aided radiation therapy, disease diagnosis, and treatment effect evaluation. One of the major challenges for this task is the blurry nature of medical images (e.g., CT, MR and, microscopic images), which can often result in low-contrast and vanishing boundaries. With the recent advances in convolutional neural networks, vast improvements have been made for image segmentation, mainly based on the skip-connection-linked encoder-decoder deep architectures. However, in many applications (with adjacent targets in blurry images), these models often fail to accurately locate complex boundaries and properly segment tiny isolated parts. In this paper, we aim to provide a method for blurry medical image segmentation and argue that skip connections are not enough to help accurately locate indistinct boundaries. Accordingly, we propose a novel high-resolution multi-scale encoder-decoder network (HMEDN), in which multi-scale dense connections are introduced for the encoder-decoder structure to finely exploit comprehensive semantic information. Besides skip connections, extra deeply-supervised high-resolution pathways (comprised of densely connected dilated convolutions) are integrated to collect high-resolution semantic information for accurate boundary localization. These pathways are paired with a difficulty-guided cross-entropy loss function and a contour regression task to enhance the quality of boundary detection. Extensive experiments on a pelvic CT image dataset, a multi-modal brain tumor dataset, and a cell segmentation dataset show the effectiveness of our method for 2D/3D semantic segmentation and 2D instance segmentation, respectively. Our experimental results also show that besides increasing the network complexity, raising the resolution of semantic feature maps can largely affect the overall model performance. For different tasks, finding a balance between these two factors can further improve the performance of the corresponding network.

Fig. 1.

Illustration of the blurry and vanishing boundaries within pelvic CT images. First row: intensity images; Second row: corresponding segmentation ground-truth.

Fig. 2.

Illustration of the structure of our proposed high-resolution multi-scale encoder-decoder network (HMEDN). The input is a set of intensity image patches and the outputs are segmentation and contour probability maps. Rectangles and triangles represent operations in the network. Three kinds of pathways, i.e., skip connection (pathway ①), distilling pathway (pathway ②) and high-resolution pathway (pathway ②) connect all kinds of operations and form the network.

Fig. 3.

Illustration of the dilated convolutional network.

Fig. 4.

Comparison of representative feature maps.

Fig. 5.

Comparison of the output activation maps of the distilling network and the high-resolution distilling network.

Fig. 6.

Influence of hyper-parameters. In these figures, the Dice ratio variation against different hyper-parameters are reported. One can see that all the hyper-parameters are effective in improving the performance of the algorithm. Setting α, δ₂, and μ to 1, 8, and 25, respectively, achieves the best performance.

Fig. 7.

Precision and robustness comparison of the compared algorithms. The sub-figures illustrate the Dice ratio of the 15 best and worst segmented samples of each algorithm.

Fig. 8.

Representative segmentation results of the compared state-of-the-art algorithms on the pelvic CT image dataset. In the first and the fourth rows, the segmentation masks and intensity images in the axial direction are provided. In the second and the fifth rows, the results in the coronal direction are provided. The yellow curves in the segmentation masks indicate the ground-truth contours of the target organs. The third and the sixth rows are the difference map and the segmentation ground-truth in 3D space. The green, red, and blue fragments are the false predictions on prostate, bladder, and rectum, respectively.

Fig. 9.

Label and intensity image patches of the brain tumor dataset. The visualized image patches (from left to right) are: (A) the whole tumor in FLAIR, (B) the tumor core in T2, (C) the enhancing tumor structures in T1c, (D) the final labels of the tumor structures (the combination of all segmentations) in T1: edema (green), non-enhancing solid core (red), enhancing core (yellow).

Fig. 10.

Segmentation results on the brain tumor dataset. In these figures, different colors indicate different tumor categorizations. The T1-weighted image is selected for visualization of the corresponding input images.

Fig. 11.

Segmentation results illustration on the nuclei segmentation dataset. In these figures, masks of different colors are corresponding to the segmented nuclei. The red arrows in the figures indicate representative segmentation results and the corresponding intensity map.