Modified U-Net (mU-Net) With Incorporation of Object-Dependent High Level Features for Improved Liver and Liver-Tumor Segmentation in CT Images

Abstract

Segmentation of livers and liver tumors is one of the most important steps in radiation therapy of hepatocellular carcinoma. The segmentation task is often done manually, making it tedious, labor intensive, and subject to intra-/inter- operator variations. While various algorithms for delineating organ-at-risks (OARs) and tumor targets have been proposed, automatic segmentation of livers and liver tumors remains intractable due to their low tissue contrast with respect to the surrounding organs and their deformable shape in CT images. The U-Net has gained increasing popularity recently for image analysis tasks and has shown promising results. Conventional U-Net architectures, however, suffer from three major drawbacks. First, skip connections allow for the duplicated transfer of low resolution information in feature maps to improve efficiency in learning, but this often leads to blurring of extracted image features. Secondly, high level features extracted by the network often do not contain enough high resolution edge information of the input, leading to greater uncertainty where high resolution edge dominantly affects the network's decisions such as liver and liver-tumor segmentation. Thirdly, it is generally difficult to optimize the number of pooling operations in order to extract high level global features, since the number of pooling operations used depends on the object size. To cope with these problems, we added a residual path with deconvolution and activation operations to the skip connection of the U-Net to avoid duplication of low resolution information of features. In the case of small object inputs, features in the skip connection are not incorporated with features in the residual path. Furthermore, the proposed architecture has additional convolution layers in the skip connection in order to extract high level global features of small object inputs as well as high level features of high resolution edge information of large object inputs. Efficacy of the modified U-Net (mU-Net) was demonstrated using the public dataset of Liver tumor segmentation (LiTS) challenge 2017. For liver-tumor segmentation, Dice similarity coefficient (DSC) of 89.72 %, volume of error (VOE) of 21.93 %, and relative volume difference (RVD) of - 0.49 % were obtained. For liver segmentation, DSC of 98.51 %, VOE of 3.07 %, and RVD of 0.26 % were calculated. For the public 3D Image Reconstruction for Comparison of Algorithm Database (3Dircadb), DSCs were 96.01 % for the liver and 68.14 % for liver-tumor segmentations, respectively. The proposed mU-Net outperformed existing state-of-art networks.

Fig. 1.

Schematic diagram of (a) conventional U-Net and (b-d) the proposed mU-Nets. (a) In the case of the conventional U-Net, full information of features passes through the skip connection and only the low resolution information is transferred to the next stage. Spatial information of small objects often disappears due to resolution loss after pooling. (b) In the mU-Net case for small objects, higher level global features can be extracted without loss of resolution by pooling. Spatial information of small objects is maintained by blocking the deconvolution path, which allows small object features to pass into the skip connection without being removed by pooling. (c) In the mU-Net case for large objects, feature information in the skip connection is restricted to edge information to avoid duplication of low resolution information. (d) A schematic diagram of the proposed network is shown. The deconvolution and activation in the residual path adaptively incorporate features of the residual path into features of the skip connection depending on the object size.

Fig. 2.

Permeation rate of the mU-Net with respect to stages. In this work, the stage is related to the matrix size of features. i.e., the same matrix size of features is extracted in the same stage. Blue dashed lines correspond to the size of 28×28 mm² at each stage.

Fig. 3.

Feature maps passing through the skip connection of the conventional U-Net (left) and feature maps passing through the skip connection before the additional convolution layer of the mU-Net (right). Red arrows show that, unlike large objects, the features of small objects are preserved in the mU-Net. The tumor sizes are represented in the label image.

Fig. 4.

The proposed network architecture.

Fig. 5.

(a) Adding a new layer in the original ones. (b) The DSC results of liver segmentation in the test cases with respect to the number of layers. The digits below each graph represent the number of layers of each network.

Fig. 6.

Target thin slice. Segmentation results from (b) Qin et al. [12], (c) Han et al. [20], (d) Men et al. [35], (e) Li et al. [36], (f) the proposed network, and (g) ground truth. (h) Target thick slice. (i-n) were acquired by the methods corresponded to (b-g), respectively. Gray regions mean liver and white regions mean liver tumor.

Fig. 7.

Absolute difference map between segmentations obtained from (a) Qin et al. [12], (b) Han et al. [20], (c) Men et al. [35], (d) Li et al. [36], (e) the proposed network and ground truth for thin slice. (f-j) were absolute difference map between the methods corresponded to (a-e) and ground truth, respectively, for thick slice. Difference from ground truth is represented with yellow color.

Fig. 8.

Contouring results of each method. (a) Liver contouring and (b) liver-tumor contouring from thin slice. (c) and (d) from thick slice correspond to (a) and (b), respectively. Blue, yellow, purple, red, white, and green lines are acquired from ground truth, Qin et al. [12], Han et al. [20], Men et al. [35], Li et al. [36], and proposed network, respectively. Each brown-square region is also magnified.

Fig. 9.

3D visualization results of (a) Qin et al. [12], (b) Han et al. [20], (c) Men et al. [35], (d) Li et al. [36], (e) proposed network, and (f) ground truth from segmentation results in Fig. 5. Liver and liver tumor are represented by pink and green color, respectively. Distance of liver tumor from surface is represented by brightness of green color.

Fig. 10.

The example of segmentation results of the mU-Net. First row shows the target slices of (a), (b), and (c). Segmentation results are shown in the second row. Yellow and white regions denote false positive error for liver and liver tumor, respectively. In contrast, gray and blue regions denote false negative error for liver and liver tumor, respectively. Third row represents the images overlaid from first and second row.