DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks

Abstract

In this paper, we propose DeepCut, a method to obtain pixelwise object segmentations given an image dataset labelled weak annotations, in our case bounding boxes. It extends the approach of the well-known GrabCut [1] method to include machine learning by training a neural network classifier from bounding box annotations. We formulate the problem as an energy minimisation problem over a densely-connected conditional random field and iteratively update the training targets to obtain pixelwise object segmentations. Additionally, we propose variants of the DeepCut method and compare those to a naïve approach to CNN training under weak supervision. We test its applicability to solve brain and lung segmentation problems on a challenging fetal magnetic resonance dataset and obtain encouraging results in terms of accuracy.

Fig. 1

CNN architecture with convolutional (conv), max-pooling (pool) layers, and fully connected layers for foreground/background classification. Layer, which inputs are subjected to 50% dropout [33] are marked with *.

Fig. 2

Naïve CNN learning versus the proposed DeepCut approach, iteratively updating the learning target classes for input patches.

Fig. 3

Example brain segmentation results for all compared methods: Top row (from left to right): (1) original image (2) manual segmentation (red), (3) initial bounding box B with halo H, (4) GrabCut [1] (GC, blue). Bottom Row: (5) naïve learning approach (CNN_naïve, yellow), (6) DeepCut from bounding boxes (DC_BB, purple), (7) DeepCut from pre-segmentation (DC_PS, pink) and (8) fully supervised CNN segmentation (CNN_FS, cyan).

Fig. 4

Example lung segmentation results for all compared methods: Top row (from left to right): (1) original image (2) manual segmentation (red), (3) initial bounding box B with halo H, (4) GrabCut [1] (GC, blue). Bottom Row: (5) naïve learning approach (CNN_naïve, yellow), (6) DeepCut from bounding boxes (DC_BB, purple), (7) DeepCut from pre-segmentation (DC_PS, pink) and (8) fully supervised CNN segmentation (CNN_FS, cyan).

Fig. 5

Comparative accuracy results for the segmentation of the fetal brain (a) and lungs (b) for all methods: Initial bounding boxes (BB), GrabCut [1] (GC), naïve CNN CNN_naïve learning approach from bounding boxes (CNN_BB), DeepCut initialised from bounding boxes (DC_BB), DeepCut initialised via pre-segmentation (DC_PS) and a fully supervised learning approach from manual segmentations (CNN_FS) as upper bound for this network architecture.

Fig. 6

Accuracy improvement in terms of DSC over DeepCut iterations in case of fetal brain segmentations. DeepCut initialisation with bounding boxes (DC_BB) (red) versus initialisation with pre-segmentation (DC_PS) (blue) in context with lower (CNN_naïve) and upper (CNN_FS) accuracy bound, depicted with mean (black) and standard deviations (grey).