GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction

Abstract

Motivation: Breast cancer is a very heterogeneous disease and there is an urgent need to design computational methods that can accurately predict the prognosis of breast cancer for appropriate therapeutic regime. Recently, deep learning-based methods have achieved great success in prognosis prediction, but many of them directly combine features from different modalities that may ignore the complex inter-modality relations. In addition, existing deep learning-based methods do not take intra-modality relations into consideration that are also beneficial to prognosis prediction. Therefore, it is of great importance to develop a deep learning-based method that can take advantage of the complementary information between intra-modality and inter-modality by integrating data from different modalities for more accurate prognosis prediction of breast cancer.

Results: We present a novel unified framework named genomic and pathological deep bilinear network (GPDBN) for prognosis prediction of breast cancer by effectively integrating both genomic data and pathological images. In GPDBN, an inter-modality bilinear feature encoding module is proposed to model complex inter-modality relations for fully exploiting intrinsic relationship of the features across different modalities. Meanwhile, intra-modality relations that are also beneficial to prognosis prediction, are captured by two intra-modality bilinear feature encoding modules. Moreover, to take advantage of the complementary information between inter-modality and intra-modality relations, GPDBN further combines the inter- and intra-modality bilinear features by using a multi-layer deep neural network for final prognosis prediction. Comprehensive experiment results demonstrate that the proposed GPDBN significantly improves the performance of breast cancer prognosis prediction and compares favorably with existing methods.

Availabilityand implementation: GPDBN is freely available at https://github.com/isfj/GPDBN.

Supplementary information: Supplementary data are available at Bioinformatics online.

Fig. 1.

Illustration of the proposed GPDBN framework

Fig.2.

ROC curves of GPDBN for breast cancer prognosis prediction

Fig. 3.

The values of Sn, Acc, Pre and F1 of GPDBN for breast cancer prognosis prediction at stringent levels of Sp = 90.0% (left) and Sp = 95.0% (right)

Fig. 4.

The values of Sn, Acc, Pre and F1 of different methods for breast cancer prognosis prediction at stringent levels of Sp = 90.0% (left) and Sp = 95.0% (right)

Fig. 5.

ROC curves of GPDBN and other methods by employing both genomic data and pathological images

Fig. 6.

Performance comparison of the proposed GPDBN and other methods using Kaplan–Meier curve

Fig. 7.

Visualization of original genomic, pathological image, combined features and the corresponding abstract features extracted by our proposed method. The red star represents shorter-term survivors, and the blue dot represents longer-term survivors