Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

Yu-Dong Zhang; Zhengchao Dong; Shui-Hua Wang; Xiang Yu; Xujing Yao; Qinghua Zhou; Hua Hu; Min Li; Carmen Jiménez-Mesa; Javier Ramirez; Francisco J Martinez; Juan Manuel Gorriz

doi:10.1016/j.inffus.2020.07.006

Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

Inf Fusion. 2020 Dec:64:149-187. doi: 10.1016/j.inffus.2020.07.006. Epub 2020 Jul 17.

Authors

Yu-Dong Zhang^{1

2}, Zhengchao Dong^{3

4}, Shui-Hua Wang^{2

5

6}, Xiang Yu¹, Xujing Yao¹, Qinghua Zhou¹, Hua Hu^{3

7}, Min Li^{3

8}, Carmen Jiménez-Mesa⁹, Javier Ramirez⁹, Francisco J Martinez⁹, Juan Manuel Gorriz^{9

10}

Affiliations

¹ School of Informatics, University of Leicester, Leicester, LE1 7RH, Leicestershire, UK.
² Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia.
³ Department of Psychiatry, Columbia University, USA.
⁴ New York State Psychiatric Institute, New York, NY 10032, USA.
⁵ School of Architecture Building and Civil engineering, Loughborough University, Loughborough, LE11 3TU, UK.
⁶ School of Mathematics and Actuarial Science, University of Leicester, LE1 7RH, UK.
⁷ Department of Neurology, The Second Affiliated Hospital of Soochow University, China.
⁸ School of Internet of Things, Hohai University, Changzhou, China.
⁹ Department of Signal Theory, Networking and Communications, University of Granada, Granada, Spain.
¹⁰ Department of Psychiatry, University of Cambridge, Cambridge CB21TN, UK.

Abstract

Multimodal fusion in neuroimaging combines data from multiple imaging modalities to overcome the fundamental limitations of individual modalities. Neuroimaging fusion can achieve higher temporal and spatial resolution, enhance contrast, correct imaging distortions, and bridge physiological and cognitive information. In this study, we analyzed over 450 references from PubMed, Google Scholar, IEEE, ScienceDirect, Web of Science, and various sources published from 1978 to 2020. We provide a review that encompasses (1) an overview of current challenges in multimodal fusion (2) the current medical applications of fusion for specific neurological diseases, (3) strengths and limitations of available imaging modalities, (4) fundamental fusion rules, (5) fusion quality assessment methods, and (6) the applications of fusion for atlas-based segmentation and quantification. Overall, multimodal fusion shows significant benefits in clinical diagnosis and neuroscience research. Widespread education and further research amongst engineers, researchers and clinicians will benefit the field of multimodal neuroimaging.

Keywords: Applications; Assessment; Fusion rules; Magnetic resonance imaging; Multimodal data fusion; Neuroimaging; PET; Partial volume effect; SPECT.