Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Yang Li; Xin Zhang; Dong Ming

doi:10.3389/fnins.2022.1062889

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Front Neurosci. 2023 Jan 9:16:1062889. doi: 10.3389/fnins.2022.1062889. eCollection 2022.

Authors

Yang Li¹, Xin Zhang^{2

3}, Dong Ming^{2

3}

Affiliations

¹ Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.
² The Laboratory of Neural Engineering and Rehabilitation, Department of Biomedical Engineering, School of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, China.
³ The Tianjin International Joint Research Center for Neural Engineering, Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.

Abstract

Introduction: Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.

Methods: In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.

Results: The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.

Keywords: EEG; fNIRS; hybrid-BCI; modality fusion; motor imagery.

Grants and funding

This study was supported in parts by the National Key Research and Development Program of China under Grant 2022YFF1202900, the National Natural Science Foundation of China under Grant 82102174, and the China Postdoctoral Science Foundation under Grant 2021TQ0243.