Modeling and classification of voluntary and imagery movements for brain-computer interface from fNIR and EEG signals through convolutional neural network

Abstract

Practical brain-computer interface (BCI) demands the learning-based adaptive model that can handle diverse problems. To implement a BCI, usually functional near-infrared spectroscopy (fNIR) is used for measuring functional changes in brain oxygenation and electroencephalography (EEG) for evaluating the neuronal electric potential regarding the psychophysiological activity. Since the fNIR modality has an issue of temporal resolution, fNIR alone is not enough to achieve satisfactory classification accuracy as multiple neural stimuli are produced by voluntary and imagery movements. This leads us to make a combination of fNIR and EEG with a view to developing a BCI model for the classification of the brain signals of the voluntary and imagery movements. This work proposes a novel approach to prepare functional neuroimages from the fNIR and EEG using eight different movement-related stimuli. The neuroimages are used to train a convolutional neural network (CNN) to formulate a predictive model for classifying the combined fNIR-EEG data. The results reveal that the combined fNIR-EEG modality approach along with a CNN provides improved classification accuracy compared to a single modality and conventional classifiers. So, the outcomes of the proposed research work will be very helpful in the implementation of the finer BCI system.

Keywords: Brain–computer interface (BCI); Convolutional neural network (CNN); Electroencephalography (EEG); Functional near-infrared spectroscopy (fNIR); Modeling and classification; Voluntary and imagery movements.

Fig. 1

Time schedule of data acquisition protocol for each participant regarding both the voluntary and imagery movements. This is a unit task performing schedule that was repeated four times in each session to complete 40 individual trials of every task

Fig. 2

Schematic illustration of MATLAB based protocol instruction aiding application for the experiment. Regarding the instruction of this application, the participant is asked to move the left hand (voluntary or imagery) by Fig. 3a and after that, Fig. 3b instructs to take rest

Fig. 3

The combined fNIRS-EEG sensor positions on the scalp and prefrontal cortex. The data of parietal lobe are acquired through the main data acquisition period but excluded for proposed offline processing

Fig. 4

Data acquisition of voluntary and imagery movements using fNIR and EEG modality

Fig. 5

Preprocessing steps of the fNIR signals

Fig. 6

The steps applied in EEG signal preprocessing

Fig. 7

Combining the fNIR and EEG data to prepare the spatiotemporal neuroimages for classification by CNN

Fig. 8

The features of the input images with the changes of layers of the CNN based classifiers

Fig. 9

The main steps of the proposed method of processing of the EEG and fNIR signal, image formation, and classification

Fig. 10

Steps regarding the manual feature extraction and classification of the combined fNIR and EEG signals by the conventional classifiers

Fig. 11

a Raw and filtered fNIR signal without baseline correction and b filtered fNIR signal after baseline correction that starts from the baseline or zero levels

Fig. 12

Step by step EEG signal pre-processing: a raw EEG signal of a single channel, b EEG signal after removing 50 Hz power line noise, c filtered EEG signal up to 45 Hz by third-order elliptical filter, and d eye-blink and EOG artifact-free EEG signal which is filtered by the EAWICA toolbox

Fig. 13

Combining the fNIR and EEG data to prepare the spatiotemporal neuroimages for classification by CNN

Fig. 14

The training and validation accuracy with loss performances with respect to the epoch iterations for 4-class problem

Fig. 15

The training and validation accuracy with loss performances with respect to the epoch iterations for 6-class problem

Fig. 16

The training and validation accuracy with loss performances with respect to the epoch iterations for 8-class problem

Fig. 17

Overall performances (mean ± SD) of the classification accuracy through SVM, LDA, and the proposed CNN method while only the fNIR data were considered

Fig. 18

Overall performances (mean ± SD) of the classification accuracy through SVM, LDA, and the proposed CNN method while combined fNIR and EEG data were considered