Toward a compact hybrid brain-computer interface (BCI): Performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels

Abstract

It has been demonstrated that the performance of typical unimodal brain-computer interfaces (BCIs) can be noticeably improved by combining two different BCI modalities. This so-called "hybrid BCI" technology has been studied for decades; however, hybrid BCIs that particularly combine electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) (hereafter referred to as hBCIs) have not been widely used in practical settings. One of the main reasons why hBCI systems are so unpopular is that their hardware is generally too bulky and complex. Therefore, to make hBCIs more appealing, it is necessary to implement a lightweight and compact hBCI system with minimal performance degradation. In this study, we investigated the feasibility of implementing a compact hBCI system with significantly less EEG channels and fNIRS source-detector (SD) pairs, but that can achieve a classification accuracy high enough to be used in practical BCI applications. EEG and fNIRS data were acquired while participants performed three different mental tasks consisting of mental arithmetic, right-hand motor imagery, and an idle state. Our analysis results showed that the three mental states could be classified with a fairly high classification accuracy of 77.6 ± 12.1% using an hBCI system with only two EEG channels and two fNIRS SD pairs.

Fig 1

Arrangement of EEG channels (blue) and fNIRS optodes (red: Sources, green: Detectors) on the frontal (left) and motor (right) areas. Channels were numbered in the same manner as in a previous study [20].

Fig 2. Illustration of a single trial of the experiment.

Each trial consisted of an introduction period of 2 s, a task period of 10 s, and an inter-trial rest period (stop and rest) of 16–18 s. During the introduction period, a random task (MA, MI, or IS) was displayed to the participant. After a short beep, the participant performed the task displayed in the introduction period while looking at a fixation cross. When a STOP sign was displayed with a second short beep, the participants stopped performing the task and relaxed during the random-length inter-trial rest period.

Fig 3. Data processing procedure.

CSP and sLDA stand for common spatial pattern and shrinkage linear discriminant analysis, respectively. To perform meta-classification, we concatenated the outputs of individual EEG and fNIRS classifiers to construct feature vectors for the meta-classifier. The “one-versus-one” block represents the strategy used to solve the three-class classification problem by dividing it into three binary classification problems and employing majority voting (VOTE) based on the result of each binary classification to predict the class.

Fig 4

(a) MI vs. IS EEG classification accuracies as a function of the number of EEG channels. Green bars indicate the grand average classification accuracies calculated using the sequential backward selection algorithm. The color map indicates the statistical significance between the differences in classification accuracy calculated according to the number of EEG channels (*p < 0.05). (b) MI vs. IS EEG classification accuracies calculated using the four optimal configurations with two EEG channel, which were (c) (Cz, CP3; the best), (d) (Cz, C3), (e) (CP3, CP4), and (f) (C3, C4; the fourth best). The red and green horizontal dashed lines indicate the threshold for an effective BCI (70%) and the value corresponding to the two EEG channels (x-axis) shown in (a), respectively. The error bars indicate standard deviation.

Fig 5

(a) Highest MA vs. IS fNIRS classification accuracies as a function of the number of SD pairs shown in (b). Statistical significance was calculated between the classification accuracies achieved using all the SD pairs and various numbers of SD pairs (*p < 0.05, **p < 0.01, ***p < 0.001). The error bars represent the standard deviation.