Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

Abstract

Background: Brain-Computer Interface (BCI) is becoming more reliable, thanks to the advantages of Artificial Intelligence (AI). Recently, hybrid Deep Learning (hDL), which combines different DL algorithms, has gained momentum over the past five years. In this work, we proposed a review on hDL-based BCI starting from the seminal studies in 2015.

Objectives: We have reviewed 47 papers that apply hDL to the BCI system published between 2015 and 2020 extracting trends and highlighting relevant aspects to the topic.

Methods: We have queried four scientific search engines (Google Scholar, PubMed, IEEE Xplore and Elsevier Science Direct) and different data items were extracted from each paper such as the database used, kind of application, online/offline training, tasks used for the BCI, pre-processing methodology adopted, type of normalization used, which kind of features were extracted, type of DL architecture used, number of layers implemented and which optimization approach were used as well. All these items were then investigated one by one to uncover trends.

Results: Our investigation reveals that Electroencephalography (EEG) has been the most used technique. Interestingly, despite the lower Signal-to-Noise Ratio (SNR) of the EEG data that makes pre-processing of that data mandatory, we have found that the pre-processing has only been used in 21.28% of the cases by showing that hDL seems to be able to overcome this intrinsic drawback of the EEG data. Temporal-features seem to be the most effective with 93.94% accuracy, while spatial-temporal features are the most used with 33.33% of the cases investigated. The most used architecture has been Convolutional Neural Network-Recurrent Neural Network CNN-RNN with 47% of the cases. Moreover, half of the studies have used a low number of layers to achieve a good compromise between the complexity of the network and computational efficiency.

Significance: To give useful information to the scientific community, we make our summary table of hDL-based BCI papers available and invite the community to published work to contribute to it directly. We have indicated a list of open challenges, emphasizing the need to use neuroimaging techniques other than EEG, such as functional Near-Infrared Spectroscopy (fNIRS), deeper investigate the advantages and disadvantages of using pre-processing and the relationship with the accuracy obtained. To implement new combinations of architectures, such as RNN-based and Deep Belief Network DBN-based, it is necessary to better explore the frequency and temporal-frequency features of the data at hand.

Keywords: Brain-Computer Interface (BCI); Electroencephalography (EEG); Hybrid Deep Learning; Neural Networks; review; survey.

Figure 1

Schematic representation of the review’s aim.

Figure 2

Flowchart of the selection process of the papers.

Figure 3

Distribution of the normalization methods across the reviewed papers.

Figure 4

(a) Feature extraction distribution across architectures. (b) The accuracy obtained for each feature. Note that bar “Other” grouped the following features extracted by Selective Attention Mechanism (SAM), Optical Flow from the EEG video, 425 silent physiological features from the 7 signals, and Hilbert–Huang spectrum (HHS), High-level features, Linear domain features (Autoregressive coefficient), and Non-Linear domain features (Approximate entropy, Hurst Exponent).

Figure 5

(a) Architecture percentage distribution. Note that there are papers that used more than one network; (b) the trend of architectures used across the years.

Figure 6

Average accuracy ± standard deviation for each architecture; (a) all datasets; (b) BCI Competition IV dataset.

Figure 7

(a) Optimization algorithms distribution across the reviewed papers. (b) Trend of Optimizers across years (note that N/A refers to the papers that did not state the use of the optimizer).

Figure 8

Number of layers vs. accuracy for each architecture. Note that some papers used more than one network with a different number of layers.

Figure 9

Specified vs. not specified BCI application.

Figure 10

(a) Percentage distribution of the dataset type. (b) Distribution of Datasets across the number of papers. Note: Some papers used more than one public dataset to compare the performance of their model. AI set: AI Dataset; Alcoholism: Examining EEG-Alcoholism Correlation; Auditory: auditory multi-class BCI; B.B.Y: Bashivan, Bidelman, Yeasin EEG data set; BCI II: BCI competition II; BCI III: BCI competition III; BCI IV: BCI competition IV; DEAP: DEAP dataset; D.a.a: Decoding auditory attention; EEG speech: EEG based speech dataset; eegmmidb: Physionet eegmmidb; Graz: Graz University Dataset; Local: Local Dataset; MAKAUT: MAKAUT Dataset; MIIR: OpenMIIR; P300 set: Exploiting P300 Amplitude changes; STEW: “STEW” dataset.

Figure 11

Mean ± standard deviation accuracy across tasks.

Figure 12

(a) Percentage distribution of the performance estimated by different algorithms. (b) Classification percentage accuracy.

Figure 13

Accuracy across hDL architectures and features on the BCI-Competition IV subset. (a) Accuracy of the architectures. (b) Accuracy of the extracted features.