Brain-Based Binary Communication Using Spatiotemporal Features of fNIRS Responses

Abstract

"Locked-in" patients lose their ability to communicate naturally due to motor system dysfunction. Brain-computer interfacing offers a solution for their inability to communicate by enabling motor-independent communication. Straightforward and convenient in-session communication is essential in clinical environments. The present study introduces a functional near-infrared spectroscopy (fNIRS)-based binary communication paradigm that requires limited preparation time and merely nine optodes. Eighteen healthy participants performed two mental imagery tasks, mental drawing and spatial navigation, to answer yes/no questions during one of two auditorily cued time windows. Each of the six questions was answered five times, resulting in five trials per answer. This communication paradigm thus combines both spatial (two different mental imagery tasks, here mental drawing for "yes" and spatial navigation for "no") and temporal (distinct time windows for encoding a "yes" and "no" answer) fNIRS signal features for information encoding. Participants' answers were decoded in simulated real-time using general linear model analysis. Joint analysis of all five encoding trials resulted in an average accuracy of 66.67 and 58.33% using the oxygenated (HbO) and deoxygenated (HbR) hemoglobin signal respectively. For half of the participants, an accuracy of 83.33% or higher was reached using either the HbO signal or the HbR signal. For four participants, effective communication with 100% accuracy was achieved using either the HbO or HbR signal. An explorative analysis investigated the differentiability of the two mental tasks based solely on spatial fNIRS signal features. Using multivariate pattern analysis (MVPA) group single-trial accuracies of 58.33% (using 20 training trials per task) and 60.56% (using 40 training trials per task) could be obtained. Combining the five trials per run using a majority voting approach heightened these MVPA accuracies to 62.04 and 75%. Additionally, an fNIRS suitability questionnaire capturing participants' physical features was administered to explore its predictive value for evaluating general data quality. Obtained questionnaire scores correlated significantly (r = -0.499) with the signal-to-noise of the raw light intensities. While more work is needed to further increase decoding accuracy, this study shows the potential of answer encoding using spatiotemporal fNIRS signal features or spatial fNIRS signal features only.

Keywords: binary communication; brain computer interface; functional near infrared spectroscopy (fNIRS); mental drawing; mental imagery; motor imagery; spatial navigation; yes/no decoding.

FIGURE 1

Encoding scheme for answering a binary question. The red periods require mental drawing (MD) imagery, whereas the green periods required spatial navigation (SN) imagery. If participants chose to answer “yes”, they started performing the MD task when they heard a “yes”, halted their imagery when they heard the cue “stop”, and ignored the auditory cues related to the “no” response. The hypothesized HbO response for a “yes” answer is shown by the upper white waveform. If participants chose to answer “no”, they started performing the SN task when they heard a “no”, halted their imagery when they heard the cue “stop”, and ignored the auditory cues related to the “yes” response. The hypothesized HbO response for a “no” answer is shown by the lower white waveform.

FIGURE 2

Optode setup with three source and six detector optodes, placed on nine points according to the international 10-20 EEG system. Large orange dots represent reference points of the 10-20 system, whereas small orange dots represent reference points of the extended 10-10 EEG system (Oostenveld and Praamstra, 2001). The red lines represent 14 source-detector pairs (each forming an fNIRS channel). Image created using NIRSite (v.1) software (NIRx Medizintechnik GmbH, Berlin, Germany; RRID: SCR_002491).

FIGURE 3

Schematic depiction of the fNIRS signal analyses. The two main pipelines were univariate analysis and multivariate analysis. Each pipeline resulted in four accuracy outcomes. These outcome variables are represented in gray colored boxes. CV, coefficient of variance; OD, optical density; HbO, oxygenated hemoglobin; HbR, deoxygenated hemoglobin; SVM20-20, support vector machine with 20 training trials of each task; SVM40-40, support vector machine with 40 training trials of each task.

FIGURE 4

Event-related averages of channels of interest in participant 4. The two graphs on the left are event-related averages from the first localizer run (mental drawing; MD). The two graphs on the right are event-related averages from the second localizer run (spatial navigation; SN). The top two graphs depict the oxygenated hemoglobin (HbO) response, whereas the bottom two graphs depict the deoxygenated hemoglobin (HbR) response. Each graph is the event-related average of 20 individual trials, with the darker average signal line and its standard deviation (lighter colored band surrounding the average signal line). Notice the clear and typical hemodynamic response during both tasks: a positive deflection in HbO and a negative deflection in HbR. The gray band from 0 to 10 s signifies the mental imagery time interval.

FIGURE 5

Event-related averages of channels of interest in participant 17. The two graphs on the left are event-related averages from the first localizer run (mental drawing; MD). The two graphs on the right are event-related averages from the second localizer run (spatial navigation; SN). The top two graphs depict the oxygenated hemoglobin (HbO) response, whereas the bottom two graphs depict the deoxygenated hemoglobin (HbR) response. Each graph is the event-related average of 20 individual trials, with the darker average signal line and its standard deviation (lighter colored band surrounding the average signal line). Notice the absence of a typical hemodynamic response during both tasks: there is no clear positive deflection in HbO, nor a negative deflection in HbR. The gray band from 0 to 10 s signifies the mental imagery time interval.

FIGURE 6

Decoding accuracies of individual participants and the sample mean obtained with the single-trial (light-colored bars) and the multi-trial (dark-colored bars) univariate approach. Decoding accuracies were attained through channels-of-interest, preceded by a channel exclusion step. The upper plot show results based on analysis of HbO data (red bars), the lower plot is based on HbR data (blue bars). The ◆ symbol indicates participants whose single-trial accuracy was significant, whereas the * symbol indicates those participants whose multi-trial accuracy was significant.

FIGURE 7

Event-related averages of channels-of-interest in participant 4. The two graphs on the left are event-related averages from the first answer decoding run, in which the participant encoded a “yes” answer. The two graphs on the right are event-related averages from the sixth answer decoding run, in which the participant encoded a “no” answer. The top two graphs depict the oxygenated hemoglobin (HbO) response, whereas the bottom two graphs depict the deoxygenated hemoglobin (HbR) response. Each graph is the event-related average of five individual trials, with the darker average signal line and its standard deviation (lighter colored band surrounding the average signal line). Notice the clear and typical hemodynamic response function during both tasks: a positive deflection in HbO and a negative deflection in HbR. The gray band from 0 to 10 s signifies the mental imagery time interval.

FIGURE 8

Event-related averages of channels-of-interest in participant 17. The two graphs on the left are event-related averages from the fifth answer decoding run, in which the participant encoded a “yes” answer. The two graphs on the right are event-related averages from the sixth answer decoding run, in which the participant encoded a “no” answer. The top two graphs depict the oxygenated hemoglobin (HbO) response, whereas the bottom two graphs depict the deoxygenated hemoglobin (HbR) response. Each graph is the event-related average of five individual trials, with the darker average signal line and its standard deviation (lighter colored band surrounding the average signal line). Notice the absence of a typical hemodynamic response function during both tasks: there is no clear positive deflection in HbO, nor a negative deflection in HbR. The gray band from 0 to 10 s signifies the mental imagery time interval.

FIGURE 9

Decoding accuracies of individual participants and the sample mean obtained with the single-trial (light-colored bars) and the multi-trial (dark-colored bars) multivariate approach. The upper plot shows decoding accuracies of the SVM20-20 classifier, the lower plot shows decoding accuracies of the SVM40-40 classifier. The symbol indicates participants whose accuracy reached significance, as tested with permutation testing (for evaluating single-trial accuracies), whereas the * symbol indicates those participants whose multi-trial accuracy was significant. Abbreviations: SVM20-20 = support vector machine with 20 training trials of each task; SVM40-40 = support vector machine with 40 training trials of each task.

FIGURE 10

Mean comfortability rating over time (fNIRS runs). A comfortability rating of 0 corresponds to “very uncomfortable” and 10 to “very comfortable”. The ten fNIRS runs are depicted in the order they were conducted in the experiment. The first two runs, MD1 and SN2, were localizer runs (block 1) for mental drawing (MD) and spatial navigation (SN). The following six runs, Q1, Q2, Q3, Q4, Q5, Q6, represent the answer decoding runs, with a Q as an abbreviation for “question run”. The last two runs, MD2 and SN2, were localizer runs (block 2). Error bars reflect standard deviations.