Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Abstract

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs.

Keywords: classification; engagement; fNIRS; functional connectivity; passive brain-computer-interface; wavelet coherence.

Figure 1

Airbus A320 twin-engine simulator at ISAE-SUPAERO. Pictures used with written consent.

Figure 2

Structure of the experimental session. (A) Overall Flow of the experiment with the 8 scenarios. (B) Detailed trial for the cruise and landing phases. (C) Upper view of the plane trajectory. The starting position was pseudo randomly placed on the blue arc. The blackline delimited the cruise from the landing phase where the pilot deactivate or not the autopilot regarding the condition.

Figure 3

Results of the Monte-Carlo simulation over (A) the frontal cortex, (B) the occipital cortex and (C) both cortices from a lateral view. Red dots represent the LED emitters, blue dots the photoreceptors and green lines the channels. The colorbar unit represents the spatial sensitivity of the fNIRS measurements. It is expressed in mm⁻¹ and values range from 0.01 to 1 in log10 units: −2 to 0.

Figure 4

Pilot's engagement classification pipeline.

Figure 5

Pilot's engagement classification performance function of the type of fNIRS-based feature (average across subject). Blue and red bars represents features extracted from respectively [HbR] and [HbO] signals. Error bars represents the confidence interval at 95 %. The black lines indicate the most relevant significant effect for our research question (^***p < 0.05).

Figure 6

Pilot's engagement classification performance function of couple fNIRS-based oxygenation feature used (average across subject). Blue and red bars represents features extracted from respectively [HbR] and [HbO] signals. Error bars represents the confidence interval at 95 %.

Figure 7

Pilot's engagement classification performance function of couple fNIRS-based connectivity feature used (average across subject). Blue and red bars represents features extracted from respectively [HbR] and [HbO] signals. Error bars represents the confidence interval at 95 %.