Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 Nov;110(4 Pt A):382-391.
doi: 10.1016/j.jphysparis.2017.03.001. Epub 2017 Mar 8.

Retrospectively Supervised Click Decoder Calibration for Self-Calibrating Point-And-Click Brain-Computer Interfaces

Affiliations
Free PMC article

Retrospectively Supervised Click Decoder Calibration for Self-Calibrating Point-And-Click Brain-Computer Interfaces

Beata Jarosiewicz et al. J Physiol Paris. .
Free PMC article

Abstract

Brain-computer interfaces (BCIs) aim to restore independence to people with severe motor disabilities by allowing control of acursor on a computer screen or other effectors with neural activity. However, physiological and/or recording-related nonstationarities in neural signals can limit long-term decoding stability, and it would be tedious for users to pause use of the BCI whenever neural control degrades to perform decoder recalibration routines. We recently demonstrated that a kinematic decoder (i.e. a decoder that controls cursor movement) can be recalibrated using data acquired during practical point-and-click control of the BCI by retrospectively inferring users' intended movement directions based on their subsequent selections. Here, we extend these methods to allow the click decoder to also be recalibrated using data acquired during practical BCI use. We retrospectively labeled neural data patterns as corresponding to "click" during all time bins in which the click log-likelihood (decoded using linear discriminant analysis, or LDA) had been above the click threshold that was used during real-time neural control. We labeled as "non-click" those periods that the kinematic decoder's retrospective target inference (RTI) heuristics determined to be consistent with intended cursor movement. Once these neural activity patterns were labeled, the click decoder was calibrated using standard supervised classifier training methods. Combined with real-time bias correction and baseline firing rate tracking, this set of "retrospectively labeled" decoder calibration methods enabled a BrainGate participant with amyotrophic lateral sclerosis (T9) to type freely across 11 research sessions spanning 29days, maintaining high-performance neural control over cursor movement and click without needing to interrupt virtual keyboard use for explicit calibration tasks. By eliminating the need for tedious calibration tasks with prescribed targets and pre-specified click times, this approach advances the potential clinical utility of intracortical BCIs for individuals with severe motor disability.

Keywords: Adaptive classification; Amyotrophic lateral sclerosis (ALS); Augmentative and assistive communication (AAC); Brain-Machine Interface (BMI); Spinal cord injury (SCI); Stroke.

Figures

Figure 1
Figure 1. False click rate affects typing throughput, from a prior study and participant
A. Top panels: typing rates in last 2 sessions of T6’s multi-day series, labeled with her trial day, as also shown in the last 2 panels of Fig. 6 in Jarosiewicz et al. (2015). The length of each bar corresponds to the duration of each typing block. CSPM = correct selections per minute, measured as the number of selections per minute minus the number of backspaces pressed. Bottom panels: the approximate false (unintended) click rate during these same typing periods, defined as the percentage of all clicks that occurred in zones of the radial keyboard where clicks had no effect. For context, across all blocks in the sessions leading up to these two sessions, the mean (± SEM) of the CSPM was 21.05 ± 1.01, and the average false click rate was 12.56 ± 0.84. B. The relationship between false click rate and typing rate for each block of self-paced typing in these last 2 sessions. The decrease in typing rate was correlated with an increase in false click rate (r = −0.66, p < 0.05), suggesting that the decay in click decoding quality was a contributing factor in the decline of point-and-click typing performance.
Figure 2
Figure 2. Research session setup
A. Timeline of research sessions. On day 1, decoders were initialized using a standard calibration task with prescribed targets, and then the participant used this standard set of decoders to begin free-typing. After 20 minutes of free-typing, the first set of retrospectively supervised (RS) decoders were calibrated. Thenceforth, RS decoders were recalibrated at each break using the last 20 – 60 minutes of typing data. Black arrowheads denote RS recalibrations. B. Photo of T9’s screen during the 2nd free-typing session in the series (his trial day 134). T9 chose to finish typing out the lines of a favorite poem (“If—” by Rudyard Kipling) that he had started typing on the previous day.
Figure 3
Figure 3. Retrospectively supervised (RS) decoder calibration method, shown for a representative one-minute sample of data collected during T9’s virtual point-and-click typing (trial day 134)
A. RTI decoder calibration using data acquired during free-typing. Thin blue and green lines represent the cursor x and y-position over time, respectively (in screen units, where edges are represented as [-1 1]). Thick blue and green lines represent the x and y-positions of the retrospectively inferred targets. Black and red asterisks in both Panel A and B represent time periods labeled as intended movement and intended click, respectively, for purposes of both RTI and RS decoder calibration. B. RS click decoder calibration using data acquired during free-typing. Thin blue line is the click log-likelihood, decoded using the LDA classifier. Black dashed line represents the click threshold. Intended movement periods were identified using the heuristics described in reference and summarized in Methods. Intended click periods were simply identified as periods during which the click log-likelihood was above click threshold, using the decoder that had been used in real time during that part of the collected data.
Figure 4
Figure 4. Neural signal nonstationarities
A. Means (left panel) and standard deviations (middle panel) of the neural features (spike power) for the channels that were selected for use in the click decoder across all typing blocks and sessions. Means and standard deviations are capped at 100 for visualization; mean rates actually peaked at 113.57 and standard deviations at 2006.02 (in spike power/sec). LDA click decoder coefficients (right panel) for the same blocks and sessions. (The 1st typing block of the 1st session used principal components of spike power as features for click decoding rather than spike power itself, so that block is skipped in these plots.) B. Top panel (“First session”): Real-time and offline reconstruction of the click log-likelihood signal using the click decoder from last block of the 1st session on the neural data from a small snippet of that same block (black trace = true log-likelihood obtained online; blue trace = offline reconstructed log-likelihood using the same decoder and neural data. Slight differences in these 2 traces are due to slight differences in real-time vs. offline z-scoring). Bottom panel (“Last session”): offline reconstruction of the click log-likelihood signal that would have been obtained during a sample period of the last session if the click decoder from the end of the 1st session had been used with neither feature statistics tracking nor RS calibration (red trace), or with feature statistics tracking but no RS recalibration (blue trace). Black trace shows the true click log-likelihood that had been obtained online for this sample period from the last session, using the fully self-calibrated click decoder with both statistics tracking and RS recalibration enabled.
Figure 5
Figure 5. Typing rates and false click rates in T9’s multi-day self-calibration sessions
A. Typing rates (top panel) and false click rates (middle panel) are shown for each block from all 11 multi-day self-calibration sessions, spanning approximately one month (breaks along the x-axis denote breaks between different sessions, each labeled above with T9’s trial day). There was no significant decline in typing rate or significant increase in false click rate over time across the series of sessions. The bottom panel shows, for each click in all blocks and all sessions, the delay between the cursor entering the target that was eventually clicked and the click being decoded. Lags to click are capped at 4 sec for visualization only (statistics were performed on the true lag values. The actual range of lags extended to 5.98 sec; throughout the session series, 28 click lags were above this 4 sec cap). There was a significant decrease over time across all blocks and sessions in the lag to click (p = 0.0063). B. (Left) the relationship between false click rate and typing rate for each block of self-paced typing across all sessions (each data point is one block). As for T6, the higher the false click rate, the lower the typing rate (p = 0.00045), providing further evidence that click decoding quality is an important factor in typing performance and validating the use of SPM as a good metric for relative typing performance. (Middle) the relationship between mean lag to click in each block and the typing rate for that block. There is no significant linear relationship between mean lag to click and SPM (p = 0.296). (Right) There is also no significant linear relationship between mean lag to click and false click rate (p = 0.34).

Similar articles

See all similar articles

Cited by 1 article

LinkOut - more resources

Feedback