Manipulating stored phonological input during verbal working memory

Abstract

Verbal working memory (vWM) involves storing and manipulating information in phonological sensory input. An influential theory of vWM proposes that manipulation is carried out by a central executive while storage is performed by two interacting systems: a phonological input buffer that captures sound-based information and an articulatory rehearsal system that controls speech motor output. Whether, when and how neural activity in the brain encodes these components remains unknown. Here we read out the contents of vWM from neural activity in human subjects as they manipulated stored speech sounds. As predicted, we identified storage systems that contained both phonological sensory and articulatory motor representations. Unexpectedly, however, we found that manipulation did not involve a single central executive but rather involved two systems with distinct contributions to successful manipulation. We propose, therefore, that multiple subsystems comprise the central executive needed to manipulate stored phonological input for articulatory motor output in vWM.

Figure 1. Match-mismatch task

Participants performed trials that were either in the a. match condition or b. mismatch condition. In the match condition (a), subjects first saw a visual cue (‘Match Listen’) and then 1.5 seconds later were presented with one of two words auditorily (‘kig’ or ‘pob’). After a delay (1.5 to 2 s), the participants saw another cue (‘Speak’) which instructed them to say the non-words they had heard (‘kig’ to ‘kig’ and ‘pob’ to ‘pob’). In the mismatch condition (b), participants were presented with a different visual cue (‘Mismatch Listen’) which instructed them that they were to speak the non-word they hadn’t heard (e.g. ‘kig’ to ‘pob’ and ‘pob’ to ‘kig’). This setup allowed us to isolate the sensory and motor processes and their activity during the delay period. Gray arrows indicate the average response times for the Match (832 ms) and Mismatch (773 ms). c. Electrodes with significant delay activity (1000 ms – 1500 m post Auditory Onset – see Experimental Procedures) are denoted in purple and selectively localized to the prefrontal cortex. d. The time course of delay activity electrodes aligned to the Auditory Onset. Task epochs are denoted in different colors (see legend). Note the elevated delay activity, as well as the activity in the sensory and motor epoch. Error values are SEM of power across electrodes (N = 36 electrodes), one sided p threshold = 0.01.

Figure 2. Example responses

a. Example spectrograms of the neural responses for Delay Electrodes. A Delay+Auditory electrode (top row) during the match-mismatch task: A high gamma (70 Hz +) neural response is seen in the auditory epoch as well as during the delay epoch. In an example Delay+Production electrode (second row) significant neural activity is present in the delay and the production epoch. In an example Delay+Sensory-Motor electrode (third row) significant neural activity is seen in the auditory, delay, and production epochs. Lastly, in an example Delay-Only electrode (bottom row) delay activity is present along with an absence of activity in the auditory and motor epoch. Gray bar reflects the variable delay period (1.5 – 2 seconds post auditory onset). Example electrodes without delay can be seen in Fig S2a and locations of the electrodes can be seen in Fig S2c. While representative, these kinds of neural responses can be seen across subjects (see Fig S3). b. Average high gamma power traces (70 – 160 Hz) are shown for each electrode class as shown in a: Delay+Auditory, Delay+Production, Delay+Sensory-Motor, and Delay-Only. The gray arrows indicate the onset of the cue and average response time. Error values are SEM of power across electrodes (Delay+Auditory: N = 6 Electrodes, Delay+Production: N = 9, Delay+Sensory-Motor: N = 11, Delay-Only: N = 10).

Figure 3. Subdivisions of Delay Activity

a. For each subject, we classified delay activity by a combination of an electrode’s response to a localizer task (see Experimental Procedures) and the delay response during the Sensory-Motor Mismatch Task. Delay+Auditory electrodes (green with purple outline) are active during the delay and auditory presentation of the non-word. Delay+Production electrodes (blue with purple outline) during the delay and articulation of the utterance. Delay+Sensory-Motor electrodes (red with purple outline) are active both during the auditory presentation and the articulation. The Delay-Only electrodes (purple) were active during the delay but not the sensory or motor epoch. b. Localization of electrodes with delay activity across subjects. Color convention the same as in a. One sided p threshold = 0.01

Figure 4. Idealized and representative responses

a. Idealized neural responses should demonstrate patterns in the sensory and motor epochs that represent the class that they belong to. Delay+Auditory (top row) should demonstrate a sensory neural response profile whereas Delay+Production electrodes (second row) should demonstrate a motor neural profile. Delay+Sensory-Motor (third row) should demonstrate a conjunction of a sensory representation in the sensory epoch and a motor representation in the motor epoch. If the Delay-Only responses track the abstract rule representation, they should reflect a differentiation between the ‘match’ and ‘mismatch’ conditions. b. Representative electrodes demonstrate the above hypothesized patterns during the sensory and motor epochs. Error values are SEM of power across trials ( Delay+Auditory: N = 51 Electrodes, Delay+Production: N = 48, Delay+Sensory-Motor: N = 57, Delay-Only: N = 50).

Figure 5. Representation of vWM

a. There are four possible representational models for the content of vWM: A Sensory representation in which the classifier confuses each of the decoded (x-axis) and actual tokens presented (y-axis) with their input sensory equivalent e.g. ‘kig’ to ‘kig’ is confused with ‘kig’ to ‘pob’ (Left most model). A motor model in which the classifier confuses the output production equivalent, e.g. confusing ‘kig’ to ‘kig’ with ‘pob’ to ‘kig’ (second model). A sensory-motor model in which all four sensory-motor mapping conditions are decoded separately in a sensory-motor representation (third model), and a Rule model in which the abstract rule is encoded regardless of token (right panel). b. Example confusion matrices show all four response classes were obtained when applied to the sub-processes of vWM. c. The Strength of each model was assessed for each electrode response category in each temporal epoch using an FDR corrected normalized KL index (KLn, one sided p threshold = 0.005). Each response class represents information differently. The two maintenance systems encode sensory and motor representations respectively: Delay+Auditory electrodes encode an auditory representation during the sensory epoch and the delay epoch, and the Delay+Production electrodes encode the motor plan during the delay, followed by the motor output. The manipulation systems displayed very different representations. The Delay+Sensory-Motor electrodes demonstrated a dynamic representation that switched from an auditory, to a transformation, and finally to production representation, linking perception and production representations. The Delay-Only electrodes demonstrated an abstract rule based representation. The onset of the rule response coincides with the switch from a sensory to transformation representation in the Delay+Sensory-Motor electrodes indicating these two systems work together to manipulate phonological input according to the rule.

Figure 6. Error Analysis

During the delay epoch, all four response classes differentiated between trials with a correct utterance and trials with no utterance (chance is 0.5 – dotted red lines). a) Delay+Auditory responses did not significantly differentiate error trials during the sensory epoch. b) Delay+Production responses significantly differentiated error trials during the late delay, and the production epochs. c) Delay+Sensory-Motor responses significantly distinguished error trials during all task epochs. d) Delay-Only responses significantly distinguished error trials during the cue and delay epochs. These results demonstrate that suggesting that errors are likely due to failures to encode and applying the abstract rule. All values were FDR-corrected with an alpha of 0.05, which resulted in a one sided p threshold of 0.02.