Neurally driven synthesis of learned, complex vocalizations

Ezequiel M Arneodo; Shukai Chen; Daril E Brown 2nd; Vikash Gilja; Timothy Q Gentner

doi:10.1016/j.cub.2021.05.035

Neurally driven synthesis of learned, complex vocalizations

Curr Biol. 2021 Aug 9;31(15):3419-3425.e5. doi: 10.1016/j.cub.2021.05.035. Epub 2021 Jun 16.

Authors

Ezequiel M Arneodo¹, Shukai Chen², Daril E Brown 2nd³, Vikash Gilja³, Timothy Q Gentner⁴

Affiliations

¹ Biocircuits Institute, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA; Department of Psychology, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA; IFLP-CONICET, Departamento de Física, Universidad Nacional de La Plata, CC 67, La Plata 1900, Argentina.
² Department of Bioengineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA.
³ Department of Electrical and Computer Engineering, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA.
⁴ Biocircuits Institute, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA; Department of Psychology, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA; Kavli Institute for Brain and Mind, 9500 Gilman Drive, La Jolla, CA 92093, USA; Neurobiology Section, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA. Electronic address: tgentner@ucsd.edu.

Abstract

Brain machine interfaces (BMIs) hold promise to restore impaired motor function and serve as powerful tools to study learned motor skill. While limb-based motor prosthetic systems have leveraged nonhuman primates as an important animal model,^1-4 speech prostheses lack a similar animal model and are more limited in terms of neural interface technology, brain coverage, and behavioral study design.^5-7 Songbirds are an attractive model for learned complex vocal behavior. Birdsong shares a number of unique similarities with human speech,^8-10 and its study has yielded general insight into multiple mechanisms and circuits behind learning, execution, and maintenance of vocal motor skill.^11-18 In addition, the biomechanics of song production bear similarity to those of humans and some nonhuman primates.^19-23 Here, we demonstrate a vocal synthesizer for birdsong, realized by mapping neural population activity recorded from electrode arrays implanted in the premotor nucleus HVC onto low-dimensional compressed representations of song, using simple computational methods that are implementable in real time. Using a generative biomechanical model of the vocal organ (syrinx) as the low-dimensional target for these mappings allows for the synthesis of vocalizations that match the bird's own song. These results provide proof of concept that high-dimensional, complex natural behaviors can be directly synthesized from ongoing neural activity. This may inspire similar approaches to prosthetics in other species by exploiting knowledge of the peripheral systems and the temporal structure of their output.

Keywords: bioprosthetics; birdsong; brain machine interfaces; electrophysiology; neural networks; nonlinear dynamics; speech.

Publication types

Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

Animals
Brain
Learning*
Songbirds*
Vocalization, Animal*

Grants and funding

R01 DC018446/DC/NIDCD NIH HHS/United States