Optimal Transport and Contrastive Learning for Brain Decoding of Musical Perception

Annu Int Conf IEEE Eng Med Biol Soc. 2025 Jul:2025:1-7. doi: 10.1109/EMBC58623.2025.11253498.

Abstract

Brain decoding aims to reconstruct external stimuli from brain activity, providing insights into the neural representation of cognitive experiences. Music decoding from functional magnetic resonance imaging (fMRI) is particularly challenging due to the complexity of auditory processing and the temporal limitations of fMRI signals. In this study, we introduce a novel decoding framework that improves the alignment between fMRI activity and latent musical representations extracted using a pre-trained multimodal model (CLAP). We propose a dual-loss approach combining Optimal Transport and Contrastive Learning to enhance feature mapping and retrieval accuracy. The first loss ensures structural consistency between brain-predicted and true musical embeddings, while the contrastive loss refines the embedding space by maximizing similarities between corresponding pairs and minimizing non-correspondences. Using fMRI data from five subjects listening to music tracks from the GTZAN dataset, our method achieves improved decoding performance, surpassing traditional regression-based approaches from 22.1% top-1 accuracy to 29.3%. These results highlight the potential of integrating Optimal Transport and Contrastive Learning to improve brain decoding performance, paving the way for extending the approach to different sensory domains and applications in Brain-Computer Interfaces (BCI).Clinical relevance- This study could have clinical implications for understanding auditory processing disorders and developing neurorehabilitation strategies. By elucidating how the brain encodes complex auditory stimuli, this approach may contribute to BCI applications for speech and music perception restoration in individuals with hearing impairments or neurological conditions affecting auditory cognition.

MeSH terms

  • Algorithms
  • Auditory Perception* / physiology
  • Brain Mapping / methods
  • Brain* / diagnostic imaging
  • Brain* / physiology
  • Brain-Computer Interfaces
  • Humans
  • Learning*
  • Magnetic Resonance Imaging
  • Male
  • Music*