Year: 2025
Type: preprint
Abstract: We investigate optimal strategies for decoding perceived natural speech from fMRI data acquired from a limited number of participants. Leveraging Lebel et al. (2023)'s dataset of 8 participants, we first demonstrate the effectiveness of training deep neural networks to predict LLM-derived text representations from fMRI activity. Then, in this data regime, we observe that multi-subject training does not improve decoding accuracy compared to single-subject approach. Furthermore, training on similar or different stimuli across subjects has a negligible effect on decoding accuracy. Finally, we find that our decoders better model syntactic than semantic features, and that stories containing sentences with complex syntax or rich semantic content are more challenging to decode. While our results demonstrate the benefits of having extensive data per participant (deep phenotyping), they suggest that leveraging multi-subject for natural speech decoding likely requires deeper phenotyping or a substantially larger cohort.
Source: arXiv (Cornell University)
Institutions: Commissariat à l'Énergie Atomique et aux Énergies Alternatives, Université Paris-Saclay, École Normale Supérieure - PSL, Centre National de la Recherche Scientifique, Inserm, École des hautes études en sciences sociales, Institut national de recherche en sciences et technologies du numérique
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
Funder: Agence Nationale de la Recherche
OpenAlex ID: W4415037217
Cites:
Cited by:
Related to: