Decoding Human Brain Language Activity with Whisper

Researchers used the Whisper model to analyze ECoG and speech signals from four epilepsy patients during natural conversations. Results showed that Whisper's acoustic, speech, and language embeddings accurately predicted neural activity, especially during speech production and comprehension. Speech embeddings excelled in perceptual and motor areas, while language embeddings performed better in higher-level language areas. The study reveals how speech and language information are encoded across multiple brain regions and how speech information influences language processing. It also uncovered distinct temporal dynamics of information flow during speech production and comprehension, and differences between deep learning and symbolic models in predicting neural activity.