Scientists develop AI that can translate brain scans into words, sentences

A technique based on artificial intelligence (AI) can translate brain scans into words and sentences, a team of computational neuroscientists reports. Although in the early stages and far from perfect, the new technology might eventually help individuals with brain injuries or paralysis regain the ability to communicate, researchers say.

The study “shows that, using the right methods and better models, we can actually decode what the subject is thinking,” says Martin Schrimpf, a computational neuroscientist at the Massachusetts Institute of Technology who was not involved in the work.

Other research teams have created brain-computer interfaces (BCIs) to, for example, translate a paralyzed patient’s brain activity into words. However, most of these approaches rely on electrodes implanted in the patient’s brain. Noninvasive techniques based on methods such as electroencephalogram (EEG), which measures brain activity via electrodes attached to the scalp, have fared less well. BCIs based on EEG have so far only been able to decipher phrases and can’t reconstruct coherent language, Schrimpf says. Previous BCIs also typically focused on individuals attempting to speak or thinking about speaking, so they relied on areas of the brain involved in producing speech-related movements and only worked when a person was moving or attempting to move.

Now, Alexander Huth, a computational neuroscientist at the University of Texas at Austin, and colleagues have developed a BCI based on functional magnetic resonance imaging (fMRI) that taps more directly into the language-producing areas of the brain to decipher imagined speech. This noninvasive method, commonly used in neuroscience research, tracks changes in blood flow within the brain to measure neural activity.

As with all BCIs, the goal was to associate each word, phrase, or sentence with the particular pattern of brain activity that it evokes. To gather the necessary data, researchers scanned the brains of three participants while each listened to roughly 16 hours of storytelling podcasts such as The Moth Radio Hour and The New York Times’s Modern Love. With those data, the researchers produced a set of maps for each subject that specified how the person’s brain reacts when it hears a certain word, phrase, or meaning. Because fMRI takes a few seconds to record brain activity, it captures not each specific word, but rather the general idea with each phrase and sentence, Huth says. His team used the fMRI data to train the AI to predict how the brain of a certain individual would react to language.

Initially, the system struggled to turn brain scans into language. But then the researchers also incorporated the natural language model GPT to predict what word might come after another. Using the maps generated from the scans and the language model, they went through different possible phrases and sentences to see whether the predicted brain activity matched the actual brain activity. If it did, they kept that phrase and went on to the next one.

Afterward, the subjects listened to podcasts not used in the training. And little by little, the system produced words, phrases, and sentences, eventually producing ideas that accurately matched what the person was hearing. The technology was particularly good at getting the gist of the story, even if it didn’t always get every word right.

It also worked when a subject told a story or saw a video. In one experiment, participants watched a movie without any sound while the system tried to decode what they were thinking. As an individual watched an animated movie where a dragon kicks someone down, the system spouted: “He knocks me to the ground.” All of this happened without participants being asked to speak. “That really demonstrates that what we’re getting at here is something deeper than just language,” Huth says. “[The system] works at the level of ideas.”

The system could someday aid individuals who have lost their ability to communicate because of brain injury, stroke, or locked-in syndrome, a type of paralysis in which individuals are conscious but paralyzed. However, that will require not only advancing the technology by using more training data, but also making it more accessible. Being based on fMRI makes the system expensive and cumbersome to use, but Huth says the team’s aim is to be able to do this with easier, more portable imaging techniques such as EEG.

Although it’s nowhere near being able to decode spontaneous thoughts in the real world, the advance raises concerns that, with improvement, the technology might mimic some type of mind reading. “Our thought when we actually had this working was, ‘Oh my God, this is kind of terrifying,’” Huth recalls. To begin to address these concerns, the authors tested whether a decoder trained on one individual would work on another—it didn’t. Consent and cooperation also seemed to be critical because if individuals resisted, by performing a task such as counting instead of paying attention to the podcast, the system could not decode any meaning from their brain activity.

Even so, privacy is still a big ethical concern for this type of neurotechnology, says Nita Farahany, a bioethicist at Duke University. Researchers should examine the implications of their work and develop safeguards against misuse early on. “We need everybody to participate in ensuring that that happens ethically,” she says. “[The technology] could be really transformational for people who need the ability to be able to communicate again, but the implications for the rest of us are profound.” Science