Understanding brain coupling in conversations through linguistic context

Cell PressAug 2 2024

When two people interact, their brain activity becomes synchronized, but it was unclear until now to what extent this "brain-to-brain coupling" is due to linguistic information or other factors, such as body language or tone of voice. Researchers report August 2 in the journal Neuron that brain-to-brain coupling during conversation can be modeled by considering the words used during that conversation, and the context in which they are used.

We can see linguistic content emerge word-by-word in the speaker's brain before they actually articulate what they're trying to say, and the same linguistic content rapidly reemerges in the listener's brain after they hear it."

Zaid Zada (@zaidzada), first author and neuroscientist of Princeton University

To communicate verbally, we must agree on the definitions of different words, but these definitions can change depending on the context. For example, without context, it would be impossible to know whether the word "cold" refers to temperature, a personality trait, or a respiratory infection.

"The contextual meaning of words as they occur in a particular sentence, or in a particular conversation, is really important for the way that we understand each other," says neuroscientist and co-senior author Samuel Nastase (@samnastase) of Princeton University. "We wanted to test the importance of context in aligning brain activity between speaker and listener to try to quantify what is shared between brains during conversation."

To examine the role of context in driving brain coupling, the team collected brain activity data and conversation transcripts from pairs of epilepsy patients during natural conversations. The patients were undergoing intracranial monitoring using electrocorticography for unrelated clinical purposes at the New York University School of Medicine Comprehensive Epilepsy Center. Compared to less invasive methods like fMRI, electrocorticography records extremely high-resolution brain activity because electrodes are placed in direct contact with the surface of the brain.

Next, the researchers used the large language model GPT-2 to extract the context surrounding each of the words used in the conversations, and then used this information to train a model to predict how brain activity changes as information flows from speaker to listener during conversation.

Using the model, the researchers were able to observe brain activity associated with the context-specific meaning of words in the brains of both speaker and listener. They showed that word-specific brain activity peaked in the speaker's brain around 250 ms before they spoke each word, and corresponding spikes in brain activity associated with the same words appeared in the listener's brain approximately 250 ms after they heard them.

Compared to previous work on speaker–listener brain coupling, the team's context-based approach model was better able to predict shared patterns in brain activity.

"This shows just how important context is, because it best explains the brain data," says Zada. "Large language models take all these different elements of linguistics like syntax and semantics and represent them in a single high-dimensional vector. We show that this type of unified model is able to outperform other hand-engineered models from linguistics."

In the future, the researchers plan to expand on their study by applying the model to other types of brain activity data, for example fMRI data, to investigate how parts of the brain not accessible with electrocorticography operate during conversations.

"There's a lot of exciting future work to be done looking at how different brain areas coordinate with each other at different timescales and with different kinds of content," says Nastase.

This research was supported by the National Institutes of Health.