AI-powered digital twin of mouse visual cortex

Much as a pilot might practice maneuvers in a flight simulator, scientists might soon be able to perform experiments on a realistic simulation of the mouse brain. In a new study, Stanford Medicine researchers and collaborators used an artificial intelligence model to build a "digital twin" of the part of the mouse brain that processes visual information.

The digital twin was trained on large datasets of brain activity collected from the visual cortex of real mice as they watched movie clips. It could then predict the response of tens of thousands of neurons to new videos and images.

Digital twins could make studying the inner workings of the brain easier and more efficient.

If you build a model of the brain and it's very accurate, that means you can do a lot more experiments. The ones that are the most promising you can then test in the real brain."

Andreas Tolias, PhD, Study Senior Author and Professor, Stanford Medicine 

The lead author of the study is Eric Wang, PhD, a medical student at Baylor College of Medicine.

Beyond the training distribution

Unlike previous AI models of the visual cortex, which could simulate the brain's response to only the type of stimuli they saw in the training data, the new model can predict the brain's response to a wide range of new visual input. It can even surmise anatomical features of each neuron.

The new model is an example of a foundation model, a relatively new class of AI models capable of learning from large datasets, then applying that knowledge to new tasks and new types of data - or what researchers call "generalizing outside the training distribution."

(ChatGPT is a familiar example of a foundation model that can learn from vast amounts of text to then understand and generate new text.)

"In many ways, the seed of intelligence is the ability to generalize robustly," Tolias said. "The ultimate goal - the holy grail - is to generalize to scenarios outside your training distribution."

Mouse movies

To train the new AI model, the researchers first recorded the brain activity of real mice as they watched movies - made-for-people movies. The films ideally would approximate what the mice might see in natural settings.

"It's very hard to sample a realistic movie for mice, because nobody makes Hollywood movies for mice," Tolias said. But action movies came close enough.

Mice have low-resolution vision - similar to our peripheral vision - meaning they mainly see movement rather than details or color. "Mice like movement, which strongly activates their visual system, so we showed them movies that have a lot of action," Tolias said.

Over many short viewing sessions, the researchers recorded more than 900 minutes of brain activity from eight mice watching clips of action-packed movies, such as Mad Max. Cameras monitored their eye movements and behavior.

The researchers used the aggregated data to train a core model, which could then be customized into a digital twin of any individual mouse with a bit of additional training.

Accurate predictions

These digital twins were able to closely simulate the neural activity of their biological counterparts in response to a variety of new visual stimuli, including videos and static images. The large quantity of aggregated training data was key to the digital twins' success, Tolias said. "They were impressively accurate because they were trained on such large datasets."

Though trained only on neural activity, the new models could generalize to other types of data.

The digital twin of one particular mouse was able to predict the anatomical locations and cell type of thousands of neurons in the visual cortex as well as the connections between these neurons. 

The researchers verified these predictions against high-resolution, electron microscope imaging of that mouse's visual cortex, which was part of a larger project to map the structure and function of the mouse visual cortex in unprecedented detail. The results of that project, known as MICrONS, was published simultaneously in Nature.

Opening the black box

Because a digital twin can function long past the lifespan of a mouse, scientists could perform a virtually unlimited number of experiments on essentially the same animal. Experiments that would take years could be completed in hours, and millions of experiments could run simultaneously, speeding up research into how the brain processes information and the principles of intelligence. 

"We're trying to open the black box, so to speak, to understand the brain at the level of individual neurons or populations of neurons and how they work together to encode information," Tolias said.

In fact, the new models are already yielding new insights. In another related study, also simultaneously published in Nature, researchers used a digital twin to discover how neurons in the visual cortex choose other neurons with which to form connections.

Scientists had known that similar neurons tend to form connections, like people forming friendships. The digital twin revealed which similarities mattered the most. Neurons prefer to connect with neurons that respond to the same stimulus - the color blue, for example - over neurons that respond to the same area of visual space.

"It's like someone selecting friends based on what they like and not where they are," Tolias said. "We learned this more precise rule of how the brain is organized."

The researchers plan to extend their modeling into other brain areas and to animals, including primates, with more advanced cognitive capabilities.

"Eventually, I believe it will be possible to build digital twins of at least parts of the human brain," Tolias said. "This is just the tip of the iceberg."