Study illuminates a general principle of neural processing in the brain’s serotonin system

Our lives are filled with binary decisions – choices between one of two alternatives. But what's really happening inside our brains when we engage in this kind of decision making?

A University of Ottawa Faculty of Medicine-led study published in Nature Neuroscience sheds new light on these big questions, illuminating a general principle of neural processing in a mysterious region of the midbrain that is the very origin of our central serotonin (5-HT) system, a key part of the nervous system involved in a remarkable range of cognitive and behavioral functions.

The current dominating model is that individual 5-HT neurons are acting independently one from another. While it had previously been suggested that 5-HT neurons may rather be connected with one another, it had not been directly demonstrated. That is what we did here. We also identify an intriguing processing role – or a computation – that is supported by this particular type of connectivity between 5-HT neurons."

Dr. Jean-Claude Béïque, full professor in the Faculty's Department of Cellular and Molecular Medicine and co-director of the uOttawa Brain and Mind Research Institute's Centre for Neural Dynamics and Artificial Intelligence

The international research team's work involved a mixture of several experimental approaches such as electrophysiology, cellular imaging, optogenetics and behavioral approaches, along with mathematical modeling and computer simulations.

Forging advances

So what does it mean that serotonin neurons clustered together in the brainstem are not independent actors largely keeping to themselves but are actually sending axons to the rest of the brain?

"In my view, the paper's main takeaway is that the mammalian serotonin system is far more anatomically and functionally complex than what we previously imagined. This is knowledge that could potentially help develop targeted therapeutics for mood disorders like major depressive disorder," says Dr. Michael Lynn, the study's first author and a former member of Dr. Béïque's Faculty of Medicine lab.

Dr. Lynn received his PhD in Neuroscience from the University of Ottawa in October 2023. He's now working as a postdoctoral fellow at the University of Oxford, in the Department of Physiology, Anatomy and Genetics.

He says the team's findings are important because it turns out that there are distinct groups of serotonin neurons with their own activity patterns, each controlling serotonin release in a particular region of the brain. This has implications for the "winner-takes-all" principle of neuroscience – an idea applied in computational models of neural networks in which neurons essentially compete to get activated.

"The new principles uncovered in this paper suggest that these distinct ensembles can interact in some scenarios: 'winning' serotonin ensembles with high activity can strongly reduce serotonin release from 'losing' serotonin ensembles with lower activity levels," he says. "These imply a more complex, dynamic set of rules about how and when serotonin is released throughout the brain, contrasting with an older view of a more monolithic signal."

Decisions, decisions

The research team's work has implications for how our brain – an organ with profoundly intricate wiring of neurons with multitudes of enmeshed connections – is involved in day-to-day decision making.

They determined how the lateral habenula, a region that is activated when we are frustrated and that is implicated in major depression, ultimately controls the activity of serotonin neurons. Habenular neurons are also believed to encode the level of threat that is perceived from a particular environment, or perhaps even from our actions.

Dr. Béïque explains it like this: "Do we jump from the high diving board at the pool? Or only from the low one? Do we walk down that very dark alley, or do we avoid it? When is dark too dark? Somehow our brain must compute features of our world – including how threatening a particular environment is – and come up with a binary output: you go, or you don't."

"We think we have identified a circuit that participates in that very computation that guides our everyday decisions," he says.

Next steps

What's next for the research team as they build on the advances they have forged over several years with this methodical, innovative examination of the serotonin system? They aim to focus on behavioral studies with mouse models.

"At this point, the behavioral manifestations of the computation we discovered were somewhat artificial behavior. We're currently trying to see if we can see similar things when mice are behaving in more naturalistic environments," Dr. Béïque says.

The talent-rich research team for the new Nature Neuroscience paper included the uOttawa Faculty of Medicine's Dr. Richard Naud, a computational neuroscientist who was the senior author on a recent serotonin-related study published in Nature, and Sean Geddes, director of Innovation and Partnerships at uOttawa.