Researchers use mixed realty viewer to visualize complex biological networks in 3D

They call it the "hairy ball." It's an unflattering name for two-dimensional representations of a complex biological network, a depiction of a system of linkages and connections so complex and dense that "it looks like a big mess," said Dmitry Korkin, PhD, associate professor of computer science at Worcester Polytechnic Institute (WPI) and director of the university's bioinformatics and computational biology program.

Korkin's mission is to tame this mess by developing new ways of "seeing" complex biological networks, methods that enable researchers to focus on the most important information and connections--to see the trees through the forest. The visualization tools he is developing may hold the key to finding critical links between proteins and genes related to complex disorders like cancer, diabetes, and autism spectrum disorder.

Complex biological networks illustrate the challenge of bioinformatics: vast quantities of valuable data generated through advanced research technologies undoubtedly hold clues to cures, and yet can they seem impenetrable to researchers seeking to analyze them. Korkin, whose research focuses on the bioinformatics of complex diseases, computational genomics, and systems biology, has long been interested in finding new ways to visualize biological networks, which can include everything from the neural connections in the human brain to all of the interactions between proteins within a yeast cell. He said he believes he has found the solution in a technology that was originally envisioned as enhancement for videogames and entertainment.

For more than a year, Korkin and his research team have been using the Microsoft HoloLens System to study protein interaction networks that include mutations associated with complex genetic disorders. The system, a type of mixed realty viewer, creates holographic projections that appear to float at fixed points before the viewer. As he explained this year to attendees at the Intelligent Systems for Molecular Biology Conference and the Cold Spring Harbor Conference on Systems Biology, Korkin believes the technology offers an exciting new route to deeper understanding of networks that are notoriously difficult to visualize.

"It is good that we can collect information on these networks with increasingly precise technologies, but when you try to visually analyze it, you realize that it is next to impossible with conventional means," Korkin said. The lack of adequate visualization techniques not only hinders understanding of complex networks, it constrains research, he said, since the most meaningful and important connections and relationships within these networks may not be readily apparent.

A few years ago, Korkin got his initial look at the HoloLens, a technology that, for the first time, seemed to offer the capabilities he had been searching for. The device is a set of "smart glasses" that take visual information and project it into the space in front of the user. Unlike virtual reality goggles, which are designed to block out reality and put the user into an artificial world, the HoloLens uses mixed reality to merge a computer-generated holographic visualization with the real world. Thus, the networks Korkin studies appear to float above the table in his office, allowing him to walk around and "see" them from all sides.

In 2016 Korkin received two early-release HoloLens kits designed for app developers. Since then, with postdoctoral researcher Pavel Terentiev, PhD, and a team of graduate students and undergraduates, he has been working to take data on actual biological networks his lab has been researching and represent them as a three-dimensional objects in the real world using the HoloLens.

"The network is a very common data structure," Korkin said. "Computer scientists have grappled with networks for ages. The problem with biological networks is that they are incredibly complex, with a lot of information packed into very dense structures. It is nearly impossible to properly visualize these networks with conventional, two-dimensional forms of representation. When I first saw early prototypes of the technology that would become the HoloLens, I realized that this could be the ideal way to interact with these networks."

Among the network types studied in the Korkin Lab are disease networks, made up of the genes and proteins associated with complex genetic disorders, such as cancer, diabetes, and autism spectrum disorder. "These networks can included hundreds of thousands of genes and proteins carrying a plethora of mutations," Korkin said. "Our approach is to use the system information as a framework to merge together information about the properties of these genes and proteins and their functions in an effort to see how individual mutations affect the whole system. This can help us understand the underlying molecular mechanisms affected by the mutations and the possible consequences."

Korkin says the HoloLens will make it easier to make these connections. One current project is using the technology to explore the myriad connections between proteins implicated in breast cancer--an example of the kind of "hairy ball" he is trying to tame. In the Korkin Lab, a user slips on the device, which looks a bit like a futuristic version of laboratory safety glasses. Suddenly a three-dimensional array of colored balls connected by lines appears to hover in the air in the middle of the room. The balls and lines are nodes and connections in the network, representing the proteins and the physical interactions between the proteins. While it floats in space, the network seems rooted in a spot several feet off the floor. As the user walks around the display, he can see the nodes and the connections from various angles.

Reaching out, the user can pinch his thumb and index finger together, as if taking hold of the network, and rotate it, right and left, and up and down, to bring different parts into view. Using gestures or voice commands, the user can highlight particular nodes and then call up information about them, such as the structure of individual proteins or the locations of particular mutations. When another user dons a second HoloLens, they are able to explore the network together as it seemingly sits between them.

"This is not just a visualization tool," Korkin said. "It is a tool for the interactive retrieval of information. It allows us to layer on top of the network a host of related data, including structural information and the presence of mutations. In this way it becomes an intelligent object that integrates information and feeds it back to the researcher with a number of suggested options. By incorporating algorithms that encapsulate basic rules, we can create a smart visualization that can help identify the elements of a network that the researchers should pay attention to--the critical things that they should investigate first."

Korkin says he hopes to bring this capability to bear on a current genomics research program, supported by a 2015 award of $768,000 from the National Science Foundation, in which he is searching for regions of the genome that have been conserved across species and across millions of years. A better understanding of these segments, called long identical multispecies elements, or LIMEs, which are likely to perform basic and vital functions in the cell, may provide new clues to the role of genetics in normal development and the onset of disease.

"We would like to use the HoloLens to map the information that we have discovered using our advanced algorithms, which will allow us to begin to build a comprehensive atlas of these elements--the first of its kind," he said. "There will be thousands of these elements, and we will have the means to start analyzing them. We will be able to visualize them on the chromosome and genome level, bringing in information on the function of the coding and non-coding elements, their mutual positions, the fact that some of these elements occur in multiple copies and some are unique, and so on.

"The goal is to merge different forms of information and look for patterns, for things that are unusual or abnormal. By allowing us to work with the complete set of data, which we simply cannot do in any other way because the information density is too high to comprehend, the HoloLens will make this process more intuitive and more informative."

To fully achieve the benefits of the HoloLens, Korkin says he and his team will have to find ways to work within--or develop techniques to overcome--some of the technology's current limitations, including the limit on the number of objects it can display at one time (a limit well below the tens of thousands of nodes in the networks Korkin studies). "It is typical for new technologies to have bottlenecks. I have no doubt the technology will evolve and this will become less of an issue, but in the meantime we can facilitate this by developing algorithms that allow us to do what we want to do."

Korkin says he sees great opportunities ahead for the HoloLens technology, which he believes can greatly speed up the pace of discovery in many fields by making it less cumbersome and less time-consuming to see relationships between data.

"This tool can be of particular value when it comes to working with very complex data," he said, "for which we have yet to learn how to develop the algorithms. In such cases, we need to build up our intuition, and this is where the HoloLens can become a critical tool. We are all accustomed to dealing with a 3D world, and this technology transforms complex systems into truly three-dimensional objects that we can interact with naturally and see in entirely new ways."