Using neuromorphic engineering to reinvent visual processing systems with a biological inspiration

As computer vision (CV) systems become increasingly power and memory intensive, they become unsuitable for high-speed and resource deficit edge applications - such as hypersonic missile tracking and autonomous navigation - because of size, weight, and power constraints.

At the University of Pittsburgh, engineers are ushering in the next generation of computer vision systems by using neuromorphic engineering to reinvent visual processing systems with a biological inspiration - human vision.

Rajkumar Kubendran, assistant professor in Pitt's Swanson School of Engineering and senior member at the Institute of Electrical and Electronics Engineers (IEEE), received a Faculty Early Career Development (CAREER) award from the National Science Foundation (NSF) for his research on energy-efficient and data-efficient neuromorphic systems. Neuromorphic engineering is a promising frontier that will introduce the next generation of CV systems by reducing the number of operations through event-based computation in a biology-inspired framework.

Dr. Kubendran is one of the pioneers in the new field of neuromorphic, event-driven sensors, algorithms, and processors. Our department is most excited about his research and growing success in this important field."

Dr. Alan George, R&H Mickle Endowed Chair and Professor and Department Chair of Electrical and Computer Engineering at Pitt's Swanson School of Engineering

While the current computer vision (CV) pipeline allows computers to use cameras and algorithms to enable visual perception, it consumes an overabundance of power and data. Kubendran will use his $549,795 CAREER award to reinvent traditional camera and processing architectures in the CV pipeline by replacing them with biological-inspired sensors, processors, and algorithms. By taking inspiration from biological systems, Kubendran will be able to optimize CV systems by using substantially less power while eliminating a large amount of data transfer.

"Humans receive visual stimuli through the retina in our eyes, and that data is sent through the optic nerve to our brain's visual cortex for visual perception," Kubendran explained. "The biological vision pipeline is heavily optimized to transfer the least amount of data. If we have retina-inspired camera sensors and visual cortex-inspired processors, then we can train the processors to learn and understand visual scenes and then become capable of versatile spatiotemporal pattern recognition and perception."

Eyes on the future

Kubendran's project has the potential to lead a transformative and generational shift in today's society. Critical areas such as healthcare, military defense, Internet of Things (IoT), edge computing, and industrial automation would benefit from event-based CV. 

"Enabling the use of advanced computer vision on personal electronics can revolutionize our lifestyle through technologies such as self-driving vehicles, always-on smart surveillance, and virtual and augmented reality applications," Kubendran said. "Bio-inspired vision sensors are predominantly developed by European and Asian companies with no industry or academic contribution from the United States. This project will address that national challenge and provide the U.S. with a leading advantage in the deployment of next-generation computer vision systems."

Kubendran's research will happen in three simultaneous thrusts. In Thrust One, the research will focus on creating a new class of retina-inspired vision sensors that can outperform existing cameras in terms of features, efficiency, and latency. Thrust Two will focus on the modeling, design, and implementation of scalable corticomorphic networks on hardware. These networks will be tested to examine whether they exhibit nonlinear neuromodulatory dynamics at multiple timescales using mixed feedback control. Thrust Three focuses on the implementation of network architectures and algorithms inspired by neuroscience, such as reinforcement learning and event-based temporal pattern recognition. 

In addition to the scientific contributions to the CV industry, Kubendran plans to use funding from his CAREER award to develop an educational consortium in the local Pittsburgh community, which will disseminate knowledge gained from the project and train undergraduate students at Pitt as well as students from local high schools to become future neuromorphic and computer vision engineers.