Hand-gesture decoding using data from noninvasive brain imaging

Reviewed by Lily Ramsey, LLMMay 19 2023

Researchers from University of California San Diego have found a way to distinguish among hand gestures that people are making by examining only data from noninvasive brain imaging, without information from the hands themselves. The results are an early step in developing a non-invasive brain-computer interface that may one day allow patients with paralysis, amputated limbs or other physical challenges to use their mind to control a device that assists with everyday tasks.

The research, recently published online ahead of print in the journal Cerebral Cortex, represents the best results thus far in distinguishing single-hand gestures using a completely noninvasive technique, in this case, magnetoencephalography (MEG).

"Our goal was to bypass invasive components," said the paper's senior author Mingxiong Huang, PhD, co-director of the MEG Center at the Qualcomm Institute at UC San Diego. Huang is also affiliated with the Department of Electrical and Computer Engineering at the UC San Diego Jacobs School of Engineering and the Department of Radiology at UC San Diego School of Medicine, as well as the Veterans Affairs (VA) San Diego Healthcare System. "MEG provides a safe and accurate option for developing a brain-computer interface that could ultimately help patients."

The researchers underscored the advantages of MEG, which uses a helmet with embedded 306-sensor array to detect the magnetic fields produced by neuronal electric currents moving between neurons in the brain. Alternate brain-computer interface techniques include electrocorticography (ECoG), which requires surgical implantation of electrodes on the brain surface, and scalp electroencephalography (EEG), which locates brain activity less precisely.

With MEG, I can see the brain thinking without taking off the skull and putting electrodes on the brain itself. I just have to put the MEG helmet on their head. There are no electrodes that could break while implanted inside the head; no expensive, delicate brain surgery; no possible brain infections."

Roland Lee, MD, study co-author, director of the MEG Center at the UC San Diego Qualcomm Institute, emeritus professor of radiology at UC San Diego School of Medicine, and physician with VA San Diego Healthcare System

Lee likens the safety of MEG to taking a patient's temperature. "MEG measures the magnetic energy your brain is putting out, like a thermometer measures the heat your body puts out. That makes it completely noninvasive and safe."

Rock paper scissors

The current study evaluated the ability to use MEG to distinguish between hand gestures made by 12 volunteer subjects. The volunteers were equipped with the MEG helmet and randomly instructed to make one of the gestures used in the game Rock Paper Scissors (as in previous studies of this kind). MEG functional information was superimposed on MRI images, which provided structural information on the brain.

To interpret the data generated, Yifeng ("Troy") Bu, an electrical and computer engineering PhD student in the UC San Diego Jacobs School of Engineering and first author of the paper, wrote a high-performing deep learning model called MEG-RPSnet.

"The special feature of this network is that it combines spatial and temporal features simultaneously," said Bu. "That's the main reason it works better than previous models."

When the results of the study were in, the researchers found that their techniques could be used to distinguish among hand gestures with more than 85% accuracy. These results were comparable to those of previous studies with a much smaller sample size using the invasive ECoG brain-computer interface.

The team also found that MEG measurements from only half of the brain regions sampled could generate results with only a small (2 – 3%) loss of accuracy, indicating that future MEG helmets might require fewer sensors.

Looking ahead, Bu noted, "This work builds a foundation for future MEG-based brain-computer interface development."