Scientists achieve breakthrough in eye movement detection technology

Eye tracking plays a critical role in the latest virtual and augmented reality headsets and is an important technology in the entertainment industry, scientific research, medical and behavioral sciences, automotive driving assistance and industrial engineering. Tracking the movements of the human eye with high accuracy, however, is a daunting challenge.

Researchers at the University of Arizona Wyant College of Optical Sciences have now demonstrated an innovative approach that could revolutionize eye-tracking applications. Their study, published in Nature Communications, finds that integrating a powerful 3D imaging technique known as deflectometry with advanced computation has the potential to significantly improve state-of-the-art eye tracking technology.

Current eye-tracking methods can only capture directional information of the eyeball from a few sparse surface points, about a dozen at most. With our deflectometry-based method, we can use the information from more than 40,000 surface points, theoretically even millions, all extracted from only one single, instantaneous camera image."

Florian Willomitzer, associate professor of optical sciences and principal investigator of the study

"More data points provide more information that can be potentially used to significantly increase the accuracy of the gaze direction estimation," said Jiazhang Wang, postdoctoral researcher in Willomitzer's lab and the study's first author. "This is critical, for instance, to enable next-generation applications in virtual reality. We have shown that our method can easily increase the number of acquired data points by a factor of more than 3,000, compared to conventional approaches."

Deflectometry is a 3D imaging technique that allows for the measurement of reflective surfaces with very high accuracy. Common applications of deflectometry include scanning large telescope mirrors or other high-performance optics for the slightest imperfections or deviations from their prescribed shape.

Leveraging the power of deflectometry for applications outside the inspection of industrial surfaces is a major research focus of Willomitzer's research group in the U of A Computational 3D Imaging and Measurement Lab. The team pairs deflectometry with advanced computational methods typically used in computer vision research. The resulting research track, which Willomitzer calls "computational deflectometry," includes techniques for the analysis of paintings and artworks, tablet-based 3D imaging methods to measure the shape of skin lesions, and eye tracking.

"The unique combination of precise measurement techniques and advanced computation allows machines to 'see the unseen,' giving them 'superhuman vision' beyond the limits of what humans can perceive," Willomitzer said.

In this study, the team conducted experiments with human participants and a realistic, artificial eye model. The team measured the study subjects' viewing direction and was able to track their gaze direction with accuracies between 0.46 and 0.97 degrees. When tested on the artificial eye model, the error was around just 0.1 degrees.

Instead of depending on a few infrared point light sources to acquire information from eye surface reflections, the new method uses a screen displaying known structured light patterns as the illumination source. Each of the more than 1 million pixels on the screen can thereby act as an individual point light source.

By analyzing the deformation of the displayed patterns as they reflect off the eye surface, the researchers can obtain accurate and dense 3D surface data from both the cornea, which overlays the pupil, and the white area around the pupil, known as the sclera, Wang explained.

"Our computational reconstruction then uses this surface data together with known geometrical constraints about the eye's optical axis to accurately predict the gaze direction," he said.

In a previous study, the team has already explored how the technology could seamlessly integrate with virtual reality and augmented reality systems by potentially using a fixed embedded pattern in the headset frame or the visual content of the headset itself – be it still images or video –as the pattern that is reflected from the eye surface. This can significantly reduce system complexity, the researchers say. Moreover, future versions of this technology could use infrared light instead of visible light, allowing the system to operate without distracting users with visible patterns.

"To obtain as much direction information as possible from the eye's cornea and sclera without any ambiguities, we use stereo-deflectometry paired with novel surface optimization algorithms," Wang said. "The technique determines the gaze without making strong assumptions about the shape or surface of the eye, as some other methods do, because these parameters can vary from user to user."

In a desirable "side effect," the new technology creates a dense and accurate surface reconstruction of the eye, which could potentially be used for on-the-fly diagnosis and correction of specific eye disorders in the future, the researchers added.

Aiming for the next technology leap

While this is the first time deflectometry has been used for eye tracking – to the researchers' knowledge – Wang said, "It is encouraging that our early implementation has already demonstrated accuracy comparable to or better than commercial eye-tracking systems in real human eye experiments."

With a pending patent and plans for commercialization through Tech Launch Arizona, the research paves the way for a new era of robust and accurate eye-tracking. The researchers believe that with further engineering refinements and algorithmic optimizations, they can push the limits of eye tracking beyond what has been previously achieved using techniques fit for real-world application settings. Next, the team plans to embed other 3D reconstruction methods into the system and take advantage of artificial intelligence to further improve the technique.

 "Our goal is to close in on the 0.1-degree accuracy levels obtained with the model eye experiments," Willomitzer said. "We hope that our new method will enable a new wave of next-generation eye tracking technology, including other applications such as neuroscience research and psychology."

Co-authors on the paper include Oliver Cossairt, adjunct associate professor of electrical and computer engineering at Northwestern University, where Willomitzer and Wang started the project, and Tianfu Wang and Bingjie Xu, both former students at Northwestern.