Researchers use reinforcement learning to train gliders to soar like birds

The words "fly like an eagle" are famously part of a song, but they may also be words that make some scientists scratch their heads. Especially when it comes to soaring birds like eagles, falcons and hawks, who seem to ascend to great heights over hills, canyons and mountain tops with ease. Scientists realize that upward currents of warm air assist the birds in their flight, but they don't know how the birds find and navigate these thermal plumes.

To figure it out, researchers from the University of California San Diego used reinforcement learning to train gliders to autonomously navigate atmospheric thermals, soaring to heights of 700 meters--nearly 2,300 feet. The novel research results, published in the Sept. 19 issue of Nature, highlight the role of vertical wind accelerations and roll-wise torques as viable biological cues for soaring birds. The findings also provide a navigational strategy that directly applies to the development of autonomous soaring vehicles, or unmanned aerial vehicles (UAVs).

"This paper is an important step toward artificial intelligence--how to autonomously soar in constantly shifting thermals like a bird. I was surprised that relatively little learning was needed to achieve expert performance," said Terry Sejnowski, a member of the research team from the Salk Institute for Biological Studies and UC San Diego's Division of Biological Sciences.

Reinforcement learning is an area of machine learning, inspired by behavioral psychology, whereby an agent learns how to behave in an environment based on performed actions and the results. According to UC San Diego Department of Physics Professor Massimo Vergassola and PhD candidate Gautam Reddy, it offers an appropriate framework to identify an effective navigational strategy as a sequence of decisions taken in response to environmental cues.

"We establish the validity of our learned flight policy through field experiments, numerical simulations and estimates of the noise in measurements that is unavoidably present due to atmospheric turbulence," explained Vergassola. "This is a novel instance of learning a navigational task in the field, where learning is severely challenged by a multitude of physical effects and the unpredictability of the natural environment."

In the study, conducted collaboratively with the UC San Diego Division of Biological Sciences, the Salk Institute and the Abdus Salam International Center for Theoretical Physics in Trieste, Italy, the team equipped two-meter wingspan gliders with a flight controller. The device enabled on-board implementation of autonomous flight policies via precise control over bank angle and pitch. A navigational strategy was determined solely from the gliders' pooled experiences collected over several days in the field using exploratory behavioral strategies. The strategies relied on new on-board methods, developed in the course of the research, to accurately estimate the gliders' local vertical wind accelerations and the roll-wise torques, which served as navigational cues.

The scientists' methodology involved estimating the vertical wind acceleration, the vertical wind velocity gradients across the gliders' wings, designing the learning module, learning the thermalling strategy in the field, testing the performance of the learned policy in the field, testing the performance for different wingspans in simulations and estimating the noise in gradient sensing due to atmospheric turbulence.

"Our results highlight the role of vertical wind accelerations and roll-wise torques as viable biological mechanosensory cues for soaring birds, and provide a navigational strategy that is directly applicable to the development of autonomous soaring vehicles," said Vergassola.