First smart wearable device tracks how much people use their voices

Northwestern University researchers have developed the first smart wearable device to continuously track how much people use their voices, alerting them to overuse before vocal fatigue and potential injury set in.

The first-of-its-kind, battery-powered, wireless device and accompanying algorithms could be a game-changer for professional singers, teachers, politicians, call-center workers, coaches and anyone who relies on their voices to communicate effectively and make a living. It also could help clinicians remotely and continuously monitor patients with voice disorders throughout their treatment.

Developed by an interdisciplinary team of materials scientists, biomedical engineers, opera singers and a speech-language pathologist, the research behind the new technology will be published during the week of Feb. 20 in the Proceedings of the National Academy of Sciences.

The soft, flexible, postage-stamp-sized device comfortably adheres to the upper chest to sense the subtle vibrations associated with talking and singing. From there, the captured data is instantaneously streamed via Bluetooth to the users' smartphone or tablet, so they can monitor their vocal activities in real time throughout the day and measure cumulative total vocal usage. Custom machine-learning algorithms distinguish the difference between speaking and singing, enabling singers to separately track each activity.

With the app, users can set their personalized vocal thresholds. When they near that threshold, their smartphone, smartwatch or an accompanying device located on the wrist provides real-time haptic feedback as an alert. Then, they can rest their voices before pushing it too far.

"The device precisely measures the amplitude and frequency for speaking and singing," said Northwestern's John A. Rogers, a bioelectronics pioneer who led the device's development. "Those two parameters are most important in determining the overall load that's occurring on the vocal folds. Being aware of those parameters, both at a given instant and cumulatively over time, is essential for managing healthy patterns of vocalization."

"It's easy for people to forget how much they use their voice," said Northwestern's Theresa Brancaccio, a voice expert who co-led the study. "Seasoned classical singers tend to be more aware of their vocal usage because they have lived and learned. But some people -; especially singers with less training or people, like teachers, politicians and sports coaches, who must speak a lot for their jobs -; often don't realize how much they are pushing it. We want to give them greater awareness to help prevent injury."

Rogers is the Louis Simpson and Kimberly Querrey Professor of Materials Science and Engineering, Biomedical Engineering and Neurological Surgery in the McCormick School of Engineering and Northwestern University Feinberg School of Medicine. He also is director of the Querrey Simpson Institute for Bioelectronics. A distinguished operatic performer, mezzo-soprano, Brancaccio is a senior lecturer at Northwestern's Bienen School of Music, where she teaches voice and vocal pedagogy.

Unaware of overuse

For the millions of people in the U.S. who make their livings by speaking or singing, vocal fatigue is a constant, looming threat. The common condition occurs when overused vocal folds swell, making the voice sound raspy and lose endurance. Vocal fatigue negatively affects singers, in particular, altering their abilities to sing clearly or hit the same notes as their healthy voice can. At best, one short period of vocal fatigue can briefly interrupt a singer's plans. At worst, it can lead to enough damage to derail a career.

Lack of awareness is the underlying problem. People rarely make the connection between vocal activities and how those activities affect their voices. Although one in 13 U.S. adults have experienced vocal fatigue, most people don't notice they are overusing their voices until hoarseness already has set in.

What leads people into trouble is when events stack up. They might have rehearsals, teach lessons, talk during class discussions and then go to a loud party, where they have to yell over the background noise. Then, throw a cold or illness into the mix. People have no idea how much they are coughing or clearing their throats. When these events stack up for days, that can put major stress on the voice."

Theresa Brancaccio, voice expert, Northwestern University

Cross-disciplinary connection

As an advocate for vocal health, Brancaccio has spent decades exploring ways to keep her students mindful of how much they use their voices. In 2009, she challenged her students to keep a paper budget -; physically writing down every time they spoke, sang and drank water, among other things. About 10 years later, she converted the system into Singer Savvy, an app that offers a personalized vocal budget for each user and helps users stay within that budget.

Separately, Rogers, in collaboration with researchers at the Shirley Ryan AbilityLab, had developed a wireless wearable device to track swallowing and speech in stroke patients. The bandage-like sensor measures swallowing abilities and speech patterns to monitor stroke patients' recovery processes. In the early weeks of the COVID-19 pandemic, Rogers' team modified the technology to monitor coughing, as a key symptom of the illness.

"I wanted to gather more data and make our tracking system more precise and more accurate," Brancaccio said. "So, I reached out to John to see if his sensors could help us gather more information."

"I thought it was a great opportunity for us to extend our technologies beyond our very important, but narrowly targeted, uses in health care to something that might capture a broader population of users," Rogers said. "Anyone who uses their voice extensively could benefit."

The pair also partnered with speech pathologist and voice expert Aaron M. Johnson to explore how the devices could be used to evaluate and monitor treatment for patients with vocal disorders. Johnson, who co-directs the NYU Langone's Voice Center, said the small, wireless device could help track patients' voices in the real world -; outside of a clinical setting.

"A key part of voice therapy is helping people change how -; and how much -; they use their voice," said Johnson, study co-author and associate professor in the department of otolaryngology at NYU Grossman School of Medicine. "This device will enable patients and their clinicians to understand voice use patterns and make adjustments in vocal demand to reduce vocal fatigue and speed recovery from voice disorders. Generalizing vocal techniques and exercises from therapy sessions into daily life is one of the most challenging aspects of voice therapy, and this device could greatly enhance that process."

Singer-trained algorithms

The team modified Rogers' existing devices to precisely measure vocal load over time. That includes frequency, volume, amplitude, duration and time of day. Like Rogers' previous devices for COVID-19 and stroke patients, the new device also senses vibrations rather than recording audio. This enables the device to detect vocal activity precisely from the user, rather than the ambient noise surrounding them.

The biggest challenge was to develop algorithms capable of distinguishing speaking from singing. To overcome this challenge, Brancaccio recruited voice and opera students to undertake a variety of singing exercises to train the machine-learning algorithms. A team of classical singers with different vocal ranges -; varying from bass to soprano -; wore the devices while humming, singing staccato scales and songs, reading and more. Each singer generated 2,500 one-second-long windows of singing and 2,500 one-second-long windows of speaking.

The resulting algorithm can separate singing from speaking with more than 95% accuracy. And, when used in a choir setting, the device captures only data from the wearer and not noise from nearby singers.

"Prolonged talking is one of the most fatiguing activities for people who are training to become professional singers," Brancaccio said. "By separating singing and speaking, it can help people develop more awareness around how much they are speaking. There is evidence that even brief 15- to 20-minute periods of total silence interspersed throughout the day can help vocal fold tissues recover and repair."

How to use it

To use the device, the wearer simply adheres it to the sternum, below the neck, and syncs the device with the accompanying app. Rogers' team currently is working on a method to personalize vocal budgets for each user. Here, users will press a button in the app if they experience vocal discomfort at any point during the day, effectively capturing the instantaneous and cumulative vocal load at the time. These data can serve as a personalized threshold for vocal fatigue. When the user nears or exceeds their personalized threshold, a haptic device will vibrate as an alert.

Similar in size and form to a wristwatch, this haptic device includes multiple motors that can activate in different patterns and with varying levels of intensity to convey different messages. Users also can monitor a graphical display within the app, which splits information into speaking and singing categories.

"It uses Bluetooth, so it can talk to any device that has a haptic motor embedded," Rogers said. "So, you don't have to use our wristband. You could just leverage a standard smart watch for haptic feedback."

Although other vocal-monitoring devices do exist, those use big collars, tethering wires and bulky equipment. Some also use embedded microphones to capture audible vocal data, leading to privacy concerns.

"Those don't work for continuous monitoring in a real environment," Brancaccio said. "Instead of wearing cumbersome, wired equipment, I can stick on this soft, wearable device. Once it's on, I don't even notice it. It's super light and easy."

What's next

Because Rogers' previous devices capture body temperature, heart rates and respiratory activity, the researchers included those capabilities in the vocal-monitoring device. They believe these extra data will help to explore fundamental research questions concerning vocal fatigue.

"This is more speculative, but it might be interesting to see how physical activity affects vocal fatigue," Rogers said. "If someone is dancing while singing, is that more stressful on the vocal folds compared to someone who is not physically exerting themselves? Those are the kinds of questions we can ask and quantitatively answer."

In the meantime, Brancaccio is excited for her students to have a tool that can help prevent injury. She hopes others -; including non-singers -; will see the benefit to keeping their vocal cords healthy.

"Your voice is part of your identity -; whether you are a singer or not," she said. "It's integral to daily life, and it's worth protecting."

The study, "Closed-loop network of skin-interfaced wireless devices for quantifying vocal fatigue and providing user feedback," was supported by the Querrey Simpson Institute for Bioelectronics at Northwestern University.