A one-star disparity on health care facility Yelp reviews could indicate a 60-death-peryear difference between some United States counties where those facilities are located, according to researchers at the Penn Medicine Center for Digital Health. Their study shows that counties holding health care facilities with the greatest share of 1-star Yelp reviews had the highest death rates, and a difference of just one point – roughly one star – between counties' average scores could indicate a mortality rate that is better or worse by dozens of lives. This work was published today in the JAMA Network Open.
Many of the facilities that provide essential care may not otherwise have standardized measures or approaches to collect data about patients' experience of care. This is a missed opportunity. Much of the focus in health care is on quality and outcomes – patient experience is also critically important and should be factored into how to improve care across the board. This appears to be one novel data source for doing that."
Raina Merchant, MD, study's senior author, director, Center for Digital Health and a professor of Emergency Medicine, Perelman School of Medicine at the University of Pennsylvania
More than 95,000 different facilities that provided some form of care recognized by the Affordable Care Act were included in the study led by Merchant and its lead author, Daniel Stokes, MD, a researcher with the Center for Digital Health and an internal medicine resident at UCLA Health. Each entity included in the study had at least three reviews between 2015 and 2019 on Yelp, a review website which uses a five-star rating system. Each health care facility's ratings were also coded to the specific United States county it was located in, resulting in more than 1,300 counties (roughly a third of the country) being represented in the work.
Overall, health care facilities achieved an average 2.9 score (out of 5 stars), but reviews were weighted very heavily to either side of the scale: five-star reviews account for 52.9 percent of all reviews, while one-stars made up 33.3 percent.
When the researchers looked at the county-level data of reviews, though, they found that five-star reviews within the group with the lowest death rates made up 55.6 percent of their total, while one-star reviews were at just 29.1 percent. In the group of counties with the highest death rates, five-star reviews made up only 42.9 percent of the total, compared to 38.8 percent one-stars.
The researchers then found that if a county's health facilities' reviews were a star higher than their average – one point on the scale – models indicated that it translated to 18 fewer deaths per 100,000 residents. But when the study was refined to include counties with three or more health care facilities, the impact was greater, indicating a reduction in roughly 53 deaths per 100,000. Refined even further to counties with five health care facilities or more, the impact grew to approximately 60 preventable deaths.
"Patient experiences of care are often ignored when it comes to measuring healthcare quality, but how patients feel about the care they are receiving has an impact on how they engage with health care and, in turn, their own health and well-being," said Stokes. "Improving patient satisfaction is not a secondary outcome, but a primary one. If patients don't feel as though they are being heard or respected in the care they receive, it doesn't matter how closely we adhere to evidence-based treatments because the mutual trust and partnership on which quality healthcare depends will continue to erode."
The fact that the Yelp reviews provide narratives is especially useful. Using natural language processing algorithms, the researchers were able to gain special, categorical insights. They showed that the types of words most associated with one-star reviews related to time (such as "hours" and "waiting"), payment ("money" and "pay") and interpersonal interactions ("rude" and "told").
Common language in five-star reviews changed depending on location. In high-mortality counties, "friendly," "nice," and "staff" were all typical, while low-mortality counties were associated with "Dr.," "helpful," "question," and, surprisingly, "pain."
"'Pain' could reflect that pain management is an important patient experience metric and that may be better addressed in some facilities than others," said Merchant, who added that it might be a good topic for further study.
The researchers believe their work adds evidence that unfiltered online repositories like review sites and social media contain valuable patient feedback and are an untapped resource for informing health care providers about what they do.
"With the ubiquity of social media, it is now commonplace for individuals to use the internet to share with others about their experiences – we rely on this for restaurants, clothing purchases, everything," said Merchant. "These mediums are now similarly being used for sharing experiences about health care. Ideally this democratization of information can help us to improve health care for all and reduce our blindspots."
Moving forward, Merchant, Stokes, and their team hope to look into how to codify this online information to make it easier to use as a tool for health care entities looking to improve care.
"Online reviews of healthcare facilities provide direct insight into patients' experiences of care and can be a powerful force in shaping the care we provide to be more patient-centered," Stokes said. "This has important implications for both individual and community health."
Study shows large language models susceptible to misinformation