Medicare's Hospital Compare portal underestimates AMI mortality rates, study finds

Apr 19 2017

New analysis of Medicare's Hospital Compare portal shows the statistical methodology used to rate and compare hospitals underestimates mortality rates of acute myocardial infarction (AMI) at small hospitals. The research, titled "Mortality Rate Estimation and Standardization for Public Reporting: Medicare's Hospital Compare," appears in the Journal of the American Statistical Association.

Hospital Compare collects data from Medicare claims and hospitals. That data is then processed through a statistical model known as a "random effects logit model," which unfortunately shrinks mortality rates from small hospitals to resemble the national average.

"Hospital Compare's finding of average risk at small hospitals is a mistake because the current model is not properly calibrated," notes Edward I. George, professor of statistics at the University of Pennsylvania's Wharton School and one of the study's authors. "It's a mistake that has implications for patients."

Low volume hospitals, by definition, have little data regarding AMI mortality. In using the current model, a poorly performing small volume hospital will look average, and even a large hospital that performs far better than average will yield results that underestimate their quality. "The underestimation of AMI mortality rates at small hospitals, as seen in Hospital Compare, contradicts previously established research and consistent findings that mortality rates are typically higher at low volume hospitals," adds Jeffrey H. Silber, professor of pediatrics at the Children's Hospital of Philadelphia, professor of health care management at The Wharton School, and one of the study's authors.

George, Silber, and fellow researchers Paul R. Rosenbaum, also a professor of statistics at The Wharton School; Veronika Ročková, assistant professor of econometrics and statistics at The University of Chicago's Booth School of Business; and Ville Satopää, assistant professor of technology and operations management at INSEAD, propose an alternative model that yields more accurate and appropriately calibrated predictions.

This model would incorporate additional factors such as hospital volume, nursing staff, medical residents, and the hospital's ability to perform cardiovascular procedures into the calculation process. Combined with a different statistical method known as "direct standardization," the researchers' approach is seen to be more effective for adjusting mortality rate comparisons between hospitals.

"Patients deserve to have the most accurate information available so they can make well-informed health care decisions. Our analysis demonstrates that further modeling with additional hospital attributes has the clear potential to shed more light on the rankings of such hospitals," the authors conclude.

Evaluating and assessing hospital performance, while challenging, is nothing new to the statistical community. In 2011, the Committee of Presidents of Statistical Societies (COPSS) published "Statistical Issues in Assessing Hospital Performance," which examined statistical issues identified by the Centers for Medicare and Medicaid Services (CMS) and other stakeholders about the government agency's approach to modeling hospital quality based on outcomes. Then chair of COPSS, Thomas Louis, a professor in the department of biostatistics at Johns Hopkins University's Bloomberg School of Public Health, knows this issue well.

"Adopting the modeling enhancements proposed by the authors in this new Hospital Compare research will improve predicting outcomes and choosing a hospital," said Louis. "I urge the CMS and all researchers working on profiling methods to seriously consider issues the authors have identified and their state-of-the-art approaches to many of them."