HealthGrades identifies major flaws in hospital ratings study

HealthGradesMay 20 2011

The findings of a journal article published in the May issue of the Archives of Surgery are fundamentally flawed and could undermine patients ability to find high quality hospital care, according to HealthGrades. The article, Evaluating Popular Media and Internet-Based Hospital Quality Ratings for Cancer Surgery, compared outcomes for gastrointestinal cancer patients who underwent surgery at hospitals on HealthGrades' "America's 50 Best Hospitals" and US News and World Report's "America's Best Hospitals" lists. The study asserts that hospitals with the best outcomes for three surgical procedures were often not among those listed as top performers by HealthGrades and US News. HealthGrades America's 50 Best Hospital designation is based on patient outcomes across 27 common medical procedures and diagnoses and is intended to reflect a broad spectrum of clinical excellence in patient care.  

"While we're encouraged by the study authors' attempts to help patients find accurate quality information, in this case this study may have the opposite effect and might actually discourage patients from getting the information they need to make critically important health care decisions," said Dr. Rick May, HealthGrades Vice President, Accelerated Clinical Excellence. "More than 10 million patients a month visit HealthGrades.com looking for information on hospitals and doctors and they trust us to provide the most accurate information on health care quality. We take that responsibility seriously and that's why, for the last decade, we have conducted the most rigorous statistical analyses of hospital quality in America. We stand firmly behind the accuracy of our hospital ratings, including our list of America's 50 Best Hospitals."

HealthGrades identified major flaws in the methodology upon which the Archives of Surgery study findings are based.

• The study authors compared two disparate groups of patients: Cancer patients undergoing cancer surgery were compared to non-cancer patients with a variety of other diagnoses. For example, the study essentially compares the outcomes of cancer patients to the outcomes of patients having knee replacements or being treated for a stroke. HealthGrades "America's 50 Best Hospitals" analyzes outcomes data from 27 different procedures and diagnoses, none of which are the three focused on in this study, pancreatectomy, esophagectomy and colectomy. HealthGrades hospital ratings for gastrointestinal procedures and diagnoses exclude patients with primary and metastatic cancers because tumor staging and grading information is not available through the Medicare administrative data set.  Thus, it is not possible to adequately risk adjust this data.
• The study included only a small fraction of the Medicare patient records analyzed by HealthGrades in its America's 50 Best Hospitals study: The 2008 HealthGrades America's 50 Best Hospitals study recognizes hospitals that have demonstrated superior performance spanning eight years of Medicare patient data (1999 to 2006), across common 27 diagnoses and procedures. The HealthGrades analysis includes over 100 million patient records. The study authors used only one year of data (2005 – 2006) and outcomes for just three procedures, which accounts for 85,000 patient records.
• The study methodology tries to compare two different treatment time frames and different outcome measures: HealthGrades America's 50 Best Hospitals study uses in-patient mortality and complications, whereas the study authors only looked at 30-day mortality.

HealthGrades America's 50 Best Hospitals are determined by independently and objectively analyzing more than 100 million Medicare patient records for 27 medical procedures and conditions. To be included in the analysis, hospitals must have met minimum thresholds in terms of patient volumes, quality ratings, and the range of services provided. Individuals may compare their local hospitals online at HealthGrades.com. HealthGrades' hospital ratings are independently created; no hospital can opt-in or opt-out of being rated, and no hospital pays to be rated. Mortality and complication rates are risk adjusted, which takes into account differing levels of severity of patient illness at different hospitals and allows for hospitals to be compared equally.

Complete procedure-by-procedure comparisons of the HealthGrades America's 50 Best Hospitals and all others can be found at HealthGrades.com. A comprehensive methodology for this award and all HealthGrades hospital ratings and rankings is also available online.