In the era of value-based reimbursement, measuring the “cost” part of the equation has been easy, but measuring “value” has proven to be a bit trickier. I’m a big fan of ratings: I like Rotten Tomatoes for movies, Consumer Reports for washing machines, and TripAdvisor for restaurants. But I know that measurement statistics of all types can be fraught with challenges. To quote Mark Twain, “There are lies, damned lies, and statistics.”
The new Centers for Medicare & Medicaid Services (CMS) STAR ratings for hospitals offers a timely example of these kinds of challenges. Released on July 27, the new Overall Hospital Quality Star Rating summarizes data from 64 existing quality measures publicly reported on Hospital Compare for effectiveness of care, efficient use of imaging, mortality, patient experience, re-admissions, safety, and timeliness of care (star ratings will be updated each quarter). These ratings will be boiled down into a single star rating for each hospital — in theory giving consumers a simple measure to understand (see Centers For Medicare & Medicaid Services Presentation On Overall Hospital Quality Star Ratings On Hospital Compare).
As early as January, CMS predicted only 2.4% of hospitals in the new system would earn the top five-star ranking (see Hospitals Prepare To See Stars In April). It turns out they weren’t that far off. Here’s a breakdown of each group by rating:
- 102 (2.7%) hospitals earned a five-star rating
- 934 (25.5%) hospitals earned a four-star rating
- 1,770 (48.3%) hospitals earned a three-star rating
- 723 (19.7%) hospitals earned a two-star rating
- 133 (3.6%) hospitals earned a one-star rating
- 937 hospitals received no ranking because they did not meet the minimum reporting thresholds
So what are the issues? On the plus side, “Consumers need a simple way to objectively gauge quality” – a succinct statement from the recent Miami Herald coverage of the new ratings, Which Three Miami Hospitals Scored Lowest In New Medicare Rating System?. Kate Goodrich, M.D., MHS, director of the Center for Clinical Standards and Quality at CMS said that while CMS will continue to work with hospitals to make the star ratings better, consumers have previously found CMS rating systems to be useful and that the top-ranked hospitals in the new ratings “tended to have lower death and readmission rates” (see Helping Consumers Make Care Choices Through Hospital Compare). Traditionally, hospital systems have not been particularly transparent about their clinical performance or fees (see Uncovering Hospital Charges). So the new rating system represents a step in that direction.
On the minus side are the politics and methodology of the new rating system. Kaiser Health News reports that few of the “top-ranked” hospitals are actually considered among the best in the U.S., as previously ranked by private ratings sources such as U.S. News & World Report, and teaching hospitals and hospitals with large numbers of low-income and chronically ill patients scored lower under this new system (see Many Well-Known Hospitals Fail To Score 5 Stars In Medicare’s New Ratings). Speaking with Kaiser Health News as part of that story, Steven Lipstein, the president of BJC HealthCare said that the ratings were more a reflection of consumer affluence and not the quality of the hospital; and Rick Pollack, president of the American Hospital Association, called the ratings confusing for consumers.
The Medicare Payment Advisory Commission (MedPAC) had more substantial comments about the mechanics of the new rating system, noting that its analysis of the star rating results showed that of the 102 five-star hospitals, only 57 have a rating based on all four of the outcome groups and of the 129 one-star hospitals, 126 were rated using all four of the outcome groups. This disparity means that only 56% of hospitals with the highest rating were rated on a full set of outcome measures, and that MedPAC is concerned that due to the missing data for the higher rated hospitals, the ratings do not reflect a true balanced comparison of hospital quality performance (see MedPAC Questions Validity Of CMS Hospital Star Ratings).
I’m hopeful that the response of executives in the field to the comments and criticisms is not one of “killing the measurement system.” One way or another, consumers and health plans are going to make value judgments about provider organization performance. I would rather have those initiatives be completely transparent and hope to see more information available to the public like CMS’ Open Data Initiative (see CMS Announces Data And Information Initiative). For more on the CMS ratings, check out these resources from the OPEN MINDS Industry Library:
- Consumer Star Ratings For Hospitals – It’s Only Going To Get Harder To Earn Those 5 Stars
- When It Comes To Performance Measurement, The “Work” Is Never Done
- The Next Evolution Of Nursing Home Performance Ratings
- The Dominant Performance Standards Of Today – CMS Stars & NCQA HEDIS
- Medicare Performance Measurement Initiatives: Bringing Transparency To Consumer Choice
These ratings aren’t going away. As CMS becomes a larger and larger payer in the U.S health care system, their efforts to improve value through ratings system will continue. For even more on performance management, join me on February 16, 2017, at The 2016 OPEN MINDS Performance Management Institute for the session, “Building Relationships & Managing Performance — Best Practices In Population Health Management.”