Judge not, that ye be not judged. Irked by the growing number of report cards assessing the quality of hospitals, a New York state hospital association has taken this biblical admonition to heart by putting out a report card grading the quality of hospital graders. Five of the 10 report cards that were evaluated were given low marks.
Photo by Phil Jern via Flickr
“HANYS’ Report On Report Cards,” published Monday by the Healthcare Association of New York State, used nine criteria to assess report cards issued by government or private organizations. These principles included whether the report card used measures grounded in scientific evidence, whether the data were recent and whether hospitals had a chance to review the findings before publication to pinpoint errors.
“We had been hearing more and more from members their general frustration of all the different report cards,” said Kathleen Ciccone, director of HANYS’ Quality Institute. “It’s so time consuming for them to be able to respond to the reports, to be able to see what’s useful about them. They’re really looking for some guidance on how to use the information.”
The scoring ranged from three stars, for the report cards that met all or nearly all of the criteria, to half a star for the report cards that met few or none. U.S. News & World Report did the worst, earning half a star for its hospital rankings, which incorporate surveys of the opinions of doctors around the country. “A subjective perception of hospital reputation is not a scientifically proven measure to evaluate hospitals’ processes of care,” HANYS wrote.
Four other report cards received only one star. HANYS dinged the Leapfrog Hospital Safety Score, which gives out grades of A to F, for relying “heavily on unvalidated survey data” that it collects directly from hospitals.
Consumer Reports’ Hospital Safety Ratings also received a sole star. HANYS faulted the nonprofit for rolling up data that came from different sources and spanned different time frames. Some of the Consumer Reports’ data use insurance billing records, which HANYS considers inferior to data taken from medical charts. Seven of the 10 report cards, including Medicare’s own Hospital Compare website, were faulted for using billing records, even though there are no other publicly available measurements in many areas of hospital quality such as mortality.
Healthgrades’ America’s Best Hospitals and Truven Health Analytics Top 100 Hospitals each received one star. HANYS said Healthgrades does not detail its rating method with enough specificity for others to replicate the calculations. Healthgrades and Truven violated another HANYS’ criteria because they include measures that are either not used by the federal government or not endorsed by the National Quality Forum, a nonprofit that seeks industry consensus on which quality measures are reliable.
HANYS gave two stars to the Niagara Health Quality Coalition New York State Hospital Card, which puts out a patient safety health card for state hospitals.
HANYS awarded three stars to The Joint Commission Quality Check, which evaluates how hospitals do in meeting benchmarks on a variety of measures. Medicare’s Hospital Compare and two New York State Department of Health ratings also garnered three stars. “In general, government and accrediting organizations are more successful than other report card organizations in meeting HANYS’ criteria for evaluating hospital performance,” the group wrote in its report card.
By at least one criterion, the HANYS report on report card falls short of its own standards: HANYS did not give the ratings groups an opportunity to preview the report before publication.
