Leapfrog Measures Show Grade Inflation

Quantifying hospital performance has become an increasingly important trend. One of the pioneers was The Leapfrog Group, who introduced a self-reported Safe Practice Score (SPS) for hospitals in 2004. Since then, measures of hospital quality have proliferated, including mandatory reports like Medicare’s Hospital Compare.  In response, Leapfrog bundled SPS with other measures to produce a Hospital Safety Score (HSS), which provides hospitals with a letter grade meant to be consumer friendly.

The simplicity of such systems has come under attack questioning whether these types of scores are valid. This study investigates how SPS results correspond with Hospital Compare’s, and, for non-reporting hospitals, the potential effect of various SPS scores on HSS grades.

Little association was found between SPS and Hospital Compare’s measures, with the exception of a small negative correlation between SPS’s culture of measurement, feedback, and intervention and Hospital Compare’s penalties for readmission or hospital-acquired conditions. Nevertheless, this was a relatively small effect, as one standard deviation increase in the culture measure averaged out to an absolute 2.7% decrease in penalties.

SPS measures are positively skewed – all but one measure had more than half of hospitals reporting perfect scores. That positive skew also has important ramifications. Low SPS scores have a strong negative effect on hospital grades while high scores SPS have a much weaker positive effect, strengthening the natural inclination to avoid reporting low scores.

As such, this study highlights the drawbacks of self-report in hospital quality measures. Hospitals naturally are not incentivized to report less than ideal results, especially when these are used to produce a letter grade that subsequently is marketed to the public. The skewed distribution created by this effect reduces the value of the resulting data and can further discourage poor performers from reporting. Consequently, compulsory reporting appears to be best way forward for hospital quality data. Nevertheless, the example of Leapfrog should remind policymakers of the importance of considering the incentives created by any quality reporting system, mandatory or otherwise.

This Policy Prescriptions® review is written by Matthew Stampfl, a second year medical student at Baylor College of Medicine.

 

Abstract

Background: Voluntary Leapfrog Safe Practices Score (SPS) measures were among the first public reports of hospital performance. Recently, Medicare’s Hospital Compare website has reported compulsory measures. Leapfrog’s Hospital Safety Score (HSS) grades incorporate SPS and Medicare measures. We evaluate associations between Leapfrog SPS and Medicare measures, and the impact of SPS on HSS grades. 

Methods: Using 2013 hospital data, we linked Leapfrog HSS data with central line–associated bloodstream infection (CLABSI) and catheter-associated urinary tract infection (CAUTI) standardized infection ratios (SIRs), and Hospital Readmission and Hospital- Acquired Condition (HAC) Reduction Program penalties incorporating 2013 performance. For SPS-providing hospitals, we used linear and logistic regression models to predict CLABSI/ CAUTI SIRs and penalties as a function of SPS. For hospitals not reporting SPS, we simulated change in HSS grades after imputing a range of SPS. 

Results: In total, 1089 hospitals reported SPS; >50% self-reported perfect scores for all but 1 measure. No SPS measures were associated with SIRs. One SPS (feedback) was associated with lower odds of HAC penalization (odds ratio, 0.86; 95% confidence interval, 0.76–0.97). Among hospitals not reporting SPS (N = 1080), 98% and 54% saw grades decline by 1+ letters with first and 10th percentile SPS imputed, respectively; 49% and 54% saw grades improve by 1+ letter with median and highest SPS imputed. 

Conclusions: Voluntary Leapfrog SPS measures skew toward positive self-report and bear little association with compulsory Medicare outcomes and penalties. SPS significantly impacts HSS grades, particularly when lower SPS is reported. With increasing compulsory reporting, Leapfrog SPS seems limited for comparing hospital performance. 

PMID: 28288072  Smith, SN, et al. Med Care. 2017; 55 (6): 606-614.