Hospitals that perform well praise the report as confirmation of their institution’s quality. Poor-performing hospitals claim to treat the sickest patients and criticize the report’s severity adjustment methodology as inadequate to account for their riskier patient mix. The report floats hospital charge data—not actual costs paid by insurers or other sources—and the press notes how widely those charges vary across institutions. Individual physicians are identified as outliers by the report and some are given the opportunity in press reports to account for their performance.

This periodic ritual raises a number of important questions, which are rarely addressed in press accounts, about what the data mean; whether the data are used; and what can be done to make their interpretation more appropriate and useful for hospitals, physicians, patients, purchasers, health insurers and regulators.

The Pennsylvania Health Care Cost Containment Council (PHC4) has released six such reports since 1992, the most recent one using 2000 data to rate each of the 55 hospitals in the state that perform the CABG procedure. In addition to data about in-hospital patient mortality, average post-surgical length of stay and average hospital charge, the new report includes several new measures for the first time: 30-day post-surgical mortality, 7-day and 30-day readmission rates.

Central to the report’s performance ratings is a CABG severity model known as the MediQual Atlas Outcomes System, which all Pa. hospitals are required to use to abstract patient severity information and report it to PHC4. The Atlas system is essentially a proprietary "black box" that ranks patients based on severity of illness on admission using Key Clinical Findings abstracted from clinical data on the patient’s chart, notes David Nash, M.D., chair of PHC4’s Technical Advisory Group and associate dean and director of Thomas Jefferson University’s Office of Health Policy and Clinical Outcomes. The system risk-adjusts the hospital- and physician-specific outcomes and predicts probabilities of mortality for each of the reported patients receiving CABG surgery in 2000.

"The 2000 CABG report is the state of the art in the whole country," says Nash, whose technical advisory committee reviews comparable reports nationwide. He notes that the severity adjustment system used by PHC4 is constantly improved and, unlike many other systems, uses clinical variables from patient charts.

For the most recent report, PHC4 asked MediQual to design a model focusing specifically on the CABG population, says PHC4 Executive Director Marc P. Volavka. He notes that the Council revisits its risk adjustment methodology system on a regular basis and recently reviewed 22 alternative models before re-endorsing the MediQual Atlas system.

Criticisms Detailed

Most hospitals in the report are shown to perform at their expected levels for in-hospital and 30-day mortality, and for 7-day and 30-day readmissions. Two hospitals, however, scored better-than-expected in one category and worse-than-expected in the other, muddying how one should interpret the quality of CABG care at those institutions.

UPMC Presbyterian was one of only three hospitals in the state with lower-than-expected mortality (both categories), but was also one of only four hospitals with higher-than expected readmissions (both categories). Reading Hospital showed the opposite: higher-than-expected in-hospital mortality, but lower-than-expected 7- and 30-day readmissions.

"Since we don’t fully understand what the reasons are, we don’t really know what the implications of it are," says Mark Schmidhofer, M.D., director of the Institute for Performance Improvement and assistant vice president for medical services, UPMC Health System.

Schmidhofer notes that the report has led Presbyterian to study more closely its readmissions to ferret out explanations for its higher rate, e.g., maybe vigilant readmitting is reducing mortality, maybe medications that prevent certain kinds of complications are not being given, or maybe some other non-fatal element during the course of treatment is not being handled as well as it should.

Consumers may have difficulty interpreting discrepant mortality and readmission data from a single institution. "The issue of readmission is one that we would not suggest is a consumer-friendly measure," says Volavka, noting that, like the report itself, it is designed to engage the provider community in serious efforts to learn reasons for its patterns, and to offer physicians the opportunity to educate their patients.

The introduction of the report itself cautions readers that the report is not intended to be a sole source of information in making decisions about CABG surgery, nor should it be used to generalize about the overall quality of care provided by a hospital or a surgeon. "Readers of this report," it reads, "should use it in discussions with their physician who can answer specific questions and concerns about CABG surgery."

Several hospitals have submitted comments to accompany the CABG report on PHC4’s website (www.PHC4.org), taking issue with specific aspects of the report’s methodology.

Reading Hospital’s Surgery Department Director Robert A. Brigham, M.D., says that PHC4’s severity adjustment methodology wrongly excluded several co-morbidity factors from its CABG patient mortality predictions, including diabetes, chronic obstructive pulmonary disease (COPD), cardiomyopathy and gender, which he says are recognized by other reviewing organizations such as the Society of Thoracic Surgeons as significant, incremental open-heart patient risk factors. Brigham adds that Reading Hospital draws patients from Berks and Schuylkill counties, for whom COPD is a significant factor, and whose exposures to chemicals and other environmental and industrial substances could have contributed to a higher mortality.

Disagreements over patient severity methodologies matter less than they seem, counters Nash, noting that research literature shows that severity of illness probably only accounts for 20 to 30 percent of the variation in outcomes for CABG patients. The remaining 70 percent, he adds, is affected by the process of care.

Pinnacle Health, the third-largest provider of CABG surgery in Pa., showed higher-than-expected in-hospital and 30-day mortality in PHC4’s latest report, but has been well within the expected range for 30-day mortality as predicted by the Society of Thoracic Surgeons (STS) risk model for 2000, as well as for 1998, 1999 and 2001, says Donald C. Durbeck, M.D., chair of Pinnacle Health’s Cardiovascular Services. Whereas the PHC4 risk adjustment methodology relies on four discrete ranks of illness severity, the STS rating system is more precise, Durbeck argues, in that it uses a continuous percentage ranking and compares institutions to a national data set.

The PHC4 report identified two of Pinnacle’s heart surgeons as having higher-than-expected patient mortality in 2000. "If we look at them over five years of Society of Thoracic Surgeon data, those two surgeons fall into the category of being our best surgeons, as far as mortality goes. Why should they end up having their names on an Internet site that says that they have above-expected mortality for one year?" asks Durbeck.

Differences between the two mortality ranking systems are to be expected and there is no conclusive way to assess which is better, says Nash, "Which is why mortality, in general terms, is probably not a fine enough measure to really use as a key screen. Regrettably, this is the state-of-the-art of the science of severity adjustment. Mortality is a reasonably straightforward endpoint and it’s collectible information," he adds, noting that other variables—such as infection rate, functional status and quality of life, are much harder to collect.

Main Line Health’s Bryn Mawr Hospital also showed higher-than-expected in-hospital and 30-day mortality on PHC4’s latest report. Like many other health systems, Main Line Health aggregates quality data real-time and analyzes events and trends gleaned from the Atlas system internally to produce periodic performance reports for its physicians and institutional mortality/morbidity reports, says K. Bobbi Traber, M.D., MBA, Main Line Health’s senior vice president of medical affairs.

Bryn Mawr’s rating for 2000 data was changed by enhancements made to the Atlas risk adjustment methodology that was used in the PHC4’s latest published report, says Traber. The hospital’s concurrent look at 2000 data—using the previous Atlas methodology—showed it to be within expected mortality levels, while the revised Atlas methodology put Bryn Mawr into the above-expected range of mortality in the report, she explains, adding that time will tell whether the changes have enhanced the methodology’s precision.

Bryn Mawr was one of the relatively few hospitals that had lower-than-expected in-hospital mortality on PHC4’s previous CABG report, which used 1994-1995 data. Reading Hospital and the two hospitals subsumed by Pinnacle were within the expected mortality range in the previous report.

Nash agrees that the different methodology used in the new report makes comparisons to previous reports difficult and notes that assessment of trends based on those comparisons may not be useful.

UPMC Lee Regional hospital, which had higher-than-expected 30-day mortality in the recent report, claims that one less mortality would have resulted in it being ranked well within the expected range. Nash concedes that, in certain circumstances, a single case could have influenced the outcome for an institution, given small enough numbers.

The report also appears to have some quirky statistical outcomes. After accounting for patient risk, the number of years of experience surgeons had in performing open heart surgery was an important predictor in patient survival, the likelihood of readmission and post-operative length of stay, as one would expect. But the report found that patient mortality increased as surgeon experience increased beyond eight years, and decreased when surgeon experience exceeded 23 years. Surgeons produced shorter lengths of stay for their patients as their experience approached ten years, then produced longer stays between 10 to 25 years of experience, beyond which they again produced shorter stays.

"We’re looking at that. It may be a statistical issue. I don’t hold eight years as being a magic number," Volavka responds.

Report’s Impact

Despite the report’s limitations, it appears to be taken seriously by hospitals and purchasers, and to some degree by health insurers.

Several hospitals submitting comments to accompany the PHC4 report say that the information in the report encourages them to examine their care processes and make quality improvements. Some institutions maintain that those efforts take place in advance of the report’s publication, while others rely on data other than PHC4’s.

The Western Pennsylvania Hospital, in its submitted comments, wrote that it uses both national and regional comparative data to benchmark outcomes and continuously enhance current CABG surgery practices. West Penn said it will incorporate the latest PHC4 report’s readmission data into its quality improvement review processes in the future.

The report’s data have made UPMC Presbyterian aware that its readmission rates are different and have induced it to focus on searching for common themes in the care process related to readmissions, such as the training of home care nurses who make readmission decisions, says Schmidhofer.

The latest PHC4 report was the first time that Reading Hospital had a higher-than-expected mortality rate for CABG patients and has induced the institution to accelerate the study of its of 2001 data, a preliminary analysis which indicates significant improvement over 2000 data, according to Brigham. Reading has initiated changes in 2000 and 2001 as part of its ongoing quality program that Brigham believes account for that improvement, including performing an increasing number of cases without the use of the heart/lung machine, adding new monitoring equipment to calculate cardiac and hemodynamic measurements in real-time, and changing anesthetic techniques used during open-heart surgery, he adds.

Pinnacle Health looks at its own mortality data periodically and presents it to its surgeons for quality improvement purposes, says Durbeck. The health system was concerned whether organizational changes it made in 2000 would impact on outcomes, he adds, noting that Pinnacle had merged the medical staffs of Harrisburg and Polyclinic hospitals, both of which had open heart centers, and built a new plant to house operating rooms, post-operative care and cath labs.

Durbeck says that the Society of Thoracic Surgeons mortality data for Pinnacle’s CABG surgeries showed no statistical difference in its 2000 performance from other years, and he notes that 2001 data show improvement in the wake of care protocol changes, such as changing the way a patient’s skin is prepped, using antibiotics in patients’ nasal passages, changing the vein harvesting technique to reduce infections and altering the physical therapy process to reduce pulmonary complications. These changes, Durbeck stresses, have been part of continuous quality improvement efforts independent of the PHC4 data.

Bryn Mawr Hospital has seen some improvement in its 2001 data and is now examining trends on a quarterly, rather than only on a case-by-case basis, says Traber. Main Line Health has also recently begun to participate in the STS database as another validating source, she adds.

Some hospitals use their positive report scores as an affirmation of their quality of care, and state as much in their comments submitted to accompany the report, including Susquehanna Health System, Butler Memorial Hospital, UPMC Passavant, Wyoming Valley Health Care System, Mercy Hospital Scranton and Altoona Hospital.

Although no hospital interviewed for this story affirmed whether they would use their performance on the report in marketing materials, a 1996 survey of how Pa. hospitals have responded to publicly-released CABG reports, co-authored by Nash and published in the Journal of the Joint Commission on Accreditation of Healthcare Organizations, suggested that hospitals do implement new approaches to marketing on the basis of those reports, and that some use the performance information to recruit staff thoracic surgeons and residents.

Some, like UPMC’s Schmidhofer, dismiss the report’s charge data as misleading, while others use the data to boast that their hospitals are providing the same or better care than larger-volume hospitals, at a lower cost. Butler Memorial Hospital wrote in its submitted comments that "...the costs are generally less at community hospitals, which is an important finding for insurers or business purchasers of health care seeking to control costs while maintaining quality."

Purchasers appear to take the PHC4 CABG report seriously as a snapshot of quality, if not of cost. SMC Business Councils, a coalition representing 5,000 small companies in southwestern Pa., discusses the report at membership meetings, distributes hard copies to members and cites excerpts of the report in its internal publications to members, says SMC’s President Cliff Shannon. He regards the report as accurate and dismisses any assertion by a hospital that PHC4’s methodology is significantly flawed as "poppycock" and "the same tired harping."

"I tell people they need to be informed consumers—that the results come from information that hospitals themselves report," notes Shannon, adding that he would single out consistent outliers over time in the reports, but would not steer members to, or away from, specific hospitals or physicians. "I wouldn’t hesitate to tell people that they need to be looking for physicians who perform at least average or better-than-average in areas of acute concern to patients," he says.

Shannon dismisses the report’s hospital charge data. "Who the hell pays charges? Nobody does. It’s the discounted rate that insurance companies pay. I say the information is at best, uninterpretable and at worst, misleading. So let’s dump it." It is less-than-optimum care, says Shannon, that translates to cost that his member companies shouldn’t have to bear.

The report is also disseminated through the PHC4 itself. Volavka notes that employers and employer groups have requested 13,000 hard copies of the report for their employees. As of mid-May, the PHC4 had over 5,000 persons on its mailing list for the report, including 800 libraries, and had over 5,000 downloads of the report from its website. The Council also conducts ongoing community outreach events at which it publicizes its reports.

Pennsylvania AARP said it does not disseminate information about the PHC4 CABG report to its members, preferring instead to communicate national research from its office in Washington.

Insurers use the report as one part of larger efforts to monitor cardiac care quality. "If something in the report stands out, good or bad, then we can look at our own data and take action, if necessary," says Joseph Reilly, manager of health information analysis for Highmark, which he says purchases the PHC4 inpatient database—covering all patients treated at all Pa. hospitals—to get a global picture of inpatient care from an all-payor perspective. For Highmark, PHC4 reports "confirm and corroborate patterns and trends which we have seen with our own data programs or condition management programs," adds Reilly. He notes that Highmark does not have a position, per se, on the accuracy of the CABG report.

Says Don Liss, M.D., senior medical director of Aetna’s mid-Atlantic region, "Should we have some issue around a cardiac surgeon, we would consider the information in this report in our credentialing process, but Aetna does not identify the worst performers based on this report and bring them before a peer review committee to determine continued participation in our plans or to request specific plans of correction."

Aetna encourages hospitals to use the report for performance improvement initiatives, and its members to engage in dialogue about it with their referring physicians. Adds Liss, "I wouldn’t want to make sweeping conclusions based on either one report, or on this sort of database reporting, without having a very good understanding of the detail. Even the best models of severity- or risk-adjustment don’t account for all of the potential nuances if a particular surgeon does the highest of high risk cases."