Healthcare outcome boost needs better studies

Jun 25 2009

Evidence suggests that outcomes in many clinical settings leave a lot to be desired, which means that research into quality improvement (QI) in clinical care has the potential to greatly improve the lot of patients. Now a study in the journal Medical Care Research and Review published by SAGE suggests that both theoretical and practical improvements in QI effectiveness studies could make these into much more powerful tools for positive change.

Evidence suggests that one in four hospital deaths may be preventable, a third of certain clinical procedures expose patients to risk without improving their health, a third of drugs are prescribed erroneously and one third of abnormal laboratory results are not chased up by clinicians in the US. QI focuses on understanding, controlling, and improving work processes, analysing problems' root causes, making work processes predictable, and then continuously improving process performance. Patients and clinicians need QI.

University of Michigan researchers Jeffrey Alexander and Larry Hearld reviewed 185 recent articles from clinical journals on QI's effects. QI effectiveness research is a developing area, and the authors aimed to find out just how useful this research is in providing managers and policy makers with evidence on the real impact of quality improvements.

Nearly 62% of the articles Alexander and Hearld reviewed focused on hospital QI interventions, the majority in university teaching hospitals. Most hospital admissions are actually in community hospitals and not teaching hospitals, raising questions about the extent of research relevance to those settings.

Physician practices and other health care organizations represented around a quarter and one tenth respectively of further studies the authors reviewed. Nursing home research (3.8%) tended to focus on relationships between organisation and quality, with less attention paid to interventions or organizational changes that may result in improved quality.

More than 30% of studies focused on multiple interventions. This meant that the effects of the interventions or intervention components could not be independently identified. This was particularly true in QI studies set in hospitals and physician practice settings.

Information technology and consumer engagement in healthcare are current hot topics in QI studies. Yet similar to nursing home studies, much of the literature in these areas focuses on describing the development or implementation of information technology or consumer engagement, with researchers paying less attention to evaluating of how these changes relate to QI.

Physicians argue that quality-of-care outcomes based on administrative databases don't provide subtle enough data to be of value. However, research gleaned from individual patient charts is expensive. Researchers evaluating QI effectiveness have to weigh up these considerations. It is also easy to mistakenly draw conclusions about how effective a particular QI intervention is by comparing studies that draw upon different sample types. Only QI studies in nursing homes, physician practices, and other health delivery organizations tend to be large enough to statistically test how effective QI changes are across certain organizational conditions. Most hospital QI studies are too small.

In terms of study design, the majority (62%) of studies employed observational techniques, and 38% used experimental/randomised control trial (RCT) designs. RCT studies are the gold standard in research, to establish internal validity and causality, but these are not always appropriate in practice: For example, it may be difficult to randomise patients or providers to experimental and control groups when necessary treatment may be withheld.

Many QI studies were opportunistic, in other words unfunded studies of a change in practice going on in an organisation. These often do not allow sufficient time for careful design or even selection of adequate control groups: fully 30% of the QI studies reviewed contained no comparison group, making it hard to tell whether outcomes associated with the QI change may have occurred anyway. Many studies did not take place over a sufficient time period, which means that they infer a correlation between the change and the outcome rather than causation. The lack of longer-term results also does not tell researchers whether the quality of outcomes for patients reverts to 'normal' after the dust of change has settled.

Researchers in only 14 of the 185 studies assessed how well their intervention was implemented. Without a formal assessment of intervention implementation, it is unclear whether an intervention failed to yield results because it was ineffective or just because it was poorly carried out. Cost benefit analysis was also lacking.

"The impact of QI changes may be overstated," says Alexander. "Our review highlights issues of inadequate study duration, potential selection bias of study participants, and difficulty making generalizations to other organizations due to unique study contexts or variation in intervention characteristics."

Short-term studies simply do not provide enough information to ascertain whether changes are going to have long-term effects on quality in these organizational settings. Also, many complex QI interventions, such as those requiring simultaneous, multiple changes in organizational process and structure, may need extended periods to actually demonstrate intended results.

The authors suggest that QI research needs to make use of theoretical and conceptual frameworks relevant to group relationships, organizational change, organizational learning, or innovation adoption and implementation to get on track. They recommend applying systems theory to QI research, and say that study designs would benefit from a multidisciplinary approach bringing onboard economists, organizational behaviourists, or other related disciplines. Today's narrow focus on medical aspects of QI ignores the critical roles of organizational context, cost-effectiveness, and perhaps most important, the value added by the QI intervention to the patient or organization.

Full bibliographic information: Medical Care Research and Review (published by SAGE)
2009; 66; 235

The article will be free to access online for a limited period from
http://mcr.sagepub.com/cgi/reprint/66/3/235