Study finds that ACS NSQIP Surgical Risk Calculator has excellent calibration

May 20 2016

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator accurately estimates the chance of a patient experiencing postoperative complications, and its performance can improve with recalibration of the tool according to research findings appearing online in the Journal of the American College of Surgeons in advance of print publication.

Created in 2013, the NSQIP Surgical Risk Calculator is a web-based decision aid and informed consent tool widely used by surgeons and their patients. It currently receives an estimated 1,500 hits a day according to study coauthor Clifford Y. Ko, MD, MS, MSHS, FACS, Director of the ACS Division of Research and Optimal Patient Care, which administers ACS NSQIP.

This online tool provides customized risk estimates for patients planning to have an operation, thus helping them decide whether to undergo the procedure.

In the study, the investigators analyzed the Surgical Risk Calculator for its calibration. Calibration is the agreement between the predicted risk and actual outcomes on average, and across the spectrum of predicted risk--from low to high, said Mark E. Cohen, PhD, one of the study's authors and a statistical manager in the ACS Division of Research and Optimal Patient Care.

"Our study demonstrates that the NSQIP Surgical Risk Calculator has excellent calibration," said Dr. Cohen, a codeveloper of the tool. "Although no such tool can provide perfect predictions, the Surgical Risk Calculator does what it is intended to do--accurately estimate a patient's probabilities for important adverse events postoperatively."

Dr. Cohen and his colleagues based their statistical analyses on 2.7 million individual surgical records collected between 2010 and 2014 from 586 hospitals that participate in ACS NSQIP. NSQIP is the leading nationally validated, risk-adjusted, outcomes-based program to measure and improve the quality of surgical care in hospitals.

Furthermore, the investigators statistically recalibrated the Surgical Risk Calculator's prediction equations, using an independent subset of the data, to determine if it would improve the calculator's performance.

"We are continuously able to make the predictions more accurate because we add more and more recent NSQIP data," Dr. Cohen said. "But this is the first time we evaluated the benefits of recalibration."

Before recalibration, the researchers found that the Surgical Risk Calculator had a slight tendency to overestimate surgical risk for the lowest and highest risk patients and to underestimate risk for patients at moderate risk of postoperative complications. Recalibration eliminated this minor distortion, Dr. Cohen said.

Recalibration will be incorporated into the NSQIP Surgical Risk Calculator this fall or early next year, and the recalibration will be invisible to the user, said Dr. Ko, who is also a professor of surgery at the University of California-Los Angeles David Geffen School of Medicine.

An update of the risk calculator scheduled later this month will add predictions of several postoperative complications. Dr. Cohen said the new risk outcomes are readmission to the hospital; ileus, a type of bowel obstruction; and leak of an intestinal anastomosis, a surgical connection of two formerly distant parts of the intestine after removal of diseased bowel.

The investigators did not include these three risk predictors in their analysis. They used the original risk outcomes, which are any complication, pneumonia, cardiac arrest or heart attack, surgical site infection, urinary tract infection, blood clot, kidney failure, or death. Their analysis also included three postoperative complications that were more recent additions to the risk calculator: discharge destination (skilled nursing facility rather than home), a return to the operating room, and any serious complication.

The Surgical Risk Calculator estimates the chance of these potential postoperative complications, compared with an average person's risk, for more than 1,500 different surgical procedures. First, the surgeon or other user enters, into blank data fields, the planned surgical procedure and 19 patient-specific preoperative risk factors, including age, body mass index obtained from height and weight, smoking status, and health conditions such as high blood pressure and diabetes. Additionally, the surgeon can adjust the risk score based on his or her subjective assessment of a patient.

"The NSQIP Surgical Risk Calculator is one way that the ACS is helping surgeons and patients communicate better and is helping patients gain more information about the surgical procedure so their care is individualized, patient-centered, and informed," Dr. Ko stated.

Another way that Dr. Ko said the ACS is working to improve surgical patients' preoperative readiness is through support of the "Strong for Surgery" initiative. This program provides preoperative checklists focusing on risk factors that patients can modify before undergoing an operation.

With support from the John A. Hartford Foundation, the ACS is also leading a program dedicated solely to improving the surgical care of older adults, called the Coalition for Quality in Geriatric Surgery (CQGS). CQGS is working with more than 50 stakeholder organizations to develop standards based on best scientific evidence that will focus on delivering optimal surgical care for a rapidly growing older patient population.

Source: American College of Surgeons