Study finds better way to evaluate operating skills of orthopaedic residents

Jun 30 2016

In a small study to determine the best way to assess the operating skills of would-be orthopaedic surgeons, Johns Hopkins researchers found that tracking the trainees’ performance on cadavers using step-by-step checklists and measures of general surgical skills works well but should be coupled with an equally rigorous system for tracking errors.

“The takeaway message is that checklists of procedural steps are a good way to assess the technical skills of these surgical residents. But they don’t measure quality, highlighting a need to measure and give feedback on errors as part of the training,” says Dawn LaPorte, M.D., professor of orthopaedic surgery at the Johns Hopkins University School of Medicine and an author of a report on the study in the Journal of Surgical Education, published this month.

Under most current training models, LaPorte says, case numbers often serve as a proxy for a resident’s mastery of a procedure but really only measure volume, not skill level. What’s more, learning opportunities have been reduced with recent caps on resident work hours, implemented to prevent mistakes due to exhaustion.

Under the current system, she says, residents rarely receive formal feedback on their motor skills or errors, allowing mistakes to perpetuate. Evaluating residents as they work on cadavers helps them identify areas in need of improvement and learn from mistakes, she says. “They can practice repeatedly in a safe environment before they’re holding the knife to operate on real people.”

The new research grows out of her team’s bid to address shortcomings in the assessment of motor skills by using a well-established checklist called the Objective Structured Assessment of Technical Skills, or OSATS, that she and her colleagues modified to fit certain surgical procedures of the shoulder.

The team then tested the checklist in residents practicing three different approaches to access a shoulder in need of repair: from the front, back or side. The procedures are common first steps for shoulder surgeries to address injuries to the bicep, shoulder bones or rotator cuffs. The shoulder, says LaPorte, was a good test case because it’s a common surgical target, and the steps for initially opening it up are easy to break down.

Three orthopaedic specialists from Johns Hopkins then used the grading systems to evaluate the work of 23 Johns Hopkins medical residents ranging from their first to fifth year of residency as they performed each of the procedures on cadavers. Under OSATS, the residents received a point for the successful completion of each step in the checklist and a zero for failed or incomplete steps. They also received zeros if they performed the steps out of order. The crucial steps in this case involved making the incision in the right place and avoiding important structures, such as nerves and arteries.

Besides the OSATS checklists, the faculty surgeons rated residents using the so-called Global Rating Scale and a simple pass/fail system. The Global Rating Scale was developed by Richard Reznick in 2006 and is applicable to all surgical procedures, taking a holistic view of the surgical process and offering residents feedback on such things as whether or not they used the surgical instruments correctly and if they display a broad understanding of the procedure.

Under the pass/fail system, residents receive a failing grade if they commit an egregious error — in the case of shoulder surgery, severing a nerve or blood vessel.

The Johns Hopkins team found that OSATS and the Global Rating Scale provided good, objective ways of measuring resident performance, while the pass/fail system gave residents unambiguous feedback.

As expected, more advanced residents received higher OSATS and Global Rating Scale scores than those just beginning their residencies. Scores on all three procedures ranged from the low single to double digits for first-year residents to the high double digits for fifth-year students. Across all three procedures, examiners observed 11 incidents in which residents damaged the nerves or veins. First- and second-year residents were responsible for nine of those mistakes.

Crucially, however, the Johns Hopkins team found that none of the three evaluations adequately captured those mistakes. The pass/fail scheme came closest, in that residents received a failing score for severing a nerve or major blood vessel, but it is not set up to subsequently inform residents of the precise nature of their mistakes. And residents could theoretically perform well on the OSATS checklist even if they made an egregious error because points are not deducted, only earned in the grading system. To mitigate this, the researchers propose adding safety steps, such as identifying and protecting important structures, to the checklist.

Moving forward, LaPorte and her team hope to implement these training protocols for other common surgical procedures. Next up is developing an OSATS checklist for treating compartment syndrome, an injury in which bleeding or swelling prevents blood from flowing to affected tissues. Compartment syndrome is among the most serious of orthopaedic injuries, LaPorte notes, as improper handling can lead to amputation of the injured limb or even death.

The residents themselves report that they like an objective evaluation system like the checklist because it gives them the confidence they need before entering an actual operating theater, LaPorte says. “The goal,” she adds, “is to make objective evaluations standard for all procedures expected of residents.”