Deploying artificial intelligence to prevent delayed and missed care

Medical imaging is one of the most widely used diagnostic tools in health care with providers ordering dozens of imaging studies for their patients every day. These studies are used to gain information about certain symptoms a patient has or a condition they are being treated for, but frequently have incidental findings unrelated to the original indication for the study.

While incidental findings are noted in the radiology report, these often go unnoticed by the ordering provider as they focus on addressing the original intent of the study. While many of these findings are benign, some may develop into malignancies making timely follow-up essential to prevent harm to the patient's health long-term.

Delayed and missed follow-up on incidental diagnostic imaging findings is one of the most prolific problems in health care harming millions of patients across the country and driving up costs. Nationwide, hospitals and health systems spend an estimated $43 million annually to settle lawsuits for missed follow-up on lung findings alone.

This problem is a case study of preventable harm: there is a documented, but unrecognized finding that could have led to a meaningful intervention, but instead the patient's disease develops unchecked."

Mozziyar Etemadi, MD, PhD, medical director of advanced technologies, Northwestern Medicine

Recognizing the serious threat to patient safety and financial risk, Northwestern Medicine mobilized a multidisciplinary team with members from Radiology, Quality, Patient Safety, Process Improvement, Primary Care, Nursing, Informatics and others to address the issue. The team created a system utilizing artificial intelligence (AI) integrated into the electronic health record (EHR) that runs on nearly every imaging studying ordered in the 11-hospital health system. When required, the AI triggers an alert in the EHR that clearly displays the findings and the recommended follow-ups directly in the ordering physician's workflow.

Once the system triggers an alert, it then tracks completion of recommended follow-up for both incidental and expected findings. If action is not taken to schedule additional imaging or other follow-up, another alert is sent to prevent delays in care and improve patient outcomes.

Northwestern Medicine first implemented the technology in December 2020 to search for lung and adrenal findings, then prospectively tested it for more than a year. The results of that study are published today in NEJM Catalyst Innovations in Care Delivery, which is among the first prospective studies of an AI tool deployed in clinical practice.

The Catalyst study looks at a year of data following live implementation of the AI system in December 2020. More than 460,000 imaging studies were screened and 23,000 were flagged as containing lung follow-up recommendations representing a 5% rate of lung finding occurrence on relevant imaging studies – that's 68 findings per day that required follow-up.

"Our data shows just how common these incidental findings are in diagnostic imaging studies and further supports the need to find a scalable solution," said Dr. Etemadi. "Our team developed an electronic health record (EHR) integrated natural language processing (NLP) system to automatically identify radiographic findings requiring follow-up. Once identified, the AI triggers automatic alerts to both physicians and patients to schedule and track completion of recommended follow-ups."

An example of how the system works is a patient has a chest x-ray for pneumonia and a lung nodule is identified by the radiologist who makes a note in the radiology report acknowledging the lung finding and recommending a chest CT as follow-up. The AI models detect the radiologist's note about the lung finding and recommended follow-up, so the integrated framework then triggers a Best Practice Advisory (BPA) to alert the ordering physician of the finding. The BPA also presents workflows where follow-up studies can be ordered as appropriate. This all occurs directly in the EHR and does not require the physician to open any additional programs.

"A common misconception is that AI deployed to make clinical decisions, but that is not the case – clinical decisions are made by the radiologist and ordering physician," said the study's lead author Jane Domingo, MBA, MS, CCC-SLP, program manager, System Clinical Performance Improvement Office at Northwestern Medicine. "This system was put in place to identify the reports with findings and streamline the associated workflow, which was previously nonexistent. It facilitates physician decision-making rather than displacing the physician. An important part of our rollout was meeting with physicians and helping them understand how it would support them in making clinical decisions."

In addition to the physician alert, the patient also receives a notification in their online patient portal with the imaging study result. Patients who do not utilize the online patient portal or do not have an established Northwestern Medicine physician are directly managed by the team of follow-up nurses to assure appropriate care is received.

"As patients, we often believe that 'no news, is good news' and if we don't hear from our physician after a test, we tend to assume everything is fine and don't proactively follow-up," said Domingo. "By building patient notification into our process, we are adding another layer of safety by creating informed patients who can advocate for themselves."

All elements of this system were built entirely in-house at Northwestern Medicine. One of the most arduous elements of building an AI is the "labeling" process, which requires huge amounts of data to be sorted through by clinical experts.

"Typically, data labeling is handled by third-party vendors and requires extensive time and financial resources often with mixed results," explained Dr. Etemadi. "At Northwestern Medicine, we used our own resources to complete this portion of the project – nurses and other front-line staff who were on light-duty restrictions and unable to fulfill their clinical responsibilities. These staff were trained as annotators and tasked with selecting relevant text from the curated radiology reports. This resulted in higher quality, expert reviewed data in a cost-effective manner."

Along with the Catalyst study, Northwestern Medicine is releasing a companion website that outlines the process of building and deploying this system so that other hospitals and health systems can implement similar results management programs. The health system is currently working on expanding the system to hepatic, thyroid and ovarian findings requiring follow-up.