When an AI system misclassified fictitious patients, physicians continued to follow its guidance, even as treatment outcomes showed the classifications were incorrect.
Study: Doctors vs. Algorithms: Physicians, too, struggle to learn from evidence that contradicts AI suggestions. Image Credit: Toey Andante / Shutterstock
In a recent study published in the journal PLOS Digital Health, researchers investigated whether physicians could override erroneous artificial intelligence (AI)-based patient classifications when they had access to relevant treatment-outcome information.
Physicians determined whether to administer a fictitious drug to individuals incorrectly classified by an AI system as more or less sensitive to the treatment. In the second experiment, the treatment served as a laboratory model of pseudomedicine, presented as potentially useful but not proven effective and used primarily to study perceptions of treatment effectiveness.
The participants generally trusted the AI classification to inform clinical judgments and struggled to use patient outcomes to correct errors, even when relevant information was available. The drug was completely ineffective in the second experiment, yet participants generally judged it to be effective.
These findings suggest that physicians should critically evaluate AI-generated patient classifications, although the simplified online experiments do not directly establish how physicians would behave in everyday clinical practice.
With a continuous shift toward digital systems and computerized workflows, AI-based models are increasingly used in clinical practice. These models can assist healthcare professionals (HCPs) in classifying patients and estimating their disease risk.
AI algorithms can also predict treatment response, potentially guiding resource allocation to improve care and reduce the burden on the healthcare system. However, these models may draw inferences from outdated, incorrect, or incomplete data. The models may be trained on homogeneous data, yet they may need to be applied to diverse populations. As a result, AI-based estimates may be inaccurate. HCPs are generally expected to recognize and correct AI errors, particularly when critical information is available.
About the Study
In the present study, researchers investigated whether HCPs can identify and override AI errors. To do so, they conducted two experiments with separate final samples of 105 and 118 participants, yielding a combined sample of 223 self-reported physicians recruited via the Prolific survey platform.
The participants spoke fluent English and self-reported as physicians practicing in healthcare settings. They also reported their specialty and years of work experience. In Experiment 1, they rated the extent to which they perceived the AI system as reliable. They also completed the General Attitudes toward AI Scale (GAAIS), which measured their attitudes toward AI use. These measures were not included in Experiment 2.
In each experiment, participants interacted with an AI system that falsely classified 60 fictitious patients in the less sensitive (n=30) and more sensitive (n=30) groups for an investigational drug used to manage a rare disease known as Lyndsay syndrome. Despite being placed into different AI-defined categories, patients responded similarly to treatment within each experiment. In Experiment 1, seven in ten patients recovered after receiving the drug, compared with two in ten who recovered without treatment. In Experiment 2, 7 of 10 patients recovered whether or not they received the drug, indicating that the treatment was completely ineffective.
The participating physicians decided whether to provide the drug to these patients and subsequently received immediate patient-level feedback indicating whether each patient had recovered. Across the trials, these outcomes allowed participants to infer that both AI-classified groups responded similarly and that the classification was therefore incorrect. In the initial experiment, the drug was moderately effective for both patient types. In the following experiment, the drug was ineffective in both groups.
Results
Participants had a mean age of 38.6 years in Experiment 1 and 36.6 years in Experiment 2. Their mean professional experience was 13.1 and 10.6 years, respectively. The most common specialties in Experiment 1 were General Medicine and Pediatrics, and in Experiment 2, General Medicine and Internal Medicine. In Experiment 1, participants generally rated the AI system as fairly reliable, with a mean score of 3.65 out of 5. The average positive attitude score was 3.7, indicating generally favorable views of AI. While participants who viewed AI more positively also tended to rate it as more reliable, having concerns about AI did not necessarily make them view it as less reliable.
In both experiments, the participating physicians largely trusted the AI classification, despite receiving patient outcomes that could have been used to challenge it. They found it difficult to update their judgments based on patient-level evidence, which consistently showed the same treatment response pattern across both AI-classified groups. The participants appeared not to fully integrate the patient recovery outcomes needed to recognize that the AI classification was inaccurate.
In Experiment 1, participants administered the treatment to 89.3% of patients classified as highly sensitive, compared with 55.7% of those classified as lowly sensitive. In Experiment 2, they administered the ineffective treatment to 78.3% of patients in one group and 36.8% in the other.
Participating physicians incorrectly judged that the drug would be more effective among individuals classified as being more sensitive. In fact, in the second experiment, they failed to recognize that the drug was ineffective, raising concerns about patient safety if similar patterns occur in clinical practice. However, the study used a highly controlled fictitious task and did not test decisions made in real clinical environments.
Conclusions
In Experiment 2, the findings demonstrate that participants generally overestimated the effectiveness of a treatment that was, in fact, completely ineffective. They did so despite receiving outcome information from each patient. These findings suggest that erroneous AI-based classification models can influence physicians’ treatment decisions and effectiveness judgments under experimentally controlled conditions. HCPs should therefore critically evaluate such systems as AI use in healthcare expands.
The findings should nevertheless be interpreted cautiously. The participants’ professional status was self-reported, and the experiments involved a fictitious disease, a fictitious treatment, limited patient information, and a binary treatment decision. The study also did not compare AI classifications with identical recommendations from a human expert or a conventional clinical system.
In further studies, research efforts should investigate whether automation bias, confirmation bias, and causal illusions contributed to the findings. The authors also noted that participants may have deferred to an apparently authoritative classification system because they lacked prior knowledge about the fictitious disease and treatment. The experiments could not fully distinguish cognitive bias from this potentially adaptive response to uncertainty.
Although AI use is expanding in clinical practice, such systems should be used cautiously and judiciously by HCPs as adjuncts, not as substitutes for clinical decision-making in real-world scenarios. While AI systems can analyze large volumes of multidimensional data, reviewing relevant clinical information using human reasoning and critical thinking remains imperative for patient care.
Human-AI teams improve healthcare only when clinicians stay in control