As problems grow with Abbott’s fast COVID test, FDA standards are under fire

In mid-May, the Food and Drug Administration issued a rare public warning about an Abbott Laboratories COVID-19 test that for weeks had received high praise from the White House because of its speed: Test results could be wrong.

The agency at that point had received 15 "adverse event reports" about Abbott's ID NOW rapid COVID test suggesting that infected patients were wrongly told they did not have the coronavirus, which had led to the deaths of tens of thousands of Americans. The warning followed multiple academic studies showing higher "false negative" rates from the Abbott device, including one from New York University researchers who found it missed close to half of the positive samples detected by a rival company's test.

But then, in a move that confounded lab officials and other public health experts, a senior FDA official later that month said coronavirus tests provided outside lab settings would be considered useful in fighting the pandemic even if they miss 1 in 5 positive cases — a worrisome failure rate.

The FDA has now received a total of 106 reports of adverse events for the Abbott test, a staggering increase. The agency has not received a single adverse event report for any other point-of-care tests meant to diagnose COVID-19, an agency spokesperson said.

In a statement, Abbott Laboratories said the NYU research was "flawed" and "an outlier," citing studies with higher accuracy rates.

Though the Abbott rapid test is one of over 100 COVID-19 diagnostic tests to receive FDA emergency use authorization during the pandemic, President Donald Trump has featured the product in the White House Rose Garden and the Health and Human Services Department's preparedness and response division has issued more than $205 million worth of contracts to buy the test, according to federal contract records.

"Everybody was raving about it," a former administration official said, speaking on the condition of anonymity to discuss internal deliberations. "It's an amazing test, but it has limitations which are now being better understood."

In its own COVID-19 testing policy for labs and commercial manufacturers, the FDA says a diagnostic test should correctly identify at least 95% of positive samples.

But medical professionals are split over the lower 80% threshold for the Abbott and other point-of-care tests' "sensitivity" — a metric showing how often a test correctly generates a positive result. They are debating whether it's sufficient, given the risks that an infected person unwittingly spreads COVID-19 after receiving a negative result.

False negatives increase the risk that patients will not self-isolate or exercise other precautions — such as wearing a mask — and make more people sick than if they had had an accurate diagnosis. Evaluations of the Abbott test have been among the most mixed, with some researchers finding that the test has bigger accuracy problems, but others saying it isn't likely to miss sicker patients.

"There's no way I would be comfortable missing 2 out of 10 patients," said Susan Whittier, director of clinical microbiology at NewYork-Presbyterian/Columbia University Medical Center. Whittier and co-authors found that the Abbott test correctly identified 74% of positive samples compared with a rival test from Roche, another diagnostics giant. A point-of-care test from Cepheid, a rival company, correctly identified 99% of positives.

An FDA official cited the 80% accuracy minimum for point-of-care tests in late May even after two White House aides tested positive for the virus. The Executive Office of the President has spent roughly $140,000 on Abbott test kits, according to contract records.

In a statement, Abbott said when its test is used as intended it "is delivering reliable results and is helping to reduce the spread of infection in society by detecting more positive results than would otherwise be found." Studies from University Hospitals Cleveland Medical Center and OhioHealth found that its test detected at least 91% of positives.

In March, HHS officials announced that Cepheid would receive approximately $3.7 million through its Biomedical Advanced Research and Development Authority for coronavirus diagnostic development work; the Strategic National Stockpile also made a one-time $2.3 million purchase of Cepheid's point-of-care tests, according to an agency spokesperson.

"Knowing the true performance of such a point-of-care test and knowing that it may be less sensitive than a central lab molecular test is important, but also can play a role in triaging patients who are suspected of having COVID-19," Dr. Timothy Stenzel, director of the FDA's Office of In Vitro Diagnostics and Radiological Health, said on the call when discussing the Abbott warning. "If we are able to determine that sensitivity of the assay in controlled trial circumstances is at least 80%, we feel like that test has a valuable place going forward in this pandemic."

Until the FDA can complete post-market studies to verify performance, he said, negative results with the Abbott rapid test will be treated as "presumptive" negatives.

Despite that note of caution, governors were also told during a June 3 call with Vice President Mike Pence that the fast Abbott test should be used to test residents and staff in long-term care settings, according to two sources with knowledge of the discussions. As of May 31, more than 95,000 people in nursing homes have tested positive for COVID-19 and nearly 32,000 have died, according to the Centers for Medicare & Medicaid Services. If 20% of tests are false negatives, personnel with COVID-19 could be going about their normal activities, spreading the virus.

Spokespeople for Pence did not respond to requests for comment.

No test is perfect, whether it's for a common illness like the flu or for COVID-19, which has killed nearly 120,000 Americans. Federal officials contend that the trade-off with point-of-care tests — especially ones as fast as Abbott's, which can turn around a positive result in as little as five minutes and a negative one in 13 minutes — is that the tests can be used in spots where traditional lab tests aren't as accessible. There's also a greater risk of operator error when administering the test in the real world given the way patient specimens are collected and handled.

An FDA spokesperson said officials' "general expectation" is that companies' test validation data indicate a sensitivity of at least 95%; however, "based on the available information, FDA has issued EUAs [emergency use authorizations] to some tests that presented data indicating a sensitivity below 95%."

"Rapid and reliable detection of positive patients can be important for public health," the spokesperson said.

Cepheid would not comment on FDA's 80% standard but pointed to a Northwell Health Laboratories study finding its test was 98% accurate in detecting positives. The company has shipped approximately 6 million tests since getting FDA authorization in March.

"Experience has shown that in COVID testing, pre-analytical variables such as the site of sample collection (nasal, nasopharyngeal, throat, saliva) and the quality of the sample collected can have a large impact on test performance," Dr. David Persing, Cepheid's chief medical and technology officer, said in a statement. "Tests with higher sensitivity have a natural advantage in this setting."

As far as Abbott's test, "there are certainly some elements of it that could be improved, but I think it's a great assay," said Michael Mina, an assistant professor of epidemiology at the Harvard T.H. Chan School of Public Health, who said the NYU study was "flawed" in part because of the number of patients researchers included with such low viral loads that they wouldn't be infectious. An NYU spokesperson has defended the research.

Mina and others also took issue with the FDA's threshold.

"For them to just say 80%, it lets people game the system," he said. Companies presented with a single figure for sensitivity could manipulate which patients they test to exceed the minimum — for example, by including only very sick patients, which most tests would have an easier time detecting as a positive.

"They really need to fix this issue," Mina said.

Abbott has already made several revisions to its materials for how its test should be performed. It removed prior language in its instructions saying swabs could be placed in a type of liquid — known as viral transport medium — before the test is run because doing so caused patient specimens to be too diluted. Now, the company says only direct swabs from patients should be inserted into the machine. It also revised its instructions for handling patient specimens following a KHN story in which lab professionals voiced safety concerns.

Christopher Polage, the medical director of Duke University Health System's clinical microbiology lab, said experts have known for years that point-of-care tests are not as good at identifying known positives, but there are still legitimate situations in which they are used clinically.

"The difference now," he said, "is that people are so fearful and tolerance for false negatives is just zero."

Abbott's rapid COVID-19 test isn't the only point-of-care test to receive FDA authorization during the pandemic, but Trump has touted it the most by far, hailing the speed at which results can be given. To date, Abbott has delivered more than 3.6 million of the rapid tests to customers in all 50 states, including urgent care clinics, physicians' offices, the federal government and hospital emergency departments.

In Washington, D.C., the city government has distributed 11 Abbott testing instruments across homeless shelters, jails, long-term care facilities, a clinic that largely serves lower-income Latinos and two public hospitals in the district's poorest neighborhoods. Mobile testing units were also equipped with five Abbott machines, using them to test 40 to 50 people per site mostly at long-term care facilities, according to city officials.

The district has not found that the Abbott test has a higher rate of false negatives compared to other tests, said Dr. Jenifer Smith, director of the city's Department of Forensic Services, which oversees the city's public health lab.

"We test every test before we put it online," she said. "We didn't find the successively high rate of false negatives. In fact, we found we were getting the same results."

But several labs have found the test with the presidential seal of approval to be less accurate than some of its competitors in detecting known positives.

"Without confirmation of negatives, I wouldn't want to use it in hospitals," said Gregory Berry, director of molecular diagnostics at Northwell Health Laboratories in New York. Berry co-authored the analysis finding that the Abbott ID Now test detected nearly 88% of positive samples compared with 98% from Cepheid, whose point-of-care test takes 45 minutes. Abbott contested the findings of the Northwell and Columbia papers because it said the tests were run in a way that diluted the specimens.

Berry and other lab personnel also said that a key metric Abbott reported as part of the process for receiving FDA authorization — known as the limit of detection, which specifies how little of the virus is needed for the test to reliably detect it — doesn't match up with their own findings, and, in at least one instance, by a huge margin. A higher limit of detection means more viral material is needed for the test to detect the coronavirus. And a higher limit of detection increases the risk of false-negative results if a patient's viral burden isn't high.

Berry's analysis determined that the lowest quantity of the virus needed to identify 100% of positive cases was 20,000 copies per milliliter. Abbott's self-reported standard is a fraction of a percent of that.

Abbott said the limit of detection it provided to the FDA is accurate.

Of Abbott's stated limit, Polage said: "I'm not sure what they were thinking."