Oregon recorded its first COVID-19 death on March 14. Over the next eight weeks, 6,799 Oregonians would die — well above the five-year average for that period.
All told, Oregon exceeded its average death toll by more than 400. It's an alarming number, particularly since traffic deaths are down 32% compared to last year. And yet, only 132 deaths in this period were attributed to the new coronavirus.
“I think that number is concerning,” State Epidemiologist Dr. Dean Sidelinger said. “I wish, you know, I knew what the reason was for all those deaths.”
But Sidelinger doesn’t know. No one does.
Oregon started to reopen last week, and to do that safely, the state needs widely available, accurate testing to catch new outbreaks of the disease. And so it's critical that Oregon's COVID-19 numbers can be trusted.
Whether or not Oregon’s count is missing cases comes down to three things: Are enough people being tested? Are the right people being tested? And are those tests accurate?
Compared with the number of cases per capita in places with similar populations — like Colorado or hard-hit states like New York, Massachusetts and Illinois — Oregon has, so far, been spared the worst of the COVID-19 pandemic. People largely complied with Gov. Kate Brown's stay-home order. The number of hospitalized COVID-19 patients in Oregon has been manageable and in decline in most parts of the state.
But that above-average number of deaths is like ripples on a pond — a sign that something disturbed the water, even if you didn’t see what broke the surface.
Bradley W. Parks / OPB
Until recently, there weren't enough COVID-19 tests, including in Oregon. So it's a given that many cases were missed as testing capacity ramped up in April and May.
But even if you got a test, there was no guarantee the results were accurate. When a diagnostic test is created, it’s tested with samples that are known to have the virus, to see if it’s accurate. That’s in the lab, where a test might be 100% accurate.
“But it’s important to remember that there’s a difference between laboratory accuracy, and analytical accuracy or clinical accuracy,” said Dr. Yassmine Akkari, the director of molecular pathology at Legacy Health in Portland. “Analytical sensitivity is the ability of a test to detect the virus when it’s actually present in the specimen.”
Basically: In the real world, things get messy. A lot of little things can go wrong.
Several studies of COVID-19 tests have found high rates of false negatives, far above the recommended 5%.
Right now, Oregon is relying on two types of tests: RT-PCR tests, considered the gold standard in disease diagnostics, and rapid tests, which have come online in large numbers more recently.
To understand how a test that seems accurate can result in false negatives, it helps to understand the tests and the way they’re conducted.
RT-PCR: an imperfect gold-standard test
The first COVID-19 tests available were RT-PCR tests. They require a very small amount of genetic material (DNA or RNA), copy that material, and then read the sequence.
For COVID-19, the test usually starts with a swab, stuck far up the nose. Then, the RNA is extracted from the virus and other material, like snot and proteins from the viral shell, are removed. The RNA forms a little pellet at the bottom of a tube.
Then that purified sample is put into the RT-PCR machine. RT stands for reverse transcription and PCR is short for polymerase chain reaction. A polymerase is a molecule that makes copies of genetic material. Scientists add a “primer” to the mix, and add the polymerase. The primer tells the polymerase what part of a gene to copy.
The machine cycles until eventually there’s enough RNA to sequence. The different letters in the genetic code are tagged with pigments that glow different colors. The machine reads those colors and turns it into genetic code — a string of letters unique to that virus.
“If somebody tests positive, that's a very clear indicator that they almost definitely have a virus,” said Thomas Jeanne, the Oregon Health Authority’s senior adviser on testing.
And if somebody isn’t sick, they won’t have viral RNA in their nose, so there won’t be any genes to amplify. That means false positives are extremely rare.
But false negatives are a different story.
In a health care setting, a lot can increase the chances of a false negative: how the person is tested, how the sample is stored, how it’s shipped, how long it takes for the sample to be processed, and even when a person is tested during the course of their illness.
“These tests should be done by labs with expertise,” Akkari said.
Akkari’s hospital system, like many others, has avoided tests run by major corporations like LabCorps and Quest. Instead many larger hospitals and health systems have created RT-PCR tests they control from start to finish. But some potential errors can’t be avoided. And they start the second a swab is stuck several inches up a person’s nose.
How are we gathering samples?
RT-PCR tests have one big problem, Jeanne said: “You can have the virus, but if you don’t get enough of it in the sample, it won’t show up on the tests.”
The way samples are taken can have a big impact on that. Think back to a time you jumped into a pool and got water up your nose. Now imagine a swab reaching up into that same, painful place. You might flinch or draw back before there’s enough sample of your mucus on the swab.
Bradley W. Parks / OPB
And then there are health care workers, who might be hesitant to stick a swab far enough up someone’s nose to get a proper result.
“It’s not just about the lab running the test, it’s about the person who’s performing it, how long they swab for, what techniques they have. Are they swabbing both sides of the nose?” Jeanne said.
For most of March and April, testing swabs were in short supply. Some hospitals reported using a nasal wash for the sample, increasing the chance of a false negative.
Research shows that samples collected with a certain type of deep-reaching swabs, called "nasopharyngeal swabs," give the most accurate test results. Specimens collected with combined front-of-the-nose and throat swabs or nasal washes are considered less effective.
OHA was not able to collect data on how samples for tests were taken, Jeanne said, so it’s unclear how widespread sampling problems were, or how that could have impacted our overall test count.
Was the sample handled properly?
Once a sample is taken, more potential problems arise. Until mid-April, many Oregonians reported waiting over a week to learn the results of their COVID-19 tests. It was an early sign that those results might not be accurate.
Most of the delayed tests were shipped to far-away laboratories or were conducted by commercial labs like Quest Diagnostics, which has been processing tests from around the country.
A long delay shouldn’t necessarily impact the accuracy of a test, but that all depends on how the sample is handled, Akkari said.
“I don’t know what most commercial labs are doing, but here at Legacy, we recommend that a test be run, at most, 72 hours after it was collected,” Akkari said. And it needs to be kept at near-freezing temperatures until then. That’s because viral RNA is extremely fragile.
“More than fragile, actually, and much more fragile than DNA,” she said. “The whole world is full of these RNases” — special molecules that break up RNA, sort of like a cleaning service in your cells.
Jeanne said that if a viral sample is going to take more than 72 hours to be tested, it should be frozen at 70 degrees below zero celsius. That’s colder than the average temperature during sunless Antarctic winters.
But given the myriad ways RT-PCR specimens are shipped to commercial labs (ideally, by courier, but occasionally by mail), Jeanne said it’s entirely possible samples aren’t being frozen before they’re shipped, or they thaw in-transit.
But he doesn't know if that actually happened. That information hasn't been passed on to public health departments like his. Health care workers across the country have voiced concerns about how test samples are shipped. But lacking hard data, it's just one more thing that could go wrong.
Bradley W. Parks / OPB
Once the samples arrive at the lab, they need to be stored — again, at temperatures well below freezing. At one point, Quest reported a backlog of 160,000 tests, received and waiting to be tested.
“If Quest and LabCorps [had] a big backlog — and they did — I would wonder if they were freezing samples or how,” Jeanne said.
And then there's the process of extracting the RNA from the virus and purifying it. Once RNA is purified it's much more stable. But the chemicals to extract RNA are in short supply, contributing to the testing shortages, said Ben Dalziel, an epidemiologist at Oregon State University. He is involved in a survey that aims to track the spread of COVID-19 in Corvallis.
If a lab doesn’t have the supplies to extract RNA, the samples could have been slowly degrading.
In recent weeks, commercial labs seem to have caught up with the flood of COVID-19 tests coming in. Jeanne said turnarounds are back to what he considers a reasonable amount of time: about four days.
Who is getting tested, and when?
Until recently, testing capacity for COVID-19 was very limited in Oregon. Even as that capacity increased, strict criteria kept the number of people who qualified for tests low.
Very few people with mild or moderate symptoms were getting tested. And many people were told to stay home and ask for a test again if they got sicker. Most tests were going to the sickest patients, and that’s a problem, because they’re the most likely to get false negatives. That’s because of a weird fluke in how the new coronavirus behaves.
As COVID-19 progresses, the coronavirus takes up residence in different places in your body, Akkari explained. And as the virus moves through your body, it’s less likely to be detected by a standard nasopharyngeal swab, because it’s just not there anymore.
According to a scientific literature review, an RT-PCR test for COVID-19 is at its most accurate about three days after symptoms appear, with a false negative rate of about 22%.The false negative rate climbs slowly as the disease progresses. By the time patients are feeling very ill, 16 days after symptoms start, as many as 66% of swabs come back with false negatives.
The Oregon Health Authority is continually updating its testing guidelines. Patients with mild symptoms can now get tested. Jeanne said that’s caused the number of tests conducted to climb rapidly.
But most people with a sniffle, cough or headache wait it out and see if they get better — they don’t want to seem like hypochondriacs and rush straight to their doctor’s office. But the sooner people get tested, the more accurate their test results will be.
Rapid tests: fairly new, dropped by some clinics
There’s one way to avoid false negatives produced by poorly handled samples: Collect the sample and do the test right away on-site. That’s why most hospitals with laboratories run their RT-PCR tests in-house.
But many hospitals and doctor’s offices don’t have in-house laboratories. And that’s led to an increased call for rapid tests, which don’t require a lab and can be completed before a checkup is even finished. The federal government only just sent Oregon the supplies needed to run these tests.
Related: Here's How Oregon's Reopening Depends On Testing And Tracing Benchmarks
The Trump administration touted the coronavirus rapid tests manufactured by Abbott. Its machines and kits were sent to health departments around the country. In Oregon, those tests are being deployed to rural areas without labs close by or are being used in emerging COVID-19 hotspots. Deploying rapid tests where they're needed is a key part of Oregon's plan to reopen.
COVID-19 rapid tests are fairly new, and they haven’t been held up to the same standard of performance that tests for other diseases have. Abbott claims that the false negative rate for their ID NOW rapid test is as low as .02%.
But the Cleveland Clinic conducted a study that found that the tests could miss up to 48% of infections. It also found issues with the accuracy of other COVID-19 rapid tests. Although Abbott says the false negatives were due to user error, the Cleveland Clinic and others have stopped using their tests.
The high false negative rates with rapid tests for other, more established diseases, has further fueled skepticism. The Food and Drug Administration has officially cautioned the public about the accuracy of Abbott's tests for COVID-19 and is investigating further.
The bottom line
Combined, all these factors can lead to a high number of false negatives that undermine the accuracy of COVID-19 testing. But as companies, clinicians and hospitals enter the pandemic’s third month, some accuracy issues have resolved. The rise of in-house and close-to-home tests is an example.
“The in-state testing from clinical hospital labs tends to be very fast,” Jeanne said. “They’re turning around within a day.”
Because turnaround times can’t be guaranteed by commercial labs, which process tests from across the country, OHA isn’t including them in their testing goals: Good contact tracing relies on fast test results. It’s the number of tests that can be processed in-state that really matters. Before Oregon could reopen, the Oregon Health Authority wanted to be able to process at least 15,000 tests in-state per week. That goal has been reached.
Related: Contact Tracer Hiring Is A Challenge For Portland Metro Leaders
As Oregon continues to reopen, models will depend on the accuracy and availability of tests going forward. Those models are supposed to warn us before an outbreak becomes uncontrollable.
But those models also depend on the accuracy of Oregon’s past tests. As the coronavirus pandemic progressed in Oregon, OHA deployed a common strategy: retroactively updating Oregon’s models to make future projections more accurate.
In a few more months, once all the death certificates have been processed, the OHA’s Center for Health Statistics may start to review the deaths in Oregon from the first quarter of this year. If they do, they’ll look for patients who died with symptoms similar to those caused by COVID-19. Maybe some questions will be answered.
“That would be a good project for our team to take on,” said Jennifer Woodward, the center’s state registrar. “But not right now.”
Like so many other things during this pandemic, the data just isn’t there yet.