If we ever hope to break out of our current pandemic isolation, we need tests to determine who has COVID-19 and who doesn’t. No test is perfect, however, which means any test is going to result in false positives, especially if you are trying to measure a very weak signal. The purpose of this post is to review some very basic statistics that help you interpret a positive test result. The question is, if you test positive, what are the chances you actually have the disease?
I’m going to borrow an example that you’ll find in most of the literature around the subject of Bayes’ Theorem. In those discussions, the issue is usually cancer screening or looking for some kind of genetic mutation, but I’ll try to generalize from that.
Nate Silver in his book The Signal and The Noise suggests the following scenario:
Consider a test with the following properties: if you have the disease, the test detects it 75% of the time. If you don’t have it, it still says you do 10% of the time. The first group are true positives — people who actually have the disease and tested positive, and the second are called false positives — people who don’t have the disease but test positive anyway. That’s a pretty lousy test, but it’s a start.
Say you administer that test to 1000 people, and the actual incidence of the disease is only 1.4%, meaning of those thousand people only 14 of them actually have it.
That means the test will detect 11 of the actual positives (75% of the 14 who have it). It will also report 99 false positives (10% of the 986 who don’t). So the total number of positives is 11 + 99 = 110. That means if you get a positive test, the probability that you have the disease is
p(disease given +) = p(disease and +) / p(+) = 11 / 110 = 0.1, or 10%
where p() is the probability and the + indicates a positive test. The “given” part means a conditional probability, i.e., the probability that you actually have the disease given that you tested positive for it. The bottom line is that if you get a positive test, there’s still only a 1 in 10 chance you actually have the disease. At worst, it’s time for another test.
You may argue that the test itself was bad, and there’s something to that, but the real key isn’t the test, is the frequency of the disease in the population. A disease that only shows up in 1.4% of the population is a very weak signal, and any test is going to struggle with that.
To show that, let’s take a more extreme example. Say the incidence of the disease is only a tenth of a percent, so
p(disease) = 0.001
and now assume we have a much better test that is 95% percent accurate. If you have the disease, it picks it up 95% of the time. We’ll also assume that if you don’t have the disease, the test says you do anyway (false positive) only 5% of the time.
Now we test 100,000 people, to make the math easier. That means only 100 actually have the disease, and our test will detect 95 of them. On the other hand, of the other 99,900 people, our test will report that 5%, or 4995 people, test positive anyway. That’s a lot of false positives, but with a weak signal, if you test enough people the false positives add up.
The result of that is:
p(disease given +) = p(disease and +)/p(+) = 95/5090 = 0.0187, or 1.87%
In other words, even given a test with a 95% accuracy rate and only 5% false positives, if you test positive your chance of actually having the disease is less than 2%. Amazing, but that happens because the overall incidence is so tiny.
Say instead that the incidence of the disease is much higher, like 40%. Now everything changes. That means in our population of 100,000, we have 40,000 with the disease, and our highly accurate test picks up 38,000 of them. Good job.
Of the 60,000 who don’t have it, our test reports only 5% of them as positive, which is 3000. So the same probability calculation now says
p(disease given +) = 38,000 / (38,000 + 3000) = 0.927, or 92.7%
Because the disease is so prevalent, if you get a positive test, there’s almost a 93% chance you have it. The test hasn’t changed at all, but signal is so much stronger that the true positives overwhelm the false positives.
That’s the problem with testing for a weak signal instead of a strong one. No matter how good your test is, your false positives are going to overwhelm the true ones.
What does all this mean for the current pandemic? Just that the more prevalent the disease becomes, the more likely a positive test will mean you have it. The disease is very contagious, which is why we’re in a pandemic in the first place. Experts suggest well over 40% of the population will eventually be exposed, so a positive test is likely to be really meaningful. In the early days, however, the same test may be a lot less certain.
Don’t be surprised, therefore, that if you test positive, your doctor suggests that you get a follow up test as well. If I wanted to double the length of this post I could discuss how Bayes’ Rule can be used to update estimates based on existing data, but at this point I’ll just say that if you test positive again, it’s a lot more likely you have the disease.
FYI, this post is based on a talk I used to give on the No Fluff, Just Stuff conference tour called Bayes’ Rule Says You’ll Attend This Talk. Who knew it would suddenly be so relevant? I should probably post the slides somewhere.
Good luck in your quarantining. Sigh.