Abrazolica

Medical tests are not perfect. Sometimes a person with the disease will test negative and sometimes a healthy person will test positive. The COVID-19 tests are no exception. They are by no means perfect. In fact some are pretty bad. So if I take one of these tests and it comes back positive, what are the chances I actually have the disease? That's the question I want to answer.

How well a medical test performs is specified by two parameters called the sensitivity and specificity of the test. To see what these parameters mean, suppose you give the test to a group of people you know are sick and a group of people you know are healthy. Let

N(S) = number of people in the sick group
N(H) = number of people in the healthy group

The results can then be summarized in the following table.

N(S+) = number of sick people who test positive
N(H+) = number of healthy people who test positive
N(S-) = number of sick people who test negative
N(H-) = number of healthy people who test negative
N(S+) + N(S-) = N(S)
N(H+) + N(H-) = N(H)
N(+) = N(S+) + N(H+) = number of positive test results
N(-) = N(S-) + N(H-) = number of negative test results

If the test was perfect you would get N(S-) = N(H+) = 0 i.e. there would be no false negatives or positives. From the numbers in the table you can get an estimate of the sensitivity and specificity of the test by looking at the fraction of sick people who test positive and the fraction of healthy people who test negative.

sensitivity = N(S+)/N(S)
specificity = N(H-)/N(H)

In other words, the sensitivity is the probability that, given someone is sick, they test positive. The specificity is the probability that, given someone is healthy, they test negative. We will call these probabilities P(+|S) and P(-|H). It is important to remember that the above ratios are just estimates of these probabilities. The larger you make N(S) and N(H), the better the estimates.

Suppose for example you have N(S)=34. If you perform the test on these people and find that N(S+)=34, i.e. they all test positive, does that mean you have a perfect test, that P(+|S)=1? No, P(+|S)=1 is just an estimate. The real value will probably be close but it is almost surely not exactly equal to 1.

A medical test should specify what's called a 95% confidence interval for P(+|S) and P(-|H). This means that if you redo the testing many times, using the same N(S) and N(H), the estimates you get for P(+|S) and P(-|H) will fall within the interval 95% of the time. There are many ways to calculate the interval and many papers have been written on the subject. It comes from the part of statistics called estimation theory. If you're interested you can start with the Wikipedia article on binomial proportion confidence intervals.

A less common approach to assessing how good the estimates for P(+|S) and P(-|H) are is to look at their probability distributions (yes, a probability can have a probability distribution). For the N(S)=34 example mentioned above what you have essentially is 34 tosses of a coin with two outcomes + or -. Since the population is known to be sick you can assume P(+) is much greater than P(-). In other words we have a highly biased coin but we don't know exactly what the probabilities are. We show in our book Coin Tossing: the Hydrogen Atom of Probability that one way you can model these probabilities is with a Beta distribution. See also our video of a Dancing Beta Distribution.

Let p=P(+|S) then the Beta distribution for p is

\[f(n,k,p)=(n+1)\binom{n}{k}p^k(1-p)^{n-k}\]

where n=N(S) and k=N(S+). So for k=n=34 you have

\[f(34,34,p)=35p^{34}\]

A plot of this distribution is shown below for 0.8 ≤ p ≤ 1. Note that the peak is at p=1 which is obviously the most likely value given the data. But there is a nonzero probability that p < 1.

Beta distribution

To get the probability that p is in the interval a ≤ p ≤ 1 you integrate f(34,34,p) over the interval. The result is

\[F(a)=P(a \lt p \lt 1) = 1 - a^{35}\]

A plot of F(a) for 0.8 ≤ a ≤ 1 is shown below. From the figure you can see that p is is almost certainly in the interval 0.8 ≤ p ≤ 1. As you narrow the interval the probability that p is contained in it gets smaller. The 95% confidence interval appears to be about 0.92 ≤ p ≤ 1.

Integral of Beta distribution

The beta distribution is not a common way to model the value of p but in my opinion it is more accurate. Before I move on I should mention that f(n,k,p) always has a maximum at p=k/n and the expectation and standard deviation for the distribution are

\[\mu = (k+1)/(n+2)\] \[\sigma = \sqrt{(k+1)*(n-k+1)/(n+3)}/(n+2)\]

Now let's look at the question I started with. Given that I get a positive COVID-19 test result, what is the probability I have the disease? I will denote this probability P(S|+). In the literature this is called the positive predictive value of the test. How do I calculate P(S|+) given the sensitivity and specificity of the test, i.e. given P(+|S) and P(-|H)?

From Bayes' Theorem we know that

\[P(S|+) = \frac{P(S+)}{P(+)} = \frac{P(+|S)P(S)}{P(+)}\]

and from the law of total probability we have:

\[P(+) = P(S+) + P(H+) = P(+|S)P(S) + P(+|H)P(H)\]

So the equation we're looking for is

\[P(S|+) = \frac{P(+|S)P(S)}{P(+|S)P(S) + P(+|H)P(H)}\]

where P(H) = 1 - P(S) and P(+|H) = 1 - P(-|H). We know P(+|S) and P(-|H), they are the sensitivity and specificity of the test. The only missing piece of information is P(S) i.e. the probability of being sick, also known as the prevalence of the disease. This will have to be estimated somehow. The FDA puts the estimate at 5% or P(S)=0.05.

As a sample calculation I'll use the sensitivity and specificity data for the Abbott Laboratories, Abbott Alinity i SARS-CoV-2 IgG test. This data along with data for other FDA approved tests can be found on the FDA's EUA Authorized Serology Test Performance web page The 95% confidence intervals for P(+|S) and P(-|H) are

P(+|S) = [0.899, 1.000]
P(-|H) = [0.946, 0.998]

Using this in the above equation for P(S|+) along with P(S)=0.05 (5% of the general population has antibodies) will give you the following 95% confidence interval for having the disease given a positive test result.

P(S|+) = [0.467, 0.963]

Note the wide range. At the low end, the chance that you have COVID-19 is no better than a coin toss.

What about the probability that you are healthy given a negative test result. This is called the negative predictive value of the test and we will denote it as P(H|-). Deriving a formula for P(H|-) in terms of P(+|S) and P(-|H) is pretty much the same as what I did above for P(S|+). Leaving out the details, the formula is

\[P(H|-) = \frac{P(-|H)P(H)}{P(-|S)P(S) + P(-|H)P(H)}\]

where P(-|S) = 1 - P(+|S). Doing the calculation using the above intervals for P(+|S) and P(-|H) will give you the following 95% confidence interval for being healthy given a negative test result.

P(H|-) = [0.994, 1.000]

The test has a much better negative predictive value than positive predictive value. This is generally true as long as the number of healthy people is much larger than the number of sick people.

The positive and negative predictive values of a test should be included in the test results. If not, you should ask for them. Also try to get the sensitivity and specificity along with information on exactly how they were estimated so you can do your own confidence interval analysis.

If you get a positive result on a test with a low positive predictive value you should have another independent test done. The positive predictive value of the two tests will usually be much higher. The simplest thing to do is to take the same test over again. While this may not exactly be independent of the first test you can treat it as such for a first approximation.

To see how this works let's assume we have a test with sensitivity and specificity equal to

P(+|S) = 0.95
P(-|H) = 0.98

and P(S)=0.01 (1% of the population is assumed to be sick) then

P(S|+) = 0.324

If you take the test again you use the same equation but the prevalence of the disease is now the value we just calculated, P(S)=0.324. The new calculation is

P(S|+) = 0.958

which is a much better result. Iterating this again you get

P(S|+) = 0.999

This is more or less common sense. If you take the test over and over again and it keeps coming up positive then you probably really do have the disease, barring the possibility that the additional tests are not at all independent and don't provide any additional information.

To summarize, always look at the positive and negative predictive values of a test. Getting a positive test result doesn't always mean that you actually have or had the disease. Always try to have more than one test done. One test by itself almost never provides a definitive result. Finally, wash your hands, wear a mask, and avoid crowds so we can beat this virus.