When is that extra 0.9% important?

The question in the post title was asked on Quora recently (in the form of “when is the difference between 99% accuracy and 99.9% accuracy important”), and while it mainly attracted some stock-standard responses, such as service level agreements, Alex Suchman told us when it’s really important: when it can stop a zombie apocalypse.

It’s 2020, and every movie buff and video gamer’s worst fear has become reality. A zombie outbreak, originating in the depths of the Amazon but quickly spreading to the rest of the world (thanks a lot, globalization) threatens the continued existence of the human race. The epidemic has become so widespread that population experts estimate one in every five hundred humans has been zombified.

The zombie infection (dubbed “Mad Human Disease” by the media) spreads through the air, meaning that anyone could succumb to it at any moment. The good news is that there’s a three day asymptomatic incubation period before the host becomes a zombie. A special task force made of the best doctors from around the world has developed a drug that cures Mad Human, but it must be administered in the 72-hour window. Giving the cure to a healthy human causes a number of harmful side effects and can even result in death. No test currently exists to determine whether a person has the infection. Without this, the cure is useless.

As a scientist hoping to do good for the world, you decide to tackle this problem. After two weeks of non-stop lab work, you stumble upon a promising discovery that might become the test the world needs.

Scenario One: The End of Mankind

Clinical trials indicate that your test is 99% accurate (for both true positives and true negatives). Remembering your college statistics course, you run the numbers and determine that someone testing positively will have Mad Human only 16.6% of the time [1]. Curse you, Thomas Bayes! You can’t justify subjecting 5 people to the negative effects of the cure in order to save one zombie, so your discovery is completely useless.

With its spread left unchecked, Mad Human claims more and more victims. The zombies have started taking entire cities, and the infection finally reaches Britain, the world’s last uncontaminated region. Small tribal groups survive by leaving civilization altogether, but it becomes clear that thousands of years of progress are coming undone. After the rest of your family succumbs to Mad Human, you try living in isolation in the hope that you can avoid the epidemic. But by this point, nowhere is safe, and a few months later you join the ranks of the undead. In 2023, the last human (who was mysteriously immune to Mad Human) dies of starvation.

Scenario Two: The Savior

Clinical trials indicate that your test is 99.9% accurate. Remembering Bayes’ Theorem from your college statistics course, you run the numbers and determine that someone testing positively will have Mad Human 66.7% of the time [2]. This isn’t ideal, but it’s workable and can help slow the zombies’ spread.

Pharmaceutical companies around the world dedicate all of their resources to producing your test and the accompanying cure. This buys world leaders precious time to develop a way to fight back against the zombies. Four months after the release of your test, the U.S. military announces the development of a new chemical weapon that decomposes zombies without harming living beings. They fill Earth’s atmosphere with the special gas for a tense 24-hour period remembered as The Extermination. The operation is successful, and the human race has been saved!

Following the War of the Dead, you gain recognition as one of the greatest scientific heroes in history. You go on to win a double Nobel Prize in Medicine and Peace. Morgan Freeman narrates a documentary about your heroics called 99.9, which sweeps the Academy Awards. Your TED Talk becomes the most-watched video ever (yeah, even more than Gangnam Style). You transition into a role as a thought leader, and every great innovator of the next century cites you as an influence.

Life is good.

That is when the difference 99% and 99.9% matters.

[1] A 99% accurate test doesn’t mean that someone who tests positive has a 99% chance of actually being positive. Because the event of having the infection is so relatively rare (only 1 in 500) and the event of not having the disease is so common (499 in 500), even though the test is rarely wrong, it turns out to be more likely that a positive test comes from a healthy person than a sick one. To compute this we use Bayes’ Theorem, which states that

P(A|B) = \frac{P(B|A)P(A)}{P(B)}

We let A be the event that the person is sick and B be the event that the person tests positive, so we have

P(sick|+ test) = \frac{P(+ test|sick)P(sick)}{P(+ test)}
In this situation,

P(+ test|sick) = .99

and

P(sick) = .002 (that’s 1 in 500)

To compute P(+ test) we have to condition on whether the person is sick or not. So

P(+ test)

= P(+ |sick)P(sick) + P(+|not)P(not)

= (.99)(.002) + (.01)(.998) = 0.01196

Plug everything in and we get

P(infected|+ test) = \frac{.99*.002}{0.01196} = 0.16555

[2] This time,

P(+ test|sick) = .999

and

P(+ test)

 = (.999)(.002) + (.001)(.998)

 = 0.002996

so

P(sick|+ test) = \frac{.999*.002}{0.002996} = 0.66689

Tagged , , . Bookmark the permalink.

Comments are closed.