Why randomness doesn't seem random

by Dan Gardner

(As published in the Ottawa Citizen, Wednesday, July 25, 2007.)

Over the weekend in Toronto, four people were murdered in separate incidents. One of the victims was an 11-year-old boy.

Murder is always tragic and terrible. The murder of a child is doubly so. That's beyond dispute. But inevitably, bursts of violence such as this lead people to ask basic questions. What does it mean? What does it tell us about safety in the city? What's wrong with society? The first answer is almost always the same: It means violence is rising. The answers to the other questions vary -- with people blaming guns, drugs, broken families or immigration, depending on their personal feelings.

But go back to the beginning. What does it mean? What does it tell us about safety in the city? People are wrong to assume it means violence is rising. It may be the result of nothing more than random distribution. Chance. And if that's the case, it means nothing. Nothing at all.

This can be hard to grasp because people generally have a poor sense of how randomness works. We can't see randomness. We don't sense it intuitively. And so if we don't learn the basic concepts, and learn to apply them, we routinely get randomness wrong. "People understand that if you flip a coin once, you're equally likely to get heads or tails," says University of Toronto mathematician Jeffrey Rosenthal. But they often fail to grasp that this remains true no matter how many times the coin is flipped or what the results are. Flip it five times, get five heads and it will be equally likely that you'll get heads or tails on the next flip. Flip it 20 times, get 20 heads and it's still true.

But that's not what our intuition tells us. Get 20 heads in a row and you will have a powerful gut feeling that tails is "due." Psychologists call this "gambler's fallacy." Casinos love it. There's no better way to keep gamblers glued to slot machines and roulette wheels.

The basic problem here is that our guts assume "random" means "evenly distributed." But that's not at all true. "You might not see anyone with purple hair for months or years and then you might see a few people with purple hair in the same week just by chance," says Rosenthal. "Or you might get three different wrong numbers on your phone in the same evening after you haven't had one for months. These things can clump up just by chance." A French mathematician by the name of Poisson determined how to calculate random distribution, so mathematicians call these "Poisson clumps." It's an ugly term for a critical insight.

In the first week of November 2003, a surge of violence in Toronto took the lives of five people. The bloodshed filled newspapers and news broadcasts. Julian Fantino, the city's police chief at the time, said it proved the justice system is so soft it "provides no apparent deterrent." Some pundits blamed cuts to social services. Others said family breakdown was the cause. A few politicians called for a ban on handguns.

People were right to be concerned. They were right to ask questions. But they were wrong to assume that there was some meaning to be gleaned from this cluster of tragedies.

"The Poisson distribution tells us that with an average of 1.5 homicides per week, there is still a 1.4 per cent probability of seeing five homicides in a given week, purely by chance," wrote Rosenthal in his book Struck by Lightning. That may not sound like much but it adds up. It means you can expect to see a week with five murders once every 71 weeks.

So a week in which there are five murders "really isn't so surprising, nor does it signify anything other than bad luck." In fact, that year in Ontario and across Canada, the homicide rate declined.

The flipside of "homicide clumping" are periods without any murders at all. "Each week has just over 22 per cent probability of being completely homicide-free, and indeed Toronto experiences many weeks without any homicides at all," Rosenthal wrote. "But I have yet to see a newspaper headline that screams 'No murders this week!' " So what does the spate of bloodshed in Toronto over the weekend mean for safety in the city? It's well within the bounds of what random distribution alone would produce. And so it may mean nothing at all.

I know people will hate me for putting it that way. These are crimes and tragedies we are talking about. It seems cold and inhuman to try to make sense of it with mathematics.

But our feelings don't change the facts. Math is an invaluable tool in making sense of crime. And it sheds light on other matters of life and death, as well.

Consider the phenomenon of "cancer clusters." Every year, officials field calls from people who notice that their father, their neighbour, and two people down the road got cancer. Doctors make these calls, too. Six patients with a rare form of cancer in my little practice, they think. Impossible. That can't be a coincidence.

And those who make these calls are usually pretty sure about the source of the cancer. If they live in farm country, they'll point to pesticides sprayed on the fields. If there's a nuclear plant in the region, they'll blame it. Or maybe the culprit is a landfill. Or a chemical factory. Whatever. The suspicious will always find something to suspect.

Officials hate these calls because they know what is almost certain to happen next. They will do the math. They will find the cluster can be explained by chance alone. They will explain this. And people will not believe them.

"People have a need to ascribe meaning to things," Rosenthal says. "But the reality is it may just be chance."

[Jeffrey Rosenthal's Home Page / contact page / Struck by Lightning page / Writing Page]