Also by Tim Harford

The Undercover Economist

The Logic of Life: The Rational Economics of an Irrational World

Dear Undercover Economist

Adapt: Why Success Always Starts with Failure

The Undercover Economist Strikes Back

Messy: The Power of Disorder to Transform Our Lives

Fifty Inventions That Shaped the Modern Economy

RIVERHEAD BOOKS

An imprint of Penguin Random House LLC

penguinrandomhouse.com

First published in Great Britain as How to Make the World Add Up: Ten Rules for Thinking Differently About Numbers by The Bridge Street Press, an imprint of Little, Brown Book Group, a division of Hachette UK Limited, London, in 2020.

First American edition published by Riverhead, 2021.

Copyright © 2020 by Tim Harford

Penguin supports copyright. Copyright fuels creativity, encourages diverse voices, promotes free speech, and creates a vibrant culture. Thank you for buying an authorized edition of this book and for complying with copyright laws by not reproducing, scanning, or distributing any part of it in any form without permission. You are supporting writers and allowing Penguin to continue to publish books for every reader.

Grateful acknowledgment is made for permission to reprint the following images: 1: © Nigel Holmes, Time magazine, 1982; 2: Larry Buchanan, The New Yorker, 2013; 3: Andy Cotgreave and Simon Scarr; 4: © US National Library of Medicine/Science Photo Library; 5: Hugh Small

Riverhead and the R colophon are registered trademarks of Penguin Random House LLC.

Library of Congress Cataloging-in-Publication Data

Names: Harford, Tim, 1973– author.

Title: The data detective : ten easy rules to make sense of statistics / Tim Harford.

Description: New York : Riverhead Books, 2021. | Includes bibliographical references and index.

Identifiers: LCCN 2020024095 | ISBN 9780593084595 (hardcover) | ISBN 9780593084670 (ebook)

Subjects: LCSH: Statistics—Methodology. | Social sciences—Statistical methods.

Classification: LCC HA29 .H2498 2021 | DDC 001.4/22—dc23

LC record available at https://lccn.loc.gov/2020024095

Cover design: Grace Han

pid_prh_5.6.1_c0_r0

To teachers everywhere, and my teachers in particular; in fond memory of Peter Sinclair

CONTENTS

Introduction • HOW TO LIE WITH STATISTICS

Rule One • SEARCH YOUR FEELINGS

Rule Two • PONDER YOUR PERSONAL EXPERIENCE

Rule Three • AVOID PREMATURE ENUMERATION

Rule Four • STEP BACK AND ENJOY THE VIEW

Rule Five • GET THE BACKSTORY

Rule Six • ASK WHO IS MISSING

Rule Seven •  DEMAND TRANSPARENCY WHEN THE COMPUTER SAYS NO

Rule Eight •  DON’T TAKE STATISTICAL BEDROCK FOR GRANTED

Rule Nine •  REMEMBER THAT MISINFORMATION CAN BE BEAUTIFUL, TOO

Rule Ten • KEEP AN OPEN MIND

The Golden Rule • BE CURIOUS

Acknowledgments

Notes

Credits

Index

INTRODUCTION

How to Lie with Statistics

The truly genuine problem . . . does not consist of proving something false but in proving that the authentic object is authentic.

• Umberto Eco[1]

You know the old story about storks delivering babies? It’s true. I can prove it with statistics. Take a look at the estimated population of storks in each country, and then at the number of babies born each year. Across Europe, there’s a remarkably strong relationship. More storks, more babies; fewer storks, fewer babies.

The pattern is easily strong enough to pass a traditional hurdle for publication in an academic journal. In fact, a scientific paper has been published with the title “Storks Deliver Babies (p = 0.008).” Without getting too technical, all those zeros tell us that this is not a coincidence.[2]

Perhaps you have already guessed the trick. Large European countries such as Germany, Poland, and Turkey are home to many babies and many storks. Small countries such as Albania and Denmark have few babies and few storks. While there’s a clear pattern in the data, that pattern does not mean that storks cause babies to appear.

You can “prove” anything with statistics, it seems—even that storks deliver babies.

You’d certainly have gotten that impression from reading How to Lie with Statistics. Published in 1954 by a little-known American freelance journalist named Darrell Huff, this wisecracking, cynical little book immediately received a rave review from the New York Times and went on to become perhaps the most popular book on statistics ever published, selling well over a million copies.

The book deserves the popularity and the praise. It’s a marvel of statistical communication. It also made Darrell Huff a nerd legend. Ben Goldacre, an epidemiologist and bestselling author of Bad Science, has written admiringly of how “The Huff” had written a “ripper.” The American writer Charles Wheelan describes his book Naked Statistics as “an homage” to Huff’s “classic.” The respected journal Statistical Science organized a Huff retrospective fifty years after its publication.

I used to feel the same way. As a teenager, I loved reading How to Lie with Statistics. Bright, sharp, and illustrated throughout with playful cartoons, the book gave me a peek behind the curtain of statistical manipulation, showing me how the swindling was done so that I would not be fooled again.

Huff is full of examples. He begins by pondering how much money Yale graduates make. According to a 1950 survey, the class of 1924 had an average income of close to $500,000 a year in today’s terms. That is just plausible enough to believe—this is Yale, after all—but half a million dollars a year is a lot of money. Is that really the average?

No. Huff explains that this “improbably salubrious” figure comes from self-reported data, which means we can expect people to exaggerate their income for the sake of vanity. Furthermore, the survey is only of people who bothered to respond—and only those alumni Yale could find. And who are easily found? The rich and famous. “Who are the little lost sheep down in the Yale rolls as ‘address unknown’?” asks Huff. Yale will keep track of the millionaire alumni, but some of the also-ran graduates might easily have slipped through the net. All this means that the survey will present a grossly inflated view.

Huff briskly moves on through a vast range of statistical crimes, from toothpaste advertisements based on cherry-picked research to maps that change their meaning depending on how you color them in. As Huff wrote, “The crooks already know these tricks; honest men must learn them in self-defense.”

If you read How to Lie with Statistics, you will come away more skeptical about the ways numbers can deceive you. It’s a clever and instructive book.

But I’ve spent more than a decade trying to communicate statistical ideas and fact-check numerical claims—and over the years, I’ve become more and more uneasy about How to Lie with Statistics and what that little book represents. What does it say about statistics—and about us—that the most successful book on the subject is, from cover to cover, a warning about misinformation?

—

Darrell Huff published How to Lie with Statistics in 1954. But something else happened that very same year: two British researchers, Richard Doll and Austin Bradford Hill, produced one of the first convincing studies to demonstrate that smoking cigarettes causes lung cancer.[3]

Doll and Hill could not have figured this out without statistics. Lung cancer rates had increased sixfold in the UK in just fifteen years; by 1950 the UK had the highest in the world, and deaths from lung cancer exceeded deaths from tuberculosis for the first time. Even to realize that this was happening required a statistical perspective. No single doctor would have formed more than an anecdotal impression.

As for showing that cigarettes were to blame—again, statistics were essential. A lot of people thought that motorcars were the cause of the rise in lung cancer. That made perfect sense. In the first half of the twentieth century, motorcars became commonplace, with their exhaust fumes and the strangely compelling vapor from the tar in new roads. Lung cancer increased at the same time. Figuring out the truth—that it was cigarettes rather than cars that caused lung cancer—required more than simply looking around. It required researchers to start counting, and comparing, with care. More concisely, it required statistics.

The cigarette hypothesis was viewed with skepticism by many, although it was not entirely new. For example, there had been a big research effort in Nazi Germany to produce evidence that cigarettes were dangerous; Adolf Hitler despised smoking. The Führer was no doubt pleased when German doctors discovered that cigarettes caused cancer. For obvious reasons, though, “hated by Nazis” was no impediment to the popularity of tobacco.

So Doll and Hill decided to conduct their own statistical investigations. Richard Doll was a handsome, quiet, and unfailingly polite young man. He had returned from the Second World War with a head full of ideas about how statistics could revolutionize medicine. His mentor, Austin Bradford Hill, had been a pilot in the First World War before nearly dying of tuberculosis.* Hill was a charismatic man, had a sharp wit, and was said to be the finest medical statistician of the twentieth century.[4] Their work together as data detectives was to prove lifesaving.

The pair’s first smoking-and-cancer study began on New Year’s Day 1948. It was centered around twenty hospitals in northwest London, and Richard Doll was in charge. Every time a patient arrived in a hospital with cancer, nurses would—at random—find someone else in the same hospital of the same sex and about the same age. Both the cancer patients and their counterparts would be quizzed in depth about where they lived and worked, their lifestyle and diet, and their history of smoking. Week after week, month after month, the results trickled in.

In October 1949, less than two years after the trial began, Doll stopped smoking. He was thirty-seven, and had been a smoker his entire adult life. He and Hill had discovered that heavy smoking of cigarettes didn’t just double the risk of lung cancer, or triple the risk, or even quadruple the risk. It made you sixteen times more likely to get lung cancer.[5]

Hill and Doll published their results in September 1950, and promptly embarked on a bigger, longer-term, and more ambitious trial. Hill wrote to every doctor in the UK—all 59,600 of them—asking them to complete a “questionary” about their health and smoking habits. Doll and Hill figured that the doctors would be capable of keeping track of what they smoked. They would stay on the medical register, so they’d always be easy to find. And when a doctor dies, you can expect a good diagnosis as to the cause of death. All Hill and Doll had to do was wait.

More than 40,000 doctors responded to Hill’s request, but not all of them were delighted. You have to understand that smoking was extremely common at the time, and it was no surprise to find that 85 percent of the male doctors in Doll and Hill’s initial sample were smokers. Nobody likes to be told that they might be slowly killing themselves, especially if the suicide method is highly addictive.

One doctor buttonholed Hill at a London party. “You’re the chap who wants us to stop smoking,” he pointedly declared.

“Not at all,” replied Hill, who was still a pipe smoker himself. “I’m interested if you go on smoking to see how you die. I’m interested if you stop because I want to see how you die. So you choose for yourself, stop or go on. It’s a matter of indifference to me. I shall score up your death anyway.”[6]

Did I mention that Hill originally trained as an economist? It’s where he learned his charm.

The study of doctors rolled on for decades, but it wasn’t long before Doll and Hill had enough data to publish a clear conclusion: Smoking causes lung cancer, and the more you smoke the higher the risk. What’s more—and this was new—smoking causes heart attacks, too.

Doctors aren’t fools. In 1954, when the research was published in the doctors’ own professional magazine, the British Medical Journal, they could draw their own conclusions. Hill quit smoking that year, and many of his fellow doctors quit, too. Doctors became the first identifiable social group in the UK to give up smoking in large numbers.

In 1954, then, two visions of statistics had emerged at the same time. To the many readers of Darrell Huff’s How to Lie with Statistics, statistics were a game, full of swindlers and cheats—and it could be amusing to catch the scoundrels at their tricks. But for Austin Bradford Hill and Richard Doll, statistics were no laughing matter. Their game had the highest imaginable stakes, and if it was played honestly and well, it would save lives.

—

By the spring of 2020—as I was putting the finishing touches to this book—the high stakes involved in rigorous, timely, and honest statistics had suddenly become all too clear. A new coronavirus was sweeping the world. Politicians had to make the most consequential decisions in decades, and fast. Many of those decisions depended on data detective work that epidemiologists, medical statisticians, and economists were scrambling to conduct. Tens of millions of lives were potentially at risk. So were billions of people’s livelihoods.

As I write these words, it is early April 2020: countries around the world are a couple of weeks into lockdowns, global deaths have just passed 60,000, and it is far from clear how the story will unfold. Perhaps, by the time this book is in your hands, we will be mired in the deepest economic depression since the 1930s and the death toll will have mushroomed. Perhaps, by human ingenuity or good fortune, such apocalyptic fears will have faded into memory. Many scenarios seem plausible. And that’s the problem.

An epidemiologist, John Ioannidis, wrote in mid-March that COVID-19 may be “a once-in-a-century evidence fiasco.”[7] The data detectives are doing their best—but they’re having to work with data that are patchy, inconsistent, and woefully inadequate for making life-and-death decisions with the confidence we’d like.

Details of the fiasco will, no doubt, be studied for years to come. But some things already seem clear. At the beginning of the crisis, for example, politics seem to have impeded the free flow of honest statistics—a problem we’ll return to in the eighth chapter. Taiwan has complained that in late December 2019 it had given important clues about human-to-human transmission to the World Health Organization—but as late as mid-January, the WHO was reassuringly tweeting that China had found no evidence of human-to-human transmission. (Taiwan is not a member of the WHO, because China claims sovereignty over the territory and demands that it should not be treated as an independent state. It’s possible that this geopolitical obstacle led to the alleged delay.)[8]

Did this matter? Almost certainly; with cases doubling every two or three days, we will never know what might have been different with an extra couple of weeks of warning. It’s clear that many leaders took their time before appreciating the potential gravity of the threat. President Trump, for instance, announced in late February, “It’s going to disappear. One day, it’s like a miracle, it will disappear.” Four weeks later, with 1,300 Americans dead and more confirmed cases in the United States than any other country, Mr. Trump was still talking hopefully about getting everybody to church at Easter.[9]

As I write, debates are raging. Can rapid testing, isolation, and contact tracing contain outbreaks indefinitely, or only delay their spread? Should we worry more about small indoor gatherings or large outdoor gatherings? Does closing schools help prevent the spread of the virus, or do more harm as children go to stay with vulnerable grandparents? How much does wearing masks help? These and many other questions can be answered only by good data on who has been infected, and when.

But a vast number of infections were not being registered in official statistics, due to a lack of tests—and the tests that were being conducted were giving a skewed picture, being focused on medical staff, critically ill patients, and—let’s face it—rich, famous people. At the time of writing, the data simply can’t yet tell us how many mild or asymptomatic cases there are—and hence how deadly the virus really is. As the death toll rose exponentially in March—doubling every two days—there was no time to wait and see. Leaders put economies into an induced coma—more than three million Americans filed jobless claims in a single week in late March, five times the previous record. The following week was even worse: another six and a half million claims were filed. Were the potential health consequences really catastrophic enough to justify sweeping away so many people’s incomes? It seemed so—but epidemiologists could only make their best guesses with very limited information.

It’s hard to imagine a more extraordinary illustration of how much we usually take accurate, systematically gathered numbers for granted. The statistics for a huge range of important issues that predate the coronavirus have been painstakingly assembled over the years by diligent statisticians, and often made available to download, free of charge, anywhere in the world. Yet we are spoiled by such luxury, casually dismissing “lies, damned lies, and statistics.” The case of COVID-19 reminds us how desperate the situation can become when the statistics simply aren’t there.

—

Darrell Huff made statistics seem like a stage magician’s trick: all good fun but never to be taken seriously. Long before the coronavirus, I’d started to worry that this isn’t an attitude that helps us today. We’ve lost our sense that statistics might help us make the world add up. It’s not that we feel every statistic is a lie, but that we feel helpless to pick out the truths. So we believe whatever we want to believe (more on that in the next chapter), and for the rest we adopt Huff’s response: a harsh laugh, a shrug, or both.

This statistical cynicism is not just a shame—it’s a tragedy. If we give in to a sense that we no longer have the power to figure out what’s true, then we’ve abandoned a vital tool. It’s a tool that showed us that cigarettes are deadly. It’s our only real chance of finding a way through the coronavirus crisis—or, more broadly, understanding the complex world in which we live. But the tool is useless if we lapse into a reflexive dismissal of any unwelcome statistical claim. Of course, we shouldn’t be credulous—but the antidote to credulity isn’t to believe nothing, but to have the confidence to assess information with curiosity and a healthy skepticism.

Good statistics are not a trick, although they are a kind of magic. Good statistics are not smoke and mirrors; in fact, they help us see more clearly. Good statistics are like a telescope for an astronomer, a microscope for a bacteriologist, or an X-ray for a radiologist. If we are willing to let them, good statistics help us see things about the world around us and about ourselves—both large and small—that we would not be able to see in any other way.

My main aim with this book is to persuade you to embrace Doll and Hill’s vision, not Huff’s cynicism. I want to convince you that statistics can be used to illuminate reality with clarity and honesty. To do that, I need to show you that you can use statistical reasoning for yourself, sizing up the claims that surround you in the media, on social media, and in everyday conversation. I want to help you evaluate statistics from scratch, and just as important, to figure out where to find help that you can trust.

The good news is that this is going to be fun. There’s a real satisfaction in getting to the bottom of the statistical story. You gain in confidence and feed your curiosity along the way, and end up feeling that you’ve mastered something. You understand rather than sneer from the sidelines. Darrell Huff’s approach is junk food: superficially appealing but tedious after a while. And it’s bad for you. But the opposite of statistical junk food isn’t raw oats and turnips; it’s a satisfying and delightfully varied menu.

—

In this book I’ll be describing what I’ve learned myself since 2007, when the BBC asked me to present a radio program called More or Less, a show about numbers in the news and in life. The show’s creators, the journalist Michael Blastland and the economist Andrew Dilnot, were moving on. I was less well qualified for the role than the BBC might have imagined: I trained in economic theory, not statistics. Yes, that training gave me some self-assurance when it came to numbers, but it was mostly defensive. I’d learned to spot flaws and tricks, but couldn’t do much beyond that.

It was there that my journey away from the viewpoint of Darrell Huff began.

Week after week, my colleagues and I would evaluate the statistical claims that had emerged out of the mouths of politicians or been printed in large type in the newspapers. Those claims often stretched the truth, but by itself a simple fact-check never seemed like a satisfying response. We would find that behind each claim—true, false, or borderline—was a fascinating world to explore and explain. Whether we were evaluating the prevalence of strokes, the evidence that debt damages economic growth, or even the number of times in The Hobbit that the word “she” is used, the numbers could illuminate the world as well as obscure it.

As the coronavirus pandemic has so starkly illustrated, we depend on reliable numbers to shape our decisions—as individuals, as organizations, and as a society. And just as with coronavirus, the statistics have often been gathered only when we’ve been faced with a crisis. Consider the unemployment rate—a measure of how many people want jobs but do not have them. It’s now a basic piece of information for any government wanting to understand the state of the economy, but back in 1920, nobody could have told you how many people were searching for work.[10] Only when severe recessions made the question more politically pertinent did governments begin to collect the data that could answer it.

Our big, bewildering world is full of questions that only careful attention to the numbers can answer. Does Facebook tend to make us happy or sad, and can we predict why different people react in different ways? How many species are in danger of extinction, is that a big proportion of the total, and is the cause climate change, the spread of human agriculture, or something else entirely? Is human innovation speeding up, or slowing down? How serious is the impact of the opioid crisis on the health of middle America? Is teenage drinking becoming less common—and if so, why?

I grew increasingly uneasy when fans of More or Less complimented the way we “debunked false statistics.” Sure, we did that, and it was fun. But slowly, learning on the job, I came to appreciate that the real joy was not in shooting down falsehoods but in trying to understand what was true.

Working on More or Less, I learned that commonsense principles can get you a surprisingly long way as a data detective. It’s these principles I shall sum up in this book. Most of the team of researchers and producers, like me, lacked any serious training in how to handle numbers. But even in highly technical areas, some simple questions—and perhaps an internet search or two—would often produce very rewarding answers. Yes, sometimes an advanced degree in statistics would have been useful, but we never needed it to ask the right questions. You don’t need it either.

—

Just before Christmas in 1953, senior tobacco executives met at the Plaza Hotel in New York. Doll and Hill’s big study wouldn’t be published until the following year, but the cigarette companies were already aware that the science was starting to look pretty bad for them. They met to figure out how to respond to this looming crisis.

Their answer was—alas—quite brilliant, and set the standard for propaganda ever since.

They muddied the waters. They questioned the existing research; they called for more research; they funded research into other things they might persuade the media to get excited about, such as sick building syndrome or mad cow disease. They manufactured doubt.[11] A secret industry memo later reminded insiders that “doubt is our product.”[12]

Understandably, when we think about persuasion, we think about people being tricked into believing something that they shouldn’t—and we’ll discuss this problem in the next chapter. But sometimes the problem is not that we are too eager to believe something, but that we find reasons not to believe anything. Smokers liked smoking, were physically dependent on nicotine, and wanted to keep smoking if they could. A situation where smokers shrugged and said to themselves, “I can’t figure out all these confusing claims,” was a situation that suited the tobacco industry well. Their challenge was not to convince smokers that cigarettes were safe, but to create doubt about the statistical evidence that showed they were dangerous.

And it turns out that doubt is a really easy product to make. A couple of decades ago, psychologists Kari Edwards and Edward Smith conducted an experiment in which they asked people in the United States to produce arguments in favor of and against the politically fraught positions of the day such as abortion rights, spanking children, allowing homosexual couples to adopt, quotas for hiring minorities, and the death penalty for those under sixteen years of age.[13] Unsurprisingly, they found that people had biases: the study participants found it hard to construct the kind of arguments that their opponents would use to defend their views. More strikingly, Edwards and Smith showed that those biases tended to appear more clearly in negative arguments. Disbelief flowed more fluidly than belief. The experimental subjects found it much easier to argue against positions they disliked than in favor of those they supported. There was a special power in doubt.

Doubt is also easy to sell because it is a part of the process of scientific exploration and debate. Most of us are—or should be—taught at school to question the evidence. The motto of one of the oldest scientific societies, the Royal Society, is Nullius in verba—“Take nobody’s word for it.” A lobby group seeking to deny the statistical evidence will always be able to point to some aspect of the current science that is not settled, note that the matter is terribly complicated, and call for more research. And these claims will sound scientific, even rather wise. Yet they give a false and dangerous impression: that nobody really knows anything.

The techniques of the tobacco industry have been widely embraced.[14] They are used today most obviously by climate change deniers, but they have spread beyond scientific questions and into politics. Robert Proctor, a historian who has spent decades studying the tobacco industry, calls modern politics “a golden age of ignorance.” Much as many smokers would like to keep smoking, many of us are fondly attached to our gut instincts on political questions. All politicians need to do is persuade us to doubt evidence that would challenge those instincts.

As Donald Trump’s former right-hand man Steve Bannon infamously told the writer Michael Lewis: “The Democrats don’t matter. The real opposition is the media. And the way to deal with them is to flood the zone with shit.”[15]

The history of another term associated with Donald Trump—“fake news”—is instructive here. Originally, it described a very specific phenomenon: websites publishing false articles in the hope of getting clicks from social media and thus advertising dollars. The iconic example was the claim that the pope endorsed Trump’s presidential bid. When Trump won, for a while there was a moral panic, serious commentators worried that gullible voters had been lured into voting for Trump because they believed these outrageous lies.

That panic was a mistake. Academic studies found that fake news was never widespread or influential; most of it was consumed by a small number of highly conservative, elderly voters who were likely Trump supporters all along. These false stories quickly became much less of a problem, as social media websites woke up to the threat.[16]

But the idea of “fake news” became a powerful one—an excuse to dismiss any inconvenient claim from any source, a modern version of the cynical aphorism about “lies, damned lies, and statistics.” Mr. Trump, with his twisted talent for turning a complex issue into a political cudgel, deployed the term to demonize regular journalists. So did many other politicians, including Theresa May, then prime minister of the UK, and her opposite number, the Labour Party leader Jeremy Corbyn.

“Fake news” resonated because it tapped into an unfortunate truth: there is plenty of slapdash journalism even in mainstream outlets, as we shall see. But there are also serious and responsible journalists who carefully source their claims, and they found themselves being tossed into the same mental trash can as the pope-endorses-Trump merchants.

I worry about a world in which many people will believe anything, but I worry far more about one in which people believe nothing beyond their own preconceptions.

—

In the spring of 1965, a US Senate committee was pondering the life-and-death matter of whether to put a health warning on packets of cigarettes. An expert witness wasn’t so sure about the scientific evidence, and so he turned to the topic of storks and babies. There was a positive correlation between the number of babies born and the number of storks in the vicinity, he explained.[17] That old story about babies being delivered by storks wasn’t true, the expert went on; of course it wasn’t. Correlation is not causation. Storks do not deliver children. But larger places have more room both for children and for storks. Similarly, just because smoking was correlated with lung cancer did not mean—not for a moment—that smoking caused cancer.

“Do you honestly think there is as casual a relationship between statistics linking smoking with disease as there is about storks?” asked the committee chair. The expert witness replied that the two “seem to me the same.”[18]

The witness’s name was Darrell Huff.

He’d been paid by the tobacco lobby to do what he did best: weave together witty examples, some statistical savvy, and a certain amount of cynicism to cast doubt on the idea that cigarettes are dangerous. He was even working on a sequel to his masterpiece—although it was never published. The sequel’s name was How to Lie with Smoking Statistics.[19]

Doubt is a powerful weapon, and statistics are a vulnerable target. That target needs defenders. Yes, it’s easy to lie with statistics—but it’s even easier to lie without them.*

And more important, without statistics it’s impossible to tell the truth—to understand the world so that we can try to change it for the better, like Richard Doll and Austin Bradford Hill. What they did took some insight and determination, but it required neither genius nor incomprehensible mathematical techniques. They counted what mattered—smokers, nonsmokers, cases of lung cancer, cases of heart disease. They counted them methodically and patiently, and, on the basis of the evidence they gathered, drew their conclusions with care. Over the years, those conclusions have saved the lives of tens of millions of people, perhaps including their own: after Hill gave up his pipe and joined Doll as a nonsmoker, both men lived into their nineties.

When we use statistics with assurance and wisdom, we see trends that would otherwise be too subtle to discern. The modern world is very big, very complex, and very, very interesting. Almost 8 billion people live here. Trillions of dollars change hands daily in our economy. In the typical human brain, there are 86 billion neurons.[20] On the internet, there are around 2 billion websites. And a new virus can spread from a single person to thousands, millions—even billions—of others. Whatever we’re trying to understand about the world, one another, and ourselves, we won’t get far without statistics—any more than we can hope to examine bones without an X-ray, bacteria without a microscope, or the heavens without a telescope.

There’s a popular story about Galileo’s telescope: that even as the father of astronomy stood accused of heresy by the Roman Catholic Church, senior cardinals would not look through the instrument he had made, proclaiming it to be a magician’s trick. Galileo said he had seen mountains on the moon? Surely the lens of the telescope was dirty. He had seen the moons of Jupiter? Pah! The moons were in the telescope itself. They refused to look.

Four centuries later, it is easy for us to laugh at the story—which, by the way, has been exaggerated over the years.[21] We shouldn’t be so self-satisfied. Many of us refuse to look at statistical evidence because we’re afraid of being tricked. We think we’re being worldly-wise by adopting the Huff approach of cynically dismissing all statistics. But we’re not. We’re admitting defeat to the populists and propagandists who want us to shrug, give up on logic and evidence, and retreat into believing whatever makes us feel good.

I want us to do something different. I want to give you the confidence to pick up the telescope of statistics and use it to scrutinize the world. I want to help you understand the logic behind statistical truths and escape the grip of the flawed logic, emotions, and cognitive biases that shape the falsehoods.

So look through the statistical telescope and gaze around. You will be amazed at how clearly you will be able to see.

RULE ONE

Search Your Feelings

Luke Skywalker: “No . . . No. That’s not true. That’s impossible!”

Darth Vader: “Search your feelings, you know it to be true!”

• The Empire Strikes Back (1980)[1]

Abraham Bredius was nobody’s fool. An art critic and collector, he was the world’s leading scholar on Dutch painters, and particularly the seventeenth-century master Johannes Vermeer. As a young man in the 1880s, Bredius had made his name by spotting works wrongly credited to Vermeer. At the age of eighty-two, in 1937, he was enjoying something of a retirement swan song. He had just published a highly respected book in which he had identified two hundred fakes or imitations of Rembrandt.[2]

It was at this moment in Bredius’s life that a charming lawyer named Gerard Boon paid a visit to his Monaco villa. Boon wanted to ask Bredius’s opinion of a newly rediscovered work, Christ at Emmaus, thought to have been painted by Vermeer himself. The exacting old man was spellbound. He sent Boon away with his verdict: Emmaus was not only a Vermeer, it was the Dutch master’s finest work.

“We have here—I am inclined to say—the masterpiece of Johannes Vermeer of Delft,” wrote Bredius in a magazine article shortly after. “Quite different from all his other paintings and yet every inch a Vermeer.”

“When this masterpiece was shown to me I had difficulty controlling my emotion,” he added, noting reverently that the work was ongerept—Dutch for “virginally pure” or “untouched.” It was an ironic choice of words: Emmaus could hardly have been more corrupt. It was a rotten fraud of a painting, stiffly applied to an old canvas just a few months before Bredius caught sight of it, and hardened with Bakelite.

Yet this crude trickery caught out not only Bredius, but the entire Dutch art world. Christ at Emmaus soon sold for 520,000 guilders to the Boijmans Museum in Rotterdam. Compared with the wages of the time, that is well over $10 million today. Bredius himself contributed to help the museum buy the picture.

Emmaus became the centerpiece of the Boijmans Museum, drawing admiring crowds and rave reviews. Several other paintings in a similar style soon emerged. Once the first forgery had been accepted as a Vermeer, it was easier to pass off these other fakes. They didn’t fool everyone, but like Emmaus they fooled the people who mattered. Critics certified them; museums exhibited them; collectors paid vast sums for them—a total of more than $100 million in today’s money. In financial terms alone, this was a monumental fraud.

But there was more. The Dutch art world revered Vermeer as one of the greatest painters who ever lived. Painting mostly in the 1660s, he had been rediscovered only in the late 1800s. Fewer than forty of his works survive. The apparent emergence of half a dozen Vermeers in just a few years was a major cultural event.

It was also an event that should have strained credulity. But it did not. Why?

Don’t look to the paintings themselves for an answer. If you compare a genuine Vermeer with the first forgery, Emmaus, it is hard to understand how anyone was fooled—let alone anyone as discerning as Abraham Bredius.

Vermeer was a true master. His most famous work is Girl with a Pearl Earring, a luminous portrait of a young woman: seductive, innocent, adoring, and anxious all at once. The painting inspired a novel, and a movie starring Scarlett Johansson as the unnamed girl. In The Milkmaid, a simple scene of domesticity is lifted by details such as the rendering of a copper pot, and a display of fresh-baked bread that looks good enough to grab out of the painting. Then there’s Woman Reading a Letter. She stands in the soft light of an unseen window. Is she, perhaps, pregnant? We see her in profile as she holds the letter close to her chest, eyes cast down as she reads. There’s a dramatic stillness about the image—we feel that she’s holding her breath as she scans the letter for news; we hold our breath, too. A masterpiece.

And Christ at Emmaus? It’s a static, awkward image by comparison. Rather than seeming to be an inferior imitation of Vermeer, it doesn’t look like a Vermeer at all. It’s not a terrible painting, but it’s not a brilliant one either. Set alongside Vermeer’s works it seems dour and clumsy. And yet it, and several others, fooled the world—and might have continued to fool the world to this day, had not the forger been caught out by a combination of recklessness and bad luck.

In May 1945, with the war in Europe at an end, two officers from the Allied Art Commission knocked on the door of 321 Keizersgracht, one of Amsterdam’s most exclusive addresses. They were met by a charismatic little man called Han van Meegeren. The young van Meegeren had enjoyed some brief success as an artist. In middle age, as his jowls had loosened and his hair had silvered, he had grown rich as an art dealer.

But perhaps he had been dealing art with the wrong people, because the officers came with a serious charge: that van Meegeren had sold Johannes Vermeer’s newly discovered masterpiece, The Woman Taken in Adultery, to a German Nazi. And not just any Nazi, but Hitler’s right-hand man, Hermann Göring.

Van Meegeren was arrested and charged with treason. He responded with furious denials, trying to bluster his way to freedom. His forceful, fast-talking manner was usually enough to get him out of a sticky situation. Not this time. A few days into his incarceration, he cracked. He confessed not to treason but to a crime that caused astonishment across the Netherlands and the art world as a whole.

“Fools!” he sneered. “You think I sold a priceless Vermeer to Göring? There was no Vermeer! I painted it myself.”[3]

Van Meegeren admitted painting not only the work that had been found in Nazi hands, but Christ at Emmaus and several other supposed Vermeers. The fraud had unraveled not because anyone spotted these flawed forgeries, but because the forger himself confessed. And why wouldn’t he? Selling an irreplaceable Vermeer masterpiece to the Nazis would have been a hanging offense, whereas selling a forgery to Hermann Göring wasn’t just forgivable, it was admirable.

But the question remains: How could a man as expert as Abraham Bredius have been fooled by so crass a forgery? And why begin a book about statistics with a tale that has nothing at all to do with numbers?

The answer to both questions is the same: when it comes to interpreting the world around us, we need to realize that our feelings can trump our expertise. When Bredius wrote, “I had difficulty controlling my emotion,” he was, alas, correct. Nobody had more skill or knowledge than Bredius, but van Meegeren understood how to turn Bredius’s skill and knowledge into a disadvantage.

Working out how van Meegeren fooled Bredius teaches us much more than a footnote in the history of art; it explains why we buy things we don’t need, fall for the wrong kind of romantic partner, and vote for politicians who betray our trust. In particular, it explains why so often we buy into statistical claims that even a moment’s thought would tell us cannot be true.

Van Meegeren wasn’t an artistic genius, but he intuitively understood something about human nature. Sometimes, we want to be fooled.

—

We’ll return to the cause of Abraham Bredius’s error in a short while. For now, it’s enough to understand that his deep knowledge of Vermeer’s paintings proved to be a liability rather than an asset. When he saw Christ at Emmaus, Bredius was undone by his emotional response. The same trap lies in wait for any of us.

The aim of this book is to help you be wiser about statistics. That means I also need to help you be wiser about yourself. All the statistical expertise in the world will not prevent your believing claims you shouldn’t believe and dismissing facts you shouldn’t dismiss. That expertise needs to be complemented by control of your own emotional reactions to the statistical claims you see.

In some cases there’s no emotional reaction to worry about. Say I tell you Mars is more than 50 million kilometers, or 30 million miles, from Earth. Very few people have a passionately held belief about that claim, so you can start asking sensible questions immediately.

For example: Is 30 million miles a long way? (Sort of. It’s more than a hundred times farther than the distance between Earth and the moon. Other planets are a lot farther away, though.) Hang on, isn’t Mars in a totally different orbit? Doesn’t that mean the distance between the Earth and Mars varies all the time? (Indeed it does. The minimum distance between the two planets is a bit more than 30 million miles, but sometimes Mars is more than 200 million miles away.) Because there is no emotional response to the claim to trip you up, you can jump straight to trying to understand and evaluate it.

It’s much more challenging when emotional reactions are involved, as we’ve seen with smokers and cancer statistics. Psychologist Ziva Kunda found the same effect in the lab, when she showed experimental subjects an article laying out the evidence that coffee or other sources of caffeine could increase the risk to women of developing breast cysts. Most people found the article pretty convincing. Women who drank a lot of coffee did not.[4]

We often find ways to dismiss evidence that we don’t like. And the opposite is true, too: when evidence seems to support our preconceptions, we are less likely to look too closely for flaws.

The more extreme the emotional reaction, the harder it is to think straight. What if your doctor told you that you had a rare form of cancer, and advised you not to look it up? What if you ignored that advice, consulted the scientific literature, and discovered that the average survival time was just eight months?

Exactly that situation confronted Stephen Jay Gould, a paleontologist and wonderful science writer, at the age of forty. “I sat stunned for about fifteen minutes . . .” he wrote in an essay that has become famous. You can well imagine his emotions. Eight months to live. Eight months to live. Eight months to live. “Then my mind started to work again, thank goodness.”[5]

Once his mind did start to work, Gould realized that his situation might not be so desperate. The eight months wasn’t an upper limit. It was the median average, which means that half of sufferers live longer than that. Some, possibly, live a great deal longer. Gould had a good chance: he was fairly young; his cancer had been spotted early; he’d get good treatment.

Gould’s doctor was being kind in trying to steer him away from the literature, and many of us will go to some lengths to avoid hearing information we suspect we might not like. In another experiment, students had a blood sample taken and were then shown a frightening presentation about the dangers of herpes; they were then told that their blood sample would be tested for the herpes virus. Herpes can’t be cured, but it can be managed, and there are precautions a person can take to prevent transmitting the virus to sexual partners—so it would be useful to know whether or not you have herpes. Nevertheless, a significant minority, one in five, not only preferred not to know whether they were infected but were willing to pay good money to have their blood sample discarded instead. They told researchers they simply didn’t want to face the anxiety.[6]

Behavioral economists call this “the ostrich effect.” For example, when stock markets are falling, people are less likely to log in to check their investment accounts online.[7] That makes no sense. If you use information about share prices to inform your investment strategy, you should be just as keen to get it in bad times as good. If you don’t, there’s little reason to log in at all—so why check your account so frequently when the market is rising?

It is not easy to master our emotions while assessing information that matters to us, not least because our emotions can lead us astray in different directions. Gould realized he hadn’t been thinking straight because of the initial shock—but how could he be sure, when he spotted those signs of hope in the statistics, that he wasn’t now in a state of denial? He couldn’t. With hindsight, he wasn’t: he lived for another twenty years, and died of an unrelated condition.

We don’t need to become emotionless processors of numerical information—just noticing our emotions and taking them into account may often be enough to improve our judgment. Rather than requiring superhuman control over our emotions, we need simply to develop good habits. Ask yourself: How does this information make me feel? Do I feel vindicated or smug? Anxious, angry, or afraid? Am I in denial, scrambling to find a reason to dismiss the claim?

I’ve tried to get better at this myself. A few years ago, I shared a graph on social media that showed a rapid increase in support for same-sex marriage. As it happens, I have strong feelings about the matter and I wanted to share the good news. Pausing just long enough to note that the graph seemed to come from a reputable newspaper, I retweeted it.

The first reply was “Tim, have you looked at the axes on that graph?” My heart sank. Five seconds looking at the graph would have told me that it was a mess, with the timescale a jumble that distorted the rate of progress. Approval for marriage equality was increasing, as the graph showed, but I should have clipped it for my “bad data visualization” file rather than eagerly sharing it with the world. My emotions had gotten the better of me.

I still make that sort of mistake—but less often, I hope.

I’ve certainly become more cautious—and more aware of the behavior when I see it in others. It was very much in evidence in the early days of the coronavirus epidemic, as helpful-seeming misinformation spread even faster than the virus itself. One viral post—circulating on Facebook and in email newsgroups—all-too-confidently explained how to distinguish between COVID-19 and a cold, reassured people that the virus was destroyed by warm weather, and incorrectly advised that ice water was to be avoided, while warm water kills any virus. The post, sometimes attributed to “my friend’s uncle,” sometimes to “Stanford hospital board” or some blameless and uninvolved pediatrician, was occasionally accurate but generally speculative and misleading. Yet people—normally sensible people—shared it again and again and again. Why? Because they wanted to help others. They felt confused, they saw apparently useful advice, and they felt impelled to share. That impulse was only human, and it was well-meaning—but it was not wise.[8]

Before I repeat any statistical claim, I first try to take note of how it makes me feel. It’s not a foolproof method against tricking myself, but it’s a habit that does little harm and is sometimes a great deal of help. Our emotions are powerful. We can’t make them vanish, nor should we want to. But we can, and should, try to notice when they are clouding our judgment.

—

In 2011, Guy Mayraz, then a behavioral economist at the University of Oxford, conducted a test of wishful thinking.[9]

Mayraz showed his experimental subjects a graph of a price rising and falling over time. These graphs were actually historical snippets from the stock market, but Mayraz told people that the graphs showed recent fluctuations in the price of wheat. He asked each person to make a forecast of where the price would move next—and offered them a reward if their forecasts came true.

But Mayraz had also divided his experiment participants into two categories. Half of them were told that they were “farmers,” who would be paid extra if wheat prices were high. The rest were “bakers,” who would earn a bonus if wheat was cheap. So the subjects might earn two separate payments: one for making an accurate forecast, and the second a windfall if the price of wheat happened to move in their direction. Yet Mayraz found that the prospect of the windfall influenced the forecast itself. The farmers hoped that the price of wheat would rise, and they also predicted that the price of wheat would rise. The bakers hoped for—and predicted—the opposite. This is wishful thinking in its purest form: letting our reasoning be swayed by our hopes.

Another example was produced by economists Linda Babcock and George Loewenstein, who ran an experiment in which participants were given evidence from a real court case about a motorbike accident.[10] They were then randomly assigned to play the role of plaintiff’s attorney (arguing that the injured motorcyclist should receive $100,000 in damages) or defense attorney (arguing that the case should be dismissed or the damages should be low).

The experimental subjects were given a financial incentive to argue their side of the case persuasively and to reach an advantageous settlement with the other side. They were also given a separate financial incentive to accurately guess what the damages the judge in the real case had actually awarded. Their predictions should have been unrelated to their role-playing, but again, their judgment was strongly influenced by what they hoped would be true.*

Psychologists call this “motivated reasoning.” Motivated reasoning is thinking through a topic with the aim, conscious or unconscious, of reaching a particular kind of conclusion. In a football game, we see the fouls committed by the other team but overlook the sins of our own side. We are more likely to notice what we want to notice.[11]

Perhaps the most striking example of this is among people who deny that the human immunodeficiency virus, HIV, causes AIDS. Some deny that HIV exists at all, but in any case HIV denialism implies rejecting the standard, and now highly effective, treatments. Some prominent believers in this idea have, tragically, doomed themselves and their children to death—but it must have been a comforting belief, particularly in the years when treatments for the condition were less effective and carried more severe side effects than they do today. One might assume that such a tragic belief would be vanishingly rare, but perhaps not. One survey of gay and bisexual men in the United States found that almost half believed HIV did not cause AIDS and more than half believed the standard treatments did more harm than good. Other surveys of people living with AIDS found the prevalence of denialist views at 15 to 20 percent. These surveys weren’t rigorous randomized samples, so I would not take the precise numbers too seriously. However, it’s clear evidence that large numbers of people reject the scientific consensus in a way that could put them in real danger.[12]

I could see wishful thinking in operation in March 2020, too, when researchers at the University of Oxford published a “tip of the iceberg” model of the pandemic. That model suggested that the coronavirus might be much more widespread but less dangerous than we thought, which had the joyful implication that the worst would soon be over. It was a minority view among epidemiologists, because the data detective work being done at that point saw little evidence that the vast majority of people had negligible symptoms. Indeed, one of the central points of the Oxford group was that we desperately needed better data to figure out the truth. That, however, was not the message that caught on. Instead, people widely shared the “good news” because it was the kind of idea we all wanted to be true.[13]

Wishful thinking isn’t the only form of motivated reasoning, but it is a common one. We believe in part because we want to. A person who is HIV-positive would find it comforting to believe that the virus does not lead to AIDS and cannot be passed to breastfeeding children. A “farmer” wants to be accurate in his forecast of wheat prices, but he also wants to make money, so his forecasts are swayed by his avarice. A political activist wants the politicians she supports to be smart and witty and incorruptible. She’ll go to some effort to ignore or dismiss evidence to the contrary.

And an art critic who loves Vermeer is motivated to conclude that the painting in front of him is not a forgery, but a masterpiece.

—

It was wishful thinking that undid Abraham Bredius. The art historian had a weak spot: his fascination with Vermeer’s religious paintings. Only two existed. He had discovered one of them himself: The Allegory of Faith. He still owned it. The other, Christ in the House of Martha and Mary, was the only Vermeer known to portray a scene from the Bible. Bredius had assessed it in 1901 and concluded quite firmly that it was not a Vermeer. Other critics disagreed, and eventually everyone reached the conclusion that Bredius had been wrong, including Bredius himself.

Stung by that experience, Bredius was determined not to repeat his mistake. He knew and loved Vermeer better than any man alive, and was always on the lookout for a chance to redeem himself by correctly identifying the next discovery of a Vermeer masterpiece.

And Bredius had become fascinated by the gap between the early, biblical Martha and Mary and Vermeer’s more characteristic works, which were painted some years later. What lurked undiscovered in that gap? Wouldn’t it be wonderful if another biblical work were found after all these years?

Bredius had another pet theory about Vermeer. The idea was that the Dutch master had, as a young man, traveled to Italy and been inspired by the religious works of the great Italian master Caravaggio. This was conjecture; not much was known about Vermeer’s life. Nobody knew if he had ever seen a Caravaggio.

Van Meegeren knew all about Bredius’s speculations. He painted Emmaus as a trap. It was a big, beautiful canvas, on a biblical theme, and—just as Bredius had argued all along—the composition was an homage to Caravaggio. Van Meegeren had planted some Vermeer-like touches in the painting, using seventeenth-century props. The bread that Christ is breaking is highlighted, just like that famous pearl earring, with thick dots of white paint called pointillés. And the paint was hard and cracked with age.

Bredius had no doubts. Why would he? Van Meegeren’s stooge, Gerard Boon, wasn’t just showing Bredius a painting: Boon was showing him evidence that he had been right all along. In the final years of his life, the old man had found the missing link at last. Bredius wanted to believe, and because he was an expert, he had no trouble summoning up reasons to support his conclusion.

Those telltale pointillés on the bread, for instance: the white dots seem a bit clumsy to the untrained eye, but they reminded Bredius of Vermeer’s highlights on that tempting loaf of bread in The Milkmaid. The fact that the composition echoed Caravaggio would have been lost on a casual viewer, but leaped off the canvas under Bredius’s gaze. He would have picked up other clues that Emmaus was the real thing. He would have noted the genuine seventeenth-century vase that van Meegeren had used as a prop. There were seventeenth-century pigments, too, or as close as possible. Van Meegeren had expertly duplicated Vermeer’s color palette. There was the canvas itself: an expert such as Bredius could spot a nineteenth- or twentieth-century forgery simply by looking at the back of the painting and noting that the canvas was too new. Van Meegeren knew this. He had painted his work on a seventeenth-century canvas, carefully scrubbed of its surface pigments but retaining the undercoat and its distinctive pattern of cracking.

And then there was the simplest test of all: Was the paint soft? The challenge for anyone who wants to forge an old master is that oil paints take half a century to dry completely. If you dip a cotton swab into some pure alcohol and gently rub the surface of an oil painting, then the cotton may come away stained with pigments. If it does, the painting is a modern fake. Only after several decades will the paint harden enough to pass this test.

Bredius had identified fakes using this method before—but the paint on Emmaus stubbornly refused to yield its pigment. This gave Bredius an excellent reason to believe that Emmaus was old, and therefore genuine. Van Meegeren had fooled him with a brilliant piece of amateur chemistry, the result of many months of experimentation. The forger had figured out a way to mix seventeenth-century oil paints with a brand-new material: phenol formaldehyde, a resin that when heated at 105°C for two hours turned into one of the first plastics, Bakelite. No wonder the paint was hard and unyielding: it was infused with industrial plastic.

Bredius had half a dozen subtle reasons to believe that Emmaus was a Vermeer. They were enough to dismiss one glaring reason to believe otherwise: that the picture doesn’t look like anything else Vermeer ever painted.

Take another look at that extraordinary statement from Abraham Bredius: “We have here—I am inclined to say—the masterpiece of Johannes Vermeer of Delft . . . quite different from all his other paintings and yet every inch a Vermeer.”

“Quite different from all his other paintings”—shouldn’t that be a warning? But the old man desperately wanted to believe that this painting was the Vermeer he’d been looking for all his life, the one that would provide the link back to Caravaggio himself. Van Meegeren set a trap into which only a true expert could stumble. Wishful thinking did the rest.

—

Abraham Bredius bears witness to the fact that experts are not immune to motivated reasoning. Under some circumstances their expertise can even become a disadvantage. The French satirist Molière once wrote, “A learned fool is more foolish than an ignorant one.” Benjamin Franklin commented, “So convenient a thing it is to be a reasonable creature, since it enables us to find or make a reason for everything one has a mind to do.”

Modern social science agrees with Molière and Franklin: people with deeper expertise are better equipped to spot deception, but if they fall into the trap of motivated reasoning, they are able to muster more reasons to believe whatever they really wish to believe.

One recent review of the evidence concluded that this tendency to evaluate evidence and test arguments in a way that’s biased toward our own preconceptions is not only common, but just as common among intelligent people. Being smart or educated is no defense.[14] In some circumstances it may even be a weakness.

One illustration of this is a study published in 2006 by two political scientists, Charles Taber and Milton Lodge. Taber and Lodge were following in the footsteps of Kari Edwards and Edward Smith, whose work on politics and doubt we encountered in the introduction. As with Edwards and Smith, they wanted to examine the way Americans reasoned about controversial political issues. The two they chose were gun control and affirmative action.

Taber and Lodge asked their experimental participants to read a number of arguments on either side and to evaluate the strength and weakness of each argument. One might hope that being asked to review these pros and cons would give people more of a shared appreciation of opposing viewpoints; instead, the new information pulled people further apart. This was because people mined the information they were given for ways to support their existing beliefs. When invited to search for more information, people would seek out data that backed their preconceived ideas. When invited to assess the strength of an opposing argument, they would spend considerable time thinking up ways to shoot it down.

This isn’t the only study to reach this sort of conclusion, but what’s particularly intriguing about Taber and Lodge’s experiment is that expertise made matters worse.* More sophisticated participants in the experiment found more material to back up their preconceptions. More surprisingly, they found less material that contradicted them—as though they were using their expertise actively to avoid uncomfortable information. They produced more arguments in favor of their own views, and picked up more flaws in the other side’s arguments. They were vastly better equipped to reach the conclusion they had wanted to reach all along.[15]

Of all the emotional responses we might have, the most politically relevant are motivated by partisanship. People with a strong political affiliation want to be on the right side of things. We see a claim, and our response is immediately shaped by whether we believe “That’s what people like me think.”

Consider this claim about climate change: “Human activity is causing the Earth’s climate to warm up, posing serious risks to our way of life.” Many of us have an emotional reaction to a claim like that; it’s not like a claim about the distance to Mars. Believing it or denying it is part of our identity. It says something about who we are, who our friends are, and the sort of world we want to live in. If I put a claim about climate change in a news headline, or in a graph designed to be shared on social media, it will attract attention and engagement not because it is true or false but because of the way people feel about it.

If you doubt this, ponder the findings of a Gallup poll conducted in 2015. It found a huge gap between how much Democrats and Republicans in the United States worried about climate change. What rational reason could there be for that? Scientific evidence is scientific evidence. Our beliefs around climate change shouldn’t skew left and right. But they do.[16]

This gap became wider the more education people had. Among those with no college education, 45 percent of Democrats and 22 percent of Republicans worried “a great deal” about climate change. Yet among those with a college education, the figures were 50 percent of Democrats and 8 percent of Republicans. A similar pattern holds if you measure scientific literacy: more scientifically literate Republicans and Democrats are further apart than those who know very little about science.[17]

If emotion didn’t come into it, surely more education and more information would help people to come to an agreement about what the truth is—or at least, the current best theory? But giving people more information seems actively to polarize them on the question of climate change. This fact alone tells us how important our emotions are. People are straining to reach the conclusion that fits with their other beliefs and values—and, like Abraham Bredius, the more they know, the more ammunition they have to reach the conclusion they hope to reach.

Psychologists call one of the processes driving this polarization “biased assimilation.” Imagine that you happen to encounter a magazine article that is discussing what we know about the effects of the death penalty. You’re interested in the topic and so you read on, encountering the following brief account of a study:

Researchers Palmer and Crandall compared murder rates in 10 pairs of neighboring states with different capital punishment laws. In 8 of the 10 pairs, murder rates were higher in the state with capital punishment. This research opposes the deterrent effect of the death penalty.

What do you think? Does that seem plausible?

If you’re opposed to the death penalty, then it probably does. But if you’re in favor of the death penalty, doubts might start to creep in—those kinds of doubts that we’ve already seen were so powerful in the case of tobacco. Was this research professionally conducted? Did they consider alternative explanations? How did they handle their data? In short, do Palmer and Crandall really know what they’re doing, or are they a pair of hacks?

Palmer and Crandall won’t be offended by your doubts. The duo do not exist. They were dreamed up by three psychologists, Charles Lord, Lee Ross, and Mark Lepper. In 1979, Lord, Ross, and Lepper conducted an experiment that was designed to explore how people thought through arguments they felt passionately about. The researchers rounded up experimental subjects with strong views in favor of, or against, the death penalty. They showed the experimental subjects summaries of two imaginary studies. One of these made-up studies demonstrated that the death penalty deterred serious crime; the other, by the fictitious researchers Palmer and Crandall, showed the opposite.[18]

As one might expect, the experimental subjects were inclined to dismiss studies that contradicted their cherished beliefs. But Lord and his colleagues discovered something more surprising: the more detail people were presented with—graphs, research methods, commentary by other fictional academics—the easier they found it to disbelieve unwelcome evidence. If doubt is the weapon, detail is the ammunition.

When we encounter evidence that we dislike, we ask ourselves, “Must I believe this?” More detail will often give us more opportunity to find holes in the argument. And when we encounter evidence that we approve of, we ask a different question: “Can I believe this?” More detail means more toeholds on to which that belief can cling.[19]

The counterintuitive result is that presenting people with a detailed and balanced account of both sides of the argument may actually push people away from the center rather than pull them in. If we already have strong opinions, then we’ll seize upon welcome evidence, but we’ll find opposing data or arguments irritating. This biased assimilation of new evidence means that the more we know, the more partisan we’re able to be on a fraught issue.

—

Maybe this sounds absurd. Don’t we all want to figure out the truth? We certainly should when it will affect us personally—and the tragic case of HIV/AIDS denialism indicates that some people will go to extraordinary lengths to reject ideas that are uncomfortable and unwelcome, even if those ideas could save their lives. Wishful thinking can be astonishingly powerful.

But often, being right doesn’t have such profound consequences. On many questions, reaching a factually incorrect conclusion causes us no harm at all. It can even help us.

To see why, ponder an issue where most people would agree that there is no objective “truth” at all: the moral difference between eating beef, eating pork, and eating dog. Which of these practices you think is right and which you think is wrong depends mostly on your culture. Few people will care to discuss the underlying logic of the matter. It’s better to fit in.

Less obviously, the same is often true of arguments where there is a correct answer. In the case of climate change, there is an objective truth even if we are unable to discern it with perfect certainty. But as you are one individual among nearly 8 billion on the planet, the environmental consequences of what you happen to think are irrelevant. With a handful of exceptions—say, if you’re the president of China—climate change is going to take its course regardless of what you say or do. From a self-centered point of view, the practical cost of being wrong is close to zero.

The social consequences of your beliefs, however, are real and immediate.

Imagine that you’re a barley farmer in Montana and that hot, dry summers are ruining your crop with increasing frequency. Climate change matters to you. And yet rural Montana is a conservative place, and the words “climate change” are politically charged. Anyway, what can you personally do about it? Here’s how one farmer, Erik Somerfeld, threads that needle:

In the field, looking at his withering crop, Somerfeld was unequivocal about the cause of his damaged crop—“climate change.” But back at the bar, with his friends, his language changed. He dropped those taboo words in favor of “erratic weather” and “drier, hotter summers”—a not-uncommon conversational tactic in farm country these days.[20]

If Somerfeld lived in Portland, Oregon, or Brighton, England, he wouldn’t need to be so circumspect at his local tavern—he’d be likely to have friends who took climate change very seriously indeed. But then those friends would quickly ostracize someone else in the social group who went around loudly claiming that climate change is a Chinese hoax.

So perhaps it is not so surprising after all to find educated Americans poles apart on the topic of climate change. Hundreds of thousands of years of human evolution have wired us to care deeply about fitting in with those around us. This helps to explain the findings of Taber and Lodge that better-informed people are actually more at risk of motivated reasoning on politically partisan topics: the more persuasively we can make the case for what our friends already believe, the more our friends will respect us.

HIV denialism shows we’re capable of being tragically wrong even in matters of life and death. But it’s far easier to lead ourselves astray when the practical consequences of being wrong are small or nonexistent, while the social consequences of being “wrong” are severe. It’s no coincidence that this describes many controversies that divide along partisan lines.

—

It’s tempting to assume that motivated reasoning is just something that happens to other people. I have political principles; you’re politically biased; he’s a fringe conspiracy theorist. But we’d be wiser to acknowledge that we all think with our hearts rather than our heads sometimes.

Kris De Meyer, a neuroscientist at King’s College London, shows his students a message describing an environmental activist’s problem with climate change denialism:

To summarize the climate deniers’ activities I think we can say that:

(1) Their efforts have been aggressive while ours have been defensive.

(2) The deniers’ activities are rather orderly—almost as if they had a plan working for them.

I think the denialist forces can be characterized as dedicated opportunists. They are quick to act and seem to be totally unprincipled in the type of information they use to attack the scientific community. There is no question, though, that we have been inept in getting our side of the story, good though it may be, across to the news media and the public.[21]

The students, all committed believers in climate change, outraged at the smokescreen laid down by the cynical and anti-scientific deniers, nod in recognition. Then De Meyer reveals the source of the text. It’s not a recent email. It’s taken, often word for word, from an infamous internal memo written by a cigarette marketing executive in 1968. The memo is complaining not about “climate deniers” but about “anti-cigarette forces,” but otherwise few changes were required. You can use the same language, the same arguments, and perhaps even have the same conviction that you’re right, whether you’re arguing (rightly) that climate change is real or (wrongly) that the cigarette–cancer link is not.

(Here’s an example of this tendency that, for personal reasons, I can’t help but be sensitive about. My left-leaning, environmentally conscious friends are justifiably critical of ad hominem attacks on climate scientists. You know the kind of thing: claims that scientists are inventing data because of their political biases or because they’re scrambling for funding from big government. In short, smearing the person rather than engaging with the evidence. Yet the same friends are happy to embrace and amplify the same kind of tactics when they’re used to attack my fellow economists: that we’re inventing data because of our political biases, or scrambling for funding from big business. I tried to point out the parallel to one thoughtful person, and got nowhere. She was completely unable to comprehend what I was talking about. I’d call this a double standard, but that would be unfair—it would suggest that it was deliberate. It’s not. It’s an unconscious bias that’s easy to see in others and very hard to see in ourselves.*)

Our emotional reaction to a statistical or scientific claim isn’t a side issue. Our emotions can, and often do, shape our beliefs more than any logic. We are capable of persuading ourselves to believe strange things, and to doubt solid evidence, in service of our political partisanship, our desire to keep drinking coffee, our unwillingness to face up to the reality of our HIV diagnosis, or any other cause that invokes an emotional response.

But we shouldn’t despair. We can learn to control our emotions—that is part of the process of growing up. The first simple step is to notice those emotions. When you see a statistical claim, pay attention to your own reaction. If you feel outrage, triumph, denial, pause for a moment. Then reflect. You don’t need to be an emotionless robot, but you could and should think as well as feel.

Most of us do not actively wish to delude ourselves, even when that might be socially advantageous. We have motives to reach certain conclusions, but facts matter, too. Lots of people would like to be movie stars, billionaires, or immune to hangovers, but very few people believe that they actually are. Wishful thinking has limits. The more we get into the habit of counting to three and noticing our knee-jerk reactions, the closer to the truth we are likely to get.

For example, one survey, conducted by a team of academics, found that most people were perfectly able to distinguish serious journalism from fake news, and also agreed that it was important to amplify the truth, not lies. Yet the same people would happily share headlines such as “Over 500 ‘Migrant Caravaners’ Arrested with Suicide Vests” because the moment at which they clicked “share,” they weren’t stopping to think. They weren’t thinking, “Is this true?” and they weren’t thinking, “Do I think the truth is important?” Instead, as they skimmed the internet in that state of constant distraction that we all recognize, they were carried away with their emotions and their partisanship. The good news is that simply pausing for a moment to reflect was all it took to filter out a lot of the misinformation. It doesn’t take much; we can all do it. All we need to do is acquire the habit of stopping to think.[22]

Another study found that people who were best able to distinguish real from fake news were also the people who scored highly on what is called a cognitive reflection test.[23] These tests—created by Shane Frederick, a behavioral economist, and made famous by Daniel Kahneman’s book Thinking, Fast and Slow—ask questions such as:

A bat and ball cost $1.10, and the bat costs a dollar more than the ball. How much does the ball cost?

and:

A lake contains a patch of lily pads which doubles in size each day. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?*

Many people get the answers to these questions wrong the first time they hear them, but what’s required to reach the correct solution isn’t intelligence or mathematical training, but pausing for a moment to double-check your gut reaction. Shane Frederick points out that noticing your initial error is usually all that’s necessary to solve the problem.[24]

The cognitive reflection questions invite us to leap to the wrong conclusion without thinking. But so, too, do inflammatory memes or tub-thumping speeches. That’s why we need to be calm. And that is also why so much persuasion is designed to arouse us—our lust, our desire, our sympathy, or our anger. When was the last time Donald Trump, or for that matter Greenpeace, tweeted something designed to make you pause in calm reflection? Today’s persuaders don’t want you to stop and think. They want you to hurry up and feel.

Don’t be rushed.

—

Han van Meegeren had been arrested almost immediately after German occupation ended. He should have been prosecuted and punished for collaboration with the Nazis.

The wily forger had prospered mightily under Nazi occupation. He owned several mansions. While Amsterdam starved during the war, he hosted regular orgies at which prostitutes helped themselves to fistfuls of jewels. If he wasn’t actually a Nazi himself, he went to extraordinary lengths to behave like one. He was friends with Nazis, and he bent over backward to celebrate Nazi ideology.

Van Meegeren illustrated and published a lavishly evil book called Teekeningen 1, full of grotesque anti-Semitic poetry and illustrations, using the Nazi iconography and colors. He spared no expense in the printing of the book, and no wonder, given who he imagined might read it. A copy was hand-delivered to Adolf Hitler, with a handwritten dedication in artist’s charcoal: “To my beloved Führer in grateful tribute—Han van Meegeren.”

It was found in Hitler’s library.

To understand what happened next, we need to understand emotion rather than logic. The Dutch were disillusioned with themselves after five years of German occupation. Anne Frank was just the most famous of the huge number of Jews to have been deported from the Netherlands and murdered, but it is less well known that a far higher proportion of Dutch Jews were deported than those living in France or Belgium.[25] Van Meegeren, of course, was yet another collaborator. But in the wake of the war, the Dutch had become tired of parading such men through their courts, month after month. They desperately wanted a more inspiring story—just as Abraham Bredius desperately wanted to find a Caravaggesque Vermeer. Yet again, van Meegeren produced what was wanted: this time, a lighthearted tale of boldness and trickery in which a Dutchman had struck back against the Nazis.

The men responsible for prosecuting van Meegeren soon became his unwitting accomplices. They arranged an absurd publicity stunt where he “proved” that he was a forger rather than a traitor by painting a picture in the style of Emmaus. One breathless headline reported: “He Paints for His Life.” Newspapers in the Netherlands and around the world couldn’t tear their gaze away from the great showman.

Then came the trial, a media circus in which the charismatic van Meegeren was the ringmaster. He spun his story: that he had only forged the art to prove his worth as an artist, and to unmask the art experts as fools. When the judge reminded him that he had sold the fakes for high prices, he replied, “Had I sold them for low prices, it would have been obvious they were fake.” The courtroom laughed; van Meegeren had them all spellbound. A man who should have been viewed as a traitor reshaped his reputation into that of a patriot, even a hero. He manipulated the emotions of the Dutch people, as he had manipulated the emotions of Abraham Bredius before the war.

It wasn’t just the Dutch who swallowed the story of the man who played Göring for a fool. Van Meegeren found plenty of people who were delighted to play up the deliciousness of the story. Early biographers of van Meegeren made him out to be a misunderstood trickster, hurt by the unjust rejections of his own art, but happy to outsmart his country’s occupiers. One oft-reported story is that Göring, awaiting trial in Nuremberg, when told that he had been duped by van Meegeren, “looked as if for the first time he had discovered there was evil in the world.” When you hear that anecdote it’s almost impossible to resist repeating it. But like the pointillés on the bread, it’s a telling detail that is just as false.

If only Hitler’s personally inscribed copy of Teekeningen 1 had been discovered before van Meegeren’s trial, the story of the daring little forger would have dissolved. The truth about van Meegeren would have been obvious. Or would it?

The discomfiting truth about Teekeningen 1 is that the dedicated copy in Hitler’s library had been found almost immediately. De Waarheid, a Dutch resistance newspaper, had announced the discovery on July 11, 1945. It just didn’t matter; nobody wanted to know. Van Meegeren waved the truth away, claiming that he had signed hundreds of copies of the book and the dedication must have been added by someone else. In a modern setting he might have dismissed the newspaper report as fake news.

It was a ludicrous excuse, but van Meegeren had managed to hypnotize his prosecutors just as he had hypnotized Bredius, by distracting them with interesting details and selling them a story they wanted to believe.

In his closing statement to the court, he claimed again that he hadn’t done it for the money, which had brought him nothing but trouble. It was a bold claim: we should remember that while wartime Amsterdam went hungry, van Meegeren liked to accessorize his mansions with prostitutes, jewels, and prostitutes draped with jewels. No matter: the newspapers and the public lapped up his story.

After being found guilty of forgery, Van Meegeren was cheered as he left the courtroom. He had pulled off an even more audacious con—a fascist and a fraud successfully presented himself as a cheeky hero of the Dutch people. Abraham Bredius desperately wanted a Vermeer. The Dutch public desperately wanted symbols of resistance to the Nazis. Han van Meegeren knew how to give people what they wanted.

Before serving a day of his sentence, van Meegeren died, on December 30, 1947, of a heart attack. An opinion poll conducted a few weeks earlier had found him to be (except for the prime minister) the most popular man in the country.

—

If wishful thinking can turn a rotten fake into a Vermeer, or a sleazy Nazi into a national hero, then it can turn a dubious statistic into solid evidence, and solid evidence into fake news. But it doesn’t have to. There is hope. We’re about to go on a journey of discovery, finding out how numbers can make the world add up. The first step, then, is to stop and think when we’re being presented with a new piece of information, to examine our emotions, and to notice if we’re straining to reach a particular conclusion.

When we encounter a statistical claim about the world and are thinking of sharing it on social media or typing a furious rebuttal, we should instead ask ourselves, “How does this make me feel?”*

We should do this not just for our own sake, but as a social duty. We’ve seen how powerful social pressure can be in influencing what we believe and how we think. When we slow down, control our emotions and our desire to signal partisan affiliation, and commit ourselves to calmly weighing the facts, we’re not just thinking more clearly—we are also modeling clear thinking for others. It is possible to take a stand not as a member of a political tribe but as someone who is willing to reflect and reason in a fair-minded manner. I want to set that sort of example. I hope that you do, too.

Van Meegeren understood all too well that how we feel shapes what we think. Yes, expertise and technical knowledge matter, but the technical side of dealing with numbers will come in the chapters that follow. If we don’t master our emotions, whether they are telling us to doubt or telling us to believe, we’re in danger of fooling ourselves.

RULE TWO

Ponder Your Personal Experience

In a bird’s-eye view you tend to survey everything . . . In a worm’s-eye view you don’t have that advantage of looking at everything. You just see whatever is close to you.

• Muhammad Yunus[1]

As I settled into presenting More or Less, I felt I had a dream gig. Debunking numerical nonsense in the news was fun, and by looking through the statistical telescope I was constantly seeing new and interesting things. There was, however, a snag: every time I traveled to the BBC studios to record the program, I felt that my personal experience was contradicting some credible-seeming statistics.

Let me explain. The commute wasn’t the world’s most glamorous journey. To get to White City in West London from Hackney in East London, I’d scurry across a busy road, hop onto a busy double-decker bus, and watch the traffic as we moved slowly toward Bethnal Green, the underground station. If the bus had been busy, the tube train was busier. It made a can of sardines look roomy. I’d join a crowd of hopeful passengers on the platform, waiting for a Central Line train to arrive with enough space to squeeze on. That was by no means a certainty. We’d often have to wait for the second or third train before being able to wriggle between the less-than-delighted passengers who’d ridden in from farther east. Getting a seat was out of the question.

It was this experience that challenged my view that numbers make the world add up, because when I looked at the statistics about how busy London’s public transport actually was, they flatly contradicted the evidence of my own eyes—and on warmer, sweatier days, my own nose. Those statistics showed that the average occupancy of a London bus was around twelve people, a tiny number compared with the sixty-two seats available on the double-decker bus I rode every morning.[2] That felt completely wrong. Some days I felt there were more than twelve people within arm’s reach, let alone on the bus.

The tube occupancy rates made even less sense. According to Transport for London, the “crush capacity” of one of those tube trains is more than a thousand people.[3] But the average occupancy? Less than 130.[4] What? You could lose 130 people on a Central Line tube train. You could squeeze them onto a single carriage and leave the other seven completely empty. And that’s not the occupancy at quiet moments—it’s the average. Was I really supposed to believe that these statistics—twelve people on a bus, 130 people on a train—reflected reality? Surely not, not when every single time I took a trip to work I could not only barely get onto the train, I would sometimes struggle to get onto the platform. The trains must be busier than the statistics showed.

In the studio, I was singing the praises of statistical thinking. But on the way to the studio, my everyday experience told me that these particular statistics must be wrong.

The contradiction between what we see with our own eyes and what the statistics claim can be very real. In the previous chapter we discovered that it is important not to be fooled by our personal feelings. As I’m a self-confessed data detective, you might expect me to say the same about our personal experiences, too. After all, who are you going to believe? A trusty spreadsheet, or your own lying eyes?

The truth is more complicated. Our personal experiences should not be dismissed along with our feelings, at least not without further thought. Sometimes the statistics give us a vastly better way to understand the world; sometimes they mislead us. We need to be wise enough to figure out when the statistics are in conflict with everyday experience—and in those cases, which to believe.

So what should we do when the numbers tell one story and day-to-day life tells us something different? That’s what this chapter is about.

—

We might start by being curious about where the statistics come from. In the case of my commute, the numbers are published by Transport for London (TfL), the government organization that oversees London’s roads and public transport. But how do the fine folk of TfL know for sure how many people are on a bus or a tube train? It’s a good question, and the answer is: They don’t. They can, however, make a good guess. Years ago, estimates were based on paper surveys, carried out by researchers standing at bus stops or in tube stations with a clipboard or handing out questionnaires. Clearly this was a ponderous method, although it is unlikely that it introduced enough errors to explain the huge disparity between my experience and the official occupancy figures.

In any case, in the age of contactless payments it’s much easier to estimate passenger numbers. The vast majority of bus journeys are made by people tapping an identifiable contactless chip on a bank card, a TfL Oyster card, or a smartphone. The data scientists at TfL can see where and when these devices are being used. They still have to make an educated guess as to when you get off the bus, but this is often possible—for example, they might see you make the return journey from the same area later. Or they might see that you had used your card on a connecting service: whenever I tapped into the tube network at Bethnal Green, one minute after the bus I’d been riding on arrived in the area, TfL could conclude with confidence that I’d been on the bus until the stop at Bethnal Green, but no farther.

On the London Underground, people tap in and out, but TfL still does not know what route commuters took across the network, which often offers several plausible alternatives. TfL thus still doesn’t know how busy particular trains are. Again, they can make an educated guess, using occasional paper-based surveys to supplement their judgment as to how passengers are choosing to get around.

The estimates will soon be more accurate yet. On July 8, 2019, TfL switched on a system to use Wi-Fi networks to measure how crowded different parts of the London Underground are. The more phones are trying to connect to Wi-Fi, the busier the pinch point in a particular station. This system promises to let TfL spot overcrowding and other problems in real time. (I spoke to the data team at TfL the day after this system was switched on. They were adorably excited.[5])

The statistics, then, are at least plausible. We can’t simply dismiss them as mistaken.

The next step is to look for reasons why our personal experience might be so different. In the case of my commute, the obvious starting point is that I was traveling at a busy time of day, on one of the busiest sections of the tube network. No wonder it was crowded.

But this particular rabbit hole goes a little deeper. It’s perfectly possible that most trains aren’t crowded, and yet most people travel on crowded trains. For an extreme illustration, imagine a hypothetical train line with ten trains a day. One rush-hour train has a thousand people crammed onto it. All the other trains carry no passengers at all. What’s the average occupancy of these trains? A hundred people—not far off the true figure on the London Underground. But what is the experience of the typical passenger in this scenario? Every single person rode on a crowded train.

The real situation on the London Underground isn’t as extreme. There aren’t many completely empty trains, but trains do sometimes run with very few passengers on them, particularly when they’re running counter to the flow of commuters. Whenever they do, very few passengers will be around to witness it. Those statistics are telling the truth—but not the whole truth.

Of course, there are alternative ways to gauge the problem of overcrowding. Rather than measure the occupancy of the average train, you could measure the situation faced by the average passenger: Out of a hundred passenger journeys, how many are on overcrowded trains? That would be a better way to measure the passenger experience—and indeed TfL is now refocusing its data collection and reporting to produce statistics that reflect the situation not of the trains, but of the passengers.

Yet there’s no single objective measure of how busy the public transport network is. As a passenger, it seems to me that all the buses I’m on are well used. But TfL’s statistics show, truthfully, that many buses are driving around largely empty. This is because buses don’t just appear in the busiest areas by magic; when they reach the end of the route they have to turn around and go back again. TfL cares about the low average occupancy of buses because those buses cost money, take up space on the roads, and emit pollution. The average occupancy is therefore an important metric for them.

In short, my own eyes told me something important and true about London’s transport network. But the statistics told me something else, something equally important and equally true—and something I couldn’t have known in any other way. Sometimes personal experience tells us one thing, the statistics tell us something quite different, and both are true.

—

That’s not always the case, of course. Think back to the discovery that heavy cigarette smoking increased the risk of lung cancer by a factor of sixteen. Many people would have found reason from their personal experience to be skeptical of this finding. Perhaps your chain-smoking nonagenarian grandma is as fit as a fiddle, whereas the only person you know who died from lung cancer is your next-door neighbor’s uncle and he never smoked a cigarette in his life.

On the face of it, this seems no different from the experience of my daily commute appearing to contradict TfL’s statistics. But on closer inspection, in this case we do find reason to discard our personal experience and trust the statistical view. Though a factor of sixteen is hardly a small effect, lung cancer is itself scarce enough to confuse our intuition. The world is full of patterns that are too subtle or too rare to detect by eyeballing them, and a pattern doesn’t need to be very subtle or rare to be hard to spot without a statistical lens.

This is true of many medical conditions and treatments. When we feel bad—anything from a headache to depression, a sore knee to an unsightly spot—we seek solutions. My wife recently suffered from a sharp pain in her shoulder whenever she raised her arm; it was bad enough to make it hard for her to get dressed or reach something on a high shelf. After a while, she went to a physiotherapist, who diagnosed the problem and prescribed some uncomfortable exercises, which my wife diligently performed every day. After a few weeks, she told me, “I think my shoulder is getting better.”

“Wow—looks like the physiotherapy worked!” I said.

“Maybe,” said my wife, who can spot me setting a statistical trap a mile off. “Or maybe it would have got better by itself anyway.”

Indeed. From my wife’s point of view, it didn’t really matter. What she wanted was for her shoulder to heal, and the evidence of her own senses was the only relevant yardstick. But for the question of whether the exercises had caused the recovery, her personal experience wasn’t much use—and from the point of view not of my wife but of future shoulder-pain sufferers, it’s the question of causation that matters. We need to know whether those exercises tend to help, or whether there might be a better approach.

The same is true of any other treatment for any other problem, whether it’s diet, therapy, exercise, antibiotics, or painkillers: it’s nice if we feel better, but future generations need to know whether we feel better because of the steps we’ve taken, or whether they were empty rituals that did no good, cost money, wasted time, and produced unwelcome side effects. For this reason, we rely on randomized trials of any treatment, ideally compared against the best available treatment, or against a fake treatment called a placebo. It’s not that our personal experience is irrelevant, it’s that it can’t give us the information we need to help those who come after us.

When personal experience and statistics seem to be in conflict, a closer look at the situation may reveal particular reasons why personal experience is likely to be an unreliable guide. Consider the idea that the vaccination against measles, mumps, and rubella (MMR) increases a child’s risk of autism. It doesn’t, but fewer than half of us are convinced of that.[6]

We can say with confidence that there is no such link thanks to the statistical perspective. Since autism is not common, we need to compare the experiences of many thousands of children who have received the vaccination and those who have not. One major study, in Denmark, did exactly that. It followed 650,000 children. Most of them received an MMR vaccine at the age of fifteen months and a follow-up at four years, but about 30,000 did not. About 1 percent of children were then diagnosed with autism, and that was true of both the vaccinated and the unvaccinated children. (The unvaccinated children, of course, were at higher risk of contracting these dangerous diseases.)[7]

So why do many people remain skeptical? Part of the answer is a sad history of reckless publishing around the issue. But in part the doubts persist because many people have heard of children whose autism was diagnosed soon after an MMR vaccine, and whose parents think the vaccine was to blame. Imagine taking your child for the vaccine and soon afterward receiving a diagnosis of autism. Would you connect the two? It would be hard not to wonder.

In fact, the prevalence of such anecdotes is not surprising because autism tends to be diagnosed at one of two ages: early signs of the condition are observable by pediatric nurses at around the age of fifteen months; if not picked up then, diagnosis often follows a child’s starting school.[8] And the two doses of the MMR vaccine are routinely given close to these ages. When we find a convincing explanation for why our personal experience sits uneasily with the statistical view, it should reassure us to set aside our doubts and trust the numbers.

A less fraught example is our relationship with television and other media. Many people on television are richer than you and me. Almost by definition, they are more famous than you and me. It’s very likely that they are better-looking than you and me; they are certainly better-looking than me (I am on the radio for a reason). When we reflect on how attractive, famous, and rich the typical person is, we can’t help but have our assessment skewed by the fact that many of the people we know, we know through the media; they are attractive, famous, and rich. Even if, on reflection, we realize that TV personalities aren’t a random sample of the global population, it’s hard to shake the feeling that they are.

Psychologists have a name for our tendency to confuse our own perspective with something more universal: it’s called “naive realism,” the sense that we are seeing reality as it truly is, without filters or errors.[9] Naive realism can lead us badly astray when we confuse our personal perspective on the world with some universal truth. We are surprised when an election goes against us: Everyone in our social circle agreed with us, so why did the nation vote otherwise? Opinion polls don’t always get it right, but I can assure you they have a better track record of predicting elections than simply talking to your friends.

Naive realism is a powerful illusion. Consider the findings of a survey from the opinion pollster Ipsos Mori. Mori asked nearly 30,000 people across 38 countries about a range of social issues, finding them—and, presumably, most of the rest of us—badly out of step with what credible statistics showed:[10]

(a) We’re wrong about the murder rate. We think it’s been rising since the year 2000. In most of the countries surveyed, it’s been falling.

(b) We think deaths from terrorism have been higher in the past fifteen years than in the fifteen years before; they’re down.

(c) We think that 28 percent of prisoners are immigrants. Ipsos Mori reckons the true rate across the countries surveyed is 15 percent.

(d) We think that 20 percent of teenage girls give birth each year. This number strains biological credibility when you think about it. An eighteen-year-old has been a teenager for six years, so if each year she has a 20 percent chance of having a baby, most eighteen-year-olds are mothers. (Those who aren’t are balanced by the eighteen-year-olds who are mothers several times over.) Look around; is that really true? The correct figure, says Ipsos Mori, is that 2 percent of teenage girls give birth each year.*

(e) We think that 34 percent of people have diabetes; the true figure is 8 percent.

(f) We think that 75 percent of people have a Facebook account. The correct figure at the time of asking, 2017, was 46 percent.

Why are our perceptions of the world so mistaken? It’s hard to be sure, but a plausible first guess is that we’re getting our impressions from the media. It’s not that a reputable newspaper or TV channel would actually give us the wrong data—although it has been known. The problem is that the news carries tales of lottery wins and fairy-tale romances, terrorist atrocities and gruesome assaults by strangers, and of course the latest trends, which are often not nearly as popular as they seem. None of these stories reflect everyday life; all of them are viscerally memorable and seem to take place in our living rooms. We form our impressions accordingly.

As the great psychologist Daniel Kahneman explained in Thinking, Fast and Slow: “When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.” Rather than asking, “Are terrorists likely to kill me?” we ask ourselves, “Have I recently seen a news report about terrorism?” Instead of saying, “Out of all the teenage girls I know, how many are already mothers?” we say, “Can I think of a recent example of a news story about teenage pregnancy?”

These news reports are data, in a way. They’re just not representative data. But they certainly influence our views of the world. To adapt Kahneman’s terminology, they’re “fast statistics”—immediate, intuitive, visceral, and powerful. “Slow statistics,” those based on a thoughtful gathering of unbiased information, aren’t the ones that tend to leap into our minds. But as we shall see, there are ways to consume more of the slow stuff and have a more balanced diet of information as a result.

—

So far we’ve seen cases in which the ponderous-and-careful slow statistics are more trustworthy than the quick-and-dirty fast statistics, and situations in which both give us a useful angle on the world. But are there also cases where we should trust our personal impressions more than the data?

Yes. There are certain things that we cannot learn from a spreadsheet.

Consider Jerry Z. Muller’s book The Tyranny of Metrics. It’s 220 pages long. The average chapter is 10.18 pages long and contains 17.76 endnotes. There are four cover endorsements and the book weighs 421 grams. But of course none of these numbers tells us what we want to know—which is what does the book say, and should we take it seriously? To understand the book you will need to read it, or trust the opinion of someone who has.

Jerry Muller takes aim at the problem with a certain kind of slow statistics—those used as management metrics or performance targets. Statistical metrics can show us facts and trends that would be impossible to see in any other way, but often they’re used as a substitute for relevant experience, by managers or politicians without specific expertise or a close-up view. For example, if a group of doctors collect and analyze data on clinical outcomes, they are likely to learn something together that helps them to do their jobs. But if the doctors’ bosses then decide to tie bonuses or professional advancement to improving these numbers, unintended consequences will predictably occur. For example, several studies have found evidence of cardiac surgeons refusing to operate on the sickest patients for fear of lowering their reported success rates.[11]

In my book Messy, I spent a chapter discussing similar examples. There was the time the UK government collected data on how many days people had to wait for an appointment when they called their doctor, which is a useful thing to know. But then the government set a target to reduce the average waiting time. Doctors logically responded by refusing to take any advance bookings at all; patients had to phone up every morning and hope they happened to be among the first to get through. Waiting times became, by definition, always less than a day.

What happened when a widely consulted ranking of US colleges, by U.S. News & World Report, rewarded more selective institutions? Oversubscribed universities scrambled to attract fresh applicants they could reject, and thereby appear to be more selective.

Then there is the notorious obsession with the “body count” metric, which was embraced by US defense secretary Robert McNamara during the Vietnam War. The more of the enemy you kill, reasoned McNamara, the closer you are to winning. This was always a dubious idea, but the body count quickly became an informal metric for ranking units and handing out promotions, and was therefore often exaggerated. And since it was sometimes easier to count enemies who were already dead than to kill anyone new, counting bodies became a military objective in itself. It was risky, and it was useless, but it responded to the skewed incentive McNamara had set.

This episode shows that statistics aren’t always worth gathering—but you can appreciate why McNamara wanted them. He was trying to understand and control a distant situation, one he had no experience of as a soldier. A few years ago I interviewed General H. R. McMaster, an expert on the mistakes made in Vietnam. He told me that the army used to believe that “situational understanding could be delivered on a computer screen.”

It could not. Sometimes you have to be there to understand—especially when a situation is fast-moving or contains soft, hard-to-quantify details, as is typically the case on the battlefield. The Nobel laureate economist Friedrich Hayek had a phrase for the kind of awareness that is hard to capture in metrics and maps: the “knowledge of the particular circumstances of time and place.”

Social scientists have long understood that statistical metrics are at their most pernicious when they are being used to control the world, rather than try to understand it. Economists tend to cite their colleague Charles Goodhart, who wrote in 1975: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”[12] (Or, more pithily: “When a measure becomes a target, it ceases to be a good measure.”) Psychologists turn to Donald T. Campbell, who around the same time explained: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”[13]

Goodhart and Campbell were onto the same basic problem: a statistical metric may be a pretty decent proxy for something that really matters, but it is almost always a proxy rather than the real thing. Once you start using that proxy as a target to be improved, or a metric to control others at a distance, it will be distorted, faked, or undermined. The value of the measure will evaporate.

—

In 2018, I visited China with my family. The trip taught me that I shouldn’t need to favor either fast or slow statistics; the deepest understanding comes from melding them together.

The slow statistics tell a familiar story—familiar, at least, to economics geeks like me. Real income per person in China has increased tenfold since 1990. Since the early 1980s, the number of extremely poor people there has fallen by more than three-quarters of a billion—well over half the entire population of the country. China consumed more cement in a recent three-year period than the United States used in the entire twentieth century. On paper, it is the most dramatic explosion of economic activity in human history.

Yet seeing it with your own eyes is another experience entirely. Nothing in the statistics truly prepared me for a journey across Guangdong, the southern province of China that has been at the forefront of this growth. We started at Hong Kong—the ultimate high-rise city—and walked into its mainland twin, Shenzhen. Then in the shadow of the Ping An skyscraper, which dwarfs the Empire State Building, we caught a bullet train across the province.

Where London’s tower blocks often stand alone or in groups of two or three, Shenzhen will have a cluster of a dozen identical monoliths, crammed with apartments, shoulder to shoulder. Next to that cluster, another dozen of a different design. Then another, and another. Here and there, in the distance across the haze, would be a Manhattanesque cluster of larger skyscrapers. The towers marched on and on, all the way (or so it seemed to me) to the city of Guangzhou—forty-five minutes or so of high-speed travel through an infinite vista of concrete.

We ended the day much deeper into China, in the picture-postcard landscape of Yangshuo. But despite the idyllic surroundings, I couldn’t sleep. The endless tower blocks scrolled through my mind. What if we had lost our six-year-old son in the middle of Guangdong? And my sleepless anxieties flitted back and forth between my family and the world. So many people. So much concrete. How could the planet possibly survive this?

Of course, there was nothing in this experience to contradict the economic data; the two perspectives on China’s growth were perfectly complementary. But they felt very different. The slow statistics required me to reflect and calculate, taking some effort to process the numbers and follow the logic of what they implied for modern China. The train journey delivered fast statistics instead. It tapped into a different and more intuitive way of thinking, as I swiftly and automatically formed my impressions, compared Guangzhou with the cities I knew back home, and anxiously sensed the danger to those I love.*

Both ways of understanding the world have their own advantages, and their own traps. Muhammad Yunus, an economist, microfinance pioneer, and winner of the Nobel Peace Prize, has contrasted the “worm’s-eye view” of personal experience with the “bird’s-eye view” that statistics can provide. The worm and the bird see the world very differently, and Professor Yunus is right to emphasize the advantage of seeing it up close.

But birds see a lot, too. Professor Yunus, paying close attention to the lives of poor women around him in Bangladesh, saw an opportunity to improve their lives by giving them access to less expensive loans, unleashing a generation of microentrepreneurs. But that up-close intuition needs to be cross-checked with some statistical rigor. The microcredit schemes that Yunus did so much to popularize have now been examined more thoroughly, using randomized trials in which a group of otherwise similar people applying for small loans are either approved or rejected at random. (This is like a clinical trial in which some patients get a new drug while others get a placebo.) These experiments tend to find that the benefits of receiving a small loan are quite modest, and temporary. Applying the same rigorous test to other approaches—for example, giving microentrepreneurs small cash payments along with advice from a mentor—finds that the cash-and-mentor scheme is more likely to boost the income from these tiny businesses than providing loans would.[14]

Statistical evidence can feel dry and thin. It doesn’t touch us in the same memorable and instinctive way as our personal experience. Yet our personal experience is limited. My trip to China took in tourist spots, airports, and high-speed rail links. It would be a serious mistake to believe I saw everything that mattered.

There is no easy answer to the balance between the bird’s-eye view and the worm’s-eye view, between the broad and rigorous but dry insight we get from the numbers and the rich but parochial lessons we learn from experience. We must simply keep reminding ourselves what we’re learning and what we might be missing. In statistics, as elsewhere, hard logic and personal impressions work best when they reinforce and correct each other. Ideally we’ll find a way to combine the best of both.

—

One effort to do that has been developed by Anna Rosling Rönnlund of Gapminder, a Swedish foundation that fights misconceptions about global development. She aims to close the gap between fast and slow statistics—between the worm’s-eye view and the bird’s-eye view—using an ingenious website, Dollar Street.

On Dollar Street you can compare the life of the Butoyi family in Makamba, Burundi, with that of the Bi family from Yunnan, China. Imelda Butoyi is a farmer. She and her four children get by on $27 a month. Bi Hua and Yue Hen are both entrepreneurs. Their family enjoys an income of $10,000 a month. It’s no surprise that life on $27 a month is very different from life on $10,000 a month. But the numbers alone don’t convey the difference in a way that we can intuitively feel, or compare with our own lives.

Dollar Street attempts to fix that, as far as is possible through the medium of a computer screen, by presenting short films and thousands of photographs of different rooms and everyday objects—a cooking stove; a source of light; a toy; somewhere to store salt; a phone; a bed. In each home about 150 photographs are taken of these everyday places and things—if they exist—and they’re portrayed in the same way as far as is possible. The images speak with great clarity.

The photographs of Imelda Butoyi’s home give a much more vivid impression than the precise yet thin statistic that she makes $27 a month. The house has mud walls, and a roof made of straw and mud. Light comes from an open fire. The toilet is a plank over a hole in the ground outside. The floor is packed earth. The children’s toys? There are just a couple of picture books.

The Bi family home, in contrast, boasts a modern shower, a flush lavatory, a fancy hi-fi, and a flat-screen TV. Their car is out front. The photographs show everything clearly, including the fact that the kitchen is surprisingly cramped, with just a couple of electric burners for cooking.

“We can use photos as data,” says Rosling Rönnlund.[15] What makes them useful data rather than random and potentially misleading is that they’re sortable, comparable, and connected to the numbers. The site allows you to filter so that you see only photographs of low-, middle-, or high-income households. Or only photographs from a particular country. Or only photographs of a particular item—such as toothpaste or toys.

It’s easy, for example, to look at all the images of cooking from very poor households and see that the standard method around the world is an iron pot hanging over an open fire. Wealthier households all use push-button appliances delivering controllable electricity or gas. Regardless of where you live, if you’re poor you’re likely to sleep on the floor in the same room as other family members. If you’re rich you’ll have privacy and a comfortable bed. Much of what we think of as cultural differences turn out to be differences in income.

“Numbers will never tell the full story of what life on Earth is all about,” wrote Hans Rosling, despite being the world’s most famous statistical guru. (Hans was Anna Rosling Rönnlund’s father-in-law.) Hans was right, of course. Numbers will never tell the full story—which is why, as a doctor and academic, he traveled so widely, and why he so expertly wove stories to go alongside his statistical evidence. But the stories the numbers do tell matter.

What I love about Dollar Street is that it successfully combines statistics, fast and slow—the worm’s-eye view and the bird’s-eye view. It shows us everyday images that we instinctively understand and remember. We empathize with people all around the world. But we do so in a clear statistical context—one that can show us life at $27 a month, or $500 a month, or $10,000 a month, and can make it clear how many people live in each situation.

If we don’t understand the statistics, we’re likely to be badly mistaken about the way the world is. It is all too easy to convince ourselves that whatever we’ve seen with our own eyes is the whole truth; it isn’t. Understanding causation is tough even with good statistics, but hopeless without them.

And yet, if we understand only the statistics, we understand little. We need to be curious about the world that we see, hear, touch, and smell, as well as the world we can examine through a spreadsheet.

My second piece of advice, then, is to try to take both perspectives—the worm’s-eye view as well as the bird’s-eye view. They will usually show you something different, and they will sometimes pose a puzzle: How could both views be true? That should be the beginning of an investigation. Sometimes the statistics will be misleading, sometimes it will be our own eyes that deceive us, and sometimes the apparent contradiction can be resolved once we get a handle on what is happening. Often that will require us to ask a few smart questions—including the question I’ll introduce in the next chapter.

RULE THREE

Avoid Premature Enumeration

Once you do know what the question actually is, you’ll know what the answer means.

• Deep Thought (a supercomputer in Douglas Adams’s Hitchhiker’s Guide to the Galaxy)

It was a vital question. Across the UK, mortality rates for newborn babies varied substantially for no obvious reason. Could doctors and nurses be doing anything different to save these children? Clinicians were dispatched to hospitals with better performances, instructed to think about the lessons that could be learned and to contemplate reconfiguring their own maternity services from the ground up.

But Dr. Lucy Smith of the University of Leicester had a nagging doubt.[1] So she looked in detail at the data from two hospital groups, one in the English Midlands and one in London. The hospitals served very similar communities, and yet the death rates of newborns were noticeably lower in London. Were the London hospitals really doing something different in their clinics, or labor wards, or neonatal intensive care units?

No, found Dr. Smith. The explanation for the disparity in mortality rates was quite different.

When a pregnancy ends at, say, twelve or thirteen weeks, everyone would call that a miscarriage. When a baby is born prematurely at twenty-four weeks or later, UK law requires this to be recorded as a birth. But when a pregnancy ends just before this cutoff point—say, at twenty-two or twenty-three weeks—how it should be described is more ambiguous. A fetus born at this stage is tiny, about the size of an adult’s hand. It is unlikely to survive. Many doctors call this heartbreaking situation a “late miscarriage” or a “late fetal loss,” even if the tiny child briefly had a heartbeat or took a few breaths. Dr. Smith tells me that parents who have been through this experience often feel strongly that the word “miscarriage” is inadequate. Perhaps in the hope of helping these parents to process their grief, the community of neonatal doctors in the Midlands had developed the custom of describing the same tragedy in a different way: the baby was born alive, but died shortly after.

Mercifully few pregnancies end at twenty-two or twenty-three weeks. But after doing some simple arithmetic, Lucy Smith realized that the difference in how these births were treated statistically was enough to explain the overall gap in newborn mortality between the two hospital groups. Newborns were no more likely to survive in London after all. It wasn’t a difference in reality, but a difference in how that reality was being recorded.

The same difference affects comparisons between countries. The United States has a notoriously high infant mortality rate for a rich country—6.1 deaths per thousand live births in 2010. In Finland, by comparison, it is just 2.3. But it turns out that physicians in America, like those in the UK’s Midlands, seem to be far more likely to record a pregnancy that ends at twenty-two weeks as a live birth, followed by an early death, than as a late miscarriage. Perhaps this is for cultural reasons, or perhaps it reflects different legal or financial considerations. Whatever the reason, some—by no means all—of the high infant mortality rate in the United States seems to be the result of recording births before twenty-four weeks as live when in other countries they would be recorded as miscarried pregnancies. For babies born after twenty-four weeks, the US infant mortality rate falls from 6.1 to 4.2 deaths per thousand live births. The rate in Finland barely shifts, from 2.3 to 2.1.[2]

The issue also arises when comparing trends over time in the same country. When the infant mortality rate rose between 2015 and 2016 in England and Wales, against a history of steadily falling rates, the press understandably raised the alarm. “Obesity, poverty, smoking and a shortage of midwives could all be factors, say health professionals,” said the Guardian newspaper.[3]

Indeed they could. But a group of doctors, writing to the British Medical Journal, pointed out that official statistics were also recording a dramatic rise in the number of live births at twenty-two weeks of gestation, or even earlier.[4] More and more doctors, it seems, were following the Midlands trend of changing their recording practices to record live births and early deaths, rather than late miscarriages. And this was sufficient to explain the increase in the infant mortality statistics.

There is an important lesson here. Often, looking for an explanation really means looking for someone to blame. The infant mortality rate is rising—are politicians not providing enough money for the health service, or is the problem caused by mothers smoking or getting fat? The infant mortality rate is lower in London than in the Midlands—what are hospitals in the Midlands doing wrong? In truth, there may never have been anything to blame anybody for at all.

—

When we are trying to understand a statistical claim—any statistical claim—we need to start by asking ourselves what the claim actually means.

Measuring infant mortality, at first glance, means doing something sad and simple: counting the babies who died. But think about it for a moment and you realize that the distinction between a baby and a fetus is anything but simple—it’s a deep ethical question that underlies one of the most acrimonious divides in US politics. The statisticians have to draw the line somewhere. If we want to understand what is going on, we need to understand where they drew it.

The coronavirus pandemic has raised similar questions. As I write these words, on April 9, 2020, the media are reporting that in the past twenty-four hours, 887 people died with COVID-19 on the British mainland—but I happen to know that number is wrong. Data detective work from the Scottish statistician Sheila Bird tells me that the true figure is more likely to be about 1,500.[5] Why such a huge disparity? Partly because some people died at home, and the statistics represent only those who died in a hospital. But mostly because these overstretched hospitals are reporting deaths with a delay of several days. Deaths announced today, a Thursday, probably took place on Sunday or Monday. And since the death toll has been growing exponentially, telling us about what happened three days ago understates how bad things are now.*

The whole discipline of statistics is built on measuring or counting things. Michael Blastland, co-creator of More or Less, imagines looking at two sheep in a field. How many sheep in the field? Two, of course. Except that one of the sheep isn’t a sheep, it’s a lamb. And the other sheep is heavily pregnant—in fact, she’s in labor, about to give birth at any moment. How many sheep again? One? Two? Two and a half? Counting to three just got difficult. Whether we’re talking about the number of nurses employed by a hospital (do two part-time nurses count as two nurses, or just one?) or the wealth of the super-rich (is that the wealth they declare to the taxman, or is there a way to estimate hidden assets, too?), it is important to understand what is being measured or counted, and how.

It is surprising how rarely we do this. Over the years, as I found myself trying to lead people out of statistical mazes week after week, I came to realize that many of the problems I encountered were because people had taken a wrong turn right at the start. They had dived into the mathematics of a statistical claim—asking about sampling errors and margins of error, debating if the number is rising or falling, believing, doubting, analyzing, dissecting—without taking the time to understand the first and most obvious fact: What is being measured, or counted? What definition is being used?

Yet while this pitfall is common, it doesn’t seem to have acquired a name. My suggestion is “premature enumeration.”

It’s a frequent topic of conversation with my wife. The radio that sits on top of the refrigerator will carry some statistical claim into our home over breakfast—a political sound bite, or the dramatic conclusion of some research. For example, “A new study shows that children who play violent video games are more likely to be violent in reality.” Despite having known my limitations for twenty years, my wife can’t quite rid herself of the illusion that I have a huge spreadsheet in my head, full of every statistic in creation. So she will turn to me and ask, “Is that true?” Very occasionally I happen to have recently researched the issue and know the answer, but far more often I can only reply, “It all depends on what they mean . . .”

I’m not trying to model some radical philosophical skepticism—or annoy my wife. I’m just pointing out that I don’t fully understand what the claim means, so I am hardly in a position (yet) to know whether it might be true. For example, what is meant by a “violent video game”? Does Pac-Man count? Pac-Man commits heinous acts, notably swallowing sentient creatures alive. Or what about Space Invaders? There’s nothing to do in Space Invaders but shoot and avoid being shot. But perhaps that is not quite what the researchers meant. Until I know what they did mean, I don’t know much.

And how about “play”; what does that mean? Perhaps the researchers had children* fill in questionnaires to identify those who play violent games for many hours in a typical week. Or perhaps they recruited some experimental subjects to play a game for twenty minutes in a laboratory, then did some kind of test to see if they’d become more “violent in reality”—and how is that defined, anyway?

“Many studies won’t measure violence,” says Rebecca Goldin, a mathematician and director of the statistical literacy project STATS.[6] “They’ll measure something else such as aggressive behavior.” And aggressive behavior itself is not easy to measure because it is not easy to define. One influential study of video games—I promise I’m not making this up—measured aggressive behavior by inviting people to add hot sauce to a drink that someone else would consume. This “hot sauce paradigm” was described as a “direct and unambiguous” assessment of aggression.[7] I am not a social psychologist, so perhaps that’s reasonable. Perhaps. But clearly, like “baby” or “sheep” or “nurse,” apparently commonsense words such as “violent” and “play” can hide a lot of wiggle room.

We should apply the same scrutiny to policy proposals as we do to factual claims about the world. We all know that politicians like to be strategically vague. They will often trumpet the merits of “fairness” or “progress” or “opportunity,” or say, in the most infuriating tic of all, “We’re proposing this policy because we think it’s the right thing to do.” But even specific-sounding policies can end up meaning very little if we don’t understand the claim. You’d like to increase funding for schools? Great! Is that a funding increase per pupil, after inflation—or not?

For example, a policy paper published in the UK in 2017 by the Brexit lobby group Leave Means Leave called for a “five-year freeze on unskilled immigration.”[8] Is that a good idea? Hard to say until we know what the idea really is: by now, we should know to ask, “What do you mean by ‘unskilled’?” The answer, on closer inspection, is that you’re unskilled if you don’t have a job offer on a salary of at least £35,000—a level that would rule out the majority of nurses, primary school teachers, technicians, paralegals, and chemists. Now that might be a good policy or it might be a bad policy, but most people would be surprised to hear that this freeze on “unskilled immigration” is a policy that proposes excluding people coming to work as teachers and intensive care nurses.[9] This wasn’t just a policy paper, either: in February 2020, the UK government announced new immigration restrictions using a lower cutoff (a salary of £25,600) but similar language about “skilled” and “unskilled.”[10]

Premature enumeration is an equal-opportunity blunder: the most numerate among us may be just as much at risk as those who find their heads spinning at the first mention of a fraction. Indeed, if you’re confident with numbers you may be more prone than most to slicing and dicing, correlating and regressing, normalizing and rebasing, effortlessly manipulating the numbers on the spreadsheet or in the statistical package—without ever realizing that you don’t fully understand what these abstract quantities refer to. Arguably this temptation lay at the root of the last financial crisis: the sophistication of mathematical risk models obscured the question of how, exactly, risks were being measured, and whether those measurements were something you’d really want to bet your global banking system on.

Working on More or Less, I found the problem everywhere. After working with a particular definition for years, the experts we talked to could easily forget that the ordinary listener might have something very different in mind when they heard the term. What the psychologist Steven Pinker calls the “curse of knowledge” is a constant obstacle to clear communication: once you know a subject fairly well, it is enormously difficult to put yourself in the position of someone who doesn’t know it. My colleagues and I weren’t immune. When we started researching some statistical confusion, we’d habitually start by pinning down the definitions—but as we quickly took them for granted, we always had to remind ourselves to explain them to our listeners, too.

—

Darrell Huff would be quick to point to the fact that an easy way to “lie with statistics” is to use a misleading definition. But we can often mislead ourselves.

Consider the number 39,773. It is the number of gun deaths in the United States in 2017 (this number is from the National Safety Council and is the most recent available from that source). This number, or something very like it, is repeated every time a mass shooting makes the headlines, even though the vast majority of these deaths have nothing to do with these grim spectacles.* (Not every mass shooting is headline news, of course. Using the common definition of four people killed or injured in a single incident, there is a mass shooting almost every day in the United States, and many of them would be well down the news editor’s order of priorities.)

“Gun death” doesn’t sound like a complicated concept: a gun is a gun and dead is dead. Then again, nor does “sheep,” so we should pause to check our intuition. Even the year of death, 2017, isn’t as straightforward as you might think. For example, in the UK in 2016, the homicide rate rose sharply. This was because an official inquest finally ruled that ninety-six people who died in a crush at the Hillsborough soccer stadium in 1989 had been unlawfully killed. Initially seen as accidental, those deaths officially became homicides in 2016. This is an extreme example, but there are often delays between when somebody died and when the cause of death was officially registered.

But the big question here is about the connotations of “death.” True, it’s not an ambiguous concept. But we hear the number 39,773 at the very moment we are watching news footage showing lines of ambulances and police cars at the sight of some vivid and horrifying slaughter. So we naturally associate it with murder, or even mass murder. In fact, about 60 percent of gun deaths in the United States are suicides, not homicides or rare accidents. Nobody set out to mislead us into thinking gun-related homicides are two and a half times more common than they actually are. It’s just an assumption we understandably make from the context in which we are usually presented with the number.

Having noticed our error, what conclusions we should draw from it is another question. It’s possible to spin it to support various political outlooks. Gun rights advocates will claim that it shows the fear of mass shootings is overblown. Gun control advocates will counterclaim that it weakens a common argument of the gun rights lobby—that people should be able to arm themselves to defend against an armed attacker, which is no help if the bigger risk is that people will turn their guns against themselves.

As thoughtful readers of statistics, we don’t need to rush to judgment either way. Clarity should come first; advocacy can come once we understand the facts.

—

We should also remember that behind every one of those 39,773 gun deaths is a tragic human story. There’s little evidence that Stalin ever said, “The death of one man is a tragedy, the death of millions is just a statistic,” but the aphorism has echoed down the years in part because it speaks to our profound lack of curiosity about the human stories behind the numbers. Premature enumeration is not just an intellectual failure. Not asking what a statistic actually means is a failure of empathy, too.

Staying with the grim subject of suicide, this time in the UK: “A Fifth of 17 to 19-Year-Old Girls Self-Harm or Attempt Suicide,” blares a headline in the Guardian. The article goes on to speculate that this may be because of social media, pressure to look good, sexual violence, pressure to do well on exams, difficulty finding work, moving to a new area, cuts in central government services, or iPads.[11] But while the piece is long on scapegoats, it’s short on detail about what is meant by self-harm.

So let’s turn to the original study, funded by the UK government and conducted by some respected research organizations.[12] It doesn’t take long to realize that an error has slipped into the headline, as errors often do. It’s not true that a fifth of seventeen- to nineteen-year-old girls self-harm or attempt suicide. What is true is that a fifth of them say that they have done so at some stage—not necessarily recently. But . . . “done so.” What exactly have they done? The study itself is no more illuminating than the Guardian report on it.

The National Health Service website lists a variety of self-harming behaviors, including cutting or burning your skin, punching or slapping yourself, eating or drinking poisons, taking drugs, misusing alcohol, eating disorders such as anorexia and bulimia, pulling out your hair, and even excessive exercise.[13] Is this what these young women were thinking of when they answered yes to the question? We don’t know. I asked the researchers what their question meant; they told me they wanted to “capture the entire spectrum of self-harm” and so did not provide a definition of self-harm to the young women they interviewed. Self-harm means whatever the interviewees thought it means.[14]

That’s fine; there is nothing necessarily wrong with aiming to capture the broadest possible range of behavior. It might be useful to know that a fifth of seventeen- to nineteen-year-old girls have at some point behaved in a way they subjectively consider to be self-harm. But those of us interpreting the statistic will want to bear in mind that nobody else can know precisely what they meant. All forms of self-harm are disturbing, but you may find some of them a great deal more disturbing than others. Binge drinking seems very different from anorexia.

Bearing this in mind, the headline lumping together self-harm and suicide, which at first glance seemed natural, starts to look irresponsible. There is an enormous gulf between excessive exercise and killing yourself. And while this survey suggests that self-harm is worryingly common among young women, suicide is thankfully quite rare. Out of every 100,000 girls in the UK aged between fifteen and nineteen, 3.5 kill themselves each year; that’s about seventy across the entire country.[15]

(By now I hope you are wondering what exactly the authorities mean by “suicide.” It is not always clear whether someone intended to kill themselves; sometimes people intended only to hurt themselves but died by accident. In the UK, the Office for National Statistics draws a clear line: if the child is fifteen or over, the death is assumed to be deliberate; under the age of fifteen, it is assumed to be an accident. Evidently, those assumptions will not always reflect the truth, which is sometimes impossible to know.)

Lumping together self-harm and suicide is all the more irresponsible because the headline singles out girls. The study did indeed find that seventeen- to nineteen-year-old girls are much more likely than seventeen- to nineteen-year-old boys to say they have harmed themselves, yet it is the boys who are the bigger suicide risk. Boys of this age are twice as likely as girls to kill themselves.

Awful tragedies lie behind each of these numbers. Pinning down the definitions is vital if we want to understand what is happening and, perhaps, how we might make life better. That is, after all, why we’re collecting the numbers.

—

I’d like to devote the rest of the chapter to a more detailed example, which I hope will illustrate how we might try to think through a complex problem—first by clarifying what’s being measured, and only then by breaking out the mathematics. It’s an important issue, but also an issue about which many people have very strong beliefs, yet a weak grasp of the definitions involved. That issue is inequality. Let’s start with perhaps the most famous sound bite on the topic.

“Oxfam: 85 Richest People as Wealthy as Poorest Half of the World.” That was a Guardian headline in January 2014. The Independent picked up on the same research published by the development charity Oxfam, as did many other media outlets.[16] It’s an astonishing claim. But what does it tell us?

Oxfam’s aim was publicity. They wanted to generate heat; if they shed any light on the subject, that was a secondary consideration. This isn’t just my opinion: the report’s lead author, Ricardo Fuentes, said as much when interviewed for an Oxfam blog post titled “Anatomy of a Killer Fact,” which celebrated the “biggest-ever traffic day on the Oxfam International website.”[17] The blog post focuses on all the attention the claim received. But was the “killer fact” informative, or even true? Fuentes later told the BBC that his research “has shortcomings but it was as good as it gets.”

I’m not so sure about that. Three years later, Oxfam had revised its analysis so comprehensively that the headline number had changed from eighty-five billionaires to eight billionaires. Had the inequality really become ten times worse, the billionaires ten times richer—or had the poor of the world perhaps lost nine-tenths of their wealth somehow? No, there was no such economic cataclysm. Oxfam’s measure was just a very noisy and uninformative way to think about inequality in the first place.

The dramatic change in the headline claim is one indication that this may not be a terribly educational way to think about inequality. The excited bewilderment of some of the media reporting is another sign of just how baffling the number really was. While the Guardian accurately repeated Oxfam’s headline—eighty-five people collectively have as much wealth as the poorest half of the world—the Independent published an infographic declaring that the eighty-five richest people had as much wealth as the rest of the world put together. (A trailer for a BBC documentary about the super-rich repeated the error.) That’s not even close to being the same claim, although you might have to think twice about why.

If thinking twice didn’t help: almost all global wealth is held neither by the poorest half of the world, who have little or nothing, nor by the richest eighty-five (or eight?) ultra-billionaires. It lies with a few hundred million prosperous people in the middle. You may very well be one of them. The Independent and the BBC had mixed up “the wealth of the poorest half” and “the wealth of everyone who isn’t a zillionaire.” This apparently minor confusion turns out to be between a sum of less than $2 trillion and a sum of more than $200 trillion. Not thinking hard enough about the exact claim being made introduced a hundredfold error.

In a magnificent display of statistical befuddlement, the Independent also declared “the 85 richest people—1%” to have the same wealth as the “rest of the world—99%.” This implies that the population of the world is 8,500. If the previous claim was a hundredfold error, this one is nearly a million times too small.

The hopeless confusions of the Independent are worth dwelling on for a moment. They remind us how easy it is for our emotions to run away with us. There are some people out there with extraordinary, imagination-boggling fortunes. There are other people out there with nothing. It’s not fair. And as we start to seethe at the unfairness, the risk is that we stop thinking. The Independent confused nearly 8 billion people with 8,500 people. It confused the wealth of the poorest half of the world with the wealth of everyone except the richest eighty-five people. These are ludicrous errors—but as Abraham Bredius showed us, when we stop thinking and start feeling, ludicrous errors show up very promptly.

It’s a nice little reminder to all of us to stop and think for a moment. It should not be too complicated a calculation to realize that whoever “the 1%” might be, there are more than eighty-five of them.

I can’t blame Oxfam, an organization devoted to campaigning and fundraising, for seeking the most sensational headlines possible. Nor do I hold them responsible for the fact that the claim prompted all kinds of screw-ups from the media.

The rest of us, however, might prefer some clarity. So—back to the drawing board, and that starts with being clear about what’s being measured, and how.

—

What’s being measured is net wealth—that is, assets such as houses, shares, and cash in the bank, less any debts. If you have a house worth $250,000 with a $100,000 mortgage on it, that’s $150,000 of net wealth.

The Oxfam calculations on which the headline was based took the best available estimate of the total net wealth of the poorest half of the world (accumulated by researchers paid by a bank, Credit Suisse) and compared it with the best available estimate of the total wealth of the top multibillionaires (as reported by newspaper rich lists).[18] They found that you had to total up only the eighty-five richest billionaires before you exceeded the total wealth of the poorest half of the world, about 2.4 billion adults (Credit Suisse’s researchers ignored children).

But does net wealth really tell us much? Let’s say you buy a nice $50,000 sports car with a $50,000 loan. The moment you drive it off the lot, the sports car has lost a few thousand dollars in value, and your net wealth has just fallen. If you’ve just finished an MBA, or law school, or medical school, and you’ve picked up a few hundred thousand dollars of debt, your net wealth is way below zero. But financially, a young doctor is likely to feel much more comfortable than a young subsistence farmer, even if the doctor is up to her chin in debt and the farmer owns a scrawny cow and a rusty bike for a net worth of $100.*

Net wealth is a great way to measure riches, but not such a good way to measure poverty. Lots of people have zero, or less than zero. Some of them are destitute; others, like the junior doctor, are going to be fine.

A further problem is that when you add up all those zeros and negative numbers, you’re never going to get a positive number. As a result, my young son’s piggy bank is worth more than the assets of the poorest billion people in the world put together, because a billion zeros and negative numbers never get you above the roughly $15 he had in there when we last counted it all up. Does that suggest my son is rich? No. Does it demonstrate that grinding poverty is endemic? Well, no, not directly. The fact that more than a billion people have no wealth is striking, but it’s not clear that trying to add up all those zeros tells us much more. I’m not sure it tells us anything, except that a billion times zero is zero.

Now that we’ve avoided premature enumeration—rushing to work with the numbers before we really understand what those numbers are supposed to mean—it’s the perfect time for a little light mathematics, which can be wonderfully clarifying.

Looking at the Global Wealth Report from Credit Suisse, the source of Oxfam’s claims, we can play with some of those numbers to shed more light on the topic.*

- Forty-two million people have more than a million dollars each, collectively owning about $142 trillion. A few of them are billionaires, but most are not. If you have a nice house with no mortgage in a place such as London, New York, or Tokyo, that might easily be enough to put you in this group. So would the right to a good private pension.* [19] Nearly 1 percent of the world’s adult population are in this group.

- Four hundred thirty-six million people, with more than $100,000 but less than a million, collectively own another $125 trillion. Nearly 10 percent of the world’s adult population are in this second group.

- Those two groups, collectively, have most of the cash.

- Another billion people have more than $10,000 but less than $100,000; they own about $4 trillion among them.

- The remaining 3.2 billion adults have only $6.2 trillion, less than $2,000 each on average. Many of them have much less than that average.

Very roughly speaking, the richest half a billion people have most of the money in the world, and the next billion have the rest. The handful of eighty-five staggeringly wealthy super-billionaires are still just a handful, so they own less than 1 percent of this total. All this, it seems to me, tells us a great deal more about the distribution of assets than a widely repeated “killer fact” that talks about wealth inequality while ignoring almost all the wealth in the world. And while Oxfam’s aim, understandably, is to produce such killer facts to win attention and raise money, my aim is to understand our planet and our society. Those facts were easily accessible online; it was a matter of an extra click or two. To find them, all it took was a couple of minutes and a curiosity about the world.

—

At least Oxfam was clear that it was talking about inequality of wealth. Often we hear someone make a vague assertion like “inequality has risen” and we can’t even guess that much: inequality of what, between whom, and measured how?

Perhaps they’re talking about wealth inequality, having read Oxfam’s stat updating the eighty-five billionaires to just eight. Or perhaps they mean inequality of income. If you want to understand how people live and what they are able to consume from day to day, inequality of income is a more natural thing to examine. What we eat, what we wear, and how we live tends to be related not to our wealth but to regular income from a salary, a pension, payments from the state, or the profits from a small business. Very few people have enough wealth to fund their lifestyle purely out of interest payments, and so if we want to understand how inequality manifests itself in everyday life, it makes sense to look at income rather than wealth. The other advantage of looking at income is that we do not need to confront the absurdity of suggesting that an ordinary schoolboy and his piggy bank are richer than a billion people put together.

If we look at inequality of income, inequality between whom? The obvious answer: between the rich and the poor. But there are other possibilities: one could look at inequality between countries, or between ethnic groups, or between men and women, or between the old and the young, or between different regions within a country.

But even once we’ve settled on looking at inequality of income, and between high earners and low earners, the question remains: Measured how?

Here are a couple of possibilities. You could compare the median income (the income of a person right in the middle of the distribution) with the tenth-percentile income (the income of someone near the bottom of the income distribution). This is called the 50/10 ratio, and it’s an indication of how the poor are doing relative to the middle class.

Alternatively, you could look at the income share of the highest-earning 1 percent—a decent indicator of how not just the billionaires are faring, but the millionaires, too. You don’t need to do this yourself; think tanks and academics have done these calculations and they are usually easy to find online.[20]

Both of these measures seem to tell us something important. But what if they conflict? Imagine a country where the income of the highest-earning 1 percent surged, while at the same time there was a reduction in inequality further down the income scale, as the 50/10 ratio shrank and poorer households caught up with the comfortably off. If the rich grow richer but the poor grow richer, too, relative to the median, has inequality risen? Or fallen? Or a bit of both?

This might seem like a cute hypothetical question, but as it happens it describes the situation in the United Kingdom between 1990 and 2017. After taxes, the top 1 percent saw their share of income rise, but inequality among lower-earning households fell as poorer households tended to catch up to those with middling incomes. It’s an awkward story for anyone who wants an easy answer, but in a complicated world we shouldn’t expect that the statistics will always come out neatly.

A few years ago I was invited to be the resident data geek on a TV debate about inequality in the UK. The show was an ambitious hour-long special in front of a studio audience during which various worthies would discuss why inequality in the UK mattered. In early discussions with the program’s production team, I pointed them toward the World Inequality Database, a resource that was originally put together by the economists Tony Atkinson and Thomas Piketty. Piketty, of course, was the superstar author of Capital in the Twenty-First Century; Sir Tony, who died in 2017, was one of his academic mentors. The two of them favored stiff redistributive taxes and wide-ranging government intervention in the economy. Like many economists, I’m quite wary of that sort of policy, but I recommended their database anyway. They were simply the world’s leading experts.

All seemed well until a few days before the show I had an awkward phone call with one of the production team. I mentioned in passing that the pretax income share of the top 1 percent had fallen slightly over the previous few years. As we’ve seen, that’s by no means the only way to measure inequality, but it’s a metric Piketty and Atkinson like to emphasize, and it seemed a good starting point: it was crisp, rigorous, and easy to explain on TV. Alarmed, she told me that the entire program was based on the premise that inequality had been increasing since the 2007–08 financial crisis. Why did they think this was true? The data were clear: the top 1 percent had 12 percent and rising of pretax income in 2008, but the crisis knocked that back to 10 or 11 percent.* This was hardly astonishing: a massive financial crisis is likely to temporarily hit the income of high earners such as bankers, lawyers, and corporate executives. And this was data, remember, gathered by two left-leaning economists who would have been first in line to decry the effects of bankers’ greed or government cutbacks.

But no: the idea that inequality had risen just seemed to the TV producers like the kind of thing that should be true. Perhaps they looked at the data I’d recommended and found some flaw with it. Perhaps they found some different measure that they felt was better. But the strong impression from my conversation was that the production team simply hadn’t looked at the data I’d recommended to them. I hope that isn’t so, because it takes a special lack of curiosity to be able to produce an ambitious TV program without taking the ninety seconds or so necessary to check whether the premise of the show is actually true.

I made my excuses and did not participate.

—

Statisticians are sometimes dismissed as bean counters. The sneering term is misleading as well as unfair. Most of the concepts that matter in policy are not like beans; they are not merely difficult to count, but difficult to define. Once you’re sure what you mean by “bean,” the bean counting itself may come more easily. But if we don’t understand the definition, then there is little point in looking at the numbers. We have fooled ourselves before we have begun.

The solution, then: Ask what is being counted, what stories lie behind the statistics. It is natural to think that the skills required to evaluate numbers are numerical—understanding how to compute a percentage, or how to disentangle your millions from your billions from your zillions. It’s a question of mathematics, is it not?

What I hope we’ve learned over the past few pages is that the truth is more subtle yet in some ways easier: our confusion often lies less in numbers than in words. Before we figure out whether nurses have had a pay raise, first find out what is meant by “nurse.” Before lamenting the prevalence of self-harm in young people, stop to consider whether you know what “self-harm” is supposed to mean. Before concluding that inequality has soared, ask, “Inequality of what?” Demanding a short, sharp answer to the question “Has inequality risen?” is not only unfair, but strangely incurious. If we are curious instead, and ask the right questions, deeper insight is within easy reach.

RULE FOUR

Step Back and Enjoy the View

The shortest-lived creatures on the Disc were mayflies, which barely make it through twenty-four hours. Two of the oldest zigzagged aimlessly over the waters of a trout stream, discussing history with some younger members of the evening hatching.

“You don’t get the kind of sun now that you used to get,” said one of them.

“You’re right there. We had proper sun in the good old hours. It were all yellow. None of this red stuff.”

“It were higher, too.”

“It was. You’re right.”

• Terry Pratchett, Reaper Man

The newspapers had an alarming message for Londoners in April 2018:

“London’s Murder Rate Is Higher Than New York’s for the First Time Ever!” The headlines played into a narrative of gangs gone wild. And if we ignore for a moment that the very definition of “murder” differs on either side of the Atlantic, this claim is also perfectly true. In February 2018, there were fourteen murders in New York City, but fifteen in London.[1]

But what should we conclude? Nothing.

We should conclude nothing because that pair of numbers alone tells us very little. If we want to understand what’s happening, we need to step back and take in a broader perspective.

Here are a few facts worth knowing about murders in London and New York. London had 184 murders in 1990, while New York suffered 2,262—more than ten times as many. It’s with that image in mind of New York as a murderous place that Londoners are alarmed at the idea that they might have become as rotten as the Big Apple. But London’s murder rate has fallen, not risen, since 1990. In 2017, there were 130 murders in London, including ten people killed in terrorist attacks. London was safe in 1990 and it’s a little bit safer today. As for New York, murders fell to 292 in 2017. That means New York is still more dangerous than London, but much, much safer than in 1990.

(We should really look at the murder rate per million people rather than the murder total, but the populations of New York City and London are similar, so let’s not worry about that.)

Now that New York is vastly safer, very occasionally it has a good month and London has a bad one, and New York’s monthly murder count dips below London’s. The thing about numbers is that over time, they do tend to go up and down a bit.*

So while the newspaper headlines are narrowly correct, they point us away from the truth rather than toward it: the news is good, not bad; London has become safer, not more dangerous; and London remains safer than the fast-improving New York. We get the real story only with context.

—

In 1965, two Norwegian social scientists, Johan Galtung and Mari Ruge, made a fascinating observation: What counts as “news” depends very much on the frequency with which we pay attention.[2] If media outlets know most of their audience is checking in every day, or every few hours, they will naturally tell us the most attention-grabbing event that’s happened in that time.

Consider the financial news. There is a big difference between the rolling business coverage of Bloomberg TV, the daily rhythm of the newspaper the Financial Times (my employer), and the weekly take of The Economist, even if the three outlets have a similar interest in business, economics, and geopolitics. Bloomberg might pick up on sharp market moves over the past hour. The same moves won’t merit a mention in The Economist. Weekly, daily, hourly—the metronome of the news clock changes the very nature of what is news.

Now imagine a much slower rhythm of news: a twenty-five-year newspaper, say. What would the latest edition say? It would be packed with updates, some hopeful and some grim; it would describe the rise of China, the World Wide Web and smartphones, the emergence of al-Qaeda and the collapse of Lehman Brothers. There might be a small feature article on crime, noting that the murder count had fallen in London, but not nearly as much as in New York. Nobody would spare a syllable on the idea that London was experiencing a killing spree; such an observation could only make sense in a fast-twitch media outlet.

How about a fifty-year newspaper? Max Roser, a young economist who created the Our World in Data website, proposed that idea, inspired by Galtung and Ruge. Roser imagines a newspaper published in 1918, 1968, and 2018. Topics that seemed earth-shattering to the daily newspapers of the time might not be mentioned at all, while huge changes in the world would scream from the front pages.[3]

What would the front page of the fifty-year newspaper say in 2018? One possibility might be a story about something that didn’t happen: “Phew! World Avoids Nuclear Armageddon!” Readers of the 1968 newspaper would have read anxiously about how, over the previous three decades, the atomic bomb had been invented, developed, used on Japan with catastrophic effect, then superseded by vast arsenals of much more powerful hydrogen bombs, and how the superpowers had flirted with nuclear conflict repeatedly—in the Korean War, during the Cuban missile crisis, and more than once over Berlin. For a reader picking up a newspaper in 2018 for the first time since 1968, it would be big news that the Cold War had simply ended without a nuclear exchange of any kind—even if no daily newspaper would have been tempted in the meantime to run with the headline “No H-Bombs Dropped Today.”

Or perhaps the editors would splash with a story on climate change. Since early research on the greenhouse effect probably wouldn’t have merited a mention in the 1968 edition, the 2018 newspaper would have to start with an explanation of the basic problem: burning fossil fuels such as gas, oil, and coal turns out to alter the composition of the atmosphere in a way that helps it trap heat. (Headline: “Gah! Burning Coal Turns Out to Be a Terrible Idea!”) That explanation would be illustrated by an alarming graph showing the increase in global temperatures.

Climate change is a difficult thing to report over a short time horizon. On an annual basis global temperatures bounce up and down; you can find almost as many years when they have fallen as when they have risen—which is raw material for the manufacturing of doubt. The fifty-year newspaper, however, conveys the grim news clearly: temperatures have risen by about 0.75°C (1.35°F) since the 1960s, depending on exactly what temperature measure you look at and between which years.[4] Alas, from the right perspective, the trend is clearly that the planet is heating up.

How about a hundred-year newspaper? The perspective changes again. Thinking about readers who last consulted a newspaper in 1918, you might decide to offer a leading story about the miracle of safe childhood: “Child Mortality Falls by a Factor of Eight!” Imagine a school set up to receive a hundred five-year-olds, randomly chosen at birth from around the world. In 1918, only sixty-eight children would have turned up for the first day of school; thirty-two would have died before reaching the age of five. This wasn’t some temporary catastrophe because of the terrible 1914–18 war, or the global influenza outbreak of 1918: in 1900 the statistic would have been even worse. Now ninety-six children show up safely for their first year in school; just four die before reaching school age. Remember, they’re selected from all over the world, including the poorest, most isolated, and most strife-torn of countries. That is astonishing progress.[5]

For a two-hundred-year newspaper, the editorial board might take yet another angle: “Most People Aren’t Poor!” There are still a lot of poor people, of course—between 600 million and 700 million now live in what we call extreme poverty, according to the World Bank’s definition of less than around $1.90 per day. That’s not far from one in ten of the world’s population. But in the early nineteenth century, almost everyone—nineteen people out of twenty—lived in that state of destitution. That’s wonderful progress, and it becomes apparent only if we step back and change our perspective.

—

So far I’ve talked about perspective mainly in terms of time. We can get useful context from other kinds of comparison, too.

Let’s return to our case study of income inequality from the previous chapter, where we learned there are many plausible ways to measure it—such as the 50/10 ratio, or the income share of the top 1 percent. What if we could produce some sort of composite measure that summarizes the whole of the income distribution? These composite measures exist, and we’ve already mentioned the most famous—the Gini coefficient, named after the early-twentieth-century Italian statistician Corrado Gini.

Like any other measure of inequality, the Gini coefficient doesn’t tell us everything. On a global scale, the coefficient has been falling—that is, incomes are becoming more equal. That’s because lots of previously very poor people, many in China and India, have become a lot better off—and in the mathematical calculations that go into the Gini coefficient, that outweighs inequality rising in the upper half of the income scale, with the very rich leaving the moderately well-to-do in their wake.[6] No single number could communicate all that. But the Gini coefficient does elegantly reflect the experience of everyone across the income spectrum. Moving a dollar from a billionaire to a millionaire will not change the top 1 percent share of income, since that dollar stays in the hands of someone in the top 1 percent. But moving a dollar from a richer person to a poorer person, no matter how rich or poor either of them might be, will reduce the Gini coefficient.

One big problem with the Gini coefficient, however, is getting an intuitive feel for what it actually means. It’s easy enough to picture a country with a Gini coefficient of zero—there, everyone gets exactly the same income. Likewise, we can readily imagine a country in which the Gini coefficient is 100 percent—there, the despotic president has cornered all the income and everyone else gets precisely nothing. But what would it be like to live in a country where the Gini coefficient of income is, say, 34 percent?

As it happens, if you live in the UK, you can answer that question.[7] But even a specialist in income distribution would probably understand a Gini coefficient of 34 percent only in reference to the Gini coefficients of other countries. It’s 50 in China, for example, 42 in the United States, 25 in Finland. Globally, including everyone who lives in the poorest sub-Saharan nations and the richest petrostates, the Gini coefficient of income is 65 percent, higher than in any individual nation.[8]

But we can get an even better intuitive feel for what the Gini coefficient means by doing the same calculation on things other than income. Take life itself. Like income, life is unequally distributed. Some babies die almost immediately after being born; others live for a hundred years. But these extremes are relatively unusual: most people live for at least sixty years, and few live for more than ninety. So we would expect the global Gini coefficient of life expectancy to be fairly low, and it is—less than 20 percent.

How about the height of adults? We all have an intuitive sense of how little that varies, so it can be another useful reference point. If my back-of-the-envelope calculation is correct, the Gini coefficient is less than 5 percent.

For a newspaper column, I once calculated the Gini coefficient of recent sexual activity in the UK among thirty-five- to forty-four-year-olds. I know you’re curious: it’s 58 percent, much higher than the UK income Gini of 34 percent.[9] Should we be surprised that it is higher than the income Gini? I’m not sure. But it is. It seems that a tenfold gap in sexual activity—with one person having sex once a month, and another having sex ten times a month—is far more common than a tenfold gap in income. A tenfold gap in longevity—a centenarian and a child who dies at the age of ten—is thankfully rarer still. A tenfold gap in adult height? Unheard of, even in the record books.

—

Another way to step back and enjoy the view is to give yourself a sense of scale. Faced with a statistic, simply ask yourself, “Is that a big number?” The creators of More or Less, Michael Blastland and Andrew Dilnot, made a habit of asking this unassuming but powerful question.[10]

Take, for example, the claim that Donald Trump’s border wall between the United States and Mexico would cost $25 billion to build. Is that a big number? It certainly sounds biggish, but to really understand the number you need something to compare it with. For example, the US defense budget is a little under $700 billion, or $2 billion a day. The wall would fund about two weeks of US military operations. Or, alternatively, the wall would cost about $75 a person: there are about 325 million people in the United States, and $25 billion divided by 325 million is about $75.* Big number? Small number? You can be the judge of that, but I’m guessing your judgment will be better informed having made these comparisons.

Andrew Elliott—an entrepreneur who likes the question so much he published a book with the title Is That a Big Number?—suggests that we should all carry a few “landmark numbers” in our heads to allow easy comparison.[11] A few examples:

- The population of the United States is 325 million. The population of the United Kingdom is 65 million. The population of the world is 7.5 billion.

- Name any particular age (under the age of sixty). There are about 800,000 people of that age in the UK. If a policy involves all three-year-olds, for example, there are 800,000 of them. In the United States, there are about 4 million people of any particular age (under the age of sixty).

- Distance around the Earth: 40,000 kilometers, or 25,000 miles. It varies depending on whether you go around the poles or around the equator, but not much.

- The drive from Boston to Seattle: 3,000 miles.

- Length of a bed: 2 meters (or 7 feet). As Elliott points out, this helps you visualize the size of a room: How many beds is that?

- The gross domestic product of the United States: about $20 trillion (or $20,000 billion). It’s a lot of walls, if that’s really how you want to spend it.

- 100,000 words: the length of a medium-size novel.

- 1,454 feet: the height of the Empire State Building to its tip. (It’s also about 102 stories.)

Personally, I like to carry a few of these numbers around in my head. I’m a geek that way. And I find that the more landmarks I have, the more sense all the other landmarks make. But the truth is that we don’t have to remember any of these numbers. We can look any of them up, from any number of reputable sources, using any reference book or internet connection—and in many cases it will be worth double-checking anyway.

Once we have some landmark numbers to hand, they’re easy to use. You can compare one thing with another (this 10,000-word report seems long, but an ordinary novel is ten times longer), or you can divide one thing by another (the US defense budget is more than $2,000 per American, per year). Memorize, or look up, some handy numbers, and then do some simple arithmetic—with a calculator if you want. It isn’t hard. But it is remarkably illuminating.*

—

It would be nice if we didn’t have to do this—if we could rely on the media outlets that present us with statistics to also helpfully provide all the context and perspective we need to make sense of them. The better ones will indeed try to do this. But context and perspective are never going to be on the front page above the fold.

We’ve seen one reason for this: our frequency of engagement. The splash of a daily newspaper, the lead story on a TV bulletin, and the top item on a website will all focus on the most dramatic, engaging, and significant events, since the typical news consumer will last have checked in a few hours previously. Some media critics believe there is another reason media outlets don’t emphasize context and perspective: people are drawn to bad news. Hans Rosling, coauthor of Factfulness and a wonderful campaigner for more realistic views of the world based on good data, calls this “the negativity instinct.” And it’s generally easier to make news seem bad if you omit the context.

I’m cautious about the idea that we’re biased toward bad news, because in general we tend to be rather optimistic; psychologist Tali Sharot reckons that 80 percent of us suffer from an “optimism bias,” systematically overestimating our longevity, our career prospects, and our talents while being blind to the risk of illness, incompetence, or divorce.[12] Daniel Kahneman, Nobel laureate and one of the fathers of behavioral economics, calls overconfidence “the most significant of the cognitive biases.”[13] In many ways we humans are actually pretty positive creatures—perhaps a little too positive, sometimes.

A more plausible explanation is that we are drawn to surprising news, and surprising news is more often bad than good.[14] If media outlets had a bias merely toward the negative, one might expect them to report regularly on, say, smoking-related deaths. Ten times as many US residents died from smoking-related diseases as from terrorism in September 2001, the month that saw the most deadly terrorist attack in the country’s history.[15] Even a weekly magazine could honestly have noted at the end of that terrible week that cigarettes had killed more people than al-Qaeda. The newspapers ignored the deaths from cigarettes because they had a bias toward the shocking.

It’s possible, of course, for shocking news to be positive. But the psychologist Steven Pinker has argued that good news tends to unfold slowly, while bad news is often more sudden.[16] That sounds right—it is, after all, quicker to knock something down than to build it. Following a thought experiment the great psychologist Amos Tversky once shared with a young Pinker,[17] imagine the best possible thing that could happen to you today. You could win the lottery, I suppose. (Would that really be good news?) There are certain other moments where something wonderful could happen: you could have been hoping for a baby after many months of fruitless trying, and finally the pregnancy test comes back positive; you might have applied for a promotion or a place at university, and you get it. But for most people, on most days, the possibility of some dramatic and surprising life improvement is fairly limited. Life is already good for many people; when life isn’t good, it is likely to improve slowly rather than thanks to some sudden miracle.

But the possibility of some dramatic turn for the worse? That’s easy to imagine. You, or a loved one, could be diagnosed with cancer, hit by a truck, or violently assaulted. Your house could be burgled, or it could burn down. You could be sacked from your job. You could be accused of a crime you didn’t commit. You could discover that your partner is having an affair, or wants a divorce. I didn’t have to think hard to reel those ideas off, and I’m sure you could add more without breaking sweat—or perhaps the cold sweat would break all too swiftly. The list of catastrophes could go on indefinitely.

So when media outlets want to grab our attention, they look for stories that are novel and unexpected over a short time horizon—and these stories are more likely to be bad than good.

The need to grab attention also skews the tactics of politicians, charities, and other campaigners. They know that if they want to get into the headlines, they need to make surprising claims. For example, in May 2015 the British media published the alarming news that strokes were on the increase in middle-aged people; this conclusion was based on official statistics highlighted by the Stroke Association, whose chief executive commented, “There is an alarming increase in the numbers of people having a stroke in working age.”[18] Fortunately, this is incorrect. Strokes are becoming rarer, thanks to improved diet, better treatment, and public awareness campaigns; but those same public awareness campaigns encouraged people to present themselves at hospital at the first sign of a minor stroke. As a result, hospital admissions for strokes in younger people increased—or “rocketed,” as the Stroke Association put it—and the Stroke Association was on the story. The good news is that the incidence of stroke in the UK has for a long time been falling steadily and substantially across most age groups. But how could the Stroke Association be noticed with a story like that? And if it isn’t noticed, it can’t raise money.

Or consider Oxfam’s lament, late in 2016, that “the highly successful fight against global poverty is being lost badly in one critical area—people’s minds. A new global survey . . . reveals that 87% of people around the world believe that global poverty has either stayed the same or gotten worse over the past 20 years, when the exact opposite is true—it has more than halved.”[19] This press release didn’t win nearly as much attention as the one we discussed in the previous chapter, which said that eighty-five people (or was it eight?) owned as much wealth as half the world (or was it everyone else?). When the alarmist press releases get the headlines, no wonder people think the plight of the world is getting worse.

In the UK, people are not hugely worried about issues such as immigration, teenage pregnancy, crime, and unemployment in their own areas—but they are profoundly anxious about these issues in the country as a whole. Similar results emerge if you ask people about their personal job situation versus their view of their country’s economy: most people think that all is well for them personally but are worried about the society they live in.[20] Presumably this is because we personally experience our own localities, but we rely on the news for information about the wider world. The “negativity instinct” may not be a driver of negativity in news coverage, but it certainly seems to be a result.

—

In 1993, Martyn Lewis, then the most popular news anchor in the UK, argued that the media should spend more time covering good news stories.[21] He was sneered at by fellow journalists, who caricatured his argument as simply a request for more cheery “And finally . . .” stories of skateboarding dogs, slotted in at the end of a news program to sprinkle a little sugar over the evening bulletin’s bitter offerings. This was unfair;* Lewis explicitly called for substantive good news stories rather than the precursors of today’s videos of cats surfing on Roombas.

“Good stories are there,” he wrote, “made all the more memorable by their rarity.” Happily, this is precisely wrong. Since Lewis wrote this in 1993, 154,000 people have escaped from extreme poverty every day.[22] In 1980, the vaccines for illnesses such as measles, diphtheria, and polio were given to about 20 percent of one-year-olds. Eighty percent missed out. Recently at least 85 percent of one-year-olds received these vaccines, and the challenge is to maintain this in the coronavirus age.[23] Child mortality, as we’ve seen, has fallen dramatically. The good stories are everywhere. They are not made memorable by their rarity; they are made forgettable by their ubiquity. Good things happen so often that they cannot seriously be considered for inclusion in a newspaper. “An Estimated 154,000 People Escaped from Poverty Yesterday!” True, but not news.

We don’t have daily updates on how many people escape poverty, and perhaps we never will. When I worked for the World Bank in 2004–05, we were still updating our estimates of extreme poverty only once every three years. If a newspaper decided to pick up on the story, fine, but that would be just a single story once every thousand days. No self-respecting newspaper would republish the story regularly to remind its readers, “Not news, but still true!” So the fall in the most extreme form of poverty—and dozens of other true stories we could tell about improved literacy, democracy, votes for women, education for girls, access to clean water, immunizations, agricultural yields, infant mortality, the price of solar power, the number of deaths in plane crashes, or the prevalence of hunger—goes unreported.[24]

It’s not just because it’s a happy story; it’s because the news comes at the wrong frequency. Gloomy stories that come at the wrong frequency are often ignored, too, as we’ve seen with smoking, the world’s most persistent, and thus most boring, cause of mass fatalities. Climate change is not ignored, but it is rarely reported directly; instead, the news covers deliberate attempts to get attention for it, such as protests, summits, and the occasional scientific or government report. We see it mentioned, infuriatingly, alongside reports on the weather—but we rarely see reports on slow-moving indicators such as the world’s rising temperature.

A third example is in finance. In 2004 and 2005, my Financial Times colleague Gillian Tett highlighted the development of huge financial markets in debt and derivatives, a kind of side bet on the movements of interest rates, exchange rates, and other financial indicators. The world financial system was like an iceberg: above the surface glistened the stock markets, easy to see and discuss; beneath the waves lurked the debt and derivatives markets, vast and hidden. Stock markets publish numbers continually, including a daily close-of-market update for the evening news. But one of the most important measures of the size of the derivatives market is produced by the Bank for International Settlements once every three years. The pace of information didn’t fit the frequency of the financial newspapers, and so it was systematically underreported. Of course, this was bad news worth being aware of: problems in these markets were at the center of the catastrophic financial crisis of 2007–08, and Gillian Tett was one of the few people who could honestly say she’d been paying attention beforehand.[25]

Some commentators argue that the cure for all this is simply to stop reading the newspapers. The author Rolf Dobelli—amusingly, writing in the Guardian newspaper—gives us ten reasons to stop reading the news.[26] Nassim Taleb, author of The Black Swan, puts it succinctly: “To be completely cured of newspapers, spend a year reading the previous week’s newspapers.”[27]

Because I work for a newspaper, you might expect me to protest. I have a lot of sympathy, though. I often find that my Saturday Financial Times column is unmoored from the news of the week. I’m just not very interested in producing a hot take on recent news; I find my interest far more engaged by topics that have occurred to me after reading books or academic papers—or just musing about life. And while I enjoy the way that fans of More or Less often compare it favorably to rolling radio and TV news, I sometimes feel that we’re getting credit for something that comes naturally: we operate at a different rhythm from that of the rolling news. As a weekly program we usually have a couple of days to chew over something that has been said—or missed—in the blur of a live interview. Often, we find ourselves pondering a topic for weeks or months. Why cover a story quickly when you can explore it properly? And we don’t usually have to worry about being scooped because we’re far too nerdy for anyone else to care about our stories.

Professionally, I can’t ignore the news, but I pay less attention to it than many of my colleagues—occasionally to their frustration. Daily news always seems more informative than rolling news; weekly news is typically more informative than daily news. A book is often better still. Even within a daily or a weekly newspaper, I find myself preferring the slower-paced explanation and analysis rather than the breaking news.

If you’re a news junkie I suggest that you go deeper and broader, rather than faster and faster. It is harder to do this when the news itself seems to be alarming, but it’s a good habit. Very little news requires the immediate attention that you might devote to a traffic update or a severe weather warning. If you come back in an hour—or a week—you will learn just as much. Indeed, you’ll probably learn more. You might even ask yourself: What would a weekly magazine or a weekly podcast be paying attention to that might otherwise be drowned out in the noise of rolling news?

In the crazy early days when the COVID-19 coronavirus went global, Scientific American admonished journalists, “Facts about this epidemic that have lasted a few days are far more reliable than the latest ‘facts’ that have just come out, which may be erroneous or unrepresentative and thus misleading . . . a question that today can be answered [by] only informed belief may perhaps be answered with a fact tomorrow.”[28] Sound advice, and not just for journalists but for citizens, too. So however much news you choose to read, make sure you spend time looking for longer-term, slower-paced information. You will notice things—good and bad—that others ignore.

—

What have we learned so far about how to evaluate a statistical claim? In the first chapter, I advised trying to notice your feelings about the claim; in the second chapter, constructively sense-checking the claim against your personal experience; in the third chapter, asking yourself if you really understand what the claim means. These are all simple, commonsense suggestions, and in this chapter I’ve added a fourth: Step back and look for information that can put the claim into context. Try to get a sense of the trend. “Another terrible crime has occurred!” is perfectly consistent with “Overall, crime is way down.”

Look for something that will give you a sense of scale, such as comparing the situation in one country with the situation in other countries, or figuring out the cost per person of some proposed government expenditure.

None of these methods is technical; anyone can use them. Together they can go a long way toward providing statistical illumination. But sometimes we need to dig a little deeper into how a statistic was produced. Let’s do that now.

RULE FIVE

Get the Backstory

“In each human coupling, a thousand million sperm vie for a single egg. Multiply those odds by countless generations . . . it was you, only you, that emerged. To distill so specific a form from that chaos of improbability, like turning air to gold . . . that is the crowning unlikelihood . . .”

“You could say that about anybody in the world!”

“Yes. Anybody in the world . . . But the world is so full of people, so crowded with these miracles, that they become commonplace and we forget . . .”

• Alan Moore, Watchmen

A couple of decades ago, two respected psychologists, Sheena Iyengar and Mark Lepper, set up a jam-tasting stall in an upmarket store in California. Sometimes they offered six varieties of jam, at other times twenty-four; customers who tasted the jam were then offered a voucher to buy it at a discount. The bigger display with a wider array of jams attracted more customers but very few of them actually bought jam. The display that offered fewer choices inspired more sales.[1]

The counterintuitive result went viral—it hit a sweet spot. People respond better to fewer choices! It became the stuff of pop-psychology articles, books, and TED Talks. It was unexpected yet seemed plausible. Few people would have predicted it, and yet somehow those who heard about it felt they’d known it all along.

As an economist, I’ve always found this a little strange. Economic theory predicts that people should often value extra choices, and will never be discouraged by them—but economic theory can be wrong, so that’s not what was curious about the jam study.

One puzzle was that according to the study, the measured effect of offering more choice was huge: only 3 percent of jam tasters at the twenty-four-flavor stand used their discount voucher, versus 30 percent at the six-flavor stand. This suggests that by trimming their range, retailers could increase their sales tenfold. Does anybody really believe that? Draeger’s, the supermarket that hosted the experiment, stocked 300 varieties of jam and 250 types of mustard. It seemed to be doing fine. Had it missed a trick? Starbucks boasts of offering literally tens of thousands of combinations of frothy drink; the chain seems to be doing fine, too. So I wondered just how general the finding might be. Still, it was a serious experiment conducted by serious researchers. And one should always be willing to adjust one’s views to fit the evidence, right?

Then I met a researcher at a conference who told me I should get in touch with a young psychologist called Benjamin Scheibehenne. I did. Scheibehenne had no reason to doubt Iyengar and Lepper’s discovery that people might be demotivated when faced with lots of options. But he had observed the same facts about the world that I had—that so many successful businesses offer a cornucopia of choice. How were those facts compatible with the experiment? Scheibehenne had a theory, which was that companies were finding ways to help people navigate complex choices. That seems plausible. Perhaps it was something to do with familiarity: people often go to the supermarket planning to buy whatever they bought last time, rather than some fancy new jam. Perhaps it was the way the aisles were signposted, or the way choices were organized to make them less bewildering. These all seem sensible things to investigate, so Scheibehenne planned to investigate them.[2]

He began by rerunning the jam experiment to get a baseline from which he could start tweaking and exploring different possibilities. But he didn’t get the same baseline. He didn’t get the same result at all. Iyengar and Lepper had discovered that choice dramatically demotivates. When Scheibehenne tried to repeat their experiment he found no such thing. Another researcher, Rainer Greifeneder, had rerun a similar study by Iyengar and Lepper that focused on choosing between luxury chocolates, and like Scheibehenne had failed to reproduce the original “choice is bad” result. The pair teamed up to pull together every study of the “choice is bad” effect they could find. There were plenty, but many of them had failed to find a journal that would publish them.

When all the studies, published and unpublished, were assembled, the overall result was mixed. Offering more choices sometimes motivates and sometimes demotivates people. Published research papers were more likely to find a large effect, either positive or negative. Unpublished papers were more likely not to find an effect at all. The average effect? Zero.[3]

This is unnerving. So far we’ve encountered misleading claims in the context of an agenda being pushed—Oxfam drumming up publicity, a media outlet chasing clicks—or a subtle detail being overlooked, like the use of different words to describe the tragic early end of a pregnancy. When it comes to academia, we might reasonably hope that the subtle details will be spotted and the only agenda being pursued is a search for knowledge. It makes sense to tread carefully with campaigning groups or clickbait headlines, but can’t we assume we’re on more solid ground when we pick up an academic journal? Iyengar and Lepper were, as I’ve said, highly respected. Is it possible that they were just flat-out wrong? If so, how? And what should we make of the next counterintuitive finding that sweeps the science pages or the airport bookshelves?

For an answer, let’s take a step sideways and ponder the internet’s most famous potato salad.

—

Surely there is no easier way to raise some cash than through Kickstarter? The crowdfunding website enjoyed a breakthrough moment in 2012 when the Pebble, an early smartwatch, raised over $10 million. In 2014, a project to make a picnic cooler raised an extraordinary $13 million. Admittedly, the Coolest cooler was the Swiss Army knife of cool boxes. It has a built-in USB charger, cocktail blender, and speakers, attracting a thundering herd of backers. The Pebble smartwatch had its revenge in 2015, as a fresh campaign raised more than $20 million for a new and better watch.

In some ways, though, Zack “Danger” Brown’s Kickstarter achievement was more impressive than any of these. He turned to Kickstarter for $10 to make some potato salad—and he raised $55,492 in what must be one of history’s most lucrative expressions of hipster irony.[4]

Following Zack Brown’s exploits, I wondered what exciting project I might launch on Kickstarter, looking forward to settling back to count the money as it poured in.

The same thought may have occurred to David McGregor. He was bidding for £3,600 to fund a trip across Scotland, photographing its glorious scenery for a glossy book—a lovely way to fund his art, and his holiday. Jonathan Reiter had bigger ambitions. His BizzFit looked to raise $35,000 to create an algorithmic matching service for employers and employees. Shannon Limeburner was also business-minded, but sought a mere $1,700 to make samples of a new line of swimwear she was designing. Two brothers in Syracuse, New York, even launched a Kickstarter campaign in the hope of being paid $400 to film themselves terrifying their neighbors at Halloween.

These disparate campaigns have one thing in common: they received precisely zero support. Not one of these people was able to persuade strangers, friends, or even their own families to kick in so much as a cent.

My inspiration and source for these tales of Kickstarter failure is Silvio Lorusso, an artist and designer based in Venice. Lorusso’s website, Kickended.com, searched Kickstarter for all the projects that have received absolutely no funding. (There are plenty: about 10 percent of Kickstarter projects go nowhere at all, and fewer than 40 percent raise enough money to hit their funding targets.)

Kickended performs an important service. It reminds us that what we see around us is not representative of the world; it is biased in systematic ways. Normally, when we talk of bias we think of a conscious ideological slant. But many biases emerge from the way the world presents some stories to us while filtering out others.

I have never read a media report or blog post about the attempts of the young and ambitious band Stereotypical Daydream to raise $8,000 on Kickstarter to record an album. (“Our band has tried many different ways of saving money to record a legitimate album in a professional studio. Unfortunately, we still have not saved enough.”) It probably will not surprise you to hear that the Stereotypical Daydream Kickstarter campaign brought them zero dollars closer to their goal.

On the other hand, I’ve heard quite a lot about the Pebble watch, the Coolest cooler, and even that potato salad. If I didn’t know better, I might form unrealistic expectations about what running a Kickstarter campaign might achieve.

This isn’t just about Kickstarter, of course. Such bias is everywhere. Most of the books people read are bestsellers—but most books are not bestsellers, and most book projects never become books at all. There’s a similar tale to tell about music, films, and business ventures.

Even cases of COVID-19 are subject to selective attention: people who feel terrible go to the hospital and are tested for the disease; people who feel fine do not. As a result, the disease looks even more dangerous than it really is. Even though statisticians understand this problem perfectly well, there’s no easy way to solve it without systematic testing. And in the early stages of the pandemic, when the most difficult policy decisions were being made, systematic testing was elusive.

There’s a famous story about the mathematician Abraham Wald, who was asked in 1943 to advise the US Air Force on how to reinforce its planes. The planes were returning from sorties peppered with bullet holes in the fuselage and wings; surely those spots could use some armor plating? Wald’s written response was highly technical, but the key idea is this: We observe damage only in the planes that return. What about the planes that were shot down? We rarely see damage to the engine or fuel tanks in planes that survive. That might be because those areas are rarely hit—or it might be that whenever those areas are hit, the plane is doomed. If we look only at the surviving planes—falling prey to “survivorship bias”—we’ll completely misunderstand where the real vulnerabilities are.[5]

The rabbit hole goes deeper. Even the story about survivorship bias is an example of survivorship bias; it bears little resemblance to what Abraham Wald actually did, which was to produce a research document full of complex technical analysis. That is largely forgotten. What survives is the tale about a mathematician’s flash of insight, with some vivid details added. What originally existed and what survives will rarely be the same thing.[6]

Kickended, then, provides an essential counterpoint to the breathless accounts of smash hits on Kickstarter. If successes are celebrated while failures languish out of sight (which is often the situation), then we see a very strange slice of the whole picture.

This starts to give us a clue as to what might have happened with the jam experiment. Like the Coolest cooler, it was a smash hit—but not the full story. Benjamin Scheibehenne’s role was a bit like Silvio Lorusso’s at Kickended: he had gone looking not just for the choice experiment that had gone viral, but for all the other experiments that had produced different results and had vanished into obscurity. When he did, he was able to reach a very different conclusion.

—

Bear Kickended in mind as you ponder the following story. In May 2010, a surprising paper was submitted to the Journal of Personality and Social Psychology. The author was Daryl Bem, a respected old hand in the field of academic psychology. What made the research paper astonishing was that it provided apparently credible statistical evidence of an utterly incredible proposition: that people could see into the future. There were nine experiments in total. In one, participants would look at a computer screen with an image of two curtains. Behind one curtain was an erotic photograph, they were told. They simply had to intuit which one. The participant would make a choice, and then—after the choice had already been made—the computer would randomly assign the photograph. If the participants’ guesses were appreciably better than chance, then that was evidence of precognition. They were.[7]

In another of the experiments that Bem’s research paper described, subjects were shown a list of forty-eight words and tested to see how many of the words they would remember. Then some subjects were asked to practice by retyping all the words. Normally it would be no surprise that practice helps you remember, but in this case Bem found that the practice worked even though the memory test came first and the practice came after.

How seriously should we take these results? Bear in mind that the research paper, “Feeling the Future,” was published in a respected academic journal after a process of peer review. The experiments it reported passed the standard statistical tests, which are designed to screen out fluke results. All this gives us some reason to believe that Bem found precognition.

There is a much better reason to believe that he did not, of course, which is that precognition would violate well-established laws of physics. Vigorous skepticism is justified. As the saying goes, extraordinary claims require extraordinary evidence.

Still, how did Bem accumulate all this publishable evidence for precognition? It’s puzzling. Perhaps it’s less puzzling after you connect it to the story of Kickended.

After Bem’s evidence of precognition had been published in the Journal of Personality and Social Psychology, several other studies were produced that followed Bem’s methods. None of them found any evidence for precognition, but the journal refused to publish any of them. (It did publish a critical commentary, but that’s not the same thing as publishing an experiment.) The journal’s grounds for refusal were that it “did not publish replications”—that is, once an experiment had demonstrated an effect there was no space to publish attempts to check on that effect. In theory, that might sound reasonable: Who wants to read papers confirming things they already knew? In practice, it has the absurd effect of ensuring that when something you thought you knew turns out to be wrong, you won’t hear about it. Bem’s striking finding became the last word.[8]

But it was also the first word. I strongly doubt that before Bem came along, any serious journal would have published research, no matter how rigorous, whose abstract read: “We tested several hundred undergraduates to see if they could see into the future. They couldn’t.”

This, then, is a survivorship bias as strong as press coverage of Kickstarter projects or trying to deduce the vulnerabilities of planes by examining only the ones whose vulnerabilities weren’t fatal. Out of all the possible studies that could have been conducted, it’s reasonable to guess that the journal was interested only in the ones that demonstrated precognition. This wasn’t because of a bias in favor of precognition. It was because of a bias in favor of novel and surprising discoveries. Before Bem, the fact that students didn’t seem to be able to see into the future was trivial and uninteresting. After Bem, the fact that students didn’t seem to be able to see into the future was a not-welcome-in-this-journal replication attempt. In other words, only evidence of precognition was publishable because only evidence of precognition was surprising. Studies showing no evidence of precognition are like bombers that have been shot in the engine: no matter how often such things happen, they’re not going to make it to where we can see them.

The “choice demotivates” finding is far more credible than the “students can see into the future” finding—but still, the jam experiment may have been subject to a similar dynamic. Imagine approaching a psychology journal before Iyengar and Lepper’s breakthrough result with the following study: “We set up stalls offering people different kinds of cheese. Sometimes the stalls had twenty-four types of cheese and sometimes just six. On the days when people were offered more types of cheese, they were a bit more likely to buy cheese.” Yawn! That’s not surprising at all. Who wants to publish that? It was only when Iyengar and Lepper ran an experiment showing the opposite result that the whole thing became not only publishable, but a Coolest-cooler smash hit.

If you read only the experiments published in the Journal of Personality and Social Psychology, you might well conclude that people can indeed see into the future. For obvious reasons, this particular flavor of survivorship bias is called “publication bias.” Interesting findings are published; non-findings, or failures to replicate previous findings, face a higher publication hurdle.

Bem’s finding was the $55,000 potato salad—wildly atypical, and widely reported as a result. The unpublished replications would typically have been like Stereotypical Daydream’s attempts to fund their album: nothing happened and nobody cared.

Except this time, somebody did care.

—

The paper is beautiful,” says Brian Nosek of Daryl Bem’s study. “It follows all the rules of what one does, does it in a really beautiful way.”[9]

But as Nosek, a psychologist at the University of Virginia, understood perfectly well, if Bem followed all the rules of academic psychology and ended up seeming to demonstrate that people can see into the future, something is wrong with the rules of academic psychology.[10]

Nosek wondered what would happen if you systematically reran some more respected and credible psychological experiments. How many results would come out the same? He sent around an email to like-minded researchers, and with impressive speed managed to get a global network of nearly three hundred psychologists collaborating to check studies that had recently been published in one of three prestigious academic journals. While Benjamin Scheibehenne had been digging into one particular field—the link between motivation and choice—Nosek’s network wanted to cast their net widely. They chose a hundred studies. How many did their replication attempts back up? Shockingly few: only thirty-nine.[11] That left Nosek and the rest of academic psychology with one big question on their hands: How on earth did this happen?

Part of the explanation must be publication bias. As with Daryl Bem’s study, there is a systemic bias toward publishing the interesting results, and of course flukes are more likely to seem interesting than genuine discoveries.

But there’s a deeper explanation. It’s the reason Nosek had to reach out to so many colleagues, rather than simply get his graduate student assistants to do all the checks. Since the top journals weren’t very interested in publishing replication attempts, he knew that devoting his research team full-time to a replication effort might be career suicide: they simply wouldn’t be able to accumulate the publications necessary to secure their future in academia. Young researchers must either “publish or perish,” because many universities and other research bodies use publication records as an objective basis for deciding who should get promotions or research grants.

This is another example of the Vietnam body count problem we met in the second chapter. Great researchers do indeed tend to publish lots of research that is widely cited by others. But once researchers are rewarded for the quantity and prominence of their research, they start looking for ways to maximize both. Perverse incentives take over. If you have a result that looks publishable but fragile, the logic of science tells you to try to disprove it. Yet the logic of academic grants and promotions tells you to publish at once, and for goodness’ sake don’t prod it too hard.

So not only are journals predisposed to publish surprising results, researchers facing “publish or perish” incentives are more likely to submit surprising results that may not stand up to scrutiny.

—

The illusionist Derren Brown once produced an undoctored film of him tossing a coin into a bowl and getting heads ten times in a row. Brown later explained the trick: the stunning sequence came only at the end of nine excruciating hours of filming, when the string of ten heads finally materialized.[12] There is a 1-in-1,024 chance of getting ten heads in a row if you toss a fair coin ten times. Toss it a few thousand times and a run of ten consecutive heads is almost guaranteed. But Brown could send his stunning result off to the Journal of Coin Flipping, perhaps with the delicious title (suggested by the journalists Jacob Goldstein and David Kestenbaum) “Heads Up! Coin-Flipping Bias in American Quarter Dollars Minted in 1977.”[13]

To be clear—such a research paper would be fraudulent, and nobody believes that such extreme and premeditated publication bias explains the large number of nonreplicable studies that Nosek and his colleagues unveiled. But there are shades of gray.

What if 1,024 researchers individually researched coin tossing and one of them produced the stunning result of ten heads in a row? That is mathematically the same situation, but from the point of view of the astonished researcher in question, she or he would be blameless. Now, it seems unlikely that so many researchers would have bothered to investigate coin tossing—but we don’t know how many people tried and failed to find precognition before Daryl Bem succeeded.

The shades of gray also apply within an individual researcher’s laboratory. For example, a scientist could do a small exploratory study. If he or she found an impressive result, why not publish? But if the study fell flat, the researcher could chalk it up as a learning experience and try something else. This behavior doesn’t sound especially unreasonable to the layman, and it probably doesn’t feel unreasonable to the researchers doing it—but it is publication bias nonetheless, and it means that flukes are likely to be disproportionately published.

Another possibility is that the researcher does the study, finds some promising results, but those results are not quite statistically solid enough to publish. Why not keep going, recruiting some more participants, gathering more data, and seeing if the results firm up? Again, this doesn’t seem unreasonable. What could be wrong with gathering more data? Wouldn’t that just mean that the study was getting closer and closer to the truth? There’s nothing wrong with doing a large study. In general, more data is better. But if data are gathered bit by bit, testing as we go, then the standard statistical tests aren’t valid. Those tests assume that the data have simply been gathered, then tested—not that scientists have collected some data, tested them, and then maybe collected a bit more.

To see the problem, imagine a game of basketball is about to be played and someone asks you a question: How convincing would a victory have to be before you feel confident saying that the winning team is better than the other team, rather than just luckier on the day? There’s no right answer—after all, sometimes luck can be outrageous. But you might decide that a margin of, say, ten points at the end of the game is enough to be convincing. This is, very roughly, what the standard statistical tests do to decide whether or not an effect is deemed to be “significant” enough to publish.

But now imagine the organizer of the basketball game stands to get a bonus if one of the teams turns out to be better—it doesn’t matter which—so, without telling you, she decides that if either team is ever ahead by ten points, she’ll bring the game to an early halt. And if, at the final whistle, the two teams are separated by seven, eight, or nine points, she’ll play overtime to see if the gap opens up to ten. After all, she’s just a basket or two away from demonstrating the superiority of one of the teams!

It’s obvious that would be a misuse of the test you set, but much of this kind of misuse seems to be quite common in practice.[14]

A third problem is that researchers also have choices as to how they analyze the data. Maybe the study holds up for men, but not women.* Maybe the study holds up if the researcher makes a statistical adjustment for age, or for income. Maybe there are some weird outliers and the study holds up only if they are included, or only if they are excluded.

Or maybe the scientist has a choice of different things she or he could measure. For instance, a study of how screen use affects the well-being of young people could measure both screen use and well-being in different ways. Well-being can be measured by asking people about episodes of anxiety; or it could be measured by asking people about how satisfied they are with their lives; or it could be measured by asking a young person’s parents how they think he or she is doing. Screen time could be measured directly through a tracking app, or indirectly through a survey; or perhaps rather than “screen time” one might want to measure “frequency of social media use.” None of these choices is right or wrong, but—again—the standard statistical tests assume that the researcher made the choice before collecting the data, then collected data, then ran the test. If the researcher ran several tests, then made a choice, flukes are vastly more likely.

Even if the researcher ran only one test, flukes are more likely to slip through if he or she did so after gathering the data and getting a feel for how they looked. This leads to yet another kind of publication bias: if a particular way of analyzing the data produces no result, and a different way produces something more intriguing, then of course the more interesting method is likely to be what is reported and then published.

Scientists sometimes call this practice “HARKing”—HARK is an acronym for Hypothesizing After Results Known. To be clear, there’s nothing wrong with gathering data, poking around to find the patterns, and then constructing a hypothesis. That’s all part of science. But you then have to get new data to test the hypothesis. Testing a hypothesis using the numbers that helped form the hypothesis in the first place is not OK.[15]

Andrew Gelman, a statistician at Columbia University, favors the term “the garden of forking paths,” named after a short story by Jorge Luis Borges. Each decision about what data to gather and how to analyze them is akin to standing on a pathway as it forks left and right and deciding which way to go. What seems like a few simple choices can quickly multiply into a labyrinth of different possibilities. Make one combination of choices and you’ll reach one conclusion; make another, equally reasonable, and you might find a very different pattern in the data.[16]

A year after Daryl Bem’s result was released, three psychologists published a demonstration of just how seriously researchers could go astray using standard statistical methods combined with these apparently trivial slips and fudges.[17] The researchers, Joseph Simmons, Uri Simonsohn, and Leif Nelson, “proved” that listening to “When I’m Sixty-Four” by the Beatles would make you nearly eighteen months younger.[18]

I know you’re curious: How did they do it? The researchers collected various pieces of information from each participant, including their age, their gender, how old they felt, the age of their fathers, and the age of their mothers—along with various other almost completely irrelevant facts. They analyzed every possible combination of these variables, and they also analyzed the data in sets of ten participants, stopping to check for significant results each time. In the end they found that if they statistically adjusted for the fathers’ ages, but not the mothers’, and if they stopped after twenty participants, and if they discarded the other variables, then they could demonstrate that people who had been randomly assigned to listen to “When I’m Sixty-Four” were substantially younger than a control group who had been randomly assigned to listen to a different song. All utter nonsense, of course—but utter nonsense that bore an eerie resemblance to research that had been published and taken seriously. Would genuine researchers ever push so far over the line from rigorous practice into rigged research? Probably not very often. But those who did would get more attention. And the majority who did not might unwittingly commit subtler versions of the same statistical sins.

The standard statistical methods are designed to exclude most chance results.[19] But a combination of publication bias and loose research practices means we can expect that mixed in with the real discoveries will be a large number of statistical accidents.

—

Darrell Huff’s How to Lie with Statistics describes how publication bias can be used as a weapon by an amoral corporation more interested in money than truth. With his trademark cynicism, he mentions that a toothpaste maker can truthfully advertise that the toothpaste is wonderfully effective simply by running experiments, putting all unwelcome results “well out of sight somewhere” and waiting until a positive result shows up.[20] That is certainly a risk—not only in advertising but also in the clinical trials that underpin potentially lucrative pharmaceutical treatments. But might accidental publication bias be an even bigger risk than weaponized publication bias?

In 2005, John Ioannidis caused a minor sensation with an article titled “Why Most Published Research Findings Are False.” Ioannidis is a “meta-researcher”—someone who researches the nature of research itself.* He reckoned that the cumulative effect of various apparently minor biases might mean that false results could easily outnumber the genuine ones. This was five years before the Journal of Personality and Social Psychology published Daryl Bem’s research on precognition, which sparked Brian Nosek’s replication attempt. Precognition might not exist, but Ioannidis clearly saw the crisis coming.[21]

I confess that when I first heard of Ioannidis’s research, it struck me as an extraordinary piece of hyperbole. Sure, all scientific research is provisional, everyone makes mistakes, and sometimes bad papers get published—but surely it was wrong to suggest that more than half of all the empirical results out there were false? But after interviewing Scheibehenne and learning what he’d discovered about the choice literature, I started to wonder. Then, over the years, it gradually became painfully clear to me and many others who were initially skeptical that Ioannidis was on to something important.

While Bem’s precognition study was understandably famous, many other surprising psychological findings had become well known to non-psychologists through books such as Thinking, Fast and Slow (by Nobel laureate Daniel Kahneman), Presence (by psychologist Amy Cuddy), and Willpower (by psychologist Roy Baumeister and journalist John Tierney). These findings hit the same counterintuitive sweet spot as the jam experiment: strange enough to be memorable, but plausible enough not to dismiss out of hand.

Baumeister is famous in academic psychology for studies showing that self-control seems to be a limited resource. People asked to restrain themselves by munching radishes while delicious freshly baked chocolate cookies lay within easy reach were then quicker to abandon a frustrating task later.[22] Cuddy found that asking people to adopt “power poses”—for example, hands-on-hips like Wonder Woman—boosted their levels of testosterone and suppressed their levels of the stress hormone cortisol.[23] Kahneman described the “priming” research of John Bargh. Young experimental subjects were asked to solve a word puzzle in which some of them were exposed to words that suggested old age, such as bald, retirement, wrinkle, Florida, and gray. The young subjects who had not seen these particular words then set off briskly down the corridor to participate in another task; the young subjects who had, instead, been “primed” with words suggesting old age shuffled off down the corridor at a measurably slower pace.[24]

These are extraordinary results, but as Kahneman wrote about priming research, “Disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.”

Now we realize that disbelief is an option. Kahneman does, too. Publication bias, and more generally the garden of forking paths, means that plenty of research that seems rigorous at first sight both to onlookers and often to the researchers themselves may instead be producing spurious conclusions. These studies—of willpower, of power posing, and of priming—have all proved difficult to replicate. In each case, the researchers have defended their original finding, but the prospect that they were all statistical accidents seems increasingly reasonable.

Daniel Kahneman himself dramatically raised the profile of the issue when he wrote an open letter to psychologists in the field warning them of a looming “train wreck” if they could not improve the credibility of their research.[25]

The entire saga—Ioannidis’s original paper, Bem’s nobody-believes-this finding, the high-profile struggles to replicate Baumeister’s, Cuddy’s, and Bargh’s research, and as the coup de grâce, Nosek’s discovery that (as Ioannidis had said all along) high-profile psychological studies were more likely not to replicate than to stand up—was sometimes described as a “replication crisis” or a “reproducibility crisis.”

In the light of Kickended, perhaps none of this should have been a surprise—but it is shocking nonetheless. The famous psychological results are famous not because they are the most rigorously demonstrated, but because they’re interesting. Fluke results are far more likely to be surprising, and so far more likely to hit that Goldilocks level of counterintuitiveness (not too absurd, but not too predictable) that makes them so fascinating. The “interestingness” filter is enormously powerful.

—

Little harm is done if publication bias (and survivorship bias) merely produces cute distortions in our view of the world, leading people to prepare for a job interview by finding a secluded spot to strike a Wonder Woman pose. Even if many would-be entrepreneurs are foolishly over-optimistic about their chances of raising money on Kickstarter, we all enjoy the fruits of successful new business ideas that more rational people would not have quit their jobs to pursue. And few scientists were about to embrace Daryl Bem’s apparent discovery of precognition, for reasons well summarized by Ben Goldacre, an expert in evidence-based medicine: “I wasn’t very interested, for the same reasons you weren’t. If humans really could see the future, we’d probably know about it already; and extraordinary claims require extraordinary evidence, rather than one-off findings.”[26]

But Ben Goldacre thinks the stakes are higher, and so do I. This bias may have serious consequences for both our money and our health.

Money first. Business writing—a field in which I confess to dabbling—is dripping with examples of survivorship bias. In my book Adapt, I had a little chuckle about the Tom Peters and Robert Waterman book In Search of Excellence, a blockbusting business bestseller published in 1982, which offered management lessons gleaned from studying forty-three of the most outstanding corporations of that time. If they really were paragons of brilliant management, then one might have expected their success to last. If instead they were the winners of an invisible lottery, the beneficiaries of largely random strokes of good fortune, then we would expect that the good luck would often fail to last.

Sure enough, within two years almost a third of them were in serious financial trouble. It’s easy to mock Peters and Waterman—and people did—but the truth is that a healthy economy has a lot of churn in it. Corporate stars rise, and burn out. Sometimes they have lasting qualities, sometimes fleeting ones, and sometimes no qualities at all, bar some luck. By all means look at the success stories and try to learn lessons, but be careful. It is easy, in Nassim Taleb’s memorable phrase, to be “fooled by randomness.”

Perhaps all such business writing is harmless: when daily data from the shop floor contradict the business-book wisdom, the shop floor will win. While the jam study became famous among the chattering classes, there is scant sign that many businesses took the “Choice is bad” finding seriously in the decisions they made about stocking their shelves. Still, one can’t help suspecting that where good data are rarer, major decisions are being made on the basis of survivor bias.

In finance, the problem may be worse. A Norwegian TV show illustrated this rather brilliantly in 2016 by organizing a stock-picking competition, in which investors would buy a variety of Norwegian shares to the value of 10,000 Norwegian kroner—about $1,000. The competitors were a diverse bunch: a pair of stockbrokers, who confidently opined, “The more you know, the better you’ll do”; the presenters of the show; an astrologer; two beauty bloggers who confessed to never having heard of any of the companies in question; and a cow named Gullros who would pick stocks by wandering around a field marked out in a grid of company names and expressing her conviction by defecating in the relevant square.

The astrologer fared worst; the professionals did a little better, matching the performance of Gullros the cow (both the cow and the professionals achieved a respectable 7 percent return over the three-month contest); the beauty bloggers did better still—but the standout winners were the TV presenters, with a return of nearly 25 percent over just three months. How had they done so well? Simple: they hadn’t entered their own competition just once. Secretly, they’d done so twenty times by allowing themselves to pick twenty different portfolios. They revealed only the best-performing one to the audience. They appeared to be inspired stock-pickers, until they revealed their own trick. Survivor bias conquers all.[27]

With that in mind, it is hard to evaluate an investment manager who picks stocks or other financial products. They have everything to gain by persuading us that they are a genius, but have very little to show us except a track record. “My fund beat the market last year, and the year before” is pretty much all we have to go on. The trouble is that we see only the successes, alongside the schadenfreude of the occasional high-profile implosion. Underperforming investment funds tend to be closed down, merged, or rebranded. A major investment house will offer many different funds, and will advertise the ones that have been successful in the past. The Norwegian TV show condensed and exaggerated the process, but be assured that when fund managers advertise their stellar results, those ads do not contain a random sample of the funds on offer.

Survivor bias even distorts some studies of investment performance. These studies often start by looking at “funds that exist today” without fully acknowledging or adjusting for the fact that any fund still in existence is a survivor—and that introduces a survivorship bias. Burton Malkiel, economist and author of A Random Walk down Wall Street, once tried to estimate how much survivorship bias flattered the performance of the surviving funds. His estimate—an astonishing 1.5 percent per year. That might not sound like much, but over a lifetime of investing it’s a factor of two: you expect retirement savings of (say) $100,000 and end up with $50,000 instead. Put another way, if you ignore all the investment funds that quietly disappear, the apparent performance is twice as good as the actual performance.[28] The result is to persuade people to invest in actively managed funds, which often charge high fees, when they might be better served by a low-cost, low-drama fund that passively tracks the stock market as a whole. That is a decision worth tens of billions of dollars a year across the US economy; if it’s a mistake, it’s a multibillion-dollar mistake.[29]

So much for money. What about health? Consider the life-or-death matter of which medical treatments work and which don’t. A randomized controlled trial (RCT) is often described as the gold standard for medical evidence. In an RCT, some people receive the treatment being tested while others, chosen at random, are given either a placebo or the best known treatment. An RCT is indeed the fairest one-shot test of a new medical treatment, but if RCTs are subject to publication bias, we won’t see the full picture of all the tests that have been done, and our conclusions are likely to be badly skewed.[30]

For example, in 2008 a quick survey of studies of a variety of antidepressant medications would have found forty-eight trials showing a positive effect and three showing no positive effect. This sounds pretty encouraging, until you ponder the risk of publication bias. So the researchers behind that survey looked harder, digging out twenty-three unpublished trials; of these, twenty-two had a negative result in which the drug did not help patients. They also found that eleven of the trials that seemed positive in the articles describing them had in fact produced negative results in the summaries presented to the regulator, the US Food and Drug Administration. The articles had managed to cherry-pick some good news and hand-wave away some bad news, and finish up presenting a positive-seeming picture about a drug that had not, in fact, been effective. The corrected score, then, was not 48–3 in favor of antidepressants working well, it was 38–37. Perhaps the antidepressants do work, at least sometimes or for some people, but it’s fair to say that the published results did not fairly reflect all the experiments that had been conducted.[31]

This matters. Billions of dollars are misspent and hundreds of thousands of lives lost because of survivorship bias, when we make decisions without seeing the whole story—the investment funds that folded, the Silicon Valley entrepreneurs who never got beyond the “junk in the garage” stage, the academic studies that were never published, and the clinical trials that went missing in action.

—

So far, this chapter has told a tale of catastrophe. The one bright spot is that these problems are vastly better understood and appreciated than they were even five years ago. So let’s focus on that bright spot for a moment, and ask if there’s hope for improvement.

For researchers, it’s clear what that improvement would look like: They need to come clean about the Kickended side of research. They need to be transparent about the data that were gathered but not published, the statistical tests that were performed but then set to one side, the clinical trials that went missing in action, and the studies that produced humdrum results and were rejected by journals or stuffed in a file drawer while researchers got on with something more fruitful.

Those of us who write about research have a similar responsibility: not just to report on a stunning new result, but to set it in the context of what has been published before—and, preferably, what should have been published but languishes in obscurity.

Ideally, we need to be able to rise out of Andrew Gelman’s “garden of forking paths” and see the maze from above, including the dead ends and the paths less traveled. That view from above comes when we have all the relevant information in the most user-friendly form.

We are a long way from achieving those standards—but there are distinct signs of improvement. It is slow and incomplete, but it is improvement nonetheless. In medicine, for example, in 2005 the International Committee of Medical Journal Editors declared that the top medical journals they edited would no longer publish clinical trials that hadn’t been preregistered. Preregistration means that before conducting a trial, researchers have to explain what they plan to do and how they plan to analyze the results, posting that explanation on a public website. Such preregistration is an important fix for publication bias, because it means that researchers can easily see cases in which a trial was planned but then somehow the results went missing in action. Preregistration should also allow other researchers to read a trial write-up and then go back to check that the plan for analyzing the data was followed, rather than being changed once the data appeared.

Preregistration isn’t a panacea. It poses a particular challenge for field studies in social science, which often require academic researchers to piggyback on some project being conducted by a government or charitable organization. Such projects evolve over time in ways that researchers cannot control or predict. And even when medical journals demand preregistration, they may fail to enforce their own demands.[32] Ben Goldacre and his colleagues at Oxford University’s Centre for Evidence-Based Medicine spent a few weeks systematically monitoring the publication of new articles in the top medical journals. They identified fifty-eight articles that fell short of the reporting standards those journals had agreed to uphold—for example, clinical trials that had prespecified that they’d measure certain outcomes for patients, but then later switched to reporting different outcomes. They promptly wrote letters of correction to the journal editors but found that their letters were often rejected rather than published.[33]

It’s disappointing to realize that standards are patchily enforced, but perhaps not surprising given that the entire system is basically self-regulated by the standards of a professional community, rather than governed by some central Solomonic figure. And it does seem to me that the situation has significantly improved over the past two decades: awareness is improving, bad practice is being called out, and it is better to have patchy standards than no standards at all. We have journals such as Trials, launched in 2006, which will publish the results of any clinical trial, regardless of whether the outcome was positive or negative, fascinating or dull, ensuring that no scientific study languishes unpublished simply because it wasn’t regarded as newsworthy in the world of research. There’s an enormous opportunity to do more with automated tools, such as automatically identifying missing trials, studies that were preregistered but then not published, or spotting when later papers are citing earlier research that has since been updated, corrected, or withdrawn.[34]

In psychology, the kerfuffle over precognition may well have a positive result. Academic psychologists want to get published, of course, but most of them don’t want to produce junk science; they want to find out what’s true. The reproducibility crisis seems to be improving awareness of good research standards, as well as holding out more carrots to reward replication efforts, and more sticks to punish sloppy research.

There are encouraging signs that more researchers are welcoming replication efforts. In 2010, for instance, political scientists Brendan Nyhan and Jason Reifler published a study on what became known as “the backfire effect”—in brief, that people were more likely to believe a false claim if they’d been shown a fact-check that debunked the claim. This caused a moral panic among some journalists, particularly after the rise of Donald Trump. Fact-checking only makes matters worse! It hit that perfect counterintuitive sweet spot. But Nyhan and Reifler encouraged further studies, and those studies suggest that the backfire effect is unusual and fact-checking does help. One summary of the research concluded: “Generally debunking can make people’s beliefs in specific claims more accurate.” Nyhan himself has quoted this summary on Twitter when he sees people relying on his original paper without considering the follow-ups.[35]

Many statisticians believe the crisis points to the need to rethink the standard statistical tests themselves—that the very concept of statistical significance is deeply flawed. Mathematically, the test is simple enough. You start by assuming that there is no effect (the drug does not work; the coin is fair; precognition does not exist; the twenty-four-jam stall and the six-jam stall are equally appealing), and then you ask how unlikely the observed data is. For example, if you assume that a coin is fair and you toss it ten times, you’d expect to see heads five times, but you wouldn’t be surprised to see six heads or maybe even seven. You’d be astonished to see ten heads in a row—and given that this would happen by chance less than one time in a thousand, you might question your original assumption that the coin was fair. Statistical significance testing relies on the same principle: assuming no effect, are the data you collect surprising? For instance, when testing a drug, your statistical analysis begins with the assumption that the drug does not work; when you observe that lots of the patients taking the drug are doing much better than the patients who are taking a placebo, you revise that assumption. In general, if the chances of randomly observing data at least as extreme as you collect are less than 5 percent, the results are significant enough to overturn the assumption: we can conclude with a sufficient degree of confidence that the drug works, large displays of jam discourage people from buying jam, and that precognition exists.

The problems are obvious. Five percent is an arbitrary cutoff point—why not 6 percent, or 4 percent?—and it encourages us to think in black-and-white, pass-or-fail terms, instead of embracing degrees of uncertainty. And if you found the previous paragraph confusing, I don’t blame you. Conceptually, statistical significance is baffling, almost backward: it tells us the chance of observing the data given a particular theory, the theory that there is no effect. Really, we’d like to know the opposite, the probability of a particular theory being true, given the data. My own instinct is that statistical significance is an unhelpful concept and we could do better, but others are more cautious. John Ioannidis—he of the “Why Most Published Research Findings Are False” paper—argues that despite the flaws of the method, it’s “a convenient obstacle to unfounded claims.”

Unfortunately, there is no single clever statistical technique that would make all these problems evaporate. The journey toward more rigorous science requires many steps, and we at least are taking some of them. I recently had the chance to interview Richard Thaler, a Nobel Memorial Prize winner in economics, who has collaborated with Daniel Kahneman and many other psychologists. He struck me as well placed to evaluate psychology as a sympathetic outsider. “I think the replication crisis has been great for psychology,” he told me. “There’s just better hygiene.”[36] Brian Nosek, meanwhile, told the BBC: “I think if we do another large reproducibility project five years from now, we are going to see a dramatic improvement in reproducibility in the field.”[37]

—

In the early chapters of this book, I cited numerous psychological studies of motivated reasoning and the biased assimilation of information. You may by now be wondering: How do I know that those studies are credible?

The honest answer is that I cannot be certain. Any experimental research I cite has a chance of being the next jam experiment—or, much worse, the next discovery that listening to “When I’m Sixty-Four” will make you younger. But when I read the studies I’ve described, I try to put the advice from the last few pages into practice. I try to get a sense of whether the study fits into the broader picture of what we know, or whether it’s some strange outlier. If there are twenty or thirty studies from different academics using different methods, but all pointing to a similar conclusion—for instance, that our powers of logical reasoning are skewed by our political beliefs—then I am less concerned that an individual experiment might turn out to be a fluke. If an empirical discovery makes sense in theory and in practice as well as in the lab, that’s reassuring.

On most topics, most of us will not be digging through academic papers. We’ll rely on the media to get a digestible take on the state of scientific knowledge. Science journalism is like any other kind of journalism: There is good, and there is bad. You can find superficial, sensationalist retreads of press releases that are themselves superficial and sensationalist. Or you can find science journalism that explains the facts, puts them in a proper context, and when necessary speaks truth to power. If you care enough as a reader you can probably figure out the difference. It’s really not hard. Ask yourself if the journalist reporting on the research has clearly explained what’s being measured. Was this a study done with humans? Or mice? Or in a petri dish? A good reporter will be clear. Then: How large is the effect? Was this a surprise to other researchers? A good journalist will try to make space to explain—and the article will be much more fun to read as a result, satisfying your curiosity and helping you to understand.*

If in doubt, you can easily find second opinions: almost any major research finding in science or social science will quickly be picked up and digested by academics and other specialists, who’ll post their own thoughts and responses online. Science journalists themselves believe that the internet has improved their profession: in a survey of about a hundred European science journalists, two-thirds agreed with that idea, and fewer than 10 percent disagreed.[38] That makes sense: the internet has made it easier to read the journal articles, easier to access the systematic reviews, and easier to reach scientists for a second opinion.

If the story you’re reading is about health, there’s one place you should be sure to look for a second opinion: the Cochrane Collaboration. It’s named after Archie Cochrane, a doctor, epidemiologist, and campaigner for better evidence in medicine. In 1941, when Cochrane was captured by the Germans and became a prisoner of war, he improvised a clinical trial. It was an astonishing combination of bravery, determination, and humility. The prison camp was full of sick men—Cochrane was one of them—and he suspected that the illness was caused by a dietary deficiency, but he knew that he didn’t know enough to confidently prescribe a treatment. Rather than slump into despair or follow a hunch, he managed to organize his fellow prisoners to test the effects of different diets, discovered what they were lacking, and provided incontrovertible evidence to the camp commandant. Vitamin supplements were duly procured, and many lives were saved as a result.[39]

In 1979, Cochrane wrote that “it is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials.” After Cochrane’s death, this challenge was taken up by Iain Chalmers. In the early 1990s, Chalmers began assembling a collection of systematic reviews, at first just of the randomized trials conducted in the field of perinatal health—the care of pregnant women and their babies. The effort grew into an international network of researchers who review, rate, synthesize, and publish the best available evidence on a huge variety of clinical topics.[40] They call themselves the Cochrane Collaboration and they maintain the Cochrane Library, an online database of systematic research reviews. The full database is not freely available in every country, but the accessible research summaries are, providing short descriptions of the state of knowledge based on randomized trials.

I looked at some recent research summaries, pretty much at random, to see what came up. One of the front-page summaries promised to evaluate “yoga for treating urinary incontinence in women.” Well, I don’t practice yoga, don’t suffer from urinary incontinence, and am not a woman, so my evaluation of this report promised to be uncompromised by any actual knowledge about the topic.

Before I looked at what the Cochrane Library had to say, I typed “Can yoga cure incontinence?” into Google. WebMD was one of the top search results.[41] It reported that a new trial had shown dramatic improvements for older women, although it noted that the study was quite small. The Daily Mail picked up on the same study and reported it in a similar way: the improvements were big, but the study was small.[42] The top search result was from a private health care company:[43] it enthused about the spectacular results and did not mention how small the study was, although it did link through to the original research.[44]

None of this reporting is great, but neither is it terrible. To be honest, I expected worse. Nor is much harm likely to result. People may take up yoga with false hopes, or alternatively may take up yoga, get better, and then credit the yoga when in fact they would have gotten better anyway. But none of this would be disastrous.

Still, the media reports failed to give the backstory. They simply regurgitated the scientific research without any indication of whether it accorded with, or contradicted, anything that had already been discovered.

The Cochrane Library, by contrast, aims to provide an accessible summary of everything we know about yoga and incontinence—if anything. It’s also on the first page of Google search results. Cochrane is not a secret.

The Cochrane review, written in plain and unshowy language, is clear enough. There have only been two studies of the issue. Both of them were small. The evidence is weak, but what evidence there is suggests that for urinary incontinence yoga is better than nothing, and that mindfulness meditation is better than yoga. That’s it—the result of a quick Google search and one minute scanning a page written in plain English. (Translations into many languages are available.) It would be nice, of course, if there was a vast and credible evidence base to lean on, but in this case, there isn’t—and I’d rather know that. Thanks to the Cochrane summary we no longer have to guess if there’s a pile of important evidence that we simply weren’t told about.[45]

A related network, the Campbell Collaboration, aims to do the same thing for social policy questions in areas such as education and criminal justice. As these efforts gain momentum and resources, it will become easier for us to work out whether a study makes sense and fits into a wider pattern of discoveries—or whether it’s a $55,000 potato salad.

RULE SIX

Ask Who Is Missing

The power to not collect data is one of the most important and little-understood sources of power that governments have . . . By refusing to amass knowledge in the first place, decision-makers exert power over the rest of us.

• Anna Powell-Smith, MissingNumbers.org

Nearly seven decades ago, the noted psychologist Solomon Asch gave a simple task to 123 experimental subjects. They were shown two illustrations, one with three quite different lines and the other with a “reference line,” and Asch asked them to pick which of the three lines was the same length as the reference line. Asch had a trick up his sleeve: he surrounded each subject with stooges who would unanimously pick the wrong line. Confused, the experimental subjects were often (though not always) swayed by the errors of those around them.

The Asch experiments are endlessly fascinating, and I often find myself discussing them in my writing and talks: they are a great starting point for a conversation about the pressure we all feel to conform, and they provide a memorable window into human nature.

Or do they? The experiments are elegant and powerful, but like many psychologists Asch was working with material that came readily to hand: 1950s American college students. We shouldn’t criticize him too much for that; Asch was simply harvesting the low-hanging fruit. It would have been troublesome for him to have collected a representative sample of all Americans, even harder to study an international sample, and impossible for him to have known what the study would have shown if it had been conducted not in 1952 but in 1972. (Others were to run the follow-up experiments, which found somewhat lower levels of conformity—perhaps a sign of student rebelliousness in the Vietnam era.)

It’s all too tempting, however, to act as though Solomon Asch discovered an immutable and universal truth—to discuss the results of psychological experiments on a very specific type of person, in this case 1950s American students, as though they were experiments on the human race as a whole. I am guilty of this myself at times, especially when under the time pressure of a talk. But we should draw conclusions about human nature only after studying a broad range of people. Psychologists are increasingly acknowledging the problem of experiments that study only “WEIRD” subjects—that is, Western, Educated, and from Industrialized Rich Democracies.

By 1996, a Cochrane-style review of the literature found that Asch’s experiment had inspired 133 follow-ups. The overall finding stood up, which is encouraging in light of the previous chapter: conformity is a powerful and widespread effect, though it seemed to have weakened over time. But the obvious next question to ask is this: Does conformity vary in its power depending on who is under pressure to conform to whom?

Disappointingly, the follow-up studies were not very diverse—most had been conducted in the United States and almost all with students—but the few exceptions were illuminating. For example, a 1967 experiment conducted with the Inuit of Baffin Island in Canada found lower levels of conformity than did one conducted with the Temne people of Sierra Leone. I am no anthropologist, but reportedly the Inuit had a relaxed and individualistic culture, while the Temne society had strict social norms, at least at the time that these experiments were conducted. In general—and with some notable exceptions such as Japan—conformity in the Asch-inspired experiments has been lower in societies that sociologists viewed as individualist, and higher in societies viewed as collectivist, where social cohesion is more important.[1]

That implies Asch probably understated the power of conformity by studying subjects from America, an individualistic society. But then, accounts in both psychology textbooks and pop-science books often exaggerate how much conformity Asch found. (Asch’s experimental subjects often rebelled against group pressure. Hardly any of them buckled every single time; much more commonly they tried to equivocate by varying their actions across repeated rounds of the experiment, sometimes agreeing with the group and sometimes staking out a lonely position.) By pure luck, these two biases in the popular understanding of Asch’s findings may have effectively canceled each other out.[2]

How much of the conformity pressure came because the group being studied was a monoculture? Would a more heterogeneous group leave more space for dissent? There are some tantalizing hints of that possibility—for example, follow-up studies found that people conformed to groups of friends much more than they conformed to groups of strangers. And when Asch instructed his stooges to disagree with one another, conformity pressure evaporated: his subjects were happy to pick the correct choice, even if they were the only one doing so, as long as others were disagreeing among themselves. All this suggests that one cure for conformity is to make decisions with a diverse group of people, people who are likely to bring different ideas and assumptions to the table. But this practical tactic is hard to test, as the original experiments and many of the follow-ups were on homogeneous groups. One can’t help feeling that an opportunity has been missed.

It should, I think, make us feel uneasy that most accounts of Asch’s results completely ignore the omission of people who might have acted differently, and whom he could easily have included. Solomon Asch taught in a proudly coeducational institution, Swarthmore College in Pennsylvania. Was it really necessary that not a single one of his experimental participants, neither the stooges nor the subjects, was female?

—

As it happens, follow-up studies suggest that all-male groups are less conformist than all-female groups. So, again, you could say it’s a case of no harm, no foul: Asch might have seen even stronger evidence of conformity had he looked beyond young American males.[3] Still, gender does matter, and Asch could have studied its effects, or at least used mixed-gender groups. But it evidently didn’t occur to him, and it’s discomfiting how few subsequent reports on his experiment seem to care.

If Solomon Asch was the only researcher to have done this, we could wave it away as a historical curiosity. But Asch isn’t alone; of course he isn’t. His student Stanley Milgram conducted a notorious set of electric shock experiments at Yale University in the 1960s. Here’s how I once described his experiments in the Financial Times:[4]

[Milgram] recruited unsuspecting members of the public to participate in a “study of memory.” On showing up at the laboratory, they drew lots with another participant to see who would be “teacher” and who “learner.” Once the learner was strapped into an electric chair, the teacher retreated into another room to take control of a shock machine. As the learner failed to answer questions correctly, the teacher was asked to administer steadily increasing electric shocks. Many proved willing to deliver possibly fatal shocks—despite having received a painful shock themselves as a demonstration, despite the learner having already complained of a heart condition, despite the screams of pain and the pleadings to be released from the other side of the wall, and despite the fact that the switches on the shock machine read “Danger: Severe Shock, XXX.” Of course, there were no shocks—the man screaming from the nearby room was pretending. Yet the research exerts a horrifying fascination.

My article should have mentioned, if only in passing, that all forty of Milgram’s experimental subjects were men. But I wasn’t thinking about that particular issue at the time, and so—as with many others before me—it didn’t occur to me to check.

I hope it would now, because since writing that article I have interviewed Caroline Criado Perez about her book Invisible Women. Meeting her was fun—she strolled into the BBC with an adorable little dog who curled up in the corner of the studio and left us to talk about the gender data gap. Reading her book was less fun, because the incompetence and injustice she described was so depressing—from the makers of protective vests for police officers who forgot that some officers have breasts, to the coders of a comprehensive Apple health app who overlooked that some iPhone users menstruate.[5] Her book argues that all too often, the people responsible for the products and policies that shape our lives implicitly view the default customer—or citizen—as male. Women are an afterthought. Criado Perez argues that the statistics we gather are no exception to this rule: she makes abundantly clear how easy it is to assume that data reflect an impartial “view from nowhere” when in fact they can contain deep and subtle biases.

Consider the historical underrepresentation of women in clinical trials. One grim landmark was thalidomide, which was widely taken by pregnant women to ease morning sickness only for it to emerge that the drug could cause severe disability and death to unborn children. After this disaster, women of childbearing age were routinely excluded from trials, as a precaution. But the precaution makes sense only if one assumes that we can learn most of what we need to know by testing drugs solely in men—a big assumption.[6]

The situation has improved, but many studies still do not disaggregate data to allow an exploration of whether there might be a different effect in men and in women. Sildenafil, for example, was originally intended as a treatment for angina. The clinical trial—conducted on men—revealed an unexpected side effect: magnificent erections. Now better known as Viagra, the drug hit the market as a treatment for erectile dysfunction. But sildenafil might have yet another unexpected benefit: it could be an effective treatment for period pain. We’re not sure, as only one small and suggestive trial has yet been funded.[7] If the trial for angina had equally represented men and women, the potential to treat period pain might have been as obvious as the impact on erections.

These kinds of sex-dependent effects are surprisingly common. One review of drug studies conducted in male and female rodents found that the drug being tested had a sex-dependent effect more than half of the time. For a long time, researchers into muscle-derived stem cells were baffled by why they sometimes regenerated and sometimes didn’t. It seemed entirely arbitrary, until it occurred to someone to check whether the cells came from males or females. Mystery solved: it turned out the cells from females regenerated, while those from males did not.

The gender blind-spot has yet to be banished. A few weeks into the coronavirus pandemic, researchers started to realize that men might be more susceptible than women, both to infection and to death. Was that because of a difference in behavior, in diligent handwashing, in the prevalence of smoking, or perhaps a deep difference in the biology of the male and female immune systems? It wasn’t easy to say, particularly since of the twenty-five countries with the largest number of infections, more than half—including the UK and the United States—did not disaggregate the cases by gender.[8]

A different problem arises when women are included in data-gathering exercises but the questions they are asked don’t fit the man-shaped box in the survey designer’s head. About twenty-five years ago in Uganda, the active labor force suddenly surged by over 10 percent, from 6.5 to 7.2 million people. What had happened? The labor force survey started asking better questions.[9]

Previously, people had been invited to list their primary activity or job, and many women who held down part-time jobs, ran their own market stalls, or put in hours on the family farm simply wrote down “housewife.” The new survey asked about secondary activities, too, and suddenly women mentioned the long hours of paid work they had been doing on the side. Uganda’s labor force increased by 700,000 people, most of them women. The problem was not that the women were ignored by the earlier survey, but that it asked questions that assumed an old-fashioned division of household labor in which the husband did full-time paid work and the wife worked unpaid in the home.

An even subtler gap in the data emerges from the fact that governments often measure the income not of individuals but of households. This is not an unreasonable decision: in a world where many families pool their resources in order to cover rent, food, and sometimes all expenses, the household is a logical unit of analysis. I know several people, men and women, who spend much or most of their time doing unpaid work at home, looking after children, while their partners are earning large salaries. It would be strange to claim that, on the basis that the unpaid partner earns little or no income, they live in poverty.

And yet while many households pool their resources, we cannot simply assume that they all do: money can be used as a weapon within a household, and unequal earnings can empower abusive relationships. Collecting data on household income alone makes such abuse statistically invisible, irrelevant by definition. It is all too tempting to assume that what we do not measure simply does not exist.

As with the Asch experiments, it turns out that we don’t have to speculate that it might matter who within a household controls the purse strings. We have good evidence that it sometimes does. Economist Shelly Lundberg and colleagues studied what happened in the UK when in 1977, child benefit, a regular subsidy to families, was switched from being a tax credit (usually to the father) to a cash payment to the mother. That shift measurably increased spending on women’s and children’s clothes relative to men’s clothes.[10]

When I wrote about Lundberg’s research in the Financial Times, an outraged reader wrote to ask me how I knew that it was better to spend money on women’s and children’s clothes than on men’s clothes. Uncharacteristically for readers of the FT, this person had missed the point: it is not that any spending pattern was better, but that the spending pattern was different. Household income did not change, but when that income was paid to a different person in the household, it was spent on different things. That tells us that measuring income only at the level of the household omits important information. The UK’s new benefit system, Universal Credit, is payable to a single “head of household.” That curiously old-school decision may well favor men—but, given the data we have, it’s going to be hard to tell.

—

It’d be nice to fondly imagine that high-quality statistics simply appear in a spreadsheet somewhere, divine providence from the numerical heavens. Yet any dataset begins with somebody deciding to collect the numbers. What numbers are and aren’t collected, what is and isn’t measured, and who is included or excluded are the result of all-too-human assumptions, preconceptions, and oversights.

The United Nations, for example, has embraced a series of ambitious “Sustainable Development Goals” for 2030. But development experts are starting to call attention to a problem: we often don’t have the data we would need to figure out whether those goals have been met. Are we succeeding in reducing the amount of domestic violence suffered by women? If few countries have chosen to collect good enough data on the problem to allow for historical comparisons, it’s very hard to tell.[11]

Sometimes the choices about what data to gather are just bizarre. Will Moy, director of the fact-checking organization Full Fact, points out that in England, the authorities know more about golfers than about people who are assaulted, robbed, or raped.[12] That’s not because somebody in government sat down with a budget to commission surveys and decided it was more important to understand golf than crime. Instead, surveys tend to be bundled with other projects. Amid the excitement of London’s being awarded the 2012 Summer Olympics, the government launched the Active Lives Survey, which reaches 200,000 people, with enough geographical spread to show which sports are most popular in each local area. That’s why we know so much about golfers.

That’s no bad thing—it’s great to have such a fine-grained picture of how people keep fit. But doesn’t it suggest there’s a case for beefing up the Crime Survey of England and Wales, which reaches just 35,000 households? That’s a large enough survey to understand the national trend in common crimes, but if it were as large as the Active Lives Survey, we might be able to understand trends for rare crimes, smaller demographic groups, or particular towns. Other things being equal, a larger survey can give more precise estimates, especially when you’re trying to count something unusual.

But bigger isn’t always better. It’s perfectly possible to reach vast numbers of people while still missing out on enough other people to get a disastrously skewed impression of what’s really going on.

—

In 1936, Kansas governor Alfred Landon was the Republican nominee for president against the incumbent, President Franklin Delano Roosevelt, a Democrat. A respected magazine, the Literary Digest, shouldered the responsibility of forecasting the result. It conducted an astonishingly ambitious postal opinion poll, which reached 10 million people, a quarter of the electorate. The deluge of mailed-in replies can hardly be imagined, but the Digest seemed to be relishing the scale of the task. In late August it reported, “Next week, the first answers from these ten million will begin the incoming tide of marked ballots, to be triple-checked, verified, five-times cross-classified and totaled.”[13]

After tabulating a remarkable 2.4 million returns as they flowed in over two months, the Literary Digest announced its conclusions: Landon would win by a convincing 55 percent to 41 percent, with a few voters favoring a third candidate.

The election delivered a very different result. Roosevelt crushed Landon by 61 percent to 37 percent. To add to the Literary Digest’s agony, a far smaller survey conducted by the opinion poll pioneer George Gallup came much closer to the final vote, forecasting a comfortable victory for Roosevelt.

Mr. Gallup understood something that the Literary Digest did not: When it comes to data, size isn’t everything. Opinion polls such as Gallup’s are based on samples of the voting population. This means that opinion pollsters need to deal with two issues: sample error and sample bias.

Sample error reflects the risk that, purely by chance, a randomly chosen sample of opinions does not reflect the true views of the population. The “margin of error” reported in opinion polls reflects this risk, and the larger the sample, the smaller the margin of error. A thousand interviews is a large enough sample for many purposes, and during the 1936 election campaign Mr. Gallup is reported to have conducted three thousand interviews.

But if three thousand interviews were good, why weren’t 2.4 million far better? The answer is that sampling error has a far more dangerous friend: sampling bias. Sampling error is when a randomly chosen sample doesn’t reflect the underlying population purely by chance; sampling bias is when the sample isn’t randomly chosen at all. George Gallup took pains to find an unbiased sample because he knew that was far more important than finding a big one.

Literary Digest, in its quest for a bigger dataset, fumbled the question of a biased sample. It mailed out forms to people on a list it had compiled from automobile registrations and telephone directories—a sample that, at least in 1936, was disproportionately prosperous. Those who had a car or a telephone were generally wealthier than those who did not. To compound the problem, Landon supporters turned out more likely to mail in their answers than did those who backed Roosevelt. The combination of those two biases was enough to doom the Literary Digest’s poll. For each person George Gallup’s pollsters interviewed, Literary Digest received eight hundred responses. All that gave the magazine for its pains was a very precise estimate of the wrong answer. By failing to pay enough attention both to the missing people (the ones who were never surveyed) and to the missing responses, the Literary Digest perpetrated one of the most famous polling disasters in statistical history.

—

All pollsters know that their polls are vulnerable to the Literary Digest effect, and the serious ones try—as George Gallup tried—to reach a representative sample of the population. This has never been easy, and it seems to be getting harder: fewer people bother to answer the pollsters’ inquiries, raising the obvious question about whether the ones who do are really representative of everyone else. This is partly because people are less willing to pick up a landline telephone to speak to cold-callers, but that’s not the only explanation. For example, the first British Election Study, a face-to-face survey in which the survey team would knock on people’s doors, had a response rate of nearly 80 percent back in 1963. The 2015 version, also face-to-face, had a response rate of just over 55 percent; in almost half of the homes approached, either nobody opened the door or somebody opened the door but refused to answer the surveyor’s questions.[14]

Pollsters try to correct for this, but there is no foolproof method of doing so. The missing responses are examples of what the statistician David Hand calls “dark data”: we know the people are out there and we know that they have opinions, but we can only guess at what those opinions are. We can ignore dark data, as Asch and Milgram ignored the question of how women would respond in their experiments, or we can try desperately to shine a light on what’s missing. But we can never entirely solve the problem.

In the UK general election of 2015, opinion polls suggested that David Cameron, the incumbent prime minister, was unlikely to win enough votes to stay in power. The polls were wrong: Cameron’s Conservative Party actually gained seats in the House of Commons and secured a narrow victory. It was unclear what had gone wrong, but many polling companies presumed that there had been a last-minute swing in favor of the Conservatives. If only they had conducted a few snap polls at the last possible moment, they might have detected that swing.

But that diagnosis of what had gone wrong was incorrect. Later research showed that the real problem was dark data. Shortly after the election, researchers chose a random sample of houses and knocked on the door to ask people if and how they voted. They got the same answer as the pollsters had: not enough Conservative voters to return Cameron to office. But the pollsters then went back again to the houses where nobody had answered, or where people had turned the surveyors away. On the second attempt, more Conservative voters were in evidence. The pollsters came back to try to fill in the gaps again, and again, and again—sometimes as many as six times—and eventually got an answer from almost everyone they’d originally hoped to talk to. The conclusion: this retrospective poll finally matched the result of the election—a Conservative government.

If the problem had been a late swing, the solution would have been a bunch of quick-and-dirty, last-minute surveys. But because the real problem was that Conservative voters were harder to reach, the real solution may have to be a slower, more exhaustive method of conducting opinion polls.[15]

Both problems hit US pollsters in the notorious 2016 election, when the polls seemed to put Hillary Clinton ahead of Donald Trump in the swing states that would decide the contest. There was a late swing toward Trump, and also the same kind of non-response bias that had doomed the 2015 UK polls: it turned out to have been easier for pollsters to find Clinton supporters than Trump supporters. The polling error was not, objectively speaking, very large. It just loomed large in people’s imagination, perhaps because Trump was such an unusual candidate. But the fact remains that the polls were wrong in part because when the pollsters tried to find a representative group of voters to talk to, too many Trump supporters were missing.[16]

—

One ambitious solution to the problem of sample bias is to stop trying to sample a representative slice of the population and instead speak to everybody. That is what the census attempts to do. However, even census takers can’t assume they have counted everyone. In the 2010 US census, they received responses from just 74 percent of households. That’s a lot of people missing out or opting out.

In the UK 2011 census, the response rate was 95 percent, representing about 25 million households. That’s much better—indeed, it seems almost perfect at first glance. With 25 million households responding, random sample error is not an issue; it will be tiny. But even with just 5 percent of people missing, sample bias is still a concern. The census takers know that certain kinds of people are less likely to respond when the official-looking census form lands with a thud on the doormat: people who live in multiple-occupancy houses such as a shared student house; men in their twenties; people who don’t speak good English. As a result, the 5 percent who don’t respond may look very different from the 95 percent who do. That fact alone is enough to skew the census data.[17]

Census taking is among the oldest ways of collecting statistics. Much newer, but with similar aspirations to reach everyone, is “big data.” Professor Viktor Mayer-Schönberger of Oxford’s Internet Institute, and coauthor of the book Big Data, told me that his favored definition of a big dataset is one where “N = All”—where we no longer have to sample, because we have the entire background population.[18]

One source of big data is so mundane as to be easy to overlook. Think about the data you create when you watch a movie. In 1980 your only option would have been to go to a cinema, probably paying with cash. The only data created would have been box office receipts. In 1990 you could instead have gone to your local video rental store; they might have had a computer to track your rental, or it might have all been done with pen and paper. If it was done on a computer it would probably not have been connected to any broader database. But in the twenty-first century, when you sign up for an account with Netflix or Amazon, your data enter a vast and interconnected world—easily analyzed, cross-referenced, or shared with a data wholesaler, if the terms and conditions allow.

The same is true when you apply for a library card, pay income tax, sign up for a mobile phone contract, or apply for a passport. Once upon a time, such data would have existed as little slips of paper in a giant alphabetical catalog. They weren’t designed for statistical analysis, as a census or a survey would have been. They were administrative building blocks—data gathered in order to get things done. Over time, as administrative data have been digitized and algorithms that can interrogate the data have been improved, it has become ever easier to use them as an input for statistical analysis, a complement to or even a substitute for survey data.

But N = All is often more of a comforting assumption than a fact. As we’ve seen, administrative data will often include information about whoever in the household fills in the forms and pays the bills; the admin-shy will be harder to pin down. And it is all too easy to forget that N = All is not the same as N = Everyone who has signed up for a particular service. Netflix, for example, has copious data about every single Netflix customer, but far less data about people who are not Netflix customers—and it would be perilous for Netflix to generalize from one group to the other.

Even more than administrative data, the lifeblood of big data is “found data”—the kind of data we leave in our wake without even noticing, as we carry our smartphones around, search Google, pay online, tweet our thoughts, post photos to Facebook, or crank up the heat on our smart thermostat. It’s not just the name and credit card details that you gave to Netflix: it’s everything you ever watched on the streaming service, when you watched it—or stopped watching it—and much else besides.

When data like these are opportunistically scraped from cyberspace, they may be skewed in all sorts of awkward ways. If we want to put our finger on the pulse of public opinion, for example, we might run a sentiment analysis algorithm on Twitter rather than going to the expense of commissioning an opinion poll. Twitter can supply every message for analysis, although in practice most researchers use a subset of that vast firehose of data. But even if we analyzed every Twitter message—N = All—we would still learn only what Twitter users think, not what the wider world thinks. And Twitter users are not particularly representative of the wider world. In the United States, for example, they are more likely than the population as a whole to be young, urban, college-educated, and black. Women, meanwhile, are more likely than men to use Facebook and Instagram, but less likely to use LinkedIn. Hispanics are more likely than whites to use Facebook, while blacks are more likely than whites to use LinkedIn, Twitter, and Instagram. None of these facts is obvious.[19]

Kate Crawford, a researcher at Microsoft, has assembled many examples of when N = All assumptions have led people astray. When Hurricane Sandy hit the New York City area in 2012, researchers published an analysis of data from Twitter and a location-based search engine, Foursquare, showing that they could track a spike in grocery shopping the day before and a boom for bars and nightclubs the day after. That’s fine, as far as it goes—but those tweets about the hurricane were disproportionately from Manhattan, whereas areas such as Coney Island had been hit much harder. In fact, Coney Island had been hit so hard the electricity was out—that was why nobody there was tweeting—while densely populated and prosperous Manhattan was unusually saturated with smartphones, at least by 2012 standards, when they were less ubiquitous than today. To make this sort of big-data analysis useful, it takes a considerable effort to disentangle the tweets from the reality.[20]

Another example: In 2012 Boston launched a smartphone app, Street Bump, which used an iPhone’s accelerometer to detect potholes. The idea was that citizens of Boston would download the app and, as they drove around the city, their phones would automatically notify city hall when the road surface needed repair—city workers would no longer have to patrol the streets looking for potholes. It’s a pleasingly elegant idea, and it did successfully find some holes in the road. Yet what Street Bump really produced, left to its own devices, was a map of potholes that systematically favored young, affluent areas where more people owned iPhones and had heard about the app. Street Bump offers us N = All in the sense that every bump from every enabled phone can be recorded. That is not the same thing as recording every pothole. The project has since been shelved.

The algorithms that analyze big data are trained using found data that can be subtly biased. Algorithms trained largely on pale faces and male voices, for example, may be confused when they later try to interpret the speech of women or the appearance of darker complexions. This is believed to help explain why Google photo software confused photographs of people with dark skin with photographs of gorillas; Hewlett-Packard webcams struggled to activate when pointing at people with dark skin tones; and Nikon cameras, programmed to retake photographs if they thought someone had blinked during the shot, kept retaking shots of people from China, Japan, and Korea, mistaking the distinctive East Asian eyelid fold for a blink. New apps, launched in the spring of 2020, promise to listen to you cough and detect whether you have COVID-19 or some other illness. I wonder whether they will do better?[21]

One thing is certain. If algorithms are shown a skewed sample of the world, they will reach a skewed conclusion.[22]

—

There are some overtly racist and sexist people out there—look around—but in general what we count and what we fail to count is often the result of an unexamined choice, of subtle biases and hidden assumptions that we haven’t realized are leading us astray.

Unless we’re collecting data ourselves, there’s a limit to how much we can do to combat the problem of missing data. But we can and should remember to ask who or what might be missing from the data we’re being told about. Some missing numbers are obvious—for example, it’s clearly hard to collect good data about crimes such as sex trafficking or the use of hard drugs. Other omissions show up only when we take a close look at the claim in question. Researchers may not be explicit that an experiment studied only men—such information is sometimes buried in a statistical appendix, and sometimes not reported at all. But often a quick investigation will reveal that the study has a blind spot. If an experiment studies only men, we can’t assume it would have pointed to the same conclusions if it had also included women. If a government statistic measures the income of a household, we must recognize that we’re learning little about the sharing of that income within a household.

Big found datasets can seem comprehensive, and may be enormously useful, but “N = All” is often a seductive illusion: it’s easy to make unwarranted assumptions that we have everything that matters. We must always ask who and what is missing. And this is only one reason to approach big data with caution. Big data represents a huge and underscrutinized change in the way statistics are being collected, and that is where our journey to make the world add up will take us next.

RULE SEVEN

Demand Transparency When the Computer Says No

I know I’ve made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal. I’ve still got the greatest enthusiasm and confidence in the mission, and I want to help you.

• Hal 9000, in 2001: A Space Odyssey

In 2009, a team of researchers from Google announced a remarkable achievement in one of the world’s top scientific journals, Nature.[1] Without needing the results of a single medical checkup, they were able to track the spread of influenza across the United States. What’s more, they could do it more quickly than the Centers for Disease Control and Prevention (CDC), which relied on reports from doctors’ surgeries. Google’s algorithm had searched for patterns in the CDC data from 2003 to 2008, identifying a correlation between flu cases and what people had been searching for online in the same area at the same time. Having discovered the pattern, the algorithm could now use today’s searches to estimate today’s flu cases, a week or more before the CDC published its official view.[2]

Not only was “Google Flu Trends” quick, accurate, and cheap, it was theory-free. Google’s engineers didn’t bother to develop a hypothesis about what search terms might be correlated with the spread of the disease. We can reasonably guess that searches such as “flu symptoms” and “pharmacies near me” might better predict flu cases than searches for “Beyoncé,” but the people on the Google team didn’t care about that. They just fed in their top 50 million search terms and let the algorithms do the work.

The success of Google Flu Trends became emblematic of the hot new trend in business, technology, and science: big data and algorithms. “Big data” can mean many things, but let’s focus on the found data we discussed in the previous chapter, the digital exhaust of web searches, credit card payments, and mobile phones pinging the nearest cell tower, perhaps buttressed by the administrative data generated as organizations organize themselves.

An algorithm, meanwhile, is a step-by-step recipe* for performing a series of actions, and in most cases “algorithm” means simply “computer program.” But over the past few years, the word has come to be associated with something quite specific: algorithms have become tools for finding patterns in large sets of data. Google Flu Trends was built on pattern-recognizing algorithms churning through those 50 million search terms, looking for ones that seemed to coincide with the CDC reporting more cases of flu.

It’s these sorts of data, and these sorts of algorithms, that I’d like to examine in this chapter. “Found” datasets can be huge. They are also often relatively cheap to collect, updated in real time, and messy—a collage of data points collected for disparate purposes. As our communication, leisure, and commerce are moving to the internet, and the internet is moving into our phones, our cars, and even our spectacles, life can be recorded and quantified in a way that would have been hard to imagine just a decade ago. The business bookshelves and the pages of management magazines are bulging with books and articles on the opportunities such data provide.

Alongside a “wise up and get rich” message, cheerleaders for big data have made three exciting claims, each one reflected in the success of Google Flu Trends. First, that data analysis produces uncannily accurate results. Second, that every single data point can be captured—the “N = All” claim we met in the previous chapter—making old statistical sampling techniques obsolete (what that means here is that Flu Trends captured every single search). And finally, that scientific models are obsolete, too: there’s simply no need to develop and test theories about why searches for “flu symptoms” or “Beyoncé” might or might not be correlated with the spread of flu, because, to quote a provocative Wired article in 2008, “with enough data, the numbers speak for themselves.”

This is revolutionary stuff. Yet four years after the original Nature paper was published, Nature News had sad tidings to convey: the latest flu outbreak had claimed an unexpected victim—Google Flu Trends. After reliably providing a swift and accurate account of flu outbreaks for several winters, the theory-free, data-rich model lost its nose for where flu was going. Google’s model pointed to a severe outbreak, but when the slow-and-steady data from the CDC arrived, they showed that Google’s estimates of the spread of flu-like illnesses were overstated—at one point they were more than double the true figure.[3] The Google Flu Trends project was shut down not long after.[4]

What had gone wrong? Part of the problem was rooted in that third exciting claim: Google did not know—and it could not begin to know—what linked the search terms with the spread of flu. Google’s engineers weren’t trying to figure out what caused what. They were merely finding statistical patterns in the data, which is what these algorithms do. In fact, the Google research team had peeked into the patterns and discovered some obviously spurious correlations that they could safely instruct the algorithm to disregard—for example, flu cases turned out to be correlated with searches for “high school basketball.” There’s no mystery about why: both flu and high school basketball tend to get going in the middle of November. But it meant that Flu Trends was part flu detector, part winter detector.[5] That became a problem when there was an outbreak of summer flu in 2009: Google Flu Trends, eagerly scanning for signs of winter and finding nothing, missed the nonseasonal outbreak, as true cases were four times higher than Flu Trends was estimating.[6]

The “winter detector” problem is common in big-data analysis. A literal example, via computer scientist Sameer Singh, is the pattern-recognizing algorithm that was shown many photos of wolves in the wild, and many photos of pet husky dogs. The algorithm seemed to be really good at distinguishing the two rather similar canines; it turned out that it was simply labeling any picture with snow as containing a wolf. An example with more serious implications was described by Janelle Shane in her book You Look Like a Thing and I Love You: an algorithm that was shown pictures of healthy skin and of skin cancer. The algorithm figured out the pattern: if there was a ruler in the photograph, it was cancer.[7] If we don’t know why the algorithm is doing what it’s doing, we’re trusting our lives to a ruler detector.

Figuring out what causes what is hard—impossible, some say. Figuring out what is correlated with what is much cheaper and easier. And some big-data enthusiasts—such as Chris Anderson, author of that provocative article in Wired magazine—have argued that it is pointless to look beyond correlations. “View data mathematically first and establish a context for it later,” he wrote; the numbers speak for themselves. To rephrase Anderson’s point unkindly, “If searches for high school basketball always pick up at the same time as flu cases, it doesn’t really matter why.”

But it does matter, because a theory-free analysis of mere correlations is inevitably fragile. If you have no idea what is behind a correlation, you have no idea what might cause that correlation to break down.

After the summer flu problem of 2009, the accuracy of Flu Trends collapsed completely at the end of 2012. It’s not clear why. One theory is that the news was full of scary stories about flu in December 2012, and these stories might have provoked internet searches from people who were healthy. Another possible explanation is a change in Google’s own search algorithm: it began automatically suggesting diagnoses when people entered medical symptoms, and this will have changed what they typed into Google in a way that might have foxed the Flu Trends model. It’s quite possible that Google could have figured out what the problem was and found a way to make the algorithm work again if they’d wanted to, but they just decided that it wasn’t worth the trouble, expense, and risk of failure.

Or maybe not. The truth is, external researchers have been forced to guess at exactly what went wrong, because they don’t have the information to know for sure. Google shares some data with researchers, and indeed makes some data freely available to anyone. But it isn’t going to release all its data to you, or me, or anyone else.

—

Two good books with pride of place on my bookshelf tell the story of how our view of big data evolved over just a few short years.

One, published in 2013, is Big Data by Kenneth Cukier and Viktor Mayer-Schönberger. It reports many examples of how cheap sensors, huge datasets, and pattern-recognizing algorithms were, to paraphrase the book’s subtitle, Transforming How We Live, Work, and Think. The triumphant example the authors chose to begin their story? Google Flu Trends. The collapse became apparent only after the book had gone to print.

Three years later, in 2016, came Cathy O’Neil’s Weapons of Math Destruction, which—as you might be able to guess—takes a far more pessimistic view. O’Neil’s subtitle tells us that big data “increases inequality and threatens democracy.”

The difference is partly one of perspective: Cukier and Mayer-Schönberger tend to adopt the viewpoint of someone who is doing something with a data-driven algorithm; O’Neil tends to see things from the viewpoint of someone to whom a data-driven algorithm is doing something. A hammer looks like a useful tool to a carpenter; the nail has a different impression altogether.

But the change in tone also reflects a change in the zeitgeist between 2013 and 2016. In 2013, the relatively few people who were paying attention to big data often imagined themselves to be the carpenters; by 2016, many of us had realized that we were nails. Big data went from seeming transformative to seeming sinister. Cheerleading gave way to doomsaying, and some breathlessly over-the-top headlines. (Perhaps my favorite was a CNN story: “Math Is Racist.”) The crisis reached a shrill pitch with the discovery that a political consulting firm, Cambridge Analytica, had exploited Facebook’s lax policies on data to slurp up information on about 50 million people, without those people knowing or meaningfully consenting to this, and show them personally targeted advertisements. It was briefly supposed by horrified commentators that these ads were so effective they essentially elected Donald Trump, though more sober analysis later concluded that Cambridge Analytica’s capabilities fell short of mind control.[8]

Each of us is sweating data, and those data are being mopped up and wrung out into oceans of information. Algorithms and large datasets are being used for everything from finding us love to deciding whether, if we are accused of a crime, we go to prison before the trial or are instead allowed to post bail. We all need to understand what these data are and how they can be exploited. Should big data excite or terrify us? Should we be more inclined to cheer the carpenters or worry about our unwitting role as nails?

The answer is that it depends—and in this chapter I hope to show you what it depends on.

—

Writing in the New York Times Magazine in 2012 when the zeitgeist was still firmly with the carpenters, the journalist Charles Duhigg captured the hype about big data quite brilliantly with an anecdote about the discount department store Target.

Duhigg explained that Target had collected so much data on its customers and was so skilled at analyzing those data that its insights into consumers could seem like magic.[9] His killer anecdote was of the man who stormed into a Target store near Minneapolis and complained to the manager that the company was sending coupons for baby clothes and maternity wear to his teenage daughter. The manager apologized profusely, and later called to apologize again—only to be told that the teenager was indeed pregnant. Her father hadn’t realized. Target, after analyzing her purchases of unscented wipes and vitamin supplements, had.

Yet is this truly statistical sorcery? I’ve discussed this story with many people and I’m struck by a disparity. Most people are wide-eyed with astonishment. But two of the groups I hang out with a lot take a rather different view. Journalists are often cynical; some suspect Duhigg of inventing, exaggerating, or passing on an urban myth. (I suspect them of professional jealousy.) Data scientists and statisticians, on the other hand, yawn. They regard the tale as both unsurprising and uninformative. And I think the statisticians have it right.

First, let’s think for a moment about just how amazing it might be to predict that someone is pregnant based on their shopping habits: not very. Consider the National Health Service advice on the vitamin supplement folic acid:

It’s recommended that all women who could get pregnant should take a daily supplement of 400 micrograms of folic acid before they’re pregnant and during the first 12 weeks of pregnancy . . . If you did not take folic acid supplements before getting pregnant, you should start taking them as soon as you find out you’re pregnant . . . The only way to be sure you’re getting the right amount is by taking a supplement.

OK. With this in mind, what conclusion should I reach if I am told that a woman has started buying folic acid? I don’t need to use a vast dataset or a brilliant analytical process. It’s not magic. Clearly, the woman might well be pregnant. The Target algorithm hadn’t produced a superhuman leap of logic, but a very human one: it figured out exactly what you or I or anyone else would also have figured out, given the same information.

Sometimes we humans can be slow on the uptake. Hannah Fry, the author of another excellent book about algorithms, Hello World, relays the example of a woman shopping online at the UK supermarket Tesco.[10] She discovered condoms in the “Buy this again?” section of her online shopping cart—implying that the algorithm knew someone in her household had bought them before. But she hadn’t, and her husband had no reason to: they didn’t use condoms together. So she assumed it was a technical error. I mean, what other explanation could there possibly be?

When the woman contacted Tesco to complain, the company representatives concluded that it wasn’t their job to break the bad news that her husband was cheating on her, and went for the tactful white lie. “Indeed, madam? A computer error? You’re quite right, that must be the reason. We are so sorry for the inconvenience.” Fry tells me that this is now the rule of thumb at Tesco: apologize and blame the computer.

If customers previously bought condoms, they might want to buy them again. If someone bought a pregnancy test and then starts buying a vitamin supplement designed for pregnant women, it’s a reasonable bet that this person is a woman and that in a few months she might become interested in buying maternity wear and baby clothes. The algorithms aren’t working statistical miracles here. They’re simply seeing something (the condoms, the pregnancy vitamins) that has been concealed from the human (the puzzled wife, the angry dad). We’re awestruck by the algorithm in part because we don’t appreciate the mundanity of what’s happening underneath the magician’s silk handkerchief.

And there’s another way in which Duhigg’s story of Target’s algorithm invites us to overestimate the capabilities of data-fueled computer analytics.

“There’s a huge false positive issue,” says Kaiser Fung, a data scientist who has spent years developing similar approaches for retailers and advertisers. What Fung means is that we don’t get to hear stories about women who receive coupons for baby wear but who aren’t pregnant. Hearing the anecdote, it’s easy to assume that Target’s algorithms are infallible—that everybody receiving coupons for onesies and wet wipes is pregnant. But nobody ever claimed that it was true. And it almost certainly isn’t; maybe everybody received coupons for onesies. We should not simply accept the idea that Target’s computer is a mind reader before considering how many misses attend each hit.

There may be many of those misses—even for an easy guess such as “woman buying folic acid may be pregnant.” Folic acid purchases don’t guarantee pregnancy. The woman might be taking folic acid for some other reason. Or she might be buying the vitamin supplement for someone else. Or—and imagine the distress when the vouchers for baby clothes start arriving—she might have been pregnant but miscarried, or trying to get pregnant without success. It’s possible that Target’s algorithm is so brilliant that it filters out these painful cases. But it is not likely.

In Charles Duhigg’s account, Target mixes in random offers, such as coupons for wineglasses, because pregnant customers would feel spooked if they realized how intimately the company’s computers understood them. But Kaiser Fung has another explanation: Target mixes up its offers not because it would be weird to send an all-baby coupon book to a woman who was pregnant, but because the company knows that many of those coupon books will be sent to women who aren’t pregnant after all.

The manager should simply have said that: Don’t worry about it, lots of people get these coupons. Why didn’t he? He probably didn’t understand the Target algorithm any more than the rest of us. Just like Google, Target might be reluctant to open up its algorithm and dataset for researchers—and competitors—to understand what’s going on.

The most likely situation is this: pregnancy is often a pretty easy condition to spot through shopping behavior, so Target’s big-data-driven algorithm surely can predict pregnancy better than random guessing. Doubtless, however, it is well short of infallibility. A random guess might be that any woman between fifteen and forty-five has a roughly 5 percent chance of being pregnant at any time. If Target can improve to guessing correctly 10 or 15 percent of the time, that’s already likely to be well worth doing. Even a modest increase in the accuracy of targeted special offers would help the bottom line. But profitability should not be conflated with omniscience.

So let’s start by toning down the hype a little—both the apocalyptic idea that Cambridge Analytica can read your mind, and the giddy prospect that big data can easily replace more plodding statistical processes such as the CDC’s survey of influenza cases. When I first started grappling with big data, I called Cambridge University professor David Spiegelhalter, one of the UK’s leading statisticians and a brilliant statistical communicator. I summarized the cheerleading claims: of uncanny accuracy, of making sampling irrelevant because every data point was captured, and of consigning scientific models to the junk heap because “the numbers speak for themselves.”

He didn’t feel the need to reach for a technical term. Those claims, he said, are “complete bollocks. Absolute nonsense.”

Making big data work is harder than it seems. Statisticians have spent the past two hundred years figuring out what traps lie in wait when we try to understand the world through data. The data are bigger, faster, and cheaper these days, but we must not pretend that the traps have all been made safe. They have not.

“There are a lot of small data problems that occur in big data,” added Spiegelhalter. “They don’t disappear because you’ve got lots of the stuff. They get worse.”

It hardly matters if some of Charles Duhigg’s readers are too credulous about the precision with which Target targets onesie coupons. But it does matter when people in power are similarly overawed by algorithms that they don’t understand, and use them to make life-changing decisions.

One of Cathy O’Neil’s most powerful examples in Weapons of Math Destruction is the IMPACT algorithm used to assess teachers in Washington, DC. As O’Neil describes, much-loved and highly respected teachers in the city’s schools were being abruptly sacked after achieving very poor ratings from the algorithm.

The IMPACT algorithm claimed to measure the quality of teaching, basically by checking whether the children in a teacher’s class had made progress or slipped backward in their test scores.[11] Measuring the true quality of teaching, however, is hard to do for two reasons. The first is that no matter how good or how bad the teacher, individual student achievement will vary a lot. With just thirty students in a class, much of what the algorithm measures will be noise; if a couple of kids made some lucky guesses in their start-of-year test and then were unlucky at the end of year, that’s enough to make a difference in a teacher’s ranking. It shouldn’t be, because it’s pure chance. Another source of variation that’s out of the teacher’s control is a child’s having a serious problem outside the classroom—anything from illness to bullying to a family member being imprisoned. This isn’t the same kind of noise as lucky or unlucky guesses on a test, because it tracks something real. A system that tracked and followed up signs of trouble outside class would be valuable. But it would be foolish and unfair to blame the pupil’s struggles on the teacher.

The second problem is that the algorithm may be fooled by teachers who cheat, and the cheats will damage the prospects of the honest teachers. If the sixth-grade teacher finds a way to unfairly boost the test performance of the children he teaches—such things aren’t unheard of—then not only will he be unfairly rewarded, but the seventh-grade teacher is going to be in serious trouble next year. Her incoming class will be geniuses on paper; improvement will be impossible unless she also finds a way to cheat.

O’Neil’s view, which is plausible, is that the data are so noisy that the task of assessing a teacher’s competence is hopeless for any algorithm. Certainly this particular algorithm’s judgments on which teachers were subpar didn’t always tally with those of the teachers’ colleagues or students. But that didn’t stop the DC school district authorities’ firing 206 teachers in 2011 for failing to meet the algorithm’s standards.

—

So far we’ve focused on excessive credulity in the power of the algorithm to extract wisdom from the data it is fed. There’s another, related problem: excessive credulity in the quality or completeness of the dataset.

We explored the completeness problem in the previous chapter. Literary Digest accumulated what might fairly be described as big data. It was certainly an enormous survey by the standards of the day—indeed, even by today’s standards a dataset with 2.4 million people in it is impressive. But you can’t use Literary Digest surveys to predict election results if “people who respond to Literary Digest surveys” differ in some consistent way from “people who vote in elections.”

Google Flu Trends captured every Google search, but not everybody who gets the flu turns to Google. Its accuracy depended on “people with flu who consult Google about it” not being systematically different from “people with flu.” The pothole-detecting app we met in the previous chapter fell short because it confused “people who hear about and install pothole-detecting apps” with “people who drive around the city.”

How about quality? Here’s an instructive example of big data from an even older vintage than the 1936 US election poll: the astonishing attempt to assess the typical temperature of the human body. Over the course of eighteen years, the nineteenth-century German doctor Carl Wunderlich assembled over a million measurements of body temperature, gathered from more than 25,000 patients. A million measurements! It’s a truly staggering achievement given the pen-and-paper technology of the day. Wunderlich is the man behind the conventional wisdom that normal body temperature is 98.6°F. Nobody wanted to gainsay his findings, partly because the dataset was large enough to command respect, and partly because the prospect of challenging it with a bigger, better dataset was intimidating. As Dr. Philip Mackowiak, an expert on Wunderlich, put it, “Nobody was in a position or had the desire to amass a dataset that large.”[12]

Yet Wunderlich’s numbers were off; we’re normally a little cooler (by about half a Fahrenheit degree).[13] So formidable were his data that it took more than a hundred years to establish that the good doctor had been in error.*

So how could so large a dataset be wrong? When Dr. Mackowiak discovered one of Carl Wunderlich’s old thermometers in a medical museum, he was able to inspect it. He found that it was miscalibrated by two degrees centigrade, almost four degrees Fahrenheit. This error was partly offset by Dr. Wunderlich’s habit of taking the temperature of the armpit rather than carefully inserting the thermometer into one of the bodily orifices conventionally used in modern times. You can take a million temperature readings, but if your thermometer is broken and you’re poking around in armpits, then your results will be a precise estimate of the wrong answer. The old cliché of “Garbage in, garbage out” remains true no matter how many scraps of garbage you collect.

As we saw in the previous chapter, the modern version of this old problem is an algorithm that has been trained on a systematically biased dataset. It’s surprisingly easy for such problems to be overlooked. In 2014, Amazon, one of the most valuable companies in the world, started using a data-driven algorithm to sift résumés, hoping that the computer would find patterns and pick out the very best people, based on their similarity to previous successful applicants. Alas, previous successful applicants were disproportionately men. The algorithm then did what algorithms do: it spotted the pattern and ran with it. Observing that men had in the past been preferred, it concluded that men were preferable. The algorithm penalized the word “women’s,” as in “Under-21 Women’s Soccer International” or “Women’s Chess Club Captain”; it downgraded certain all-women’s colleges. Amazon abandoned the algorithm in 2018; it’s unclear exactly how much influence it had in making decisions, but Amazon admitted that its recruiters had been looking at the algorithm’s rankings.

Remember the “Math Is Racist” headline? I’m fairly confident that math isn’t racist. Neither is it misogynistic, or homophobic, or biased in other ways. But I’m just as confident that some humans are. And computers trained on our own historical biases will repeat those biases at the very moment we’re trying to leave them behind us.[14]

—

I hope I’ve persuaded you that we shouldn’t be too eager to entrust our decisions to algorithms. But I don’t want to overdo the critique, because we don’t have some infallible alternative way of making decisions. The choice is between algorithms and humans. Some humans are prejudiced. Many humans are frequently tired, harassed, and overworked. And all humans are, well, human.

In the 1950s, the psychologist Paul Meehl investigated whether the most basic of algorithms, in the form of uncomplicated statistical rules, could ever outperform expert human judgment. For example, a patient arrives at the hospital complaining of chest pains. Does she have indigestion, or is she suffering from a heart attack? Meehl compared the verdicts of experienced doctors with the result of working through a brief checklist. Is chest pain the main symptom? Has the patient had heart attacks in the past? Has the patient used nitroglycerin to relieve chest pain in the past? What quantifiable patterns are shown on the cardiogram?[15] Disconcertingly, that simple decision tree got the right diagnosis more often than the doctors. And it was not the only example. Remarkably often, Meehl found, experts fared poorly when compared with simple checklists. Meehl described his Clinical vs. Statistical Prediction as “my disturbing little book.”[16]

So, to be fair, we should compare the fallibility of today’s algorithms with that of the humans who would otherwise be making the decisions. A good place to start is with an example from Hannah Fry’s book Hello World.

The story starts during the London riots of 2011. Initially a protest against police brutality, demonstrations turned into violent riots as order broke down each evening across the city, and in several other cities across the country. Shops would close in the early afternoon and law-abiding citizens would hurry home, knowing that opportunistic troublemakers would be out as the light faded. In three days of trouble, more than a thousand people were arrested.

Among their number were Nicholas Robinson and Richard Johnson. Robinson was walking through the chaos and helped himself to a pack of bottled water from a shattered London supermarket. Johnson drove to a gaming store, put on a balaclava, and ran in to grab an armful of computer games. Johnson’s theft was of higher value and was premeditated rather than on the spur of the moment. And yet it was Robinson who received a six-month sentence, while Johnson’s conviction earned him no jail time at all. No algorithm could be blamed for the difference; human judges were the ones handing out those sentences, and the disparity seems bizarre.

It’s always possible that each judge made the right decision, based on some subtle detail of the case. But the most plausible answer for the inconsistent treatment of the two men was that Robinson was sentenced just two weeks after the riots, at a time when nerves were jangling and the fabric of civilization seemed easily ripped. Johnson was sentenced months later, when the memory of the riots was fading and people were asking themselves what all the fuss had been about.[17]

Would a data-driven computer program have tuned out the mood music and delivered fairer sentences? It’s impossible to know—but quite possibly, yes. There is ample evidence that human judges aren’t terribly consistent. One way to test this is to show hypothetical cases to various judges and see if they reach different conclusions. They do. In one British study from 2001, judges were asked for judgments on a variety of cases; some of the cases (presented a suitable distance apart to disguise the subterfuge) were simply repeats of earlier cases, with names and other irrelevant details changed. The judges didn’t even agree with their own previous judgment on the identical case. That is one error that we can be fairly sure a computer would not make.[18]

A more recent study was conducted in the United States, by economist Sendhil Mullainathan and four colleagues. They analyzed over 750,000 cases in New York City between 2008 and 2013—cases in which someone had been arrested and the decision had to be made to release the defendant, detain the person, or set a cash bail that had to be posted to secure release. The researchers could then see who had gone on to be rearrested.[19] They then used a portion of these cases (220,000) to train an algorithm to decide whether to release, detain, or set bail. And they used the remaining cases to check whether the algorithm had done a good job or not, relative to human judges.[20]

The humans did not do well. The researchers’ algorithm could have reduced crime-while-on-release by almost 25 percent by jailing a better-selected group of defendants. Alternatively, they could have jailed 40 percent fewer people without any increase in crime. Thousands of crimes could have been prevented, or thousands of people released pending trial, purely as a result of the algorithm outperforming the human judges.

One important error that the judges make is what American legal scholar Cass Sunstein calls “current offense bias”—that is, when they make decisions about bail, they focus too much on the specific offense the defendant has been accused of. Defendants whose track record suggests they’re a high risk are treated as low risk if they’re accused of a minor crime, and defendants whose track record suggests they’re low risk are treated as high risk if the current offense is serious. There’s valuable information here that the algorithm puts to good use, but the human judges—for all their intelligence, experience, and training—tend to overlook it.

This seems to be how we humans operate. Consider the way I described the cases of Nicholas Robinson and Richard Johnson: I told you about the offenses in question, nothing at all about Robinson and Johnson. It just seemed reasonable to me—and perhaps to you—to tell you all about the short term, about the current offense. An algorithm would have used more information if more information had been available. A human might not.

Many people have strong intuitions about whether they would rather have a vital decision about them made by algorithms or humans. Some people are touchingly impressed by the capabilities of the algorithms; others have far too much faith in human judgment. The truth is that sometimes the algorithms will do better than the humans, and sometimes they won’t. If we want to avoid the problems and unlock the promise of big data, we’re going to need to assess the performance of the algorithms on a case-by-case basis. All too often, this is much harder than it should be.

Consider this scenario. The police, or social services, receive a call from someone—a neighbor, a grandparent, a doctor, a teacher—who is worried about the safety of a child. Sometimes the child will genuinely be in danger; sometimes the caller will be mistaken, or overanxious, or even malicious. In an ideal world, we’d take no chances and send a blue-lighted vehicle around immediately to check what’s going on. But we don’t have enough resources to do this in every case—we have to prioritize. The stakes could hardly be higher: official figures in the United States show that 1,670 children died in 2015 of abuse or neglect. That’s a horrific number, but a tiny fraction of the 4 million times someone calls to report their concerns about a child.

Which reports need to be followed up, and which can be reasonably ignored? Many police and social services use algorithms to help make that decision. The state of Illinois introduced just such an algorithm, called Rapid Safety Feedback. It analyzed data on each report, compared them with the outcomes of previous cases, and produced a percentage prediction of the child’s risk of death or serious harm.

The results were not impressive. The Chicago Tribune reported that the algorithm gave 369 children a 100 percent chance of serious injury or death. No matter how dire the home environment, that degree of certitude seems unduly pessimistic. It could also have grave implications: a false allegation of child neglect or abuse could have terrible consequences for the accused and the child alike.

But perhaps the algorithm erred on the side of caution, exaggerating the risk of harm because it was designed not to miss a single case? No: in some terrible cases toddlers died after being given a percentage of risk too low to justify follow-up. In the end, Illinois decided the technology was useless, or worse, and stopped using it.[21]

The moral of this story isn’t that algorithms shouldn’t be used to assess reports about vulnerable children. Someone or something must make the decision on which cases to follow up. Mistakes are inevitable, and there’s no reason—in principle—why some other algorithm might not make fewer mistakes than a human call handler.[22] The moral is that we know about the limitations of this particular algorithm only because it spat out explicit numbers that were obviously absurd.

“It’s good they gave numerical probabilities, as this supplies the loud siren that makes us realize that these numbers are bad,” explains statistician Andrew Gelman. “What would be worse is if [the algorithm] had just reported the predictions as ‘high risk,’ ‘mid risk’ and ‘low risk.’” The problems might then never have come to light.[23]

So the problem is not the algorithms, or the big datasets. The problem is a lack of scrutiny, transparency, and debate. And the solution, I’d argue, goes back a very long time.

—

In the mid-seventeenth century, a distinction began to emerge between alchemy and what we’d regard as modern science. It is a distinction that we need to remember if we are to flourish in a world of big-data algorithms.

In 1648, Blaise Pascal’s brother-in-law, at the urging of the great French mathematician, conducted a celebrated experiment. In the garden of a monastery in the little city of Clermont-Ferrand, he took a tube filled with mercury, slid its open end into a bowl full of the liquid metal, and lifted it to a vertical position, jutting above the surface with the end submerged. Some of the mercury immediately drained into the bowl, but some did not. There was a column 711 millimeters high in the tube, and above it a space containing—what? Air? A vacuum? A mysterious ether?[24]

This was only the first stage of the experiment Pascal had proposed, and it wasn’t unprecedented. Gasparo Berti had done something similar in Rome with water—although with water, the glass tube needs to be more than 10 meters long, and making one was no easy task. Evangelista Torricelli, a student of Galileo, was the man who had the idea of using mercury instead, which requires a much shorter tube.

Pascal’s idea—or perhaps it was his friend René Descartes’s, since both of them claimed the credit—was to repeat the experiment at altitude. And so it was Pascal’s brother-in-law who had the job of lugging fragile glass tubes and several kilograms of mercury up to the top of Puy de Dôme, a striking dormant volcano in the heart of France, more than a kilometer above Clermont-Ferrand. At the top of the mountain, the mercury rose not 711 millimeters but just 627. Halfway down the mountain, the mercury column was longer than at the summit, but shorter than down in the garden. The next day, the column was measured at the top of Clermont-Ferrand’s cathedral. It was 4 millimeters shorter there than in the monastery garden. Pascal had invented what we now call the barometer—and simultaneously, the altimeter, a device that measured air pressure and, indirectly, altitude. In 1662, just fourteen years later, Robert Boyle formulated his famous gas law, describing the relationship between pressure and the volume of a gas. This was a rapid and rather modern advance in the state of scientific knowledge.

Yet it was taking place alongside the altogether more ancient practice of alchemy, the quest to find a way to turn base metals into gold and to produce an elixir of eternal life. These goals are, as far as we know, as near to impossible as makes no difference*—but if alchemy had been conducted using scientific methods, one might still have expected all the alchemical research to produce a rich seam of informative failures, and a gradual evolution into modern chemistry.

That’s not what happened. Alchemy did not evolve into chemistry. It stagnated, and in due course science elbowed it to one side. For a while the two disciplines existed in parallel. So what distinguished them?

Of course, modern science uses the experimental method, so clearly demonstrated by Pascal’s hardworking brother-in-law, by Torricelli, Boyle, and others. But so did alchemy. The alchemists were unrelenting experimenters. It’s just that their experiments yielded no information that advanced the field as a whole. The use of experiments does not explain why chemistry flourished and alchemy died.

Perhaps, then, it was down to the characters involved? Perhaps the great early scientists such as Robert Boyle and Isaac Newton were sharper, wiser, more creative men than the alchemists they replaced? This is a spectacularly unpersuasive explanation. Two of the leading alchemists of the 1600s were Robert Boyle and Isaac Newton. They were energetic, even fervent, practitioners of alchemy, which thankfully did not prevent their enormous contributions to modern science.[25]

No, the alchemists were often the very same people using the same experimental methods to try to understand the world around them. What accounts for the difference, says David Wootton, a historian of science, is that alchemy was pursued in secret, while science depended on open debate. In the late 1640s, a small network of experimenters across France, including Pascal, worked simultaneously on vacuum experiments. At least a hundred people are known to have performed these experiments between Torricelli’s in 1643 and the formulation of Boyle’s Law in 1662. “These hundred people are the first dispersed community of experimental scientists,” says Wootton.[26]

At the center of the web of knowledge was Marin Mersenne—a monk, a mathematician, and a catalyst for scientific collaboration and open competition. Mersenne was friends with Pascal and Descartes, along with thinkers from Galileo to Thomas Hobbes, and would make copies of the letters he received and circulate them to others whom he thought would be interested. So prolific was his correspondence that he became known as “the post-box of Europe.”[27]

Mersenne died in 1648, less than three weeks before the experiment on Puy de Dôme, but his ideas about scientific collaboration lived on in the form of the Royal Society in London (established 1660) and the French Academy of Sciences (established 1666), both operating along distinctly Mersennian lines. One of the virtues of the new approach, well understood at the time, was reproducibility—which, as we saw in chapter five, is a vital check on both fraud and error. The Puy de Dôme experiment could be and was repeated anywhere there was a hill or even a tall building. “All the curious can test it themselves whenever they like,” wrote Pascal. And they did.

Yet while the debate over vacuums, gases, and those tubes of mercury was being vigorously carried out through letters, publications, and meetings at Mersenne’s home in Paris, alchemical experiments were conducted in secret. It isn’t hard to see why: There is no value in turning lead into gold if everyone knows how to do it. No alchemist wanted to share his potentially instructive failures with anyone else.

The secrecy was self-perpetuating. One of the reasons that alchemy lasted so long, and that even brilliant scholars such as Boyle and Newton took it seriously, was the assumption that alchemical problems had been solved by previous generations, but were kept secret and then lost. When Newton famously declared, “If I have seen further it is by standing on the shoulders of giants,” this was true only of his scientific work. As an alchemist, he stood on nobody’s shoulders and saw little.

When Boyle did try to publish some of his findings, and seek out other alchemists, Newton warned him to stop and instead maintain “high silence.” And as it became clear that the newly open scientific community was making rapid progress, alchemy itself became discredited within a generation. In short, says Wootton:

What killed alchemy was the insistence that experiments must be openly reported in publications which presented a clear account of what had happened, and they must then be replicated, preferably before independent witnesses. The alchemists had pursued a secret learning . . . some parts of that learning could be taken over by . . . the new chemistry, but much of it had to be abandoned as incomprehensible and unreproducible. Esoteric knowledge was replaced by a new form of knowledge which depended both on publication and on public or semi-public performance.[28]

Alchemy is not the same as gathering big datasets and developing pattern-recognizing algorithms. For one thing, alchemy is impossible, and deriving insights from big data is not. Yet the parallels should also be obvious. The likes of Google and Target are no more keen to share their datasets and algorithms than Newton was to share his alchemical experiments. Sometimes there are legal or ethical reasons—if you’re trying to keep your pregnancy a secret, you don’t want Target publicly disclosing your folic acid purchases—but most obviously the reasons are commercial. There’s gold in the data that Amazon, Apple, Facebook, Google, and Microsoft have about us. And that gold will be worth a lot less to them if the knowledge that produces it is shared with everyone.

But just as the most brilliant thinkers of the age failed to make progress while practicing in secret, secret algorithms based on secret data are likely to lead to missed opportunities for improvement. Again, it hardly matters much if Target is missing out on a slightly more effective way to target onesie coupons. But when algorithms are firing capable teachers, directing social services to the wrong households, or downgrading job applicants who went to women’s colleges, we need to be able to subject them to scrutiny.

But how?

One approach is that used by a team of investigative journalists at ProPublica, led by Julia Angwin. Angwin’s team wanted to scrutinize a widely used algorithm called COMPAS (Correctional Offender Management Profiling for Alternative Sanctions). COMPAS used the answers to a 137-item questionnaire to assess the risk that a criminal might be rearrested. But did it work? And was it fair?

It wasn’t easy to find out. COMPAS is owned by a company, Equivant (formerly Northpointe), that is under no obligation to share the details of how it works. And so Angwin and her team had to judge it by analyzing the results, laboriously pulled together from Broward County in Florida, a state that has strong transparency laws.

Here’s an edited account of how the ProPublica team went about their work:

Through a public records request, ProPublica obtained two years’ worth of COMPAS scores from the Broward County Sheriff’s Office in Florida. We received data for all 18,610 people who were scored in 2013 and 2014 . . . Each pretrial defendant received at least three COMPAS scores: “Risk of Recidivism,” “Risk of Violence” and “Risk of Failure to Appear.” COMPAS scores for each defendant ranged from 1 to 10, with ten being the highest risk. Scores 1 to 4 were labeled by COMPAS as Low; 5 to 7 were labeled Medium; and 8 to 10 were labeled High. Starting with the database of COMPAS scores, we built a profile of each person’s criminal history, both before and after they were scored. We collected public criminal records from the Broward County Clerk’s Office website through April 1, 2016. On average, defendants in our dataset were not incarcerated for 622.87 days (sd: 329.19). We matched the criminal records to the COMPAS records using a person’s first and last names and date of birth . . . We downloaded around 80,000 criminal records from the Broward County Clerk’s Office website.[29]

And so it continues. This was painstaking work.

Eventually, ProPublica published its conclusions. The COMPAS algorithm did not use an offender’s race as a predictor, yet it nevertheless was producing racially disparate results. It tended to produce false positives for black offenders (predicting that they would be rearrested, but then they were not) and false negatives for white offenders (predicting that they would not be rearrested, but then they were).

That sounds very worrying: racial discrimination is both immoral and illegal when coming from a human; we shouldn’t tolerate it if it emerges from an algorithm.

But then four academic researchers, Sam Corbett-Davies, Emma Pierson, Avi Feller, and Sharad Goel, pointed out that the situation wasn’t so clear-cut.[30] They used the data laboriously assembled by ProPublica to show that the algorithm was fair by another important metric, which was that if the algorithm gave two criminals—one black, one white—the same risk rating, then the actual risk that they would be rearrested was the same. In that important respect the algorithm was color-blind.

What’s more, the researchers showed that it was impossible for the algorithm to be fair in both ways simultaneously. It was possible to craft an algorithm that would give an equal rate of false positives for all races, and it was possible to craft an algorithm where the risk ratings matched the risk of rearrest for all races, but it wasn’t possible to do both at the same time: the numbers just couldn’t be made to add up.

The only way in which an algorithm could be constructed to produce equal results for different groups—whether those groups were defined by age, gender, race, hair color, height, or any other criterion—would be if the groups otherwise behaved and were treated identically. If they moved through the world in different ways, the algorithm would, inevitably, violate at least one criterion of fairness when evaluating them. That is true whether or not the algorithm was actually told their age, gender, race, hair color, or height. It would also be true of a human judge; it’s a matter of arithmetic.

Julia Dressel and Hany Farid, also computer scientists, observed this debate over whether COMPAS was producing results with a racial bias. They thought there was something missing. “There was this underlying assumption in the conversation that the algorithm’s predictions were inherently better than human ones,” Dressel told the science writer Ed Yong, “but I couldn’t find any research proving that.”[31]

Thanks to ProPublica’s spadework, Dressel and Farid could investigate the question for themselves. Even if COMPAS itself was a secret, ProPublica had published enough of the results to allow it to be meaningfully tested against other benchmarks. One was a simple mathematical model with just two variables: the age of the offender and the number of previous offenses. Dressel and Farid showed that the two-variable model was just as accurate as the much-vaunted 137-variable COMPAS model. Dressel and Farid also tested COMPAS predictions against the judgment of ordinary, non-expert humans who were shown just seven pieces of information about each offender and asked to predict whether he or she would be rearrested within two years. The average of a few of these non-expert predictions outperformed the COMPAS algorithm.

This is striking stuff. As Farid commented, a judge might be impressed if told that a data-driven algorithm had rated a person as high-risk, but would be far less impressed if told, “Hey, I asked twenty random people online if this person will recidivate and they said yes.”[32]

Is it too much to ask COMPAS to beat the judgment of twenty random people from the internet? It doesn’t seem to be a high bar; nevertheless COMPAS could not clear it.[33]

Demonstrating the limitations of the COMPAS algorithm wasn’t hard once the ProPublica data on COMPAS’s decision making had been released to allow researchers to analyze and debate them. Keeping the algorithms and the datasets under wraps is the mind-set of the alchemist. Sharing them openly so they can be analyzed, debated, and—hopefully—improved on? That’s the mind-set of the scientist.

—

Listen to the speeches of traditional center-ground politicians, or read media commentary, and it is common to encounter a view such as “Levels of trust are declining” or “We need to rebuild trust.” Onora O’Neill, a philosopher who has become an authority on the topic, argues that such hand-wringing reflects sloppy thinking. She argues that we don’t and shouldn’t trust in general: we trust specific people or institutions to do specific things. (For example: I have a friend I’d never trust to post a letter for me but I’d gladly trust him to take care of my children.) Trust should be discriminating: ideally we should trust the trustworthy, and distrust the incompetent or malign.[34]

Just like people, algorithms are neither trustworthy nor untrustworthy as a general class. Just as with people, rather than asking, “Should we trust algorithms?” we should ask, “Which algorithms can we trust, and what can we trust them to do?”

Onora O’Neill argues that if we want to demonstrate trustworthiness, we need the basis of our decisions to be “intelligently open.” She proposes a checklist of four properties that intelligently open decisions should have. Information should be accessible: that implies it’s not hiding deep in some secret data vault. Decisions should be understandable—capable of being explained clearly and in plain language. Information should be usable—which may mean something as simple as making data available in a standard digital format. And decisions should be assessable—meaning that anyone with the time and expertise has the detail required to rigorously test any claims or decisions if they wish to.

O’Neill’s principles seem like a sensible way to approach algorithms entrusted with life-changing responsibilities, such as whether to release a prisoner, or respond to a report of child abuse. It should be possible for independent experts to get under the hood and see how the computers are making their decisions. When we have legal protections—for example, forbidding discrimination on the grounds of race, sexuality, or gender—we need to ensure that the algorithms live up to the same standards we expect from humans. At the very least that means the algorithm needs to be scrutable in court.

Cathy O’Neil, author of Weapons of Math Destruction, argues that data scientists should—like doctors—form a professional society with a professional code of ethics. If nothing else, that would provide an outlet for whistleblowers, “so that we’d have someone to complain to when our employer (Facebook, say) is asking us to do something we suspect is unethical or at least isn’t up to the standards of accountability that we’ve all agreed to.”[35]

Another parallel with the practice of medicine is that important algorithms should be tested using randomized controlled trials. If an algorithm’s creators claim that it will sack the right teachers or recommend bail for the right criminal suspects, our response should be “Prove it.” The history of medicine teaches us that plausible-sounding ideas can be found wanting when subjected to a fair test. Algorithms aren’t medicines, so simply cloning an organization such as the US Food and Drug Administration wouldn’t work; we’d need to run the trials over faster timelines and take a different view of what informed consent looked like. (Clinical trials have high standards for ensuring that people consent to participate; it’s not so clear how those standards would apply to an algorithm that rates teachers—or criminal suspects.) Still, anyone who is confident of the effectiveness of their algorithm should be happy to demonstrate that effectiveness in a fair and rigorous test. And vital institutions such as schools and courts shouldn’t be willing to use those algorithms on a large scale unless they’ve proved themselves.

Not all algorithms raise such weighty concerns, of course. It wouldn’t obviously serve the public interest to force Target to let researchers see how they decide who receives onesie coupons. We need to look on a case-by-case basis. What sort of accountability or transparency we want depends on what problem we are trying to solve.

We might, for example, want to distinguish YouTube’s algorithm for recommending videos from Netflix’s algorithm for recommending movies. There is plenty of disturbing content on YouTube, and its recommendation engine has become notorious for its apparent tendency to suggest ever more fringy and conspiratorial videos. It’s not clear that the evidence supports the idea that YouTube is an engine of radicalization, but without more transparency it’s hard to be sure.[36]

Netflix illustrates a different issue: competition. Its recommendation algorithm draws on a huge, secret dataset of which customers have watched which movies. Amazon has a similar, equally secret dataset. Suppose I’m a young entrepreneur with a brilliant idea for a new kind of algorithm to predict which movies people will like based on their previous viewing habits. Without the data to test it on, my brilliant idea can never be realized. There’s no particular reason for us to worry about how the Amazon and Netflix algorithms work, but is there a case for forcing them to make public their movie-viewing datasets, to unleash competition in algorithm design that might ultimately benefit consumers?

One concern is immediately apparent: privacy. You might think that was an easy problem to solve—just remove the names from records and the data are anonymous! Not so fast: with a rich dataset, and by cross-referencing with other datasets, it is often surprisingly easy to figure out who Individual #961860384 actually is. Netflix once released an anonymized dataset to researchers as part of a competition to find a better recommendation algorithm. Unfortunately, it turned out that one of their customers had posted the same review of a family movie on Netflix and, under her real name, on the Internet Movie Database website. Her no-longer-anonymous Netflix reviews revealed that she was attracted to other women—something she preferred to keep secret.[37] She sued the company for outing her; it settled on undisclosed terms.

Still, there are ways forward. One is to allow secure access to certified researchers. Another is to release fuzzy data where all the individual details are a little bit off, but rigorous conclusions can still be drawn about populations as a whole. Companies such as Google and Facebook gain an enormous competitive advantage from their datasets: they can nip small competitors in the bud, or use data from one service (such as Google Search) to promote another (such as Google Maps or Android). If some of that data were made publicly available, other companies would be able to learn from it, produce better services, and challenge the big players. Scientists and social scientists could learn a lot, too; one possible model is to require private big-data sets to be published after a delay and with suitable protections of anonymity. Three-year-old data are stale for many commercial purposes but may still be of tremendous scientific value.

There is a precedent for this: patent holders must publish their ideas in order to receive any intellectual property protection. Perhaps a similar bargain could be offered to, or imposed on, private holders of large datasets.

—

Big data is revolutionizing the world around us, and it is easy to feel alienated by tales of computers handing down decisions made in ways we don’t understand. I think we’re right to be concerned. Modern data analytics can produce some miraculous results, but big data is often less trustworthy than small data. Small data can typically be scrutinized; big data tends to be locked away in the vaults of Silicon Valley. The simple statistical tools used to analyze small datasets are usually easy to check; pattern-recognizing algorithms can all too easily be mysterious and commercially sensitive black boxes.

I’ve argued that we need to be skeptical of both hype and hysteria. We should ask tough questions on a case-by-case basis whenever we have reason for concern. Are the underlying data accessible? Has the performance of the algorithm been assessed rigorously—for example, by running a randomized trial to see if people make better decisions with or without algorithmic advice? Have independent experts been given a chance to evaluate the algorithm? What have they concluded? We should not simply trust that algorithms are doing a better job than humans, nor should we assume that if the algorithms are flawed, the humans would be flawless.

But there is one source of statistics that, at least for the citizens of most rich countries, I think we should trust more than we do. And it is to this source that we now turn.

RULE EIGHT

Don’t Take Statistical Bedrock for Granted

“What do you base your facts on?”

“Statistics from the International Monetary Fund and the United Nations, nothing controversial. These facts are not up for discussion. I am right, and you are wrong.”

• Interview with Hans Rosling[1]

Monday, October 9, 1974. The place: Washington, DC, near the picturesque Tidal Basin—a quiet, leafy sanctuary not far from the White House. The time: two o’clock in the morning. A car is weaving around in the darkness, fast, with its headlights off. The police pull the car over, at which point a flamboyantly dressed woman with two black eyes jumps out of the passenger’s side, runs down the road yelling alternately in English and Spanish, and leaps into the water. The police pull her out and she tries to jump in again, at which point they handcuff her. At the wheel is an elderly fellow with broken glasses and minor cuts to his face. He is steaming drunk.[2]

Just another night in DC, perhaps. Except the woman, Annabelle Battistella, was better known as Fanne Foxe, the Argentine Firecracker, an erotic dancer at the Silver Slipper nightclub. And the man was one of the most powerful men in the United States: Wilbur Mills, an Arkansas congressman since the 1930s, who as the long-serving chair of the House Ways and Means Committee effectively had veto power over most legislation. These were, however, deferential times. The police offered to drive Mills home to his wife in his own car, and he was reelected by voters just a few weeks later.

But hard on the heels of that electoral triumph, Mills—drunk again—appeared onstage with Foxe in the middle of her act, received a peck on the cheek, and made his exit, stage left. To be caught cavorting with a stripper once might be considered a misfortune. To do it twice suggested carelessness. His colleagues had a quiet word. Wilbur Mills stepped down from the Ways and Means Committee and joined Alcoholics Anonymous. Fanne Foxe rebranded herself “The Tidal Basin Bombshell,” wrote a tell-all memoir, and eventually retired into obscurity.[3]

To most people, this story might be dimly remembered as America’s third most spectacular sex scandal. But in my home country of Nerdland, it has another significance. At the time, Congress was deadlocked over a putative new agency, the Congressional Budget Office, which would provide advice to Congress about the budgetary costs of different policy proposals. One congressional dinosaur in particular objected to plans to appoint a woman as its director. But Wilbur Mills’s resignation triggered the usual game of musical chairs, the indirect consequence of which was that the deadlock was broken. The Congressional Budget Office was duly established, and with the dinosaur ambling off to graze elsewhere on Capitol Hill, there was no obstacle to its first director being the woman every sensible person would have wanted: Alice Rivlin. Forty years later, she reflected, “I owed my job to Fanne Foxe.”[4]

After this strange beginning, Alice Rivlin went on to lead the Congressional Budget Office to glory.* The CBO had been established by Congress to serve as a counterweight to what was seen as Richard Nixon’s overreaching and overmighty presidency. Congress saw the value of having better statistics and more analysis of policy issues. But Rivlin interpreted this role in a particular way: rather than churning out talking points for the majority party or running statistical errands for the powerful chairs of congressional committees, she would supply impartial, high-quality information and analysis to Congress as a whole. In the judgment of one academic, the CBO duly became “one of the most influential and well-regarded institutions in Washington . . . the authoritative source of information on the budget and the economy.”[5]

Alice Rivlin’s deputy and later one of her successors, Robert Reischauer, described the CBO as

basically a manhole in which Congress would have a bill or something, and it would lift up the manhole cover and put the bill down it, and you would hear grinding noises, and twenty minutes later a piece of paper would be handed up, with the cost estimate, the answer, on it. No visibility, [just] some kind of mechanism below the ground level doing this . . . noncontroversial, the way the sewer system is.[6]

The analogy is apt, and not just because sewers are invisible and uncontroversial. Independent statistical agencies, like sewers, are an essential part of modern life. As with sewers, we tend to take them for granted until something goes wrong. And like sewers, they can suffer badly from neglect—or because someone tries to force something unsuitable through them for their own selfish or foolish reasons.

The official statistics and analyses produced by organizations such as the CBO are more important than we might think, and more useful in the everyday lives of ordinary citizens. They are also under threat—and we should defend them. They should not have to depend on twists of fate involving drunk congressmen and strippers.

—

The CBO was established, remember, with Richard Nixon in mind. But Nixon had resigned before the CBO began operations, and the first president to object to what the CBO was doing was not a Republican like Nixon, but a Democrat: Jimmy Carter. With oil prices spiking in the late 1970s, President Carter had ambitious goals to improve America’s energy efficiency. Alice Rivlin’s CBO team evaluated the proposals and judged that they wouldn’t work as well as Carter hoped.

“It made the Carter Administration unhappy,” Rivlin later recalled. The House Speaker, Thomas P. “Tip” O’Neill, also a Democrat, wasn’t happy either. “He was fighting for the legislation and the CBO wasn’t helping.”[7]

No. It wasn’t helping. This was exactly the point: Alice Rivlin knew that the value of the CBO would lie in being impartial rather than in serving up propaganda for the party in power. It wasn’t long before the party in power was the Republicans again, and it was their turn for their grand claims to smack into the unyielding reality of the CBO’s independent opinion. In 1981, the CBO argued that the budget deficit was likely to be far higher than the Reagan White House projected. President Reagan called the CBO numbers “phony.”

In 1983, Alice Rivlin left the CBO after eight years in charge. Successive administrations continued to put pressure on it—in the 1990s, for example, leading Democrats wanted the CBO to produce a more flattering analysis of President Clinton’s health care reforms—and it continued to assert its independence.[8] The CBO certainly isn’t perfect: much of its task is to make projections of the future gap between spending and tax revenue, and—as we’ll discuss in chapter ten—such economic forecasts are hard to make; official agencies often get them wrong. The important point, though, is that they don’t make politically expedient errors, systematically warping their forecasts to fit a political agenda. Evaluations of the CBO[9] have tended to find that it produces forecasts that are as accurate as we might reasonably hope, and—crucially—unbiased.*

In the UK, the Office for Budget Responsibility (OBR) performs a similar role to the CBO. It was established as an independent agency only in 2010. Forecasts of spending, tax receipts, and other economic variables had previously been made by the Treasury, where officials are more directly answerable to politicians. That enables us to make an interesting comparison: Are the OBR forecasts better? It turns out they are, substantially so.[10] That’s encouraging for the OBR’s reputation and future work, but it also suggests there was previously a problem—that before 2010, the Treasury’s economists had been routinely shaping their forecasts to please their political overlords.

The CBO and OBR are far from the only kinds of statistical agencies that need to assert their political independence. While they project the future impacts of proposed tax or spending changes, many other agencies describe current realities. There are censuses, which try to estimate how many people live in different parts of the country, along with some basic information about those people. There are economic statistics—measuring inflation, unemployment, economic growth, trade, and inequality. There are social statistics—measuring crime, education, access to housing, migration, and well-being. There are studies of particular industries, or of issues such as environmental pollution.

Every country has its own arrangements for putting together these official statistics. In the UK, many are produced and published by one organization, the Office for National Statistics. In the United States, the task is spread across a range of agencies including the Bureau of Economic Analysis, the Bureau of Labor Statistics, the Census Bureau, the Federal Reserve, the Department of Agriculture, and the Energy Information Administration.

How useful is all this counting and measurement? Very useful indeed; it’s hard to overstate how useful. The numbers produced by such agencies are a nation’s statistical bedrock. When journalists, think tanks, academics, and fact-checkers want to know what is going on, their analysis usually rests, either directly or ultimately, on this bedrock. I’ll have more to say about the costs and varied benefits of producing professional and impartial official statistics later in the chapter. But perhaps the most vivid argument for their value is to look at attempts to distort, discredit, or suppress them.

As a presidential candidate in 2016, Donald Trump faced a problem. His campaign wanted to claim that the American economy was broken, but official statistics showed that the unemployment rate was very low—below 5 percent and falling. There could have been a thoughtful response to that—for example, that the unemployment rate doesn’t measure the quality, security, or earning power of jobs. But Trump took the simpler path of repeatedly dismissing unemployment figures as “phony” and “total fiction” and claiming that the true unemployment rate was 35 percent.

Simply inventing your own numbers is a tactic more often used by totalitarian dictators than by candidates for democratic election, but Trump evidently figured it was a tactic that would be effective. And perhaps he was right. His supporters believed him: just 13 percent of them trusted the economic data produced by the federal government, versus 86 percent of those who voted for Hillary Clinton.[11]

As president, Trump changed his mind. According to the official data, unemployment crept even lower after he had assumed office. Now, however, Trump wished to get credit for this rather than to dismiss it. His spokesman Sean Spicer declared, with a straight face, “I talked to the president prior to this, and he said to quote him very clearly. They may have been phony in the past, but it’s very real now.” Amusing as this kind of shamelessness might be, it also carries a real risk—that Trump’s opponents will start to distrust official statistics just as much as his supporters do.[12]

If you grow tired of undermining trust in your own statistical agency when it isn’t producing politically convenient figures, you could always attack the statistical agency of someone else. For example, after Germany’s leader Angela Merkel took the politically risky step of welcoming almost a million refugees into the country in 2015, Donald Trump wanted to use Germany as a cautionary tale. “Crime in Germany is way up,” he tweeted in June 2018. Look at all the crimes those refugees were causing!

Unfortunately for President Trump, one group of people stood in the way to spoil his story: German statisticians. Their latest figures, a month before Trump’s tweet, showed that not only was crime in Germany not “way up,” it was at its lowest level since 1992.[13] Unabashed, Trump had an answer. A few hours later, he tweeted: “Crime in Germany is up 10% plus (officials do not want to report these crimes).”[14]

The allegation is implausible. In part that’s because the ministry in Germany responsible for putting together the police crime statistics was run by Horst Seehofer, an immigration hawk who in the same year threatened to resign if Germany’s immigration policy wasn’t tightened up: Seehofer would hardly have wanted to pressure officials to hide uncomfortable truths about migration. It’s also implausible because Germany has not acquired a reputation for political interference in statistics.

Sadly, that’s not true for every country. Around the world, pressure to fiddle the figures is real and widespread—and the consequences for statisticians can be far more serious than grumbling from senior politicians.

—

In 2010, the economist Andreas Georgiou left a two-decade career at the International Monetary Fund, bringing his baby daughter with him from Washington, DC, to his home country, Greece. His mission was to run ELSTAT, Greece’s new statistical agency.

At the time, Greece’s statistics were in bad shape. They had never been well funded or well respected. When, in 2002, the economist Paola Subacchi visited the Greek statistical office, she found it tucked away in a residential suburb of Athens, “in a square of ordinary shops, and I had to hunt for a doorway in a 1950s apartment block that took me up some stairs to a dusty room with a handful of people. I can’t remember seeing any computers. It was extraordinary, not a professional operation at all.”[15]

But when Georgiou arrived, there was more to worry about than dust and outdated technology. The entire world had reached the conclusion that you should trust Greek official statistics about as much as you should trust their giant wooden horses. Eurostat, the statistical office of the European Union, had repeatedly complained about the credibility and quality of the official Greek economic data. The European Commission issued a blistering report about them.[16]

The basic problem was that Greece was supposed to keep its government budget deficit at a modest level. The budget deficit is the amount the government borrows each year to cover any gap between what it spends and what it receives in taxes. One of the obligations that come with membership in the eurozone is for a country to keep its deficit below 3 percent of gross domestic product, with some exemptions for various exceptional circumstances. (Economically speaking, it’s not a very sensible rule—but that’s another story for another book.) That target was onerous, so why not tweak the figures until all seemed well? One year the Greek accounts left out several billion euros of borrowing to pay for hospitals. Another year, they omitted a big chunk of the cost of the military. They also did a deal with the investment bank Goldman Sachs that effectively made borrowing look like a different kind of transaction, and thus not counting toward the deficit.[17]

In 2009, the shock of the global financial crisis was followed by the realization that Greece had been underplaying its borrowing for years. Nobody believed its debts could be repaid. The EU and IMF moved in with the customary mix of a bailout and some brutal austerity, and the Greek economy collapsed. Into this situation stepped Andreas Georgiou. He might not be able to rescue Greece’s prosperity, but there was some hope that he would save the reputation of Greek official statistics.

Georgiou’s first priority was to look at the deficit figures for 2009, the most recent available. The initial forecast, from the Greek Ministry of Finance, had been 3.7 percent of gross domestic product—not too far outside the EU’s target, but unfortunately quite implausible. Even before Georgiou’s arrival the Greek authorities had revised that to a shocking 13.6 percent. Eurostat was still unconvinced. Within a few months, Georgiou published his conclusion: the deficit had actually been 15.4 percent, a grimly large number. But it was, at least, believable—and Eurostat believed it.

It was then that Georgiou’s troubles began. First, there was an almighty row within ELSTAT. The police eventually realized that Georgiou’s email account had been hacked by his own deputy, ELSTAT’s vice president. Then the Greek Prosecutor of Economic Crimes began legal action against Georgiou, accusing him of deliberately exaggerating Greece’s deficit and causing immense damage to the Greek economy. Various other charges were added, including failing to allow ELSTAT’s board to vote on what the deficit should be. (The idea that the size of Greece’s budget deficit should be put to a vote seems more Eurovision than Eurostat.) The potential sentence for Georgiou’s “crimes” was life imprisonment. The judicial system threw out the charges six times, but they were repeatedly reinstated by the Greek supreme court. Indeed, his convictions, acquittals, and reconvictions have been so frequent that it is hard to have any confidence that any verdict will stick.[18] This is harassment worthy of a Kafka novel.

Of course it is possible that Georgiou really is a traitor. But it does not seem likely. Eighty former chief statisticians from around the world signed a letter protesting against his treatment, Eurostat repeatedly signed off on the quality of his work, and in 2018 he received a special commendation from a group of respected professional bodies including the International Statistical Association, the American Statistical Association, and the Royal Statistical Society “for his competency and strength in the face of adversity, his commitment to the production of quality and trustworthiness of official statistics and his advocacy for the improvement, integrity and independence of official statistics.”[19]

Andreas Georgiou is not the only statistician who has shown courage in adversity, as Graciela Bevacqua, a long-serving Argentine statistician, could attest. Argentina has long suffered from high inflation. The Argentine government, under a husband-and-wife pair of populist presidents, Néstor Kirchner (president 2003–07) and Cristina Fernández de Kirchner (president 2007–15), decided to solve the problem not by reducing inflation but by changing the inflation statistics. Bevacqua found herself receiving some discomfiting demands.

For example, she was instructed to round down all decimals in the monthly inflation figures—as though Argentine computers had run out of decimal points. That makes more difference than you might think, because each distortion compounds the earlier ones: compounding inflation of 1 percent a month gives 12.7 percent a year, while 1.9 percent a month is 25.3 percent a year. Funnily enough, official estimates of annual inflation in Argentina have tended to be close to the first number, and independent unofficial estimates have been closer to the second.

When Graciela Bevacqua produced a monthly figure of 2.1 percent inflation at the beginning of 2007, her supervisors weren’t happy. Hadn’t they told her to produce a number below 1.5 percent? They told her to take a vacation, then sacked her when she returned, transferring her from the statistical agency to a library and slashing her pay by two-thirds. She resigned soon afterward.[20]

With Bevacqua out of the way—and having been made an example of—Argentina’s official inflation numbers showed inflation of below 10 percent. That’s high by the standards of a developed country but still implausibly low. Most independent experts reckoned it was close to 25 percent, and a group of those experts produced their own unofficial price index, advised by none other than Graciela Bevacqua—who was promptly fined $250,000 for false advertising.

As with Georgiou, international observers stand behind Bevacqua and her methods, and with a new government in Argentina it looks like she’ll be OK. As for Georgiou, he stuck it out for five years at ELSTAT and then returned to the United States, leaving behind him an organization with a credibility it never had before he arrived. He is most unlikely to go to prison, but other Greek statisticians will have noticed the way he was persecuted for trying to tell the truth about the statistics that were his responsibility. “It will not be lost on them that their well-being—not only professional but personal—is at risk if they do the right thing and follow the law,” he told Significance magazine. He added that the Greek government was only damaging itself in the long run, by “undermining the statistics which they themselves use. They’re undermining the credibility of the country itself.” Meanwhile, the people who repeatedly understated Greece’s deficit before the crisis seem to have escaped censure.[21]

Heroic as Andreas Georgiou and Graciela Bevacqua have shown themselves to be, we would be naive to assume that every statistician has their determination, or that every attempt to exert pressure comes to public attention. One respected statistician, Professor Denise Lievesley, told me that a fellow statistician from Africa had been told that if he didn’t produce the numbers that his nation’s president required, his children would be murdered. For understandable reasons, she didn’t wish to identify him.[22] It would be equally understandable if he had decided to comply.

There are subtler ways to undermine the independence of official statisticians. In Tanzania, in late 2018, the government passed a law that made criticism of official statistics a criminal offense, punishable with fines or a minimum of three years in prison. Candidates for the presidency there will think twice before following Trump’s example of calling the jobless figures “phony.” But imprisoning anyone who finds fault with government statistics is not only an outrage against free speech, it will ensure that faults go uncorrected. Tanzania’s move—which has been criticized by the World Bank—would be the perfect prelude to distorting its own statistics for political reasons.[23]

In India, Prime Minister Narendra Modi’s government quietly stopped publishing data on unemployment in 2019. Modi had made big promises about creating jobs, but in the run-up to that year’s election (which he won comfortably) it began to look as though reality was going to prove embarrassing. The answer was simply to find an excuse to stop publishing, pending the arrival of “improvements” in the data. One Indian expert explained to the Financial Times exactly what was going on: “It’s very clear that for a long time, the objective of the government has been to keep the picture fuzzy.”[24]

Even in countries with the most solid of reputations in Nerdland, serious conflicts can arise between the politicians and the statisticians. The Canadian statistical agency, Statistics Canada, has long been admired by statistical agencies around the world for its competence and independence—but the same qualities are not always appreciated closer to home. First the government under Prime Minister Stephen Harper (2006–15) tried to abolish the traditional census, replacing it with a voluntary survey—something that would have been cheaper and more convenient but massively less robust. The chief statistician, Munir Sheikh, made his objections very public and resigned.[25] The Harper government also wanted to move IT infrastructure to an organization called Shared Services Canada; when the administration of the next prime minister, Justin Trudeau, pressed ahead with that plan, the next chief statistician, Wayne Smith, also resigned. He argued that if his data and computing power were being moved into another organization, he could not guarantee the confidentiality of the statistics he was collecting. Nor could he be sure that Canadian statisticians would remain independent, since they could be squeezed or pressured by any government official with power over Shared Services Canada.

It’s fair to say that Statistics Canada’s reputation for robust independence has only been enhanced by these episodes. But there is a risk that if one side of the political spectrum is seen as hostile to the statisticians while the other side leaps to their defense, statistics itself becomes a partisan political issue. With that in mind, perhaps we should be reassured that when the last two chief statisticians of Canada resigned in protest, they did so under two different governments.[26]

—

In Puerto Rico, the government’s response to troublesome statisticians was more radical: they attempted to disband entirely the statistical agency, PRIS, soon after the disastrous hurricane of September 2017. The ostensible reason was that PRIS was too expensive: its million-dollar budget could be better spent elsewhere.

That may not have been the real reason. You may recall that shortly after that hurricane, President Trump expressed gratitude that the death toll had been so small—sixteen or seventeen people, not a “real tragedy” like the hurricane that had flooded New Orleans twelve years earlier. That was glib, but in line with the official death toll at the time—which later rose, but only to just over fifty. It seemed suspiciously low. Numerous independent researchers attempted to figure out their own estimates, to include not just the people who had been killed outright by the storm but those who had later died because of overstretched medical services, or because they were cut off from assistance by blocked roads and downed power lines. Alexis Santos was one of these researchers. He is a demographer at Penn State, and his Puerto Rican mother was on the island when the hurricane struck. Professor Santos put out an estimate that around a thousand people had died, directly or indirectly, as a result of the hurricane. It was big news in Puerto Rico. Even graver estimates were published later.

All of these estimates were built on demographic data from PRIS. PRIS itself, meanwhile, was suing the Puerto Rican health ministry in an effort to get accurate, timely information about the dead.[27] Given the embarrassment it was causing the administration, perhaps the threat to disband PRIS was not entirely surprising.

Still, let’s take the given reason at face value: Is PRIS really worth its million-dollar budget? The question of how much value official statistics create is a valid one, and there are fewer attempts to quantify this than one might hope.

One cost-benefit exercise was conducted in the UK in the run-up to the 2011 census; it produced a long list of benefits from the census, everything from informing the debate over pension policy, to ensuring that schools and hospitals were located in the right areas, to enabling all sorts of other statistics to be calculated. After all, you can’t produce any “per capita” statistics—from crime to teen pregnancy to income to the unemployment rate—unless you know the population.

The analysts observed that “statistics in themselves don’t deliver benefits. It’s the use of statistics that delivers benefits through better, quicker decisions by governments, companies, charities and individuals.”[28] That sounds plausible, and there are some surprising examples. London’s Metropolitan Police, for example, used the census to identify streets with large numbers of elderly residents, and focused efforts on preventing fraudsters and burglars from preying on vulnerable people. Everything from public health campaigns to nuclear disaster contingency plans depends on figuring out where everyone lives.

Disappointingly, the cost-benefit analysts shrugged their shoulders and declared themselves unable to put a value on all this, except to declare that it was obviously jolly useful. Still, they did find some benefits they judged to be quantifiable, and they pegged those as being worth £500 million a year—a bit less than £10 per UK resident. Since the census itself cost less than £500 million and lasts ten years, that suggests a tenfold return is a rather conservative estimate of the benefits.

Another attempt to tot up the value of official statistics was made in New Zealand, where the census, which cost NZ$200 million to conduct (about $120 million), was reckoned to have produced a benefit of at least a billion New Zealand dollars—a fivefold return. The study reckoned that refreshing the basic knowledge provided by the census—who lives where—produced a more accurate allocation of public spending on facilities such as hospitals and roads, and better-informed policy more generally.[29] Back in Puerto Rico, researchers pointed out ways in which PRIS had paid for itself, such as enabling the introduction of new systems to prevent fraud in collecting Medicare payments.[30]

But perhaps the strongest evidence that statistics are worthwhile is how cheap they are to collect, relative to the value of the decisions they inform. Consider the CBO: it advises Congress on $4 trillion worth of annual spending, on a budget of just $50 million a year. To put it another way, for every $80,000 the US government spends, one dollar funds the CBO to shed light on the other $79,999.[31] To justify its existence, the CBO would need to improve the effectiveness of government spending decisions by a mere 0.00125 percent. It’s hard to imagine how the CBO could fail to clear that bar.

Likewise, the million-dollar budget of PRIS sounds a lot more modest when you put it in the context of the Puerto Rican government’s overall spending, which at nearly $10 billion is about 10,000 times larger. The UK’s Office for National Statistics costs about £250 million a year—less than one pound for every £3,000 the UK government spends. Between them, the thirteen principal statistical agencies in the United States cost one dollar for every $2,000 the federal government spends.[32] If serious, independently gathered data improve government decision-making even by a tiny fraction, then these agencies are well worth the small sliver of public spending that is devoted to them.

—

Without statistics, then, governments would fumble in ignorance. But there is an intriguing counterargument, which is that governments are so reliably incompetent that giving them more information is risky; it will only encourage them.

One prominent advocate of this view was Sir John Cowperthwaite. Sir John was the financial secretary of Hong Kong throughout the 1960s, at a time when it was still under the control of the British—and when it was experiencing scorchingly rapid economic growth. Exactly how rapid was hard to say, because Sir John refused to collect basic information about Hong Kong’s economy. The economist Milton Friedman, later to win the Nobel Memorial Prize in Economics, met Sir John at the time and asked him why. “Cowperthwaite explained that he had resisted requests from civil servants to provide such data because he was convinced that once the data was published there would be pressure to use them for government intervention in the economy.”[33]

There was a logic to this. Hong Kong’s rapid growth was partly thanks to an influx of immigrants from famine-struck communist China, but Cowperthwaite and Friedman also believed—with some reason—that it was flourishing thanks to a laissez-faire approach to policy. Cowperthwaite’s government levied low taxes and provided very little in the way of public services. The private sector, he argued, would tend to solve people’s problems more quickly and efficiently than the state. Why, then, collect data that would only encourage meddling from the authorities back in London? Cowperthwaite figured that the less London’s politicians did, the better—and the less they knew, the less they would try to do.

Similarly, in his magisterial book Seeing Like a State, James C. Scott argues that the statistical information that states gather is flawed, missing the local details that matter. Imagine, say, a rural community in Southeast Asia with complex customs regarding a piece of local land. Every household has some rights to farm the land, in rough proportion to its number of able-bodied members; then after each harvest, it becomes common land for grazing. Everyone can gather firewood, too, but the village baker and blacksmith are allowed to gather more. A surveyor from the new national land registry turns up, asking, “Who owns this land?” Well—it’s not so simple.

Now, it’s one thing to be wrong, or to have a view of the world that misses out on something important. But, argues Scott, because the state is powerful, its misperceptions of the world often take physical form, producing well-meaning but clumsy and oppressive modernist schemes that ignore local knowledge and stifle local autonomy.[34] Perhaps our frustrated land registry surveyor decides to write on her clipboard that the local government owns the land; then a few years later the villagers are surprised to find the land being cleared for a palm oil plantation.

One can take the argument even further: that governments can be utterly malevolent, and the worst cases are so catastrophic that they should inform our thinking about how much data any government should have. Wouldn’t it have been better if Hitler, Mao, and Stalin had understood less about their own societies? Might they have done less harm? And is it reasonable to worry that the more governments know about us, the more they will be tempted to exert control over us?

This argument seems plausible, but I’m not convinced. From communist East Germany to modern-day China, governments interested in mass surveillance and population control have tended to use very different methods from those deployed by independent statistical offices in modern democracies, and to collect very different kinds of data. And history suggests that dictators often have either little interest in the collection of solid statistics or little ability to collect them.

Consider the disastrous government-induced famine of the late 1950s caused by the Great Leap Forward in communist China, in which people were reduced to eating tree bark, bird droppings, and rats. Between 20 and 40 million people died. The catastrophe was made worse by a lack of accurate data about agricultural production. When official statistics began to make the death toll apparent, they were destroyed.[35]

Stalin, similarly, suppressed the publication of the 1937 census of the Soviet Union when it showed that the population was lower than he’d previously announced. This contradiction was an affront in its own right, but it also highlighted the millions of deaths as a result, directly and indirectly, of Stalin’s brutality. The penalty for accurately counting the Soviet population? Olimpiy Kvitkin, the statistician in charge, was arrested and shot. Several of his colleagues met the same fate.[36] This is not the act of a totalitarian leader who finds accurate statistical information to be an indispensable tool of oppression.[37]

In Nazi Germany, there was no lack of ambition to use data to support the apparatus of the state. The Reich tried to use punch-card machines, the latest technology, in an effort to track the entire population. But as Adam Tooze argues in Statistics and the German State, statistical standards actually fell apart under the Nazis: “No workable system was ever devised.”[38] The traditions of official statistics—privacy, confidentiality, and independence—were so alien to the Nazi project that the system all but collapsed under the political pressure and factional infighting.

All that said, I have a great deal of sympathy with James C. Scott’s argument (I discuss Scott’s ideas in more detail in my book Messy) and some sympathy with Sir John Cowperthwaite’s. States should be humble. Bureaucrats must recognize the limits of their knowledge. There is always a risk that the bird’s-eye view is so grand and sweeping as to induce delusions of omnipotence.

Sir John’s strategy to deny information to the British government seems to have worked for Hong Kong fifty years ago, but Hong Kong was in a very particular situation—a colonial possession of a fading imperial power in which big government was fashionable, and any government intervention would have taken place at a distance of 6,000 miles. Those are unusual circumstances.

But the tactic of simply refusing to collect basic statistics could only make sense for a libertarian, laissez-faire regime. And the truth is that very few people seem attracted by that prospect. For better or worse, we want our governments to take action, and if they are to take action they need information. Statistics collected by the state make for better-informed policies—on crime, education, infrastructure, and much else.

In poor countries, where official statistical agencies tend to be less well resourced, there is especially wide scope to improve decision making through better statistics. One example may illustrate the problem. How effective is education in improving literacy? That seems like the kind of question that might usefully help to inform education policy and spending. So researchers at the World Bank looked into statistics collated by UNESCO (the UN Educational, Scientific and Cultural Organization) and found there was an amazingly high correlation between education and literacy: without fail, countries that provided more years of formal education to more people had higher literacy rates. Clearly, education worked! They excitedly published their findings.[39]

Unfortunately, they hadn’t read the small print. UNESCO simply hadn’t had the resources to collect all the data they wanted to: they had just seventy staff covering 220 countries trying to pull together data in all kinds of areas—adult literacy was just one. (What does literacy even mean in a place such as Papua New Guinea? It has four hundred languages, some of which have no written form.) Inevitably, there would be shortcuts. UNESCO couldn’t send teams of people to assess rates of adult literacy themselves, so they looked for a proxy indicator—a best guess in a difficult situation. And they decided that if someone had fewer than five years’ formal education, they would be assumed to be illiterate. No wonder the World Bank researchers found such a close correlation between education and literacy.

If organizations like UNESCO had more resources to collect statistics, they would have less need to rely on proxies, and researchers would have greater ability to answer questions such as how well education promotes literacy. Statistical bedrock is so patchy in poor countries that already one dollar in every three hundred that is spent on international aid goes toward funding statistics. There is a case that doubling that might well produce much more value from the remaining $298.[40]

—

Sir John’s comment to Milton Friedman contains an implicit assumption: that government statistics are not just collected by government, but they are collected for government. He was unusual in believing that government would do a better job without those statistics, but otherwise that perspective is common. Congress seemed to have the same idea in mind when creating the Congressional Budget Office: the CBO was designed to provide information to Congress. The clue is in the name. And the idea goes back a long way. As the future president James Madison put it in 1790, politicians should be willing to commission accurate statistics, “in order that they might rest their arguments on facts, instead of assertions and conjectures.”[41]

There is nothing wrong with the idea that government should collect statistics to inform itself. But there is a risk that this view slips into a proprietorial sense of ownership, when politicians believe not only that they should be using statistics to run the country, but that those statistics are none of anyone else’s business, and that external scrutiny is a distraction. The facts are no longer the facts—they become the tools of the powerful.

Sir Derek Rayner was a proud proponent of the view that statistics should be managerial tools.[42] He had already been a highly successful manager at Marks & Spencer, national treasure of the British high street, before advising the UK government on how to become more efficient. In 1980, Prime Minister Margaret Thatcher asked him to review the way official statistics were collected and published in the UK. Sir Derek was happy to oblige: he saw these numbers as basically a management information system. Those that helped the government run the country could be retained; those that did not could be discarded. And there was no need to make a big fuss about publishing the numbers so that anyone could learn from them, or challenge them.

Sir Derek’s view was a mistake. Good statistics don’t just serve government planners; they are valuable to a far wider group of people. In the commercial sector, businesses rely on government-collected data to plan their production targets; the location of factories, offices, and stores; and other business activities. Data gathered by the Bureau of Labor Statistics, the Census Bureau, the Energy Information Administration, and the Bureau of Economic Analysis allow banks, real estate agents, insurance companies, auto manufacturers, construction firms, retailers, and many other businesses to make plans and to assess their own data against a broader backdrop. The multibillion-dollar turnover of data-intensive private sector companies such as Bloomberg, Reuters, Zillow, Nielsen, and IHS Markit suggests that businesses are willing to pay handsomely for useful statistics; what is less well understood is that these businesses build their statistical edifices on the foundations of government data.[43]

This isn’t just about making money; it’s about making sure that citizens have access to accurate information about the world in which they live. Government statistical agencies typically make their work available to all, free of charge. Some of that data might be impossible for a private agency to collect, at any price: governments can legally require a response that a private agency cannot, as with the case of the census. Other data could be collected but they would be offered only on an expensive subscription basis—private providers can charge tens of thousands of dollars a year for people who want data at their fingertips. Of course some data might be gathered by private firms and given away without charge, but such statistics are often just ads in the guise of information.

Publicly available statistics can be used to understand and illuminate pressing social issues. To pick just one example, W. E. B. Du Bois—historian, sociologist, and civil rights campaigner—led a remarkable data visualization effort at the end of the nineteenth century as part of the Paris Exposition of 1900.[44] His team produced beautiful, modernist graphs showing the situation of African Americans in the United States at the time, with data on demographics, wealth, inequality, and more besides. Some of them used data that Du Bois and his team had gathered at Atlanta University, but some of the most striking graphics relied on official statistical sources such as the US Census. It’s just one example of the way in which those who want to understand the world or campaign for change, or both, can turn to official statistics to help them.

With reliable statistics, citizens can hold their governments to account and those governments can make better decisions. If the government decides instead that the statistics belong to politicians, not to citizens, the quality of government decisions will not improve as a result. Neither will the esteem in which government is held.

Sir Derek Rayner’s ideas appalled many statisticians. The problem was partly the corrosive message to the British public: “These numbers aren’t for you—they’re only for important people.” But even if, like Sir Derek, one believes that statistics really are just for the important people, there’s still a good reason to make them publicly available: doing so keeps them honest. As we saw in the previous chapter, public scrutiny is vital. It’s what distinguishes science from alchemy. If statistics are published and designed to be accessible to all, they can be analyzed and examined by academics, policy wonks, and indeed anybody with a bit of time and access to a computer. Errors can be identified and corrected.

As it was, Sir Derek’s proposed reforms led to a situation where the definition of unemployment was tweaked more than thirty times in a decade, generally in a way as to lower the headline unemployment rate.[45] That is what happens when statistics are no longer regarded as a public good. And unsurprisingly, people became extremely cynical about the quality of these statistics. “Phony,” as Donald Trump might have said. Of course when official data keeps being tweaked for propaganda reasons, trust will rightly evaporate.

The UK’s statistical system, now reformed, has spent a quarter of a century trying to recover its reputation. That has taken time and hard work, because trust is easy to throw away and hard to regain. Still, the UK’s Office for National Statistics is more trusted than comparable organizations such as the Bank of England, the courts, the police, and the civil service—and vastly more trusted than politicians or the media.[46]

Sir Derek’s view—that government-collected statistics exist mainly for the convenience of government administrators, and that citizens have no particular right to see them—has thankfully fallen out of fashion around most established democracies. But one vestige clearly remains, and this was unwittingly highlighted by President Trump on Friday, June 1, 2018—the day on which the monthly jobs report was to be published.

“Looking forward to seeing the employment numbers at 8:30 this morning,” Trump tweeted, at 7:21 a.m., in an uncanny demonstration of how to wink on Twitter. Markets leaped in expectation of good news. Sixty-nine minutes later the jobs report was released, and—surprise, surprise—the news was indeed good.

Was Trump clairvoyant? No. He had simply been given advanced sight of the job numbers and decided to tell the world to expect good news.

Official statistics are often both politically and financially sensitive—for example, if the latest numbers on unemployment show that lots of jobs have been created, financial markets will respond in a different way than if the report looks grim. The numbers sometimes shape political arguments, too. For this reason, official statistics are kept confidential as they are being calculated and checked; they are then released at a particular moment, on the dot.

But in some countries, including the United States and the UK, certain people get to see certain official statistics in advance. This is called “pre-release access” and it’s a controversial practice. The justification for it is to allow ministers to prepare a response, to answer questions from journalists, and so on. For this reason various political advisers, press officers, and the like are often on the list of people given this privileged access. A self-congratulatory review of the practice by the Cabinet Office in the UK noted that press officers thought that ending the practice of pre-release “would be a disaster . . . The media would simply have their stories without any proper, official comment.” Boo hoo.[47]

It’s clear why politicians in power might find it convenient to get advance notice of statistics so they can plan to crow about them if they’re good—or if they’re bad, to get their story straight or create a distraction. But it’s far from clear that this is in the public interest. Why shouldn’t everyone, on all sides of the debate, get access to the numbers at the same time, once they’re ready?

(There is a compromise position: ministers could receive the statistics thirty minutes in advance and sit alone, without access to a cell phone, to compose a response. Quite apart from being pleasingly like sending powerful people back to sit exams, this is how journalists are sometimes given sensitive official releases. We cope. I was told a story about a Canadian statistician explaining this approach at an international gathering of colleagues. Her Russian counterpart chimed in with a question: “How does that approach work if the minister wishes to change the statistics?” Exactly.)

There’s more at stake here than a sense of fair play. In the UK, where a number of officials and advisers have routinely had pre-release access to the unemployment statistics, market watchers noticed something strange: key financial market prices such as foreign exchange rates and the price of government bonds would sometimes move sharply not long before the numbers were published. Most of the time this happened, the data would be surprising—either much better or much worse than the market had expected—and the trading would be in the direction that took advantage of the surprise.

Just to check that the market wasn’t somehow figuring out the same thing that the statisticians had, forty-five minutes in advance of publication, economist Alexander Kurov made a systematic comparison of the situation in the UK and the situation in Sweden—which is economically quite similar to the UK but which bans pre-release access to official statistics. Swedish politicians and their press officers learn about the numbers at the same time as everyone else—and traders of the Swedish krona, it seems, do not have the same weird powers of clairvoyance as traders of the British pound.[48]

It’s impossible to prove, but it seems highly likely that someone with pre-release access was giving the nod to his or her trader friends, allowing insider trading on official data. Who? Well, there were 118 people with pre-release access to the unemployment statistics, which doesn’t make it easy to identify a specific culprit. (If you are wondering why it took 118 people to prepare “proper, official comment” for the media, so am I.)

Trump’s tweet probably didn’t do much harm in itself: after all, everyone had access to the tweet at the same time. Indeed the president may unwittingly have done some good, by turning the hidden scandal of pre-release access—and the way it is an invitation to corruption, at least when the data go to subtler operators than Trump—into a widely discussed blunder.

Such privileged access facilitates insider trading—but perhaps more important is that it is corrosive of trust in official statistics. The UK press officers, keen to retain the insider perk, protested that if ministers weren’t able instantly to offer some polished patter about the data, trust in statistics would be damaged. But the truth is that the countries that are most scrupulous about forbidding pre-release access are also the countries with the strongest public confidence in official data. Those press officers might be surprised at that. I’m not.

Thankfully, the data detectives were at hand to lead the charge. In the UK, the Royal Statistical Society campaigned hard against the practice of letting ministers and other insiders sneak a peek at valuable data before the rest of us. The idea that the government needed to see the numbers so as to compose a press release, said the RSS, “is pernicious. It skews debate over the figures and perpetuates the impression that ministers control the data.” That seems right to me. In the UK, our levels of trust in official statistics are not as high as in some countries, and not as high as they should be—but they are still far higher than our levels of trust in politicians. I can see why politicians would like to get themselves wrapped up in the release of trusted statistics; it’s far from clear why any of the rest of us should want that.

So I’m delighted to report that as of July 1, 2019, the UK decided to emulate the Swedes and end pre-release access to official statistics. Under the new system the only people who will know these numbers before they’re published will be the statisticians working on them. Something tells me that trust in official statistics will survive the shock of ministers knowing the facts at the same time as everyone else.

—

This chapter has been a wholehearted defense of my fellow nerds, the ones who do an essential job in government, sometimes facing indifference from voters, interference from the powerful, and skepticism from all sides.

I wouldn’t want to suggest that any nation’s official statistical machinery is by definition unimpeachable. We’ve seen that the official statistics emerging from Argentina and Greece turned out to be deceptive, that the unemployment data emerging from the UK were tweaked every few months throughout the 1980s, and that in Canada statisticians have been forced to resign in protest at the decisions politicians have made. Some statisticians have had to endure death threats made against their families; others openly acknowledge that ministers can change data if they wish. It would be naive to assume that such problems are always exposed and that the truth always triumphs.

Even when official statistics are produced as skillfully and independently as we’d hope, they will never be perfect. Some things we care about are simply hard to measure, such as domestic violence, tax evasion, and rough-sleeping. There is, no doubt, plenty of scope for official statisticians to make the data they collect more representative, more relevant, easier to reconcile with everyday experience, and fully transparent. The more they are able to do this, the more they will deserve our trust.

Yet for all their problems and weaknesses, official statistics are still the closest we have to data bedrock. When a country picks and defends a team of skilled, professional, and independent statisticians, the facts have a way of making themselves known. When a country’s national statistics fall short, an international community of statisticians will complain. When an independent statistician is attacked or threatened by politicians, that same community will rally to his or her defense. Statisticians are capable of greater courage than most of us appreciate. Their independence is not something to take for granted, or to casually undermine.

As citizens, we need to look for that statistical bedrock. If we want to understand the situation a country is in—whether to inform our own decisions, or to hold our government to account—then we will usually start with the statistics and the analysis produced by organizations such as the Office for National Statistics, Eurostat, Statistics Canada, the Bureau of Labor Statistics, and the Congressional Budget Office.

Tough, independent-minded statistical agencies make us all smarter. So: be grateful to Andreas Georgiou, Graciela Bevacqua, and for that matter the late Alice Rivlin. And, if you like, raise a glass to Fanne Foxe.

RULE NINE

Remember That Misinformation Can Be Beautiful, Too

We are in danger of making the same statistical mistakes that we’ve always made—only prettier.

• Michael Blastland, co-creator of BBC Radio 4’s More or Less

Florence Nightingale would have needed no introduction in Victorian Britain: she was the nation’s unofficial patron saint, and was the only nonroyal woman to appear on English banknotes until 2002. Her legend continues to this day; the 4,000-bed London hospital constructed in a few days to meet the demands of the pandemic was named Nightingale Hospital.

In Florence Nightingale’s own time, the only woman more recognizable would have been Queen Victoria herself. The nation revered Nightingale for her “feminine” heroics in the Crimean War, pacing the wards of the Scutari barracks hospital in Istanbul. Here’s an editorial in The Times from February 8, 1855: “She is a ministering angel without any exaggeration in these hospitals, and as her slender form glides quietly along each corridor, every poor fellow’s face softens with gratitude at the sight of her.”

Bleurgh. I’m much more interested in her contribution as a statistician.

Nightingale was the first woman to be made a fellow of the Royal Statistical Society. When her “slender form” wasn’t too busy gliding along the corridors, causing faces to “soften with gratitude,” she was spending her time in Scutari carefully compiling data about disease and death. What she saw in the figures inspired her with a mission to change both the British Army and the British nation. Shortly after her return from Crimea, at one of the intellectual dinner parties she often attended, she met William Farr. Farr, thirteen years her senior, had been born to poor parents and lacked Nightingale’s fame, front-line experience, and political connections. But he was the best statistician in the country, and that was what mattered to her. They became friends and collaborators. One of Nightingale’s many biographers, Hugh Small, convincingly argues that the skillful way in which she and Farr wielded the data she’d assembled ended up raising life expectancy in the UK by twenty years and saving millions of lives.[1]

There’s a famous remark in a letter between Nightingale and Farr, written in the spring of 1861: “You complain that your report would be dry. The dryer the better. Statistics should be the dryest of all reading.” It is reported by several biographers as being written by Farr to Nightingale. That makes sense—the fusty middle-aged statistician advising the fiery young advocate to rein in her righteous campaigning impulses. In fact, the biographers are wrong. The letter was written by Nightingale to Farr.* The pair of them were wrestling with the problem of how best to communicate with statistics, and Nightingale was affirming that communications had to be based on hard, dry facts. (In the same letter she wrote: “We want facts. ‘Facta, facta facta’ is the motto which ought to stand at the head of all statistical work.”[2])

But that didn’t mean the communications themselves had to be dry. Nightingale could conjure up an arresting turn of phrase—she argued, for example, that the needlessly high death rate in the army in peacetime was the equivalent of taking 1,100 men out onto Salisbury Plain and shooting them.

More pertinent for our purposes, she designed an image that was a landmark in data visualization. Her “rose diagram” was one of the first-ever infographics. That makes her perhaps the first person to grasp that busy, influential people would pay far more attention to a vivid diagram than to a table of numbers. In one letter, written on Christmas Day 1857—less than three years after being beatified in The Times—she sketched out a plan to use data visualization for social change. She declared her plan to have her diagrams glazed, framed, and hung on the wall at the Army Medical Board, Horse Guards, and War Department. “This is what they do not know and did ought to!” she wrote. She even planned to lobby Queen Victoria, and she knew all too well that beautiful diagrams would be essential. As Nightingale quipped when sending one of her analytical books to the Queen, “She may look at it because it has pictures.”[3]

It’s a cynical, almost contemptuous, thing to write. But it is true. A chart has a special power. Our visual sense is potent, perhaps too potent. The word “see” is often used as a direct synonym for “understand”—“I see what you mean.” Yet sometimes we see but we don’t understand; worse, we see, then “understand” something that isn’t true at all. Done well, a picture of data is worth the proverbial thousand words. It is more than persuasive; it shows us things we could not have seen before, revealing patterns amid chaos. However, much depends on the intent of the chart’s creator, and the wisdom of the reader.

This chapter, then, will talk about what happens when we try to turn numbers into pictures. We’ll see what can go wrong. And by following the story behind Nightingale’s famous rose diagram, we’ll see how powerful data visualization can be when used clearly and honestly.

—

Much of the data visualization that bombards us today is decoration at best, and distraction or even disinformation at worst. The decorative function is surprisingly common, perhaps because the data visualization teams of many media organizations are part of the art departments. They are led by people whose skills and experience are not in statistics but in illustration or graphic design.[4] The emphasis is on the visualization, not on the data. It is, above all, a picture.

The most egregious examples of numbers as decoration are nothing more than the same old number in a large, striking font.

19

the number of words in the preceding sentence

I suppose this brightens up a page loaded with text, but it’s hardly an insightful use of ink. Also, the correct number is twenty-one. Never let zippy design distract you from the possibility that the underlying numbers simply might be wrong.

Another decorative approach is what we might call “Big Duck” graphics.[5] The Big Duck itself is a building near New York City, built by a duck farmer in the 1930s to serve as a shop from which he could sell duck eggs and ducks. It may not entirely surprise you to read that the Big Duck closely resembles a thirty-foot-long white duck. The architects Denise Scott Brown and Robert Venturi used the term “ducks” to describe any building that is designed to resemble a relevant product or service, such as a strawberry stand in the shape of a giant strawberry, or Shenzhen airport, which has the shape of an airplane.

The graphic guru Edward Tufte borrowed the “duck” term to describe a similar tendency in graphics: a graph about the NASA budget in the shape of a rocket; a graph about higher education in the shape of a mortarboard; or, in the example created for Time by Nigel Holmes, a graph about the price of diamonds in the shape of a diamond-adorned dame, her shapely fishnet-clad legs sketching out the price of a flawless 1-carat stone. Sometimes these visual puns do help people to read and remember the information they frame.[6] But they are often a poor attempt at humor, or a desperate bid to inject interest into data that seem dull. Data visualization ducks can be more than tasteless: the duckness of the graph can actually obscure—or worse, it can misrepresent—the underlying information.*

There is a curious historical parallel for this: dazzle camouflage. Dazzle was a defense mechanism for battleships in the First World War, always at risk of being torpedoed by a lurking submarine. The usual “blending in” method of camouflage wasn’t an option for a huge steel vessel that advertised its presence against an ever-changing sea and sky with bow waves and smokestacks. Dazzle camouflage flipped the idea of camouflage on its head. It was an abstract riot of squiggles and harlequin patterns—in fact, it bore enough of a resemblance to Cubist art that Picasso himself impishly tried to claim the credit.[7]

The real inventor of dazzle was Norman Wilkinson, a charismatic artist who joined the Royal Naval Reserve at the beginning of the war. He later explained, “Since it was impossible to paint a ship so that she could not be seen by a submarine, the extreme opposite was the answer—in other words, to paint her, not for low visibility, but in such a way as to break up her form and thus confuse a submarine officer as the course on which she was heading.”

Because torpedoes took some time to slice through the water to hit their target, the submarine’s periscope operator had swiftly to judge a ship’s speed and direction before firing the torpedo on an intercept course. Gazing through a tiny scope at a ship in dazzle camouflage, the operator knew he was looking at a ship, but would find it hard to make out any of the clues that mattered for aiming the torpedo accurately. The squiggles looked like bow waves, while harlequin diamonds could easily be confused with the various angled surfaces of a battleship’s hull. The result was that the lookout could easily misjudge the ship’s speed, the angle of its travel, and the size of and therefore distance to the ship. He might even see two ships rather than one, or mistake the bow for the stern and aim behind the ship rather than ahead of it. Dazzle camouflage was intended to provoke misjudgments.

More than a century later, it isn’t hard to see echoes of dazzle camouflage in infographics. From TV to newspapers, websites to social media, we are surrounded by graphical images that grab our attention, pleading to be shared and retweeted, but that also—intentionally or not—mislead, prodding us to a judgment that is often mistaken. At least the periscope operator whose eye was caught by a dazzle ship would have realized that he was looking at something odd, even if he couldn’t make sense of it. But many of us who are dazzled by infographics don’t suspect a thing.

—

All that lay far in the future when Florence Nightingale was a girl discovering a passion for data. At the age of nine she was categorizing and graphing the plants in her garden. As she grew older, she successfully pleaded with her parents to receive high-quality mathematical tuition; at dinner parties she met the likes of Charles Babbage, a mathematician and designer of a now-famous proto-computer; she was a houseguest of Ada Lovelace, Babbage’s collaborator; and she corresponded with the great Belgian statistician Adolphe Quetelet. Quetelet was the person who popularized the idea of taking the “average” or “arithmetic mean” of a group, which was a revolutionary way to summarize complex data with a single number. He also pioneered the idea that statistics could be used to analyze not just astronomical observations or the behavior of gases, but social, psychological, and medical questions such as the prevalence of suicide, obesity, or crime. Babbage and Quetelet were later to be founders of the Royal Statistical Society; Nightingale, as I have mentioned, became its first female fellow.

By her thirties, Florence Nightingale was steeped in this world of pioneering mathematicians—but her job was as nursing superintendent at a small hospital in London’s Harley Street, where she not only sorted out the bookkeeping and hospital infrastructure, but sent surveys to other hospitals all around Europe, asking them about their administrative practices and tabulating the results.

It was at this time, late in 1854, that she was persuaded by the secretary of state for war, her old friend Sidney Herbert, to lead a delegation of nurses to Istanbul to tend to wounded British soldiers from the Crimean War. The war was a bitter struggle between the Russian Empire and several other great European powers, including Britain. Nightingale’s presence with the British Army, which was unprecedented for a woman, was designed to pacify a public incensed by what they were reading about the dreadful conditions in the hospitals there. Reports in The Times turned the Crimean War into a long-running narrative of disaster with many familiar characters. By the end of it all, Florence Nightingale was perhaps the only figure to retain public support; the generals and the rest had been discredited by the catastrophe.

The Barrack Hospital in Scutari, Istanbul, was a death trap. Hundreds of soldiers from the Crimean front were succumbing to typhus, cholera, and dysentery as they tried to recover from their wounds in cramped conditions next to the sewers. Nightingale arrived to find rats and fleas everywhere she looked. Basics such as beds and blankets were missing, as were food to cook, pots to cook it in, and bowls to eat it from. All this outraged public opinion when it was reported in The Times, and Nightingale herself was quick to use the newspaper to raise funds from readers—and to pressure an ill-organized British Army to get its act together.

Less of a cause célèbre was that the hospital record-keeping was as badly organized as anything else. There were no standardized medical records and no consistent reporting between the various British Army hospitals. This may seem a relatively trivial matter, but Nightingale knew it was a big problem. Without good statistics it was impossible to understand why so many soldiers were dying, or to find a way to improve conditions. Even the dead were going uncounted, buried without their deaths being recorded. Nightingale saw all this more closely than anyone. She even took on the duty of writing to the family of each dead soldier. But she wanted the bird’s-eye view as well as personal experience, understanding that certain truths can only be perceived through the statistical lens. She tried to standardize and make sense of the hospital data.

Long after the war was over, Nightingale was still pressing to improve the standard of medical statistics. Some of this work, in partnership with Farr, was magnificently unglamorous. For example, they tried to standardize the description of different illnesses and causes of death, with Farr leading on the technical side while Nightingale campaigned for his ideas to be adopted. She wrote to the International Statistical Congress in 1860 to argue that hospitals should make use of Farr’s methods by collecting statistics according to a uniform standard. This wasn’t mere fussiness: standardizing the statistics meant that different hospitals could be compared and could learn from each other. It is this sort of statistical foundation-building that many of us overlook—but as we’ve seen many times in this book, without well-defined standards for statistical record-keeping, nothing adds up. Numbers can easily confuse us when they are unmoored from a clear definition.

Florence Nightingale may have been a savvy campaigner, but her campaigns were built on the most solid of foundations.

—

The most straightforward problem with a clever decorative idea is that the basic data may not be solid. The visualization then simply hides that fact—the shimmering icing over a moldering statistical cake.

One educational example is Debtris, an unforgettable animation produced several years ago by David McCandless, author of Information Is Beautiful.[8] It shows large blocks falling slowly against an eight-bit soundtrack in homage to the addictive computer game Tetris. Their size indicates their dollar value; “$60bn: estimated cost of Iraq war in 2003” is followed by “$3000bn: estimated total cost of Iraq war,” and then Walmart’s revenue, the United Nations’ budget, the cost of the financial crisis, and much else. As decoration this is wonderful stuff: the graph looks great, the music endlessly loops in your head, and the slow revealing of the different comparisons leaves you gasping with surprise, laughter, and anger.

But the same elements that make Debtris such a delight to watch also make it much harder to spot the underlying problems. Statistical apples are compared with statistical oranges throughout. Stocks are compared with flows. That’s the equivalent of comparing the total cost of buying a house with the annual cost of renting one; it’s not a trivial confusion. Net measures are put alongside gross ones—the equivalent of comparing a firm’s profit with its turnover.

The shocking difference between the before-and-after comparisons of the cost of the war in Iraq turns out to be based on an unfair comparison. (Admittedly, a fair comparison might also show a shocking difference.) The prewar number is a narrow estimate: the cost to the US military budget. The postwar number is very broad, including a figure for the cost of lost life, the cost of high oil prices, and a huge sum for the cost of macroeconomic instability, putting part of the blame for the 2008 financial crisis on the war. That broad estimate of cost is not unreasonable, but what is unreasonable is to put it alongside a very different kind of estimate without comment. What seems to be a pure before-and-after contrast is actually narrow-and-before versus broad-and-after, measuring a different thing at a different time. Nobody looking at the Debtris animation would realize that.

Debtris was published in 2010, and quickly became my favorite cautionary example—the visualization is so good but the data are so bad. A couple of years later I was introduced to David McCandless at a conference. I felt a bit awkward. I’d been moaning about his work while he wasn’t in the room, but I’d never done him the courtesy of sending him an email with my comments. But perhaps he hadn’t noticed them? I felt compelled to confess.

“I should probably say, David, that I have a concern about your Debtris animation.”

“I know you do,” he replied.

I squirmed. But to his credit, his more recent work is similarly striking, while also being more careful about the underlying data. For example, a visualization in a similar spirit—“The Billion Pound-O-Gram”—still mixes stocks and flows, but it is much more transparent about doing so.[9] Further to McCandless’s defense, the only reason I could find out that the data behind his Debtris animation were patchy and inconsistent is that he fully referenced it. Many don’t.

So information is beautiful—but misinformation can be beautiful, too. And producing beautiful misinformation is becoming easier than ever.

Graphics once required a great deal of time and trouble to produce and then reproduce. Even something as simple as a graphic with straight lines, precise edges, and color would have demanded expert draftsmanship and expensive printing methods. It’s telling that Edward Tufte, in a 1983 book, devotes some attention to deploring the use of diagonally shaded black-and-white patterns because they can produce an unsettling optional illusion of a flicker. “This moiré vibration [is] probably the most common form of graphical clutter,” he complains. It may have been common then; it is unheard of today. We would now invariably use color rather than diagonal shading.

No draftsmanship is required these days. A variety of powerful software tools can swiftly turn numbers into pictures. But any powerful tool should be used with care, and the very speed of the process means that impressive-seeming graphics can be created without any serious thought about the underlying data or how best to describe them.

The ease of creating pretty graphics is exceeded only by the ease of sharing them. A quick “like” on Facebook or a retweet on Twitter will speed the image on. Ideas that are best expressed in words or numbers are turned into graphics anyway, because that’s what spreads on social media. Unfortunately, the selection mechanism is often some combination of beauty and shock value, rather than pertinence and accuracy.

Consider the experience of Brian Brettschneider, a climate scientist with a fondness for gorgeous maps. He celebrated Thanksgiving in 2018 by producing a map showing “the favorite Thanksgiving Day pie by region,” including coconut cream pie for the Midwest, sweet potato pie for the West Coast, and key lime pie for the South. As a Brit, I don’t know much about Thanksgiving, and my favorite pie is a cold pork pie, but I’m told that the map seemed wrong to American eyes. No pumpkin pie? No apple pie? The map—and the outrage—went viral on Twitter. Senator Ted Cruz of Texas, a prominent Republican politician, didn’t like the suggestion that Texans favored key lime pie: “#FakeNews,” he tweeted.

And he was right. Brettschneider had made it all up. He was joking; the map was a parody of all the other bad maps that go viral on the internet. After more than a million people had seen the tweet, however, Brettschneider started to become uneasy. Did people even know he was joking? We don’t know who got the joke, who shared the map in mild outrage, and who believed it was solid fact. But we can be fairly sure that the use of a vivid graphic gave it its viral power. “We tend to place very high value in maps as holders of accurate information,” writes Brettschneider. “If it’s in a map, it must be true, right? If I had tweeted a joke list of favorite pies by region, it would be very quickly ignored. Since it was in map form, it had an air of authenticity.”[10]

Quite so. My only difference with Brettschneider is that I don’t think the problem is limited to maps. Any vivid graphic has the potential to go viral, whether true, false, or a bit of both. This book started with a warning that we should notice our emotional response to the factual claims around us. Just so: pictures engage the imagination and the emotion, and are easily shared before we have time to think a little harder. If we don’t, we’re allowing ourselves to be dazzled.

—

The situation in the Scutari hospital was catastrophic. Florence Nightingale was later to write, “To inexperienced eyes the Scutari buildings were magnificent. To ours, in their first state, they were truly whited sepulchers, pest houses.”[11] But why, exactly, were so many soldiers dying?

Poor hygiene is the obvious explanation from a modern perspective: germs were being transmitted freely in the filthy, vermin-ridden conditions. But the idea that diseases might be transmitted by microbes, and fought by using antiseptics and keeping things clean, was in its infancy. Very few doctors would even have heard of it as speculation, let alone believed it. Nightingale was no different; she thought instead that the high death toll in Scutari was due to lack of food and supplies, a problem she sought to remedy with her high-profile fundraising and campaigning through The Times.

Nevertheless, she also requested a team to help clean up the hospital, and in the spring of 1855 this “sanitary commission” arrived from the UK, whitewashed the walls, carted away filth and dead animals, and flushed out the sewers. The main hope was to make the hospital less unpleasant, but the immediate effect was to cut the death rate almost immediately from more than 50 percent to 20 percent.

Florence Nightingale wanted to understand what had happened, and why. And like Richard Doll and Austin Bradford Hill, she believed she could work out the truth if she examined the data with sufficient care. Her scrupulous record-keeping made the dramatic improvement after the sanitary commission’s work very clear indeed.

When Nightingale returned from the war, Queen Victoria summoned her for a royal audience. Nightingale persuaded Victoria to support a royal commission investigating the health of the army. She also recommended that the commission include William Farr, though Farr’s lowborn status meant that he was not treated well by the establishment; he was eventually retained only as an unpaid consultant to the commission.

Nightingale and Farr concluded that poor sanitation had caused many of the deaths in the Crimean War hospitals, and that most military and medical professionals had failed to learn this lesson. The problem was much bigger than one war: it was an ongoing public health disaster in barracks, civilian hospitals, and beyond. The pair began to campaign for better public health measures, tighter laws on hygiene in rented properties, and improvements to sanitation in barracks and hospitals across the country.*

Nightingale may have been the most famous nurse in the country, but she was a woman in a man’s world, and had to convince the country’s medical and military establishments, led by England’s chief medical officer, John Simon, that they had been doing things wrong all their lives. Dr. Simon wrote in 1858 that deaths from contagious diseases were “practically speaking, unavoidable”—that there was nothing to be done to prevent future deaths. Nightingale set herself the task of proving him wrong.

William Farr’s daughter, Mary, described eavesdropping on an early conversation between her father and Florence Nightingale. Mary recalled Farr giving Nightingale a warning about speaking out against the establishment. “‘Well, if you do it, you will make yourself enemies,’ and she drew herself up and answered, ‘After what I’ve seen, I can fire my own guns.’”[12]

Nightingale wrote to her friend the secretary of state for war Sidney Herbert, “Whenever I am infuriated, I revenge myself with a new diagram.”[13] Statistics had been the telescope through which she perceived the truth; now she needed a diagram that would compel everyone else to look at the truth, too.

—

A good chart isn’t an illustration but a visual argument,” declares Alberto Cairo near the beginning of his book How Charts Lie.[14] As the title of the book implies, Cairo has some concerns. If a good chart is a visual argument, a bad chart may be a confusing mess—or it may also be a visual argument, but a deceptive and seductive one. Either way, by organizing and presenting the data, we are inviting people to draw certain conclusions. And just as a verbal argument can be logical or emotional, sharp or woolly, clear or baffling, honest or misleading, so too can the argument made by a chart.

I should note here that not all good charts are visual arguments. Some data visualization is not intended to be persuasive, but exploratory. If you’re handling a complex dataset, you’ll learn a lot by turning it into a few different graphs to see what they show. Trends and patterns will often leap out immediately if plotted in the right way. For example, visualization expert Robert Kosara suggests plotting linear data on a spiral. If there’s a periodic pattern to the data—say, repeating every seven days or every three months—that may be concealed by other fluctuations in a conventional plot but will leap out in a spiral plot.

Similarly, certain kinds of problems make themselves known immediately when the data are turned into pictures. Imagine a dataset with the height and weight of tens of thousands of hospital patients. Some of them are fifty or sixty feet tall! That must be a typo. Hundreds of them have a weight of zero. That would be because a nurse or doctor was filling in an electronic form, didn’t take a weight measurement, so just hit “enter” and moved on to the next box. These problems won’t be apparent if you ask your computer to calculate an average or a standard deviation, or if you scan columns of data manually. If you look at a picture of the data, however, you’ll see the problem in a second.

But let’s assume you’ve explored your numbers, and now you want to turn them into a visual argument. The standard advice for management consultants and academic researchers presenting a graph is to include a title or caption that calls attention to the key features of the data and draws a conclusion.[15]

Say It with Charts, the bible of management consultants, makes this process very clear. First, says author Gene Zelazny, decide what you want to say with a graph. Once you’ve decided what you want to say, that suggests a particular kind of comparison. That, in turn, suggests a particular choice of graph—such as a scatterplot, a line graph, a stacked bar chart, or a pie chart.* Finally, underline your message by sticking it in the graph title. Don’t just write “Number of contracts, January–August.” Write something like “The number of contracts has increased” or “The number of contracts has been fluctuating,” depending on whether you’d like to call attention to the upward trend or to the variations around that trend. Zelazny’s vision is one in which the management consultant tells people what to think. Both the graphs and the annotation are chosen to support that message.

I realize there’s something unsettling about the way this process starts with the conclusion and then figures out how to package the data to support that conclusion. But let’s be fair: a lot of communication works in this way. Newspaper articles begin with a headline; the rest of the text is explanation. Even a scientific paper begins with an abstract that serves a similar purpose to a newspaper headline: it tells you what happened and what it means. A good journalist doesn’t begin reporting with the conclusion in mind; a good scientist doesn’t decide on the results before the experiment has been run. (I can’t vouch for what a good management consultant does.) But once both journalists and scientists have discovered something of interest, they want to give their audiences some pointers as to what it is. The same is true for chart designers.

Edward Tufte, the influential information designer, admires graphics that are dense and complex with a minimum of decoration or annotation. The introduction to one of his books, Envisioning Information, sternly warns readers, “The illustrations repay careful study. They are treasures, complex and witty, rich with meaning.” Look hard. Think. Pay attention at the back of the class. For Tufte, the ideal graphic invites the reader to sit down with a cup of coffee and really pore over the details. “Emaciated data-thin designs,” he warns, “provoke suspicions—and rightfully so—about the quality of measurement and analysis.”[16]

He may be right—although as we should know by now, the data density of the graph is no guarantee that the data themselves are reliable: a graph that presents a few data points in a lighthearted fashion may be unimpeachable, while an intricate graphic may be saturated with bad data.

Even if the numbers are solid, a graphic detailed enough to demand a coffee may be persuasive without also being informative. An impressive example is the New Yorker website’s 2013 presentation of data about inequality. The infographic, designed by Larry Buchanan, evokes the iconic New York City subway map. Viewers can click on different subway lines and see how median income varies along each line. This is an evocative data visualization “duck”: the graphs of rising and falling income resemble subway routes, and they carefully copy the distinctive design elements of the New York subway map and signage.[17]

What makes the infographic persuasive is that it invites us to make a natural comparison and immediately imagine the people behind it: we observe incomes varying along a chosen line as it moves through different neighborhoods, we grasp the vast inequality encompassed in a brief subway ride, and we picture the characters involved, rubbing shoulders in the subway car. The rich and poor are so close together, so similar in some ways and yet so different. The infographic carries a real emotional punch.

But is it informative? Not so much. As we click around, it is surprisingly hard to learn anything that we didn’t already know. It’s difficult to compare one subway line with another or to spot any but the most obvious patterns.

This becomes clear when we read the brief article accompanying the infographic, which is full of facts that cannot easily be discovered from the graphic itself. The highest median household income of a subway-endowed census tract in New York City was $205,192. The lowest was $12,288. The article tells us the subway lines with the largest and the smallest income ranges, and the largest gap between any two stations, although quite why any of this information is useful is unclear. The blog post notes that income inequality in Manhattan is similar to inequality in Lesotho or Namibia. Is that bad? It sounds bad. If you happened to carry around a list of the income inequality recorded in every country on the planet, you’d realize that it was bad. But do you? The goal of the graphic is not to convey information but to stir feelings. If the article compared income inequality in New York with that in other global cities such as London and Tokyo, or other US cities such as Chicago and Los Angeles, we might actually learn something worth knowing.

The result is gorgeous but far less informative than a map would have been. It is a piece of persuasive art pretending to be a piece of statistical analysis. We’ve been powerfully reminded of something we already believed. We are more passionate, more engaged, but are we truly any more informed?

There’s nothing wrong with a polemic—I write them myself, occasionally—but we should be honest with ourselves about what’s going on.

Another example is a graph by Simon Scarr, a senior designer at Thomson Reuters. The graph depicts deaths in Iraq in each month between 2003 and 2011. It’s an inverted bar graph: the larger the number of deaths that month, the longer the bar hangs down. Scarr colored his bars red, meaning the entire graph looks like blood running down from some awful gash at the top of the page. In case the message was ambiguous, the chart is titled “Iraq’s Bloody Toll.” If Larry Buchanan’s subway-inequality graph tugs at your heartstrings, Scarr’s graph rips your heart right out of your chest. Not for nothing did it win a design award.[18] And unlike the subway diagram, Scarr’s graph does give you the relevant information: it is persuasive and informative.

But when Andy Cotgreave, a data visualization expert, saw Scarr’s graph, he tried a little experiment. First, he recolored the graph, showing the same bars in a cool corporate blue-gray. Then he turned it upside down. Finally, he changed the title from “Iraq’s Bloody Toll” to “Iraq: Deaths on the Decline.” The change in the emotional impact is bracing. Scarr’s graph screams raw outrage. Cotgreave’s is sober, almost soothing. Which is the better graph? It depends on the message. Scarr’s graph wails, “Oh, the humanity!” Cotgreave’s graph calmly states, “The worst is behind us.” Both messages are fair. It’s a reminder that the simplest choices of color and alignment can change the tone of a chart and how people will perceive that chart, just as your tone of voice can dramatically alter how your words will be received.[19]

—

How could a lowborn statistician, William Farr, and a mere female, Florence Nightingale, win over the stubborn doctors and soldiers of the Victorian establishment?

First, they had to make sure their data were absolutely watertight. Facta, facta! Farr and Nightingale knew very well that their work would be pounced upon by their political enemies. In one telling exchange, Nightingale wrote to Farr warning him to prepare for an attack on his latest statistical analysis. His response showed the confidence he had in the quality of the work: “Let us wait, & keep our powder dry. We are not going to fire in the air—like people frightened out of their wits. Let them point out our ‘mistakes,’ and if they are mistakes—we will admit them freely: but shake our foundations—or blow down our walls—the fellows cannot.”[20]

Then they had to present their findings. Nightingale circulated her rose diagram in 1858 and published it early in 1859. That was just a few years after her time at Scutari hospital—and a matter of months after Dr. John Simon’s assertion that contagious diseases were practically unavoidable. The rose diagram (overleaf) is a brilliant visual argument. I’ve seen one of the original prints up close, in the library of the Royal Statistical Society. It’s breathtaking and alarming, a beautiful array of colored wedges showing deaths from infectious diseases before and after the sanitary improvements at Scutari.

If you wanted to be unkind about the diagram, you could say that it is a pie chart on steroids. Technically, it’s a polar area diagram, quite possibly the first such diagram ever created. What it isn’t is a dry presentation of statistical truth. It tells a story.

To see just how powerful a piece of visual rhetoric it is, consider the alternative presentation as a bar chart (the example below is based on a graph by Nightingale’s biographer Hugh Small, using William Farr’s data).

At first glance, Small’s bar chart is far clearer and easier to follow. But it draws the viewer to the wrong conclusion. It focuses attention on the dreadful death toll of January and February 1855, which might lead one to wonder if these deaths were basically caused by a bitter winter, and spring brought relief. It also makes the decline in deaths look dramatic but smooth—a process rather than a sharp change.

The polar area diagram, in contrast, divides the death toll into two periods—before the sanitary improvements and after. In doing so, it visually creates a sharp break that is less than clear in the raw data. Because the polar area diagram plots deaths in proportion to the area of a wedge, rather than the height of a bar, it also slightly obscures just how awful January and February 1855 were, instead lumping them together with the grim bulk of “before the sanitary commission.”

Nightingale wanted to make the importance of improved sanitation leap off the page, convincing the viewer that the Scutari experience could be repeated in hospitals, barracks, and even private dwellings across the British Empire. She created the powerful “before and after” structure of the diagram to strengthen that argument.

Is this dazzle camouflage? Perhaps. I’m inclined to say it isn’t, if only because the data are rock solid and there in plain sight. Unlike Debtris, it doesn’t rely on patchy statistics and unhelpful comparisons; unlike the subway-inequality diagram, it isn’t all sizzle and no steak. It’s more like “Iraq’s Bloody Toll,” but it is far more subtle in the way it invites readers to draw their conclusions. Few discussions of the rose diagram highlight just how clever it is at directing the reader toward one interpretation of the data and not another. Thankfully the idea was both true and important; the visual rhetoric helped people to reach a conclusion that happened to be correct.

Nightingale explained to Sidney Herbert that the diagram “is to affect thro’ the Eyes what we may fail to convey to the brains of the public through their word-proof eyes.” To get her diagram in front of as many eyes as possible, Nightingale asked the radical writer Harriet Martineau to produce a moving book about the Crimean War and the suffering of British soldiers there. Martineau had read Nightingale’s reports and praised them as “one of the most remarkable political or social productions ever seen.” Nightingale included her polar area diagram as a foldout frontispiece in Martineau’s book. It didn’t get read by as many soldiers as it might have done—the army banned it from military libraries and barracks[21]—but Nightingale had a more particular audience in mind for her diagram, as she told Herbert:

None but scientific men even look into the appendices of a Report, and this is for the vulgar public . . . Now, who is the vulgar public who is to have it? (1) The queen (2) Prince Albert . . . (7) all the crown heads in Europe, through the ambassadors or ministers of each (8) all the commanding officers in the army (9) all the regimental surgeons and medical officers . . . (10) the chief sanitarians in both houses [of Parliament] (11) all the newspapers, reviews and magazines.

The senior doctors who had argued that there was nothing to be done gradually came around to Nightingale’s argument for better sanitation. In the 1870s, Parliament passed several public health acts. Death rates in the UK began to fall, and life expectancy to rise.

What makes Florence Nightingale’s story so striking is that she was able to see that statistics could be tools and weapons at the same time. She appreciated the importance of solid foundations such as the tedious tasks of standardizing definitions and getting everyone to fill in the right forms, and of producing “the dryest of all” analyses, impervious to attack from the critics. But she also understood the need to give the data a makeover, presenting them in the most persuasive light. She produced a picture with enough power to change the world.

Florence Nightingale was on the right side of history, but many of the people who misuse catchy graphics are not. For those of us on the receiving end of beautiful visualizations, everything we’ve learned so far in this book applies.

First—and most important, since the visual sense can be so visceral—check your emotional response. Pause for a moment to notice how the graph makes you feel: triumphant, defensive, angry, celebratory? Take that feeling into account.

Second, check that you understand the basics behind the graph. What do the axes actually mean? Do you understand what is being measured or counted? Do you have the context to understand, or is the graph showing just a few data points? If the graph reflects complex analysis or the results of an experiment, do you understand what is being done? If you’re not in a position to evaluate that personally, do you trust those who were? (Or have you, perhaps, sought a second opinion?)

When you look at data visualizations, you’ll do much better if you recognize that someone may well be trying to persuade you of something. There is nothing wrong with artfully persuasive graphs, any more than with artfully persuasive words. And there is nothing wrong with being persuaded and changing your mind. That’s our next subject.

RULE TEN

Keep an Open Mind

A man with a conviction is a hard man to change. Tell him you disagree and he turns away. Show him facts or figures and he questions your sources. Appeal to logic and he fails to see your point.

• Leon Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails[1]

Irving Fisher was one of the greatest economists who ever lived.[2]

“Anywhere from a decade to two generations ahead of his time,” opined the first Nobel laureate economist, Ragnar Frisch, in the late 1940s, more than half a century after Fisher’s genius first lit up his subject. Paul Samuelson, who won the Nobel Memorial Prize the year after Frisch, said that Irving Fisher’s 1891 PhD thesis “was the greatest doctoral dissertation in economics ever written.”

That’s what Fisher’s peers thought. The public loved him, too. A hundred years ago, Irving Fisher was the most famous economist on the planet. Yet Fisher is remembered now only by economists with a sense of history. He’s no longer a household name like Milton Friedman, Adam Smith, or John Maynard Keynes, his younger contemporary. That is because something awful happened to Irving Fisher, and to his reputation—something with a lesson for us all.

Fisher’s downfall certainly wasn’t through lack of ambition. “How much there is I want to do!” he wrote to an old school friend while studying at Yale. “I always feel that I haven’t time to accomplish what I wish. I want to read much. I want to write a great deal. I want to make money.”

It was understandable that money was important to Fisher. His father had died of tuberculosis the very week that Irving arrived at Yale. Fisher’s drive and intellect kept him afloat: he won prizes in Greek and Latin, for algebra and mathematics, for public speaking (finishing second to a future US secretary of state), and was both the class valedictorian and a member of the rowing crew. Yet amid all these achievements, the young man needed to scramble for funds throughout his studies; he understood what it was to struggle financially while surrounded by wealth.

At the age of twenty-six, however, Fisher found himself with a small fortune at his disposal. He married a childhood playmate, Margaret Hazard, who was the daughter of a wealthy industrialist. Irving and Margaret’s wedding in 1893 was sumptuous enough to be covered by the New York Times, with two thousand invited guests, three ministers, an extravagant lunch, and a sixty-pound wedding cake. They commenced a fourteen-month European honeymoon and returned to a brand-new mansion at 460 Prospect Street, New Haven. It had been built in their absence as a wedding present from Margaret’s father and was furnished with a library, a music room, and spacious offices.

There are three things you should know about Irving Fisher.

The first is that he was a health fanatic. This was understandable. Tuberculosis had killed the young man’s father; fifteen years later, the disease nearly killed him, too. No wonder he adopted a fastidious health regime: he abstained from alcohol, tobacco, meat, tea, coffee, and chocolate. One dinner guest enjoyed his hospitality while noting his quirkiness: “While I ate right through my succession of delicious courses, he dined on a vegetable and a raw egg.”[3]

This wasn’t just a personal matter: he was an evangelist for health and nutrition. He founded the Life Extension Institute and persuaded William Howard Taft, who’d just finished his presidential term, to be its chairman. (It may seem an incongruous choice: Taft was obese, the heaviest man ever to be president. Taft’s weight problem did, however, prompt his interest in diet and exercise.) In 1915, when Fisher was nearly fifty years old, he published a book titled How to Live: Rules for Healthful Living Based on Modern Science. (How to live! Now that’s real ambition.) It was a huge bestseller, and it’s hilarious from a modern perspective. He advocated a “sun-bath . . . Common sense must dictate its intensity and duration.” It was “important [to] practice thorough mastication . . . chewing to the point of natural, involuntary swallowing.” He even adds a discussion of the correct angle between the feet while walking—“about seven or eight degrees of out-toeing in each foot.”[4]

And there’s a short section on eugenics. It hasn’t aged well.

But while it’s easy to laugh at the book, How to Live is in many ways as far ahead of its time as Fisher’s economic analysis. He applied scientific thinking to the question of well-being. He described detailed exercises, preached mindfulness, and at a time when the majority of doctors were smokers, correctly warned that tobacco causes cancer.

That is the second thing you need to know about Irving Fisher: he believed in the power of rational, numerical analysis, in economics and elsewhere. He calculated the net economic cost of tuberculosis. He conducted experimental investigations of vegetarianism and even of thorough mastication, which he found to increase endurance. (A 1917 advertisement for the breakfast cereal Grape-Nuts included an endorsement from Professor Fisher.) At one point in How to Live, he even pauses to inform the reader that “in the modern study of scientific clothing there is a new unit, the ‘clo.’ This is a technical unit for measuring the ‘warming power’ of clothing.”

It is arguable that his love of numbers occasionally led him astray. For example, when Fisher quantified the benefits of Prohibition, he exuberantly generalized from a small study that a stiff drink on an empty stomach made workers 2 percent less efficient. Fisher calculated that Prohibition would add $6 billion to America’s economy—which at the time was an absolutely enormous gain. We saw in the first chapter that Abraham Bredius’s art expertise had highlighted reasons to believe that Han van Meegeren’s rotten forgery was truly a Vermeer. Similarly, Fisher’s statistical expertise allowed him to produce grand calculations about Prohibition on a shaky foundation stone. His strong feelings about the evils of alcohol were undermining the rigor of his statistical reasoning.[5]

There’s also the money—that’s the third thing you need to know. Irving Fisher was rich, and not just because of his wife’s inheritance. Making money was a matter of pride for Fisher; he didn’t want to be dependent on his wife. There were the book royalties from How to Live. There were his inventions, most notably a way of organizing business cards that was the forerunner of the Rolodex. He sold that invention, Index Visible, to a stationery company for $660,000 in cash—many millions of dollars in today’s terms—a seat on its board, and a bundle of stocks.

Fisher turned his academic research into a major business operation called the Index Number Institute. It sold data, forecasts, and analysis as a syndicated package, Irving Fisher’s Business Page, to newspapers across the United States. Forecasting was a natural extension of the data and analysis. After all, if we want to make the world add up, it’s not always because the intellectual joy of understanding is an end in itself. Sometimes we’re interested in sizing up the current situation as a means to anticipate, and perhaps profit from, what will happen next.

With such a platform, Fisher was able to evangelize about his approach to investment—which, broadly speaking, was to bet on American growth by buying shares in the new industrial corporations using borrowed money. Such borrowing is often called leverage, since it magnifies both profits and losses.

But during the 1920s stock market investors had few losses to worry about. Share prices were soaring. Anyone who had made leveraged bets on that growth had every reason to feel clever. Fisher wrote to his old childhood friend to inform him that his ambition had been fulfilled. “We are all making a lot of money!”

In the summer of 1929, Irving Fisher—bestselling author, inventor, friend of presidents, entrepreneur, health campaigner, syndicated columnist, statistical pioneer, the greatest academic economist of his generation, and a millionaire many times over—was able to boast to his son that a renovation of the family mansion had been paid for not by Hazard family money but by Irving Fisher himself.

That achievement mattered to him. Fisher’s own father hadn’t lived to see his seventeen-year-old boy grow into one of the most respected figures of the age; as Irving and his son watched a mansion reshaped before them, he could, perhaps, be forgiven his pride. But he was standing on the brink of a financial precipice.

—

The stock market cracked in the autumn of 1929. The Dow Jones Industrial Average fell by more than a third between the beginning of September and the end of November. But it wasn’t the great Wall Street crash that did in Irving Fisher—at least, not immediately. The crash, of course, was a cataclysmic financial event, one far more severe even than the banking crisis of 2008. The Great Depression that followed was the greatest peacetime economic calamity to befall the Western world. Fisher was more exposed than many, since he had made his investments with leverage, magnifying both losses and gains.

But it took more than a leveraged bet on a financial bubble to ruin Fisher. It took stubbornness. The crash had its dramatic moments, but it was not simply a matter of lurches on days such as Black Thursday and Black Monday. It was best understood as a long downward grind, punctuated by brief rallies, all the way from 380 points in September 1929 to just over 40 points by the summer of 1932. If Fisher had cut his losses and stepped back from the market in late 1929, he would have been fine. He could have returned to his academic research and his many other enthusiasms, and his luxurious lifestyle funded by many years of trading profits along with his income as an author and businessman.

Instead, Fisher doubled down on his initial views. He was convinced the market would turn upward again. He made several comments about how the crash was the “shaking out of the lunatic fringe” and reflected “the psychology of panic.” He publicly declared that recovery was imminent. It was not.

Most important, he didn’t just stay invested in the market. His confidence that he was right made him continue to rely on borrowed money in the hope of bigger gains. One of Fisher’s major investments was in Remington Rand, following his sale of his Rolodex system, Index Visible. The share price tells the story: $58 before the crash, $28 within a few months. Fisher might have learned by then that leverage was terribly risky. But no: he borrowed more money to invest—and the share price soon dropped to $1. That is a sure route to ruin.

We shouldn’t be too quick to judge Fisher. Even if you’re the smartest one in the room—and Irving Fisher usually was—it simply isn’t easy to change your mind.

—

Irving Fisher’s contemporary Robert Millikan was no less distinguished a man than Fisher. His interests were a little different, however: Millikan was a physicist. In 1923, as Fisher’s stock tips were being devoured across the United States, Millikan was collecting a Nobel Prize.

For all his achievements, Millikan is most famous for an experiment so simple that a schoolkid can attempt it: the “oil drop” experiment, in which a mist of oil droplets from a perfume spritzer is given an electrical charge while floating between two electrified plates. Millikan could adjust the voltage between the plates until the droplets were suspended, without moving—and since he could measure the diameter of the droplets, he could calculate their mass, and thus also the electrical charge that was precisely offsetting the pull of gravity. This, in effect, allowed Millikan to calculate the electrical charge of a single electron.

I was one of countless students who attempted this experiment in school, but in all honesty I was unable to get my results quite as neat as Millikan’s. There are a lot of details to get right—in particular, the experiment depends on correctly measuring the diameter of the tiny oil droplet. Mismeasure that, and all your other calculations will be off.

We now know that even Millikan didn’t get his answers quite as neat as he claimed he did. He systematically omitted observations that didn’t suit him, and lied about those omissions. (He also minimized the contribution of a junior colleague, Harvey Fletcher.) Historians of science argue about the seriousness of this cherry-picking, ethically and practically. What seems clear is that if the scientific world had seen all of Millikan’s results, it would have had less confidence that his answer was right. That would have been no bad thing, because it wasn’t. Millikan’s answer was too low.[6]

The charismatic Nobel laureate Richard Feynman pointed out in the early 1970s that the process of fixing Millikan’s error with better measurements was a strange one: “One is a little bit bigger than Millikan’s, and the next one’s a little bit bigger than that, and the next one’s a little bit bigger than that, until finally they settle down to a number which is higher. Why didn’t they discover the new number was higher right away?”[7]

The answer is that whenever a number was close to Millikan’s, it was accepted without too much scrutiny. When a number seemed wrong it would be viewed with skepticism. Reasons would be found to discard it. As we saw in chapter one, our preconceptions are powerful things. We filter new information. If it accords with what we expect, we’ll be more likely to accept it.

And since Millikan’s estimate was too low, it would be rare to have a measurement that was so much lower as to be unexpected. Typically, the surprising measurements would be substantially larger than Millikan’s instead. Accepting them was a long and gradual process. It wasn’t helped by the fact that Millikan had discarded some of his measurements to make himself seem like a more accomplished scientist. But we can be confident that it would have happened anyway, because a later study found the same pattern of gradual convergence in other estimates of physical constants, such as Avogadro’s number and Planck’s constant.* Convergence continued throughout the 1950s and 1960s and sometimes into the 1970s.[8] It’s a powerful demonstration of the way that even scientists measuring essential and unchanging facts filter the data to suit their preconceptions.

This shouldn’t be entirely surprising. Our brains are always trying to make sense of the world around us based on incomplete information. The brain makes predictions about what it expects, and tends to fill in the gaps, often based on surprisingly sparse data. That is why we can understand a routine telephone conversation on a bad line—until the point at which genuinely novel information such as a phone number or street address is being spoken through the static. Our brains fill in the gaps—which is why we see what we expect to see and hear what we expect to hear, just as Millikan’s successors found what they expected to find. It is only when we can’t fill in the gaps that we realize just how bad the connection is.

We even smell what we expect to smell. When scientists give people a whiff of a scent, the reactions differ sharply depending on whether the scientists tell the experimental subjects, “This is the aroma of a gourmet cheese,” or “This is the stink of armpits.”[9] (It’s both: they’re smelling an aromatic molecule present in both runny cheese and bodily crevices.)

This process of sensing what you expect to sense is widespread. In the cheese study, it was visceral. In the case of the electron charge or Avogadro’s number, it was cerebral. In both cases, it seems to have been unconscious.

But we can also filter new information consciously, because we don’t want it to spoil our day. Back in the first chapter, we encountered students who’d pay not to have their blood tested for herpes and investors who avoided checking their stock portfolios when the news might be bad. Here’s another example—a study published in 1967, which asked undergraduates to listen to tape-recorded speeches and requested that they “judge the persuasiveness and sincerity of talks prepared by high school juniors and seniors . . . After each talk you will be given a rating sheet to rate the persuasiveness and sincerity of the speech.”

However, there was a catch. The talks were clouded with annoying static. The experimental subjects were told: “Since the talks were recorded on a small portable tape recorder there is considerable electrical interference. The interference can be ‘adjusted out’ by pressing and then immediately releasing the control button. Use of the control several times in a row reduces somewhat the static and other interference noise.”[10]

Fine. Of course, as you can guess by now, the experiment involved some deception. Some of the undergraduates were committed Christians, and others were committed smokers. One of the talks was based on an old-school atheistic pamphlet titled Christianity Is Evil, another relied on “an authoritative refutation of the arguments linking smoking to lung cancer,” and a third spoke with similar authority about the fact that smoking did cause lung cancer.

As we’ve seen, all of us are capable of metaphorically filtering the information that comes our way, discarding some ideas and clinging on to others. In this experiment, the filter was more literal: static that obscured the messages the experimental subjects were supposed to listen to and evaluate. Pressing a button could remove the crackle and hiss—but not everyone enthusiastically mashed the button for every speech. It may not surprise you to hear that the Christians were content to leave the militant atheism behind a reassuring fog of static. Smokers pressed the button repeatedly to listen to the talk explaining that their habit was perfectly safe, while allowing the static to float back in when a different taped message told them unwelcome news.

One of the reasons facts don’t always change our minds is that we are keen to avoid uncomfortable truths. These days, of course, we don’t need to mess around with a static-reducing button. On social media we can choose who to follow and who to block. A vast range of cable channels, podcasts, and streaming video lets us decide what to watch and what to ignore. We have more such choices than ever before, and you can bet that we’ll use them.

If you do have to absorb unwelcome facts, not to worry: you can always selectively misremember them. That was the conclusion of Baruch Fischhoff and Ruth Beyth, two psychologists who ran an elegant experiment in 1972. They conducted a survey in which they asked male and female students for predictions about Richard Nixon’s imminent presidential visits to China and the Soviet Union. How likely was it that Nixon and Mao Zedong would meet? What were the chances that the United States would grant diplomatic recognition to China? Would the US and USSR announce a joint space program?

Fischhoff and Beyth wanted to know how people would later remember their forecasts. They’d given their subjects every chance, since the forecasts were both specific and written down. (Usually our forecasts are rather vague prognostications in the middle of conversation. We rarely commit them to writing.) So one might have hoped for accuracy. But no—the subjects flattered themselves hopelessly. If they put some event at a 25 percent likelihood and then it happened, they might then remember they’d called it as a 50/50 proposition. If a subject had put a 60 percent probability on an event that later failed to happen, she might later recall that she’d forecast a 30 percent probability. The Fischhoff–Beyth paper was titled “I Knew It Would Happen.”

It’s yet another striking illustration of how our emotions lead us to filter the most straightforward information—our own memory of an estimate we made not long ago, and went to the trouble of committing to paper.[11] In some ways, this shows a remarkable mental flexibility. But rather than admit error and learn from it, Fischhoff and Beyth’s subjects were changing their own recollections to ensure that no painful reckoning with reality was required. As we’ve seen: admitting you’re wrong, then changing your view, is not an easy thing to do.

—

Of course, Irving Fisher wouldn’t have had to change his mind if he’d been right all along. Perhaps his real downfall was not the failure to adjust, but the failure to forecast accurately in the first place? Perhaps. It is certainly preferable to be right the first time than to learn through painful experience. But the best studies we have of forecasting ability suggest that being right the first time isn’t easy either.

In 1987, a young Canadian-born psychologist, Philip Tetlock, planted a time bomb under the forecasting industry that would not explode for eighteen years. Tetlock had been part of a rather grand project in which social scientists had been tasked with preventing nuclear war between the United States and the Soviet Union. As part of that project, he had interviewed many top experts to get their sense of what was happening in the Soviet Union, how the Soviets might respond to Ronald Reagan’s hawkish stance, what might happen next, and why.

But he found himself frustrated: frustrated by the fact that the leading political scientists, Sovietologists, historians, and policy wonks had such contradictory views about what might happen next; frustrated by their refusal to change their minds in the face of contradictory evidence; and frustrated by the many ways in which even failed forecasts could be justified. Some predicted disaster, but were happy to rationalize the lack of catastrophe: “I was nearly right but fortunately it was Gorbachev rather than some neo-Stalinist who took over the reins.” “I made the right mistake: far more dangerous to underestimate the Soviet threat than overestimate it.” Or, of course, the get-out for all failed stock market forecasts, “Only my timing was wrong.”

Tetlock’s response was patient, painstaking, and quietly brilliant. Following in the footsteps of Fischhoff and Beyth, but with more detail and on a much larger scale, he began to collect forecasts from almost three hundred experts, eventually accumulating 27,500 predictions. The main focus of the questions he asked was on politics and geopolitics, with a few from other areas such as economics thrown in. Tetlock sought clearly defined questions, enabling him with the benefit of hindsight to pronounce each forecast right or wrong. Then he simply waited while the results rolled in—for eighteen years.

Tetlock published his conclusions in 2005, in a subtle and scholarly book, Expert Political Judgment. He found that his experts were terrible forecasters. This was true in both the simple sense that the forecasts failed to materialize and in the deeper sense that the experts had little idea of how confident they should be in making forecasts in different contexts. It is easier to make forecasts about the territorial integrity of Canada than about the territorial integrity of Syria, but beyond the most obvious cases, the experts Tetlock consulted failed to distinguish the Canadas from the Syrias. Tetlock’s experts, like Fischhoff and Beyth’s amateurs, also dramatically misremembered their own forecasts, recalling some of their failures as things they’d been right about all along.[12]

Adding to the appeal of this tale of expert hubris, Tetlock found that the most famous experts made even less accurate forecasts than those outside the media spotlight. Other than that, the humiliation was evenly distributed. Regardless of political ideology, profession, and academic training, experts failed to see into the future.

Most people, hearing about Tetlock’s research, simply conclude that the world is too complex to forecast or that experts are too stupid to forecast it, or both. But there was one person who kept faith in the possibility that even for intractable human questions of macroeconomics and geopolitics, a forecasting approach might exist that would bear fruit. That person was Philip Tetlock himself.

—

In 2013, on the auspicious date of April 1, I received an email from Tetlock inviting me to join what he described as “a major new research program funded in part by Intelligence Advanced Research Projects Activity, an agency within the U.S. intelligence community.”

The core of the program, which had been running since 2011, was a collection of quantifiable forecasts much like Tetlock’s long-running study. The forecasts would be of economic and geopolitical events, “real and pressing matters of the sort that concern the intelligence community—whether Greece will default, whether there will be a military strike on Iran, etc.” These forecasts took the form of a tournament with thousands of contestants; the tournament ran for four annual seasons.

“You would simply log on to a website,” Tetlock’s email continued, “give your best judgment about matters you may be following anyway, and update that judgment if and when you feel it should be. When time passes and forecasts are judged, you could compare your results with those of others.”

I did not participate. I told myself I was too busy; perhaps I was too much of a coward as well. But the truth is that I did not participate because, largely thanks to Tetlock’s work, I had concluded that the forecasting task was impossible.

Still, more than 20,000 people embraced the idea. Some could reasonably be described as having some professional standing, with experience in intelligence analysis, think tanks, or academia. Others were pure amateurs. Tetlock and two other psychologists, Barbara Mellers (Mellers and Tetlock are married) and Don Moore, ran experiments with the cooperation of this army of volunteers. Some were given training in some basic statistical techniques (more on this in a moment); some were assembled into teams; some were given information about other forecasts; and others operated in isolation. The entire exercise was given the name Good Judgment Project, and the aim was to find better ways to see into the future.

This vast project has produced a number of insights, but the most striking is that there was a select group of people whose forecasts, while they were by no means perfect, were vastly better than the dart-throwing-chimp standard reached by the typical prognosticator. What is more, they got better over time rather than fading away as their luck changed. Tetlock, with an uncharacteristic touch of hyperbole, called this group “superforecasters.”

The cynics were too hasty: it is possible to see into the future after all.

What makes a superforecaster? Not subject-matter expertise: professors were no better than well-informed amateurs. Nor was it a matter of intelligence; otherwise Irving Fisher would have been just fine. But there were a few common traits among the better forecasters.

First, encouragingly for us nerds, it did help to have some training—of a particular kind. Just an hour of training in basic statistics improved the performance of forecasters by helping them turn their expertise about the world into a sensible probabilistic forecast, such as “The chance that a woman will be elected president of the United States within the next ten years is 25 percent.” The tip that seemed to help most was to encourage them to focus on something called “base rates.”[13]

What on earth are base rates? Well, imagine that you find yourself at a wedding, sitting at one of the back tables with the drunk school friends of the groom or the disgruntled ex-boyfriend of the bride. (Yes, that sort of wedding.) At a tedious moment during one of the speeches, the conversation at your table turns to the distasteful question: Will these two actually make it? Will the marriage last or is the relationship doomed to divorce?

The instinctive starting point is to think about the couple. It’s always hard to imagine divorce in the middle of the romance of a wedding day (although sharing a whiskey with the bride’s ex-boyfriend may shake you out of that rosy glow), but you’d naturally ponder questions such as: “Do they seem happy and committed to each other?” and “Have I ever seen them argue?” and “Have they split up and gotten back together three times already?” In other words, we make a forecast with the facts that are in front of our nose.

But it is a better idea to zoom out and find one very straightforward* statistic: In general, how many marriages end in divorce? This number is known as the “base rate.” Unless you know whether the base rate is 5 percent or 50 percent, all the gossip you’re getting from the grumpy ex doesn’t fit into any useful framework.

The importance of the base rate was made famous by the psychologist Daniel Kahneman, who coined the phrase “the outside view and the inside view.” The inside view means looking at the specific case in front of you: this couple. The outside view requires you to look at a more general “comparison class” of cases—here, the comparison class is all married couples. (The outside view needn’t be statistical, but it often will be.)

Ideally, a decision maker or a forecaster will combine the outside view and the inside view—or, similarly, statistics plus personal experience. But it’s much better to start with the statistical view, the outside view, and then modify it in the light of personal experience than it is to go the other way around. If you start with the inside view you have no real frame of reference, no sense of scale—and can easily come up with a probability that is ten times too large, or ten times too small.

Second, keeping score was important. As Tetlock’s intellectual predecessors Fischhoff and Beyth had demonstrated, we find it challenging to do something as simple as remembering whether our earlier forecasts had been right or wrong.

Third, superforecasters tended to update their forecasts frequently as new information emerged, which suggests that a receptiveness to new evidence was important. This willingness to adjust predictions is correlated with making better predictions in the first place: it wasn’t just that the superforecasters beat the others because they were news junkies with too much time on their hands, prospering by endlessly tweaking their forecasts with each new headline. Even if the tournament rules had demanded a one-shot forecast, the superforecasters would have come top of the heap.

Which points to the fourth and perhaps most crucial element: superforecasting is a matter of having an open-minded personality. The superforecasters are what psychologists call “actively open-minded thinkers”—people who don’t cling too tightly to a single approach, are comfortable abandoning an old view in the light of fresh evidence or new arguments, and embrace disagreements with others as an opportunity to learn. “For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded,” wrote Philip Tetlock after the study had been completed. “It would be facile to reduce superforecasting to a bumper-sticker slogan, but if I had to, that would be it.”[14]

And if even that is too long for the bumper sticker, what about this: superforecasting means being willing to change your mind.

The unfortunate Irving Fisher had struggled to change his mind. Not everyone had the same difficulty. The contrast with John Maynard Keynes is striking, despite the many similarities the two men shared. Keynes, like Fisher, was a colossal figure in economics. Like Fisher, he was a popular author, a regular newspaper commentator, a friend of powerful politicians, and a charismatic speaker. (After witnessing Keynes giving a speech, the Canadian diplomat Douglas LePan was moved to write: “I am spellbound. This is the most beautiful creature I have ever listened to. Does he belong to our species? Or is he from some other order?”)[15] And like Fisher, Keynes was an enthusiastic participant in financial markets—founding an early hedge fund, dabbling in currency speculation, and managing a large portfolio on behalf of King’s College, Cambridge. His ultimate fate, however, was very different. The similarities and the contrasts between the two men are instructive.

Unlike Fisher, who had had to scramble for his success, Keynes was the ultimate insider. As a schoolboy Keynes was educated at Eton College—just like Britain’s first prime minister, and nineteen others since. Like his father, he became a senior academic: a fellow of King’s College, the most spectacular of all the Cambridge colleges. His job during the First World War was managing both debt and currency on behalf of the British Empire; he’d barely turned thirty. He knew everyone. He whispered in the ear of prime ministers. He had the inside track on whatever was going on in the British economy—the Bank of England would even call him to give him advance notice of interest rate movements.

But this child of the British establishment was a very different person from his American counterpart. He loved fine wines and rich food; he gambled at Monte Carlo. His sex life was more like that of a 1970s pop star than a 1900s economist: bisexual, polyamorous, eventually settling down not with his childhood sweetheart but with a Russian ballerina, Lydia Lopokova. One of Keynes’s ex-boyfriends was the best man at their wedding.

He was adventurous in other ways, too. In 1918, for example, Keynes worked at the British Treasury. The First World War was still raging. The German army was camped outside Paris, shelling the city. But Keynes caught wind of the fact that in Paris, the great French artist Edgar Degas was about to auction his vast collection of pieces by France’s greatest nineteenth-century painters: Manet, Ingres, and Delacroix.[16]

And so Keynes launched an insane adventure. First, he persuaded the British Treasury, which was four years into fighting the most devastating war the planet had yet seen, to put together a fund for purchasing £20,000 worth of art—millions in today’s money. There was certainly a logic to the idea that it was a buyer’s market, but you’ve got to be pretty persuasive to free up funds from a wartime treasury to splurge on nineteenth-century French art.

Then, escorted by destroyers and a silver airship, Keynes crossed the Channel to France with the director of London’s National Gallery, who was wearing a fake mustache so that nobody recognized him. With the German artillery booming beyond the horizon, they showed up at the auction and cleared Degas out. The National Gallery got twenty-seven masterpieces at rock-bottom prices. Keynes even bought a few for himself.

After escaping back across the Channel, and exhausted after his adventures in Paris, Keynes showed up at the door of his friend Vanessa Bell and told her that he’d left a Cézanne outside in the hedge—could he please have a hand carrying it in? (Bell was the sister of the author Virginia Woolf and the lover of Keynes’s ex-boyfriend Duncan Grant, although she was married to someone else; Keynes’s social circle was complicated.) Keynes had gotten himself a bargain: these days a good Cézanne is worth a lot more than anything the National Gallery dared to purchase at the auction. But what Irving Fisher would have made of it all, I do not know.

At the end of the war, Keynes represented the British Treasury at the peace conference in Versailles. (He was disgusted at the outcome—and subsequent events proved him right.) Then, with currencies free-floating and volatile, Keynes set up what some historians describe as the first hedge fund to speculate on their movements. He raised capital from rich friends and from his own father, to whom he made the not entirely reassuring comment, “Win or lose, this high-stakes gambling amuses me!”

Initially Keynes made money fast—over £25,000, even more than the art fund he’d wheedled out of the Treasury. His bet, in brief, was that the currencies of France, Italy, and Germany would suffer in a bout of postwar inflation. In this he was broadly correct. Yet there’s an old saying, often attributed (without evidence) to Keynes himself: The market can stay wrong longer than you can stay solvent. A brief surge of optimism about Germany’s prospects wiped out Keynes’s fund in 1920. Undaunted, he went back to his investors. “I am not in a position to risk any capital myself, having quite exhausted my resources,” he noted. But the spellbinding Keynes persuaded others to invest and his fund was back in profit by 1922.

One of Keynes’s next investment projects—he had several—concerned the portfolio of King’s College. Five centuries old, the college had long-standing rules on its investment policy, leaving it reliant on agricultural rents and very conservative investments such as railway bonds and government gilts. In 1921 the ever-persuasive Keynes convinced the college to change these rules to give him complete discretion over a significant slice of the college portfolio.

Keynes’s strategy for this money was top-down. He would forecast booms and recessions both in the UK and abroad, and invest in shares and commodities accordingly, moving across different sectors and countries depending on the macroeconomic outlook.

Such an approach seemed to make sense. Keynes was the leading economic theorist in the country. He was receiving tips from the Bank of England. If anyone could call the ebb and flow of the British economy, it was John Maynard Keynes.

If.

Keynes, like Fisher, did not predict the great crash of 1929. Unlike Fisher, though, he recovered. Keynes died a millionaire, his reputation enhanced by his financial acumen. The reason is simple: Keynes, unlike Fisher, changed his mind, and his investment strategy.

Keynes had one advantage over Fisher: his track record as an investor had been painfully mixed. Yes, he had scored a remarkable coup in the art auction of 1918, and made a small fortune in the currency markets in 1922. But he had been wiped out in 1920, and his clever-seeming approach with the King’s College portfolio wasn’t working either. Over the course of the 1920s, Keynes’s attempts to forecast the business cycle had led him to trail the market as a whole by about 20 percent. That is not a disaster, but it is certainly an indication that all is not well.

None of this helped Keynes see the great crash of 1929, but it did help him react to it. He had already been pondering his limitations as an investor, and wondering whether a different approach might pay off. When the crash hit, Keynes shrugged and adjusted.

By the early 1930s, Keynes had abandoned business-cycle forecasting entirely. The greatest economist in the world had decided that he just couldn’t do it well enough to make money. It is a striking instance of humility from a man famous for his self-confidence. But Keynes had looked at the evidence and done something unusual: he’d changed his mind.

He moved instead to an investment strategy that required no great macroeconomic insight. Instead, he explained, “As time goes on, I get more and more convinced that the right method in investment is to put fairly large sums into enterprises which one thinks one knows something about and in the management of which one thoroughly believes.” Forget what the economy is doing; just find well-managed companies, buy some shares, and don’t try to be too clever. And if that approach sounds familiar, it’s most famously associated with Warren Buffett, the world’s richest investor—and a man who loves to quote John Maynard Keynes.

Keynes is rightly viewed today as a successful investor. At King’s College, he recovered from the poor performance of the early years. When two financial economists, David Chambers and Elroy Dimson, recently studied Keynes’s track record with the King’s College portfolio, they found it to be excellent. Keynes secured high returns with modest risks, and outperformed the stock market as a whole by an average of six percentage points a year over a quarter of a century. That’s an impressive reward for being able to change your mind.[17]

—

It all sounds so simple: things are going badly, so do something different. Why, then, did Irving Fisher struggle to adapt?

Fisher’s first problem, ironically, was his successful track record. He was seriously wealthy by the end of the 1920s, having prospered in almost every endeavor he had attempted. As an investor, he had correctly predicted the productivity boom of the 1920s and correctly judged that the stock market would soar, and his leveraged bets on those judgments had paid off handsomely. Unlike Keynes, Fisher had received very little evidence of his own fallibility. It must have been hard for him to take in the scale of the financial bloodbath. It was all too tempting to write it off as a brief spasm of lunacy, which is what Fisher did.

In contrast, when the market crashed, Keynes was able to see it—and himself—for what it was. He’d been in crashes before, and lost heavily before. He was like a physicist who’d been forewarned that Robert Millikan’s research was flawed, so his estimates shouldn’t be taken too seriously; or perhaps like an experimental subject sniffing a test tube after being told, “This might be cheese or it might be armpits, so think carefully.”

Fisher was vulnerable in a second way. He was constantly writing about his investment ideas, pinning his reputation to the idea that the stock market was on the up-and-up. There is a lot of vague prophecy in the forecasting business, so such public commitments are admirably honest. They are also dangerous. It wasn’t the concreteness of the predictions that was the problem. As we’ve seen, superforecasters tend to keep a careful record of their predictions. How else can they learn from their mistakes? No: it was the high public profile that made it harder for Fisher to change his mind.

One study of this, conducted by psychologists Morton Deutsch and Harold Gerard in 1955, asked college students to estimate the lengths of lines—a modification of the experiments conducted by Solomon Asch a few months previously, described in the sixth chapter. Some of the students did not write their estimates down. Others wrote their estimates down on an erasable pad, before erasing the result. Still others wrote their estimates down in permanent marker. As new information emerged, the students who had made this more public commitment were the least willing to change their minds.[18]

“Kurt Lewin noticed [this effect] in the 1930s,” says Philip Tetlock, referring to one of the founders of modern psychology. “Making public commitments ‘freezes’ attitudes in place. So saying something dumb makes you a bit dumber. It becomes harder to correct yourself.”[19]

And Fisher’s commitment could hardly have been more public. Two weeks before the Wall Street crash began, he was reported by the New York Times as saying, “Stocks have reached what looks like a permanently high plateau.” How do you back away from that?

Fisher’s third problem—perhaps the deepest—was his belief that, in the end, the future was knowable. “The sagacious businessman is constantly forecasting,” he once wrote. Maybe. But contrast that with John Maynard Keynes’s famous view about long-term forecasts: “About these matters there is no scientific basis on which to form any calculable probability whatever. We simply do not know.”

Fisher, a man who was happy to specify the perfect angle for the out-turn of the foot, admire the rigor of the “clo” unit of warming, and estimate the productivity gain from Prohibition, believed that with a sufficiently powerful statistical lens any problem would yield to the man of science. The statistical lens is indeed powerful. Still, I hope that I have convinced you that for any problem, it takes more than mere numbers to make the world add up.

Poor Irving Fisher had believed himself to be a man of logic and reason. He was a campaigner for education reform and the proven benefits of a vegetarian diet, and a student of “the science of wealth.” And yet he became the most famous financial basket case in the country.

He kept thinking and working, producing an incisive account of why the Depression had been so severe—including a painful reckoning with the effect of debt on the economy. But while his economic ideas are still respected today, he became a marginalized figure. He was deep in debt to the taxman and to his brokers, and toward the end of his life, a widower living alone in modest circumstances, he became an easy target for scam artists: he was always looking for the big financial break that would restore his fortune. The mansion was long gone. He avoided bankruptcy, and perhaps even prison, because his late wife’s sister covered his debts to the value of tens of millions of dollars in today’s terms. It was a kindness, but for the proud Professor Fisher it must have been the ultimate humiliation.

The economic historian Sylvia Nasar wrote of Fisher, “His optimism, overconfidence and stubbornness betrayed him.”[20] Keynes had plenty of confidence, too, but he had also learned the hard way that there are certain facts about the world that do not easily yield to logic. Recall his comment to his father—“This high-stakes gambling amuses me.” The Monte Carlo gambler knew, all along, that while investing was a fascinating game, it was a game nonetheless, and one should not take an unlucky throw of the dice too much to heart. When his early investment ideas failed, he tried something else. Keynes was able to change his mind; Fisher, alas, could not.

Fisher and Keynes died within a few months of each other, not long after the end of the Second World War. Fisher was a much-diminished figure; Keynes was the most influential economist on the planet, fresh from shaping the World Bank, the IMF, and the entire global financial system at the Bretton Woods conference in 1944.

Late in his life, Keynes reflected, “My only regret is that I have not drunk more champagne in my life.” But he is remembered far more for words that he probably never said. Nevertheless, he lived by them: “When my information changes, I alter my conclusions. What do you do, sir?”

If only he had taught that lesson to Irving Fisher.

—

Fisher and Keynes were equally expert, and they had the same statistical information at their fingertips—data they themselves had done much to collect. Just as with Abraham Bredius, the art scholar so cruelly tricked by the forger Han van Meegeren, their fates were determined not by their expertise but by their emotions.

This book has argued that it is possible to gather and to analyze numbers in ways that help us understand the world. But it has also argued that very often we make mistakes not because the data aren’t available, but because we refuse to accept what they are telling us. For Irving Fisher, and for many others, the refusal to accept the data was rooted in a refusal to acknowledge that the world had changed.

One of Fisher’s rivals, an entrepreneurial forecaster named Roger Babson, explained (not without sympathy) that while Fisher was “one of the greatest economists in the world today and a most useful and unselfish citizen,” he had failed as a forecaster because “he thinks the world is ruled by figures instead of feelings.”[21]

I hope that this book has persuaded you that it is ruled by both.

THE GOLDEN RULE

Be Curious

I can think of nothing an audience won’t understand. The only problem is to interest them; once they are interested they understand anything in the world.

• Orson Welles[1]

I’ve laid down ten statistical commandments in this book.

First, we should learn to stop and notice our emotional reaction to a claim, rather than accepting or rejecting it because of how it makes us feel.

Second, we should look for ways to combine the “bird’s eye” statistical perspective with the “worm’s eye” view from personal experience.

Third, we should look at the labels on the data we’re being given, and ask if we understand what’s really being described.

Fourth, we should look for comparisons and context, putting any claim into perspective.

Fifth, we should look behind the statistics at where they came from—and what other data might have vanished into obscurity.

Sixth, we should ask who is missing from the data we’re being shown, and whether our conclusions might differ if they were included.

Seventh, we should ask tough questions about algorithms and the big datasets that drive them, recognizing that without intelligent openness they cannot be trusted.

Eighth, we should pay more attention to the bedrock of official statistics—and the sometimes heroic statisticians who protect it.

Ninth, we should look under the surface of any beautiful graph or chart.

And tenth, we should keep an open mind, asking how we might be mistaken, and whether the facts have changed.

I realize that having ten commandments is something of a cliché. And in truth, they’re not commandments so much as rules of thumb, or habits of mind that I’ve acquired the hard way as I’ve gone along. You might find them worth a try yourself, when you come across a statistical claim of particular interest to you. Of course, I don’t expect you to run personally through the checklist with every claim you see in the media—who has the time for that? Still, they can be useful in forming a preliminary assessment of your news source. Is the journalist making an effort to define terms, provide context, assess sources? The less these habits of mind are in evidence, the louder alarm bells should ring.

Ten rules of thumb are still a lot for anyone to remember, so perhaps I should try to make things simpler. I realize that these suggestions have a common thread—a golden rule, if you like.

Be curious.

Look deeper and ask questions. It’s a lot to ask, but I hope that it is not too much. At the start of this book I begged you not to abandon the idea that we can understand the world by looking at it with the help of statistics, not to give in to the cynical distrust so temptingly offered by the likes of Darrell Huff. I believe we can—and should—be able to trust that numbers can give us answers to important questions. My colleagues and I at More or Less work hard to earn listeners’ trust that we’re coming to the same conclusions they would if they investigated the issue themselves. But of course we want listeners to be curious and to question us, too. Nullius in verba. We shouldn’t trust without also asking questions.

The philosopher Onora O’Neill once declared, “Well-placed trust grows out of active inquiry rather than blind acceptance.”[2] That seems right. If we want to be able to trust the world around us, we need to show an interest and ask a few basic questions. I hope I’ve persuaded you that those questions aren’t obscure or overly technical; they are what any thoughtful, curious person would be happy to ask. And despite all the confusions of the modern world, it has never been easier to find answers to those questions.

Curiosity, it turns out, can be a remarkably powerful thing.

—

About a decade ago, researchers led by Yale University’s Dan Kahan showed students some footage of a protest outside an unidentified building. Some of the students were told that it was a pro-life demonstration outside an abortion clinic. Others were informed that it was a gay rights demonstration outside an army recruitment office. The students were asked some factual questions. Was it a peaceful protest? Did the protesters try to intimidate people passing by? Did they scream or shout? Did they block the entrance to the building?

The answers people gave depended on the political identities they embraced. Conservative students who believed they were looking at a demonstration against abortion saw no problems with the protest: no abuse, no violence, no obstruction. Students on the left who thought they were looking at a gay rights protest reached the same conclusion: the protesters had conducted themselves with dignity and restraint.

But right-wing students who thought they were looking at a gay rights demonstration reached a very different conclusion, as did left-wing students who believed they were watching an anti-abortion protest. Both these groups concluded that the protesters had been aggressive, intimidating, and obstructive.[3]

Kahan was studying a problem we met in the first chapter: the way our political and cultural identity—our desire to belong to a community of like-minded, right-thinking people—can, on certain hot-button issues, lead us to reach the conclusions we wish to reach. Depressingly, not only do we reach politically comfortable conclusions when parsing complex statistical claims on issues such as climate change, we reach politically comfortable conclusions regardless of the evidence of our own eyes.*

As we saw earlier, expertise is no guarantee against this kind of motivated reasoning: Republicans and Democrats with high levels of scientific literacy are further apart on climate change than those with little scientific education. The same disheartening pattern holds from nuclear power to gun control to fracking: the more scientifically literate opponents are, the more they disagree. The same is true for numeracy. “The greater the proficiency, the more acute the polarization,” notes Kahan.[4]

After a long and fruitless search for an antidote to tribalism, Kahan could be forgiven for becoming jaded.[5] Yet a few years ago, to his surprise, he and his colleagues stumbled upon a trait that some people have—and that other people can be encouraged to develop—that inoculates us against this toxic polarization. On the most politically polluted, tribal questions, where intelligence and education fail, this trait does not.

And if you’re desperately, burningly curious to know what it is—congratulations. You may be inoculated already.

Curiosity breaks the relentless pattern. Specifically, Kahan identified “scientific curiosity.” That’s different from scientific literacy. The two qualities are correlated, of course, but there are curious people who know rather little about science (yet), and highly trained people with little appetite to learn more.

More scientifically curious Republicans aren’t further apart from Democrats on these polarized issues. If anything, they’re slightly closer together. It’s important not to exaggerate the effect. Curious Republicans and Democrats still disagree on issues such as climate change—but the more curious they are, the more they converge on what we might call an evidence-based view of the issues in question. Or to put it another way, the more curious we are, the less our tribalism seems to matter. (There is little correlation between scientific curiosity and political affiliation. Happily, there are plenty of curious people across the political spectrum.)

Although the discovery surprised Kahan, it makes sense. As we’ve seen, one of our stubborn defenses against changing our minds is that we’re good at filtering out or dismissing unwelcome information. A curious person, however, enjoys being surprised and hungers for the unexpected. He or she will not be filtering out surprising news, because it’s far too intriguing.

The scientifically curious people Kahan’s team studied were originally identified with simple questions, buried in a marketing survey so that people weren’t conscious that their curiosity was being measured. One question, for example, was “How often do you read science books?” Scientifically curious people are more interested in watching a documentary about space travel or penguins than a basketball game or a celebrity gossip show. And they didn’t just answer survey questions differently, they also made different choices in the psychology lab. In one experiment, participants were shown a range of headlines about climate change and invited to pick the “most interesting” article to read. There were four headlines. Two suggested climate skepticism and two did not; two were framed as surprising and two were not:

1. “Scientists Find Still More Evidence That Global Warming Actually Slowed in Last Decade” (skeptical, unsurprising)

2. “Scientists Report Surprising Evidence: Arctic Ice Melting Even Faster Than Expected” (surprising and not skeptical)

3. “Scientists Report Surprising Evidence: Ice Increasing in Antarctic, Not Currently Contributing to Sea Level Rise” (skeptical and surprising)

4. “Scientists Find Still More Evidence Linking Global Warming to Extreme Weather” (neither surprising nor skeptical)

Typically we’d expect people to reach for the article that pandered to their prejudices: the Democrats would tend to favor a headline that took global warming seriously, while Republicans would prefer something with a skeptical tone. Scientifically curious people—Republicans or Democrats—were different. They were happy to grab an article that ran counter to their preconceptions, as long as it seemed surprising and fresh. And once you’re actually reading the article, there’s always a chance that it might teach you something.

A surprising statistical claim is a challenge to our existing worldview. It may provoke an emotional response—even a fearful one. Neuroscientific studies suggest that the brain responds in much the same anxious way to facts that threaten our preconceptions as it does to wild animals that threaten our lives.[6] Yet for someone in a curious frame of mind, in contrast, a surprising claim need not provoke anxiety. It can be an engaging mystery, or a puzzle to solve.

A curious person might, at this point, have some questions. When I met Dan Kahan, the question that was most urgent in my mind was: Can we cultivate curiosity? Can we become more curious, and can we inspire curiosity in others?

There are reasons to believe that the answers are yes. One reason, says Kahan, is that his measure of curiosity suggests that incremental change is possible. When he measures scientific curiosity, he doesn’t find a lump of stubbornly incurious people at one end of the spectrum and a lump of voraciously curious people at the other, with a yawning gap in the middle. Instead, curiosity follows a continuous bell curve: most people are either moderately incurious or moderately curious. This doesn’t prove that curiosity can be cultivated; perhaps that bell curve is cast in iron. Yet it does at least hold out some hope that people can be nudged a little further toward the curious end of that curve, because no radical leap is required.

A second reason is that curiosity is often situational. In the right place, at the right time, curiosity will smolder in any of us.* Indeed, Kahan’s discovery that an individual’s scientific curiosity persisted over time was a surprise to some psychologists. They had believed, with some reason, that there was no such thing as a curious person, just a situation that inspired curiosity. In fact, it does now seem that people can tend to be curious or incurious. That does not alter the fact that curiosity can be fueled or dampened by context. We all have it in us to be curious, or not, about different things at different times.

One thing that provokes curiosity is the sense of a gap in our knowledge that needs to be filled. George Loewenstein, a behavioral economist, framed this idea in what has become known as the “information gap” theory of curiosity. As Loewenstein puts it, curiosity starts to glow when there’s a gap “between what we know and what we want to know.” There’s a sweet spot for curiosity: if we know nothing, we ask no questions; if we know everything, we ask no questions either. Curiosity is fueled once we know enough to know that we do not know.[7]

Alas, all too often we don’t even think about what we don’t know. There’s a beautiful little experiment about our incuriosity, conducted by the psychologists Leonid Rozenblit and Frank Keil. They gave their experimental subjects a simple task: to look through a list of everyday objects such as a flush lavatory, a zipper, and a bicycle, and to rate their understanding of each object on a scale of one to seven.[8]

After people had written down their ratings, the researchers would gently launch a devastating ambush. They asked the subjects to elaborate. Here’s a pen and paper, they would say; please write out your explanation of a flush lavatory in as much detail as you can. By all means include diagrams.

It turns out that this task wasn’t as easy as people had thought. People stumbled, struggling to explain the details of everyday mechanisms. They had assumed that those details would readily spring to mind, and they did not. And to their credit, most experimental subjects realized that they’d been lying to themselves. They had felt they understood zippers and lavatories, but when invited to elaborate, they realized they didn’t understand at all. When people were asked to reconsider their previous one-to-seven rating, they marked themselves down, acknowledging that their knowledge had been shallower than they’d realized.

Rozenblit and Keil called this “the illusion of explanatory depth.” The illusion of explanatory depth is a curiosity killer and a trap. If we think we already understand, why go deeper? Why ask questions? It is striking that it was so easy to get people to pull back from their earlier confidence: all it took was to get them to reflect on the gaps in their knowledge. And as Loewenstein argued, gaps in knowledge can fuel curiosity.

There is more at stake here than zippers. Another team of researchers, led by Philip Fernbach and Steven Sloman, authors of The Knowledge Illusion, adapted the flush lavatory question to ask about policies such as a cap-and-trade system for carbon emissions, a flat tax, or a proposal to impose unilateral sanctions on Iran. The researchers, importantly, didn’t ask people whether or why they were in favor of or against these policies—there’s plenty of prior evidence that such questions would lead people to dig in. Instead, Fernbach and his colleagues just gave them the same simple task: Please rate your understanding on a scale of one to seven. Then, the same devastating follow-up: Please elaborate; tell us exactly what unilateral sanctions are and how a flat tax works. And the same thing happened. People said, yes, they basically understood these policies fairly well. Then, when prompted to explain, the illusion was dispelled. They realized that perhaps they didn’t really understand at all.[9]

More striking was that when the illusion faded, political polarization also started to fade. People who would have instinctively described their political opponents as wicked, and who would have gone to the barricades to defend their own ideas, tended to be less strident when forced to admit to themselves that they didn’t fully understand what it was they were so passionate about in the first place. The experiment influenced actions as well as words: researchers found that people became less likely to give money to lobby groups or other organizations that supported the positions they had once favored.[10]

It’s a rather beautiful discovery: in a world where so many people seem to hold extreme views with strident certainty, you can deflate somebody’s overconfidence and moderate their politics simply by asking them to explain the details. Next time you’re in a politically heated argument, try asking your interlocutor not to justify herself, but simply to explain the policy in question. She wants to introduce a universal basic income, or a flat tax, or a points-based immigration system, or Medicare for all. OK, that’s interesting. So what exactly does she mean by that? She may learn something as she tries to explain. So may you. And you may both find that you understand a little less, and agree a little more, than you had assumed.

—

Figuring out the workings of a flush lavatory or understanding what a cap-and-trade scheme really is can require some effort. One way to encourage that effort is to embarrass someone by innocently inviting an overconfident answer on a scale of one to seven; but another, kinder way is to engage their interest. As Orson Welles said, once people are interested they can understand anything in the world.

How to engage people’s interest is neither a new problem nor an intractable one. Novelists, screenwriters, and comedians have been figuring out this craft for as long as they have existed. They know that we love mysteries, are drawn in by sympathetic characters, enjoy the arc of a good story, and will stick around for anything that makes us laugh. And scientific evidence suggests that Orson Welles was absolutely right: for example, studies in which people were asked to read narratives and non-narrative texts found that they zipped through the narrative at twice the speed, and recalled twice as much information later.[11]

As for humor, consider the case of the comedian Stephen Colbert’s “civics lesson.” Before his current role as the host of The Late Show, Colbert presented The Colbert Report in character as a blowhard right-wing commentator.* In March 2011, Colbert began a long-running joke in which he explored the role of money in American politics. He decided that he needed to set up a political action committee—a PAC—to raise funds in case he decided to run for president. “I clearly need a PAC but I have no idea what PACs do,” he explained to a friendly expert on the show.

Over the course of the next few weeks, Colbert had PACs—and Super PACs, and 501(c)(4)s—explained to him: from where they could accept donations, up to what limits, with what transparency requirements, and to spend on what. He was to discover that the right combination of fundraising structures could be used to raise almost any amount of money for almost any purpose, with almost no disclosure. “Clearly (c)(4)s have created an unprecedented, unaccountable, untraceable cash tsunami that will infect every corner of the next election,” he mused. “And I feel like an idiot for not having one.”

Colbert later learned how to dissolve his fundraising structures and keep the money—without notifying the taxman. By repeatedly returning to the topic and—in character—demanding advice as to how to abuse the electoral rules, Colbert explored campaign finance in far more depth than any news report could have dreamed of doing.

Did all of this actually improve viewers’ knowledge of the issue? It seems so. A team including Kathleen Hall Jamieson, who also worked with Dan Kahan on the scientific curiosity research, used the Colbert story line to investigate how much people learned amid the laughter. They found that watching The Colbert Report was correlated with increased knowledge about Super PACs and 501(c)(4) groups—how they worked, what they could legally do. Reading a newspaper or listening to talk radio also helped, but the effect of The Colbert Report was much bigger. One day a week of watching Colbert taught people as much about campaign finance as did four days a week reading a newspaper, for example—or five extra years of schooling.

Of course, this is a measure of correlation, not causation. It’s possible that the people who were already interested in Super PACs tuned in to Colbert to hear him wisecrack about them. Or perhaps politics junkies know about Super PACs and also love watching Colbert. But I suspect the show did contribute to the growing understanding, because Colbert really did go deep into the details. And large audiences stuck with him—because he was funny.[12]

You don’t have to be one of America’s best-loved comedians to pull off this trick. The NPR podcast Planet Money once shed light on the details of the global economy by designing, manufacturing, and importing several thousand T-shirts. This allowed a long-running story line investigating cotton farming; the role of automation in textiles; how African communities make new fashions out of donated American T-shirts; the logistics of the shipping industry; and strange details such as the fact that the men’s shirts, which were made in Bangladesh, attract a tariff of 16.5 percent, whereas the women’s shirts, made in Colombia, are duty-free.[13]

These examples should be models for communication, precisely because they inspire curiosity. “How does money influence politics?” is not an especially engaging question, but “If I were running for president, how would I raise lots of money with few conditions and no scrutiny?” is much more intriguing.

Those of us in the business of communicating ideas need to go beyond the fact-check and the statistical smackdown. Facts are valuable things, and so is fact-checking. But if we really want people to understand complex issues, we need to engage their curiosity. If people are curious, they will learn.*

I’ve found this in my own work with the team who make More or Less for the BBC. The program is often regarded affectionately as a myth-buster, but I feel that our best work is when we use statistics to illuminate the truth rather than to debunk a stream of falsehoods. We try to bring people along with us as we explore the world around us with the help of reliable numbers. What’s false is interesting—but not as interesting as what’s true.

—

After the referendum of 2016, in which my fellow British voters decided to leave the European Union, the economics profession engaged in some soul-searching. Most technical experts thought that leaving the EU was a bad idea—costly, complex, and unlikely to deliver many of the promised benefits or to solve the country’s most pressing problems. Yet, as one infamous sound bite put it, “The people in this country have had enough of experts.”* Few people seemed to care what economists had to say on the subject, and—to our credit, I think—professional economists wanted to understand what we had done wrong and whether we might do better in the future.

Later, at a conference about “the profession and the public,” the great and the good of the British economics community pondered the problem and discussed solutions.[14] We needed to be more chatty and approachable on Twitter, suggested one analysis. We needed to express ourselves clearly and without jargon, offered many speakers—not unreasonably.

My own perspective was slightly different. I argued that we were operating in a politically polarized environment, in which almost any opinion we might offer would be fiercely contested by partisans. Economists deal with controversial issues such as inequality, taxation, public spending, climate change, trade, immigration, and, of course, Brexit. In such a febrile environment, speaking slowly and clearly will only get you so far. To communicate complex ideas, we needed to spark people’s curiosity—even inspire a sense of wonder. After all, the great science communicators—people such as Stephen Hawking and David Attenborough—do not win people over simply by using small words, crisply spoken. They stoke the flames of our curiosity, making us burn with desire to learn more. If we economists want people to understand economics, we must first engage their interest.

What is true of economists is equally true for scientists, social scientists, historians, statisticians, and anyone else with complex ideas to convey. Whether the topic is the evolution of black holes or the emergence of Black Lives Matter, the possibility of precognition or the necessity of preregistration, the details matter—and presented in the right way, they should always have the capacity to fascinate us.

—

Awaken our sense of wonder, I say to my fellow nerd-communicators. Ignite the spark of curiosity and give it some fuel, using the time-honored methods of storytelling, character, suspense, and humor. But let’s not rely on the journalists and the scientists and the other communicators of complex ideas. We have to be responsible for our own sense of curiosity. As the saying goes, “Only boring people get bored.” The world is so much more interesting if we take an active interest in it.

“The cure for boredom is curiosity,” goes an old saying. “There is no cure for curiosity.”[15] Just so: once we start to peer beneath the surface of things, become aware of the gaps in our knowledge, and treat each question as the path to a better question, we find that curiosity is habit-forming.

Sometimes we need to think like Darrell Huff; there is a place in life for the mean-minded, hard-nosed skepticism that asks, Where’s the trick? Why is this lying bastard lying to me?[16] But while “I don’t believe it” is sometimes the right starting point when confronted with a surprising statistical claim, it is a lazy and depressing place to finish.

And I hope you won’t finish there. I hope that I have persuaded you that we should make more room both for the novelty-seeking curiosity that says “Tell me more” and the dogged curiosity that drove Austin Bradford Hill and Richard Doll to ask why so many people were dying of lung cancer, and whether cigarettes might be to blame.

If we want to make the world add up, we need to ask questions—open-minded, genuine questions. And once we start asking them, we may find it is delightfully difficult to stop.

ACKNOWLEDGMENTS

It’s been nearly fifteen years since Nicola Meyrick emailed me out of the blue to suggest that I might want to present a BBC Radio 4 program about statistics. Ever since then, I’ve been part of the More or Less family, and it’s been both a pleasure and a privilege. This book reflects everything I’ve learned over the years, and Nicola deserves the credit for starting it all.

I’m grateful to everyone at the BBC who has worked researching, producing, reporting, and mixing More or Less, doing everything in their power to make me sound good. My back-of-the-envelope estimate is that about a hundred people have been part of the team over the years, and in particular, Richard Fenton Smith and Lizzy McNeill worked on the stories about premature births and gun violence that illustrated the third chapter. I am especially lucky to have worked with my long-suffering editor, Richard Vadon, and a series of superb producers, notably Ruth Alexander, Innes Bowen, Richard Knight, Kate Lamble, and Charlotte McDonald. Andrew Dilnot, who co-created More or Less with Michael Blastland, was extremely generous when I first asked for his advice. He has been so ever since.

Under Hetan Shah’s leadership, everyone at the Royal Statistical Society made me feel every inch an honorary statistician. I’m grateful to them all. Two statistical gurus in particular have been so helpful to me in pondering this book: Denise Lievesley and David Spiegelhalter.

David Bodanis, Paul Klemperer, and Bill Leigh all made invaluable comments after reading the entire manuscript—truly a selfless act—and Bruno Giussani caught an important error in an early draft. At Pushkin, Julia Barton, Ryan Dilley, Mia Lobel, and Jacob Weisberg have been a pleasure to work with—as well as supplying comments on a podcast script that helped make the tenth chapter a lot better. Andrew Wright’s detailed and insightful editing improved the book a great deal, as it has done many times before; he’s a star and a true friend.

Thanks to every scholar and writer whose work I have relied on either through interviews or emails or by consulting their writing, in particular: Anjana Ahuja, Michael Blastland, Alberto Cairo, Andy Cotgreave, Kate Crawford, Kenn Cukier, Andrew Dilnot, Anne Emberton, Baruch Fischhoff, Walter Friedman, Hannah Fry, Kaiser Fung, Dan Gardner, Andrew Gelman, Ben Goldacre, Rebecca Goldin, David Hand, Dan Kahan, Daniel Kahneman, Eileen Magnello, Viktor Mayer-Schönberger, Lynn McDonald, David McRaney, Barbara Mellers, Errol Morris, Will Moy, Terry Murray, Sylvia Nasar, Cathy O’Neil, Onora O’Neill, Caroline Criado Perez, Robert Proctor, Jason Reifler, Alex Reinhart, Anna Rosling Rönnlund, Max Roser, Hans Rosling, Benjamin Scheibehenne, Janelle Shane, Hugh Small, Lucy Smith, Philip Tetlock, Edward Tufte, Patrick Wolfe, David Wootton, Frank Wynne, Ed Yong, and Jason Zweig.

At Little, Brown in the UK, Tim Whiting and Nithya Rae have been models of patience as I embarked on an extended coronavirus rewrite, while my editor at Riverhead Books, Jake Morrissey, has offered his welcome mix of encouragement and exacting comment, ably assisted by Jackie Shost. Thanks of course to the implacably excellent Sally Holloway, to Zoë Pagnamenta, and to everyone at Felicity Bryan Associates.

I continue to value the support and indulgence of my editors at the Financial Times, in particular Alice Fishburn, Brooke Masters, and Alec Russell. Loyal FT readers will see that some of the ideas in this book were first explored in my writing for the newspaper. I love the FT and am so glad to be part of it.

Thank you to my children, Stella, Africa, and Herbie, just for being you. And to Fran Monks; I’m not even going to try to count the ways I am grateful—they would fill another book.

NOTES

Introduction: How to Lie with Statistics

1. Umberto Eco, Serendipities: Language and Lunacy (London: Orion, 1998).

2. Robert Matthews, “Storks Deliver Babies (p = 0.008),” Teaching Statistics 22, no. 2 (June 2000), 36–38. Research papers in social science typically say that a relationship is “statistically significant” if p = 0.05, which means that if there was no relationship at all, a pattern at least as clear as the one observed would occur just one in twenty times. The stork paper boasted p = 0.008, which means that if there was in fact no relationship between storks and births, a pattern as clear as the one observed would occur just one in 125 times. The tradition of applying such a statistical significance test is regrettable, for reasons we shall not go into now.

3. Conrad Keating, Smoking Kills (Oxford, UK: Signal Books, 2009), xv.

4. See, among others, Peter Armitage, “Obituary: Sir Austin Bradford Hill, 1897–1991,” Journal of the Royal Statistical Society, Series A (Statistics in Society), 154, no. 3 (1991), 482–84, JSTOR, www.jstor.org/stable/2983156.

5. Keating, Smoking Kills, 85–90.

6. Keating, 113.

7. John P. A. Ioannidis, “A Fiasco in the Making?,” Stat, March 17, 2020, https://www.statnews.com/2020/03/17/a-fiasco-in-the-making-as-the-coronavirus-pandemic-takes-hold-we-are-making-decisions-without-reliable-data/.

8. “Taiwan Says WHO Failed to Act on Coronavirus Transmission Warning,” Financial Times, March 20, 2020, https://www.ft.com/content/2a70a02a-644a-11ea-a6cd-df28cc3c6a68.

9. Demetri Sevastopulo and Hannah Kuchler, “Donald Trump’s Chaotic Coronavirus Crisis,” Financial Times, March 27, 2020, https://www.ft.com/content/80aa0b58-7010-11ea-9bca-bf503995cd6f.

10. David Card, “Origins of the Unemployment Rate: The Lasting Legacy of Measurement without Theory,” UC Berkeley and National Bureau of Economic Research Working Paper, February 2011, http://davidcard.berkeley.edu/papers/origins-of-unemployment.pdf.

11. Naomi Oreskes and Eric Conway, Merchants of Doubt (New York: Bloomsbury, 2010), chap. 1; and Robert Proctor, Golden Holocaust (Berkeley: University of California Press, 2011).

12. Unknown—No Title—August 21, 1969, Brown & Williamson Record, https://www.industrydocumentslibrary.ucsf.edu/tobacco/docs/xqkd0134.

13. Kari Edwards and Edward Smith, “A Disconfirmation Bias in the Evaluation of Arguments,” Journal of Personality and Social Psychology 71, no. 1 (1996), 5–24.

14. Oreskes and Conway, Merchants of Doubt.

15. Michael Lewis, “Has Anyone Seen the President?,” Bloomberg, February 9, 2018, https://www.bloomberg.com/opinion/articles/2018-02-09/has-anyone-seen-the-president.

16. Brendan Nyhan, “Why Fears of Fake News Are Overhyped,” Medium, February 4, 2019; and Gillian Tett, “The Kids Are Alright: The Truth about Fake News,” Financial Times, February 6, 2019, https://www.ft.com/content/d8f43574-29a1-11e9-a5ab-ff8ef2b976c7?desktop=true&segmentId=7c8f09b9-9b61-4fbb-9430-9208a9e233c8.

17. “Health Warning Required on Cigarette Packs,” Congressional Quarterly Almanac 21 (1965), 344–51, https://library.cqpress.com/cqalmanac/document.php?id=cqal65-1259268; and Alex Reinhart, “Huff and Puff,” Significance 11, no. 4 (October 3, 2014).

18. Andrew Gelman, “Statistics for Cigarette Sellers,” Chance 25, no. 3 (2012); Reinhart, “Huff and Puff.”

19. How to Lie with Smoking Statistics is stored in the Tobacco Industry Documents library. Alex Reinhart pieced together the manuscript and various documents pertaining to the project: Reinhart, “The History of ‘How to Lie with Smoking Statistics,’” October 4, 2014, https://www.refsmmat.com/articles/smoking-statistics.html.

20. Suzana Herculano-Houzel, “What Is So Special about the Human Brain?,” TED Talk, 2013: https://www.ted.com/talks/suzana_herculano_houzel_what_is_so_special_about_the_human_brain/transcript?ga_source=embed&ga_medium=embed&ga_campaign=embedT.

21. On Galileo’s telescope: “Refusing to Look,” The Renaissance Mathematicus (blog), August 23, 2012, https://thonyc.wordpress.com/2012/08/23/refusing-to-look/; and Ryan D. Tweney, “How the Telescope Changed Our Minds,” Wired, October 2, 2008, https://www.wired.com/2008/10/how-the-telesco/; and https://web.archive.org/web/20160829064012/https://thekindlyones.org/2010/10/13/refusing-to-look-through-galileos-telescope/.

Rule One: Search Your Feelings

1. Also known as Star Wars: Episode V; screenplay by Leigh Brackett and Lawrence Kasdan.

2. The van Meegeren case is described in John Godley, The Master Forger (London: Home and Van Thal, 1951); Van Meegeren: A Case Study (London: Nelson, 1967); Noah Charney, The Art of Forgery: The Minds, Motives and Methods of Master Forgers (London: Phaidon, 2015); Frank Wynne, I Was Vermeer (London: Bloomsbury, 2007); Edward Dolnick, The Forger's Spell (New York: Harper Perennial, 2009); the BBC TV program Fake or Fortune (series 1, program 3, 2011); a series of blog posts by Errol Morris titled “Bamboozling Ourselves,” starting on the New York Times website, May 20, 2009; the Boijmans Museum film Van Meegeren’s Fake Vermeers (2010, available on YouTube at https://www.youtube.com/watch?v=NnnkuOz08GQ); and particularly Jonathan Lopez, The Man Who Made Vermeers (London: Houghton Mifflin, 2009).

3. Different accounts exist of exactly how van Meegeren made this confession—another account has van Meegeren equating himself with the Dutch master more directly: “The painting in Göring’s hands is not, as you assume, a Vermeer of Delft, but a van Meegeren!” The quotation in the text is from Frank Wynne’s book I Was Vermeer.

4. Ziva Kunda, “Motivated Inference: Self-Serving Generation and Evaluation of Causal Theories,” Journal of Personality and Social Psychology 53, no. 4 (1987), 636–47.

5. Stephen Jay Gould, “The Median Isn’t the Message,” AMA Journal of Ethics (January 2013), DOI: http://dx.doi.org/10.1001/virtualmentor.2013.15.1.mnar1-1301.

6. This experiment was described on NPR’s Hidden Brain podcast: “You 2.0: The Ostrich Effect,” August 6, 2018, https://www.npr.org/templates/transcript/transcript.php?storyId=636133086.

7. Nachum Sicherman et al., “Financial Attention,” Review of Financial Studies 29, no. 4 (April 1, 2016), 863–97, https://doi.org/10.1093/rfs/hhv073.

8. “Viral Post about Someone’s Uncle’s Coronavirus Advice Is Not All It’s Cracked Up to Be,” Full Fact, March 5, 2020, https://fullfact.org/online/coronavirus-claims-symptoms-viral/.

9. Guy Mayraz, “Wishful Thinking,” October 25, 2011, http://dx.doi.org/10.2139/ssrn.1955644.

10. Linda Babcock and George Loewenstein, “Explaining Bargaining Impasse: The Role of Self-Serving Biases,” Journal of Economic Perspectives 11, no. 1 (1997), 109–26, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.11.1.109.

11. A good summary is Dan Kahan’s blog post “What Is Motivated Reasoning? How Does It Work?,” reproduced at http://blogs.discovermagazine.com/intersection/2011/05/05/what-is-motivated-reasoning-how-does-it-work-dan-kahan-answers/#.WN5zJ_nyuUm. An excellent survey is Ziva Kunda, “The Case for Motivated Reasoning,” Psychological Bulletin 108, no. 3 (1990), 480–98, http://dx.doi.org/10.1037/0033-2909.108.3.480.

12. S. C. Kalichman, L. Eaton, and C. Cherry, “‘There Is No Proof That HIV Causes AIDS’: AIDS Denialism Beliefs among People Living with HIV/AIDS,” Journal of Behavioral Medicine 33, no. 6 (2010), 432–40, DOI: 10.1007/s10865-010-9275-7; and A. B. Hutchinson et al., “Conspiracy Beliefs and Trust in Information about HIV/AIDS among Minority Men Who Have Sex with Men,” Journal of Acquired Immune Deficiency Syndrome 45, no. 5 (August 15, 2007), 603–5.

13. Tim Harford, “Why It’s Too Tempting to Believe the Oxford Study,” Financial Times, March 27, 2020, https://www.ft.com/content/14df8908-6f47-11ea-9bca-bf503995cd6f.

14. Keith E. Stanovich, Richard F. West, and Maggie E. Toplak, “Myside Bias, Rational Thinking, and Intelligence,” Current Directions in Psychological Science 22, no. 4 (August 2013), 259–64, DOI: 10.1177/0963721413480174.

15. Charles S. Taber and Milton Lodge, “Motivated Skepticism in the Evaluation of Political Beliefs,” American Journal of Political Science 50, no. 3 (July 2006), 755–69, JSTOR, www.jstor.org/stable/3694247.

16. Kevin Quealy, “The More Education Republicans Have, the Less They Tend to Believe in Climate Change,” New York Times, November 14, 2017, https://www.nytimes.com/interactive/2017/11/14/upshot/climate-change-by-education.html.

17. Caitlin Drummond and Baruch Fischhoff, “Individuals with Greater Science Literacy and Education Have More Polarized Beliefs on Controversial Science Topics,” Proceedings of the National Academy of Sciences of the United States of America, August 21, 2017, http://www.pnas.org/content/early/2017/08/15/1704882114.

18. Charles Lord, L. Ross, and M. R. Lepper, “Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence,” Journal of Personality and Social Psychology 37, no. 11 (1979), 2098–2109.

19. Nicholas Epley and Thomas Gilovich, “The Mechanics of Motivated Reasoning,” Journal of Economic Perspectives 30, no. 3 (2016), 133–40, https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.30.3.133.

20. Ari LeVaux, “Climate Change Threatens Montana’s Barley Farmers—and Possibly Your Beer,” Food and Environment Research Network, December 13, 2017, https://thefern.org/2017/12/climate-change-threatens-montanas-barley-farmers-possibly-beer/.

21. Author correspondence with Kris De Meyer, October 27, 2018.

22. Gordon Pennycook et al., “Understanding and Reducing the Spread of Misinformation Online,” PsyArXiv, November 13, 2019, DOI: 10.31234/osf.io/3n9u8; also see Oliver Burkeman, “How to Stop the Spread of Fake News? Pause for a Moment,” Guardian, February 7, 2020, https://www.theguardian.com/lifeandstyle/2020/feb/07/how-to-stop-spread-of-fake-news-oliver-burkeman.

23. G. Pennycook and D. G. Rand, “Lazy, Not Biased: Susceptibility to Partisan Fake News Is Better Explained by Lack of Reasoning Than by Motivated Reasoning,” Cognition, 2018, https://doi.org/10.1016/j.cognition.2018.06.011.

24. Shane Frederick, “Cognitive Reflection and Decision Making,” Journal of Economic Perspectives 19, no. 4 (2005), 25–42, DOI: 10.1257/089533005775196732.

25. Diane Wolf, Beyond Anne Frank: Hidden Children and Postwar Families in Holland (Berkeley: University of California Press, 2007), Table 1, citing Raul Hilberg, The Destruction of the European Jews (1985).

Rule Two: Ponder Your Personal Experience

1. Muhammad Yunus, in “Stephen Covey Interviews Grameen Bank Founder Muhammad Yunus,” SocialBusinessPedia, last updated November 21, 2016, http://socialbusinesspedia.com/wiki/details/248.

2. Transport for London, Travel in London: Report 11 (2018), figure 10.8, 202, http://content.tfl.gov.uk/travel-in-london-report-11.pdf.

3. These numbers were revealed in a Freedom of Information request, March 28, 2011 (https://www.whatdotheyknow.com/request/journey_demand_and_service_suppl), and they are nicely summarized at the website Ian Visits, “London Tube Train Capacities,” August 5, 2106, https://www.ianvisits.co.uk/blog/2016/08/05/london-tube-train-capacities/.

4. Transport for London, Travel in London: Report 4 (2011), 5, http://content.tfl.gov.uk/travel-in-london-report-4.pdf.

5. Author interview with Lauren Sager Weinstein and Dale Campbell of TfL, July 9, 2019.

6. Ipsos Mori, Perils of Perception 2017, December 5, 2017, https://www.ipsos.com/ipsos-mori/en-uk/perils-perception-2017.

7. “‘No Link between MMR and Autism,’ Major Study Finds,” NHS News, March 5, 2019, https://www.nhs.uk/news/medication/no-link-between-mmr-and-autism-major-study-finds/.

8. National Institute of Child Health and Clinical Development, “When Do Children Usually Show Symptoms of Autism?,” January 31, 2017, https://www.nichd.nih.gov/health/topics/autism/conditioninfo/symptoms-appear.

9. David McRaney, interview with Lee Ross, “Naive Realism,” episode 62 of You Are Not So Smart, November 9, 2015, https://youarenotsosmart.com/2015/11/09/yanss-062-why-you-often-believe-people-who-see-the-world-differently-are-wrong/; and Tom Gilovich and Lee Ross, The Wisest One in the Room (New York: Free Press, 2016).

10. Ipsos Mori, Perils of Perception.

11. David Dranove et al., “Is More Information Better? The Effects of ‘Report Cards’ on Health Care Providers,” National Bureau of Economic Research Working Paper 8697 (2002), http://www.nber.org/papers/w8697.

12. Charles Goodhart, “Problems of Monetary Management: The U.K. Experience,” in Inflation, Depression, and Economic Policy in the West, ed. Anthony S. Courakis (Lanham, MD: Rowman & Littlefield, 1981), 111–46. The original paper was presented at a conference in 1975.

13. Donald T. Campbell, “Assessing the Impact of Planned Social Change,” Evaluation and Program Planning 2, no. 1 (1979). An earlier version was published in 1976 and a conference paper existed in 1974.

14. Abhijit Banerjee, Dean S. Karlan, and Jonathan Zinman, “Six Randomized Evaluations of Microcredit: Introduction and Further Steps,” American Economic Journal: Applied Economics 7, no. 1 (2015), 1–21; and Rachel Meager, “Understanding the Average Effect of Microcredit,” VoxDev, July 1, 2019, https://voxdev.org/topic/methods-measurement/understanding-average-effect-microcredit.

15. Anna Rosling Rönnlund, “See How the Rest of the World Lives, Organized by Income,” a talk presented at a TED conference, April 2017, https://www.ted.com/talks/anna_rosling_ronnlund_see_how_the_rest_of_the_world_lives_organized_by_income?language=en.

Rule Three: Avoid Premature Enumeration

1. Dr. Lucy Smith was interviewed by me and my colleague Richard Fenton-Smith for an episode of More or Less broadcast on BBC Radio 4, June 8, 2018, https://www.bbc.co.uk/programmes/p069jd0p. The account here is based on our broadcast interview, discussions over email, and a phone interview I conducted with Dr. Smith on August 12, 2019. Dr. Smith’s interviews with people who had lost a baby at between twenty and twenty-four weeks of pregnancy are at “Losing a Baby at 20-24 Weeks of Pregnancy,” healthtalk.org.

2. Merian F. MacDorman et al., “International Comparisons of Infant Mortality and Related Factors: United States and Europe, 2010,” National Vital Statistics Reports, September 24, 2014.

3. Denis Campbell, “Concern at Rising Infant Mortality Rate in England and Wales,” Guardian, March 15, 2018, https://www.theguardian.com/society/2018/mar/15/concern-at-rising-infant-mortality-rate-in-england-and-wales.

4. Peter Davis et al., “Rising Infant Mortality Rates in England and Wales—We Need to Understand Gestation Specific Mortality,” BMJ (May 8, 2018), 361.

5. More or Less, BBC, April 8, 2020, https://www.bbc.co.uk/programmes/m000h6cb.

6. Author interview with Rebecca Goldin, December 12, 2017.

7. Paul J. C. Adachi and Teena Willoughby, “The Effect of Video Game Competition and Violence on Aggressive Behavior: Which Characteristic Has the Greatest Influence?,” Psychology of Violence 1, no. 4 (2011), 259–74, DOI: 10.1037/a0024908.

8. Steven Woolfe, “Immigration Post-Brexit,” Leave Means Leave research paper, https://static1.squarespace.com/static/5d08d13a2a109b000173bc76/t/5d47277f8fefae000124db10/1564944257844/FINAL-Low-Res-LML-Steven-Woolfe-migration-report-.pdf.

9. Jonathan Portes, “Who Are You Calling Low-Skilled?,” UK in a Changing Europe, April 12, 2017, https://ukandeu.ac.uk/who-are-you-calling-low-skilled/.

10. Robert Wright, “Brexit Visa Changes to Hit Sectors in Need of Low-Skilled Labour,” Financial Times, February 18, 2020, https://www.ft.com/content/890e84ce-5268-11ea-90ad-25e377c0ee1f.

11. Denis Campbell, “Fifth of 17 to 19-Year-Old Girls Self-Harm or Attempt Suicide,” Guardian, November 22, 2018, https://www.theguardian.com/society/2018/nov/22/concern-over-rise-in-suicide-attempts-among-young-women.

12. Mental Health of Children and Young People in England, 2017, NHS Digital, November 22, 2018, https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2017/2017.

13. “Self-Harm,” NHS, page last reviewed May 25, 2018, https://www.nhs.uk/conditions/self-harm/.

14. Email correspondence with the NatCen Social Research (UK) press office, November 29, 2018.

15. Data from official sources such as the Office for National Statistics: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/suicidesintheunitedkingdom/2017registrations#suicide-patterns-by-age.

16. Graeme Wearden, “Oxfam: 85 Richest People as Wealthy as Poorest Half of the World,” Guardian, January 20, 2014, https://www.theguardian.com/business/2014/jan/20/oxfam-85-richest-people-half-of-the-world. Nigel Morris, “World’s 85 Richest People Have as Much as Poorest 3.5 Billion,” Independent, January 20, 2014, https://www.independent.co.uk/news/world/politics/oxfam-warns-davos-of-pernicious-impact-of-the-widening-wealth-gap-9070714.html.

17. “Anatomy of a Killer Fact: The World’s 85 Richest People Own as Much as the Poorest 3.5 Billion,” From Poverty to Power (blog), January 31, 2014, https://oxfamblogs.org/fp2p/anatomy-of-a-killer-fact-the-worlds-85-richest-people-own-as-much-as-poorest-3-5-billion/. For the BBC interview with Fuentes, see “Do 85 Rich People Have Same Wealth as Half the World?,” March 19, 2014, https://www.bbc.com/news/magazine-26613682.

18. The underlying data come from the Global Wealth Report, which is published each year by Credit Suisse. The 2013 version supplied the data for Oxfam’s original “killer fact.”

19. On pensions as discussed in the footnote, see “Social Protection for Older Persons: Policy Trends and Statistics 2017–19,” International Labour Office, Social Protection Department (2018).

20. For the UK, the Institute for Fiscal Studies’ Living Standards, Poverty and Inequality in the UK. For global top incomes, the World Inequality Report. Another good source is Our World in Data. More specific references are provided in the notes below.

Rule Four: Step Back and Enjoy the View

1. For more reporting on this issue, listen to the June 8, 2018, episode of More or Less, presented by me and researched by my colleagues Richard Fenton-Smith and Richard Vadon, https://www.bbc.co.uk/programmes/p069jd0p.

2. Johan Galtung and Mari Holmboe Ruge, “The Structure of Foreign News: The Presentation of the Congo, Cuba and Cyprus Crises in Four Norwegian Newspapers,” Journal of Peace Research 2, no. 1 (1965), 64–90.

3. Max Roser, “Stop Saying That 2016 Was the Worst Year,” Washington Post, December 29, 2016, https://www.washingtonpost.com/posteverything/wp/2016/12/29/stop-saying-that-2016-was-the-worst-year/?utm_term=.bad894bad69a; see also NPR’s Planet Money, “The Fifty Year Newspaper,” December 29, 2017, https://www.npr.org/templates/transcript/transcript.php?storyId=574662798.

4. Colin P. Morice et al., “Quantifying Uncertainties in Global and Regional Temperature Change Using an Ensemble of Observational Estimates: The Hadcrut4 Dataset,” Journal of Geophysical Research (2012), 117, D08101, DOI: 10.1029/2011JD017187, describing data from the Met Office Hadley Centre. The data are charted by and downloadable from Our World in Data, https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions. In the 1960s, global temperatures were typically around 0.1°C below the average of 1961–1990. In the twenty-first century they’ve typically been about 0.6°C above that average, and more recently above 0.7°C. Increase in temperatures, then, over the past fifty years, has been 0.7–0.8°C.

5. Max Roser, “The Short History of Global Living Conditions and Why It Matters That We Know It” (2018), published online at OurWorldInData.org, retrieved from https://ourworldindata.org/a-history-of-global-living-conditions-in-5-charts; for child mortality, Roser cites data from Gapminder and the World Bank.

6. See Figure E4 in the World Inequality Lab’s Executive Summary of the World Inequality Report 2018, https://wir2018.wid.world/files/download/wir2018-summary-english.pdf.

7. An excellent source is the Institute for Fiscal Studies Report on Living Standards, Poverty and Inequality in the UK. I’ve used the 2018 edition, the most recent available at the time of writing, https://www.ifs.org.uk/uploads/R145%20for%20web.pdf.

8. A good summary article on inequality around the world is on the Our World in Data website, written by Joe Hasell, an authority on the subject: https://ourworldindata.org/income-inequality-since-1990.

9. Author calculations, based on Natsal-3, the third National Survey of Sexual Attitudes and Lifestyles: http://timharford.com/2018/09/is-twitter-more-unequal-than-life-sex-or-happiness/.

10. Michael Blastland and Andrew Dilnot, The Tiger That Isn’t (London: Profile Books, 2008).

11. Andrew C. A. Elliott, Is That a Big Number? (Oxford: Oxford University Press, 2018).

12. Tali Sharot, “The Optimism Bias,” TED Talk, 2012, https://www.ted.com/talks/tali_sharot_the_optimism_bias/transcript#t-18026.

13. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2010).

14. Ross A. Miller and Karen Albert, “If It Leads, It Bleeds (and If It Bleeds, It Leads): Media Coverage and Fatalities in Militarized Interstate Disputes,” Political Communication 32, no. 1 (2015), 61–82, DOI: 10.1080/10584609.2014.880976; Barbara Combs and Paul Slovic, “Newspaper Coverage of Causes of Death,” Journalism Quarterly 56, no. 4 (1979), 837–43, 849.

15. “Smoking and Tobacco Use, Fast Facts,” Centers for Disease Control and Prevention, page last reviewed May 21, 2020, https://www.cdc.gov/tobacco/data_statistics/fact_sheets/fast_facts/. There are 1,300 deaths a day from smoking-related diseases, about 40,000 a month; almost 3,000 people were killed in the attacks of September 11, 2001.

16. “Steven Pinker on the Case for Optimism,” interview by Chris Anderson, TED2018 conference, https://www.ted.com/talks/the_ted_interview_steven_pinker_on_why_our_pessimism_about_the_world_is_wrong/transcript?language=en.

17. Steven Pinker mentions in the endnotes of Enlightenment Now (New York: Viking, 2018) that this correspondence took place in 1982.

18. Quoted in Denis Campbell, “Stroke Association Warns of Alarming Rise in Number of Victims,” Guardian, May 12, 2015, https://www.theguardian.com/society/2015/may/12/stroke-association-warns-of-alarming-rise-in-number-of-victims; see also More or Less, May 17, 2015, with the analysis of this claim: https://www.bbc.co.uk/programmes/b05tpz78.

19. “Public Pessimism and Misunderstanding Undermining Fight against Global Poverty,” Oxfam press release, September 22, 2016, http://oxfamapps.org/media/ppdwr.

20. A useful survey of various relevant graphs is Max Roser and Mohamed Nagdy, “Optimism and Pessimism,” published online at OurWorldInData.org, 2018, retrieved from https://ourworldindata.org/optimism-pessimism, particularly Section I.1 with graphs from Eurobarometer and Ipsos Mori.

21. Martyn Lewis, “Not My Idea of Good News,” Independent, April 26, 1993, https://www.independent.co.uk/voices/not-my-idea-of-good-news-at-the-end-of-a-week-of-horrifying-events-martyn-lewis-bbc-presenter-argues-1457539.html.

22. Roser, “The Short History of Global Living Conditions and Why It Matters That We Know It”; underlying data from the World Bank and from Bourguignon and Morrisson, “Inequality among World Citizens: 1820–1992,” American Economic Review 92, no. 4 (2002), 727–48. In 1993 there were 1.94 billion people living in extreme poverty; by 2015 that had fallen to 0.7 billion (705.55 million). The rate of improvement averages 153,600 a day, although of course we have no way of measuring the daily rate as it fluctuates.

23. Samantha Vanderslott, Bernadeta Dadonaite, and Max Roser, “Vaccination,” published online at OurWorldInData.org (2020), retrieved from https://ourworldindata.org/vaccination.

24. Anna Rosling Rönnlund, Hans Rosling, and Ola Rosling, Factfulness (New York: Flatiron, 2018).

25. Gillian Tett, “Silos and Silences,” Derivatives: Financial Innovation and Stability, Banque de France Financial Stability Review 14 (July 2010), https://publications.banque-france.fr/sites/default/files/medias/documents/financial-stability-review-14_2010-07.pdf.

26. Rolf Dobelli, “News Is Bad for You—and Giving Up Reading It Will Make You Happier,” Guardian, April 12, 2013, https://www.theguardian.com/media/2013/apr/12/news-is-bad-rolf-dobelli.

27. Nassim Nicholas Taleb, The Bed of Procrustes (London: Penguin Books, 2010).

28. Bill Hanage and Mark Lipsitch, “How to Report on the COVID-19 Outbreak Responsibly,” Scientific American, February 23, 2020, https://blogs.scientificamerican.com/observations/how-to-report-on-the-covid-19-outbreak-responsibly/.

Rule Five: Get the Backstory

1. Sheena Iyengar and Mark Lepper, “When Choice Is Demotivating: Can One Desire Too Much of a Good Thing?,” Journal of Personality and Social Psychology 79, 2000.

2. Author interview with Benjamin Scheibehenne, October 2009. (I’d like to claim I was ahead of the curve on this one.)

3. B. Scheibehenne, R. Greifeneder, and P. M. Todd, “Can There Ever Be Too Many Options? A Meta-Analytic Review of Choice Overload,” Journal of Consumer Research 37 (2010), 409–25, http://scheibehenne.de/ScheibehenneGreifenederTodd2010.pdf.

4. Nicole Lyn Pesce, “Ten Kickstarter Products That Raised the Most Money,” MarketWatch, June 22, 2017, https://www.marketwatch.com/story/10-kickstarter-products-that-raised-the-most-money-2017-06-22-10883052.

5. The story is well told in Jordan Ellenberg’s book How Not to Be Wrong (New York: Penguin, 2014), with the relevant extract here: “Abraham Wald and the Missing Bullet Holes,” Medium, July 14, 2016, https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d.

6. A technical summary (along with some grumbling about how the story has been exaggerated) is in Bill Casselman, “The Legend of Abraham Wald,” American Mathematical Society, http://www.ams.org/publicoutreach/feature-column/fc-2016-06.

7. An excellent account of the controversy is Daniel Engber, “Daryl Bem Proved ESP Is Real Which Means Science Is Broken,” Slate, May 17, 2017, https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html.

8. Chris French, “Precognition Studies and the Curse of the Failed Replications,” Guardian, March 15, 2012, https://www.theguardian.com/science/2012/mar/15/precognition-studies-curse-failed-replications.

9. Nosek was speaking on the Planet Money podcast, episode 677, “The Experiment,” https://www.npr.org/sections/money/2018/03/07/591213302/episode-677-the-experiment-experiment.

10. Brian Nosek has given useful interviews to several podcasts, including You Are Not So Smart (episode 100), https://youarenotsosmart.com/2017/07/19/yanss-100-the-replication-crisis/; Planet Money (episode 677); EconTalk, November 16, 2015, http://www.econtalk.org/brian-nosek-on-the-reproducibility-project/; The Hidden Brain (episode 32), https://www.npr.org/templates/transcript/transcript.php?storyId=477921050; as well as BBC Analysis, “The Replication Crisis,” November 12, 2018, https://www.bbc.co.uk/programmes/m00013p9.

11. This figure of thirty-nine is based on the subjective opinion of the replicating researchers. Did their results basically back up the original study, or not? That’s a judgment call. An alternative metric is to ask how many of the replication studies produced results that passed the standard (but rather problematic) hurdle of “statistical significance.” Only thirty-six did; ninety-seven of the original studies had cleared that hurdle. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 28, no. 6251 (August 2015), 349, DOI: 10.1126/science.aac4716.

12. Brief film on YouTube: “Derren Brown—Ten Heads in a Row,” ThinkSceptically, April 8, 2012, https://www.youtube.com/watch?v=n1SJ-Tn3bcQ.

13. Planet Money, episode 677.

14. F. J. Anscombe, “Fixed-Sample-Size Analysis of Sequential Observations,” Biometrics 10, no. 1 (1954), 89–100, JSTOR, www.jstor.org/stable/3001665; and Andrew Gelman’s blog post “Statistical Inference, Modeling and Social Science,” May 2, 2018, https://statmodeling.stat.columbia.edu/2018/05/02/continuously-increased-number-animals-statistical-significance-reached-support-conclusions-think-not-bad-actually/.

15. David J. Hand, Dark Data (Princeton, NJ: Princeton University Press, 2020).

16. Andrew Gelman and Eric Loken, “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘P-Hacking’ and the Research Hypothesis Was Posited Ahead of Time,” working paper, November 14, 2013, http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.

17. J. P. Simmons, L. D. Nelson, and U. Simonsohn, “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” Psychological Science 22, no. 11 (2011), 1359–66, https://doi.org/10.1177/0956797611417632.

18. Kai Kupferschmidt, “More and More Scientists Are Preregistering Their Studies. Should You?,” Science, September 21, 2018.

19. Anjana Ahuja, “Scientists Strike Back against Statistical Tyranny,” Financial Times, March 27, 2019, https://www.ft.com/content/36f9374c-5075-11e9-8f44-fe4a86c48b33.

20. Darrell Huff, How to Lie with Statistics (New York: W. W. Norton, 1993), 40.

21. John Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (August 2005), e124, DOI: 10.1371/journal.pmed.0020124, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/.

22. R. F. Baumeister et al., “Ego Depletion: Is the Active Self a Limited Resource?,” Journal of Personality and Social Psychology 74, no. 5 (1998), 1252–65, http://dx.doi.org/10.1037/0022-3514.74.5.1252; “The End of Ego Depletion Theory?,” Neuroskeptic (blog), July 31, 2016, http://blogs.discovermagazine.com/neuroskeptic/2016/07/31/end-of-ego-depletion/#.XGGyflz7SUk.

23. Amy Cuddy, “Your Body Language May Shape Who You Are,” TED Talk, 2012, https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are/transcript?language=en.

24. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2010), 53–57.

25. Ed Yong, “Nobel Laureate Challenges Psychologists to Clean Up Their Act,” Nature News, October 3, 2012, https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535.

26. Ben Goldacre, “Backwards Step on Looking into the Future,” Guardian, April 23, 2011, https://www.theguardian.com/commentisfree/2011/apr/23/ben-goldacre-bad-science.

27. Robin Wrigglesworth, “How a Herd of Cows Trampled on Human Stockpickers,” Financial Times, January 21, 2020, https://www.ft.com/content/563d61dc-3b70-11ea-a01a-bae547046735?.

28. Burton Malkiel, “Returns from Investing in Equity Funds,” working paper, Princeton University, 1994; published as “Returns from Investing in Equity Funds: 1971–1991,” Journal of Finance 50 (1995), 549–72.

29. Eric Balchunas, “How the Vanguard Effect Adds Up to $1 Trillion,” Bloomberg .com, August 30, 2016, https://www.bloomberg.com/opinion/articles/2016-08-30/how-much-has-vanguard-saved-investors-try-1-trillion.

30. For an accessible overview, see Ben Goldacre, “What Doctors Don’t Know about the Drugs They Prescribe,” TED Talk, 2012, https://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe/footnotes?language=en.

31. Erick Turner et al., “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy,” New England Journal of Medicine, January 17, 2008, https://www.nejm.org/doi/full/10.1056/NEJMsa065779.

32. Ben Goldacre, “Transparency, Beyond Publication Bias,” talk given at the International Journal of Epidemiology Conference, 2016; available at https://www.badscience.net/2016/10/transparency-beyond-publication-bias-a-video-of-my-super-speedy-talk-at-ije/.

33. Ben Goldacre et al., “COMPare: A Prospective Cohort Study Correcting and Monitoring 58 Misreported Trials in Real Time,” Trials 20, no. 118 (2019), https://doi.org/10.1186/s13063-019-3173-2.

34. Ben Goldacre, “Transparency, Beyond Publication Bias.”

35. Amy Sippett, “Does the Backfire Effect Exist?,” Full Fact (blog), March 20, 2019, https://fullfact.org/blog/2019/mar/does-backfire-effect-exist/; Brendan Nyhan, “Read this! Widespread neglect of more recent studies since my paper w/ @JasonReifler,” Twitter, March 20, 2019, 10:40 a.m., https://twitter.com/BrendanNyhan/status/1108377656414879744.

36. Author interview with Richard Thaler, July 17, 2019.

37. “The Replication Crisis,” BBC Analysis, November 12, 2018, https://www.bbc.co.uk/programmes/m00013p9.

38. Antonio Granado, “Slaves to Journals, Serfs to the Web: The Use of the Internet in Newsgathering among European Science Journalists,” Journalism 12, no. 7 (2011), 794–813.

39. A. L. Cochrane, “Sickness in Salonica: My First, Worst, and Most Successful Clinical Trial,” British Medical Journal (Clinical Research Edition) 289, no. 6460 (1984): 1726–27, DOI: 10.1136/bmj.289.6460.1726.

40. Cochrane Community, “A Brief History of Cochrane,” https://community.cochrane.org/handbook-sri/chapter-1-introduction/11-cochrane/112-brief-history-cochrane [inactive].

41. Alan Mozes, “Yoga May Be Right Move Versus Urinary Incontinence,” WebMD, May 22, 2018, https://www.webmd.com/urinary-incontinence-oab/news/20180522/yoga-may-be-right-move-versus-urinary-incontinence#1.

42. Emma Innes, “Could Yoga Cure Incontinence?,” DailyMail.com, May 12, 2014, https://www.dailymail.co.uk/health/article-2626209/Could-yoga-cure-INCONTINENCE-Exercise-strengthens-pelvic-floor-muscles-reducing-leakage.html.

43. “Yoga and Incontinence,” Home Care Delivered (blog), https://www.hcd.com/incontinence/yoga-incontinence/.

44. Alison J. Huang et al., “A Group-Based Yoga Therapy Intervention for Urinary Incontinence in Women,” Female Pelvic Medicine & Reconstructive Surgery 20, no. 3 (2014), 147–54, DOI: 10.1097/SPV.0000000000000072, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4310548/.

45. L. S. Wieland et al., “Yoga for Treating Urinary Incontinence in Women,” Cochrane Database of Systematic Reviews 2019 2, art. no. CD012668, DOI: 10.1002/14651858.CD012668.pub2.

Rule Six: Ask Who Is Missing

1. Rod Bond and Peter B. Smith, “Culture and Conformity: A Meta-Analysis of Studies Using Asch’s (1952b, 1956) Line Judgment Task,” Psychological Bulletin 119, no. 1 (1996), 111–37, http://dx.doi.org/10.1037/0033-2909.119.1.111.

2. Tim Harford, “The Truth about Our Norm-Core,” Financial Times, June 12, 2015, http://timharford.com/2015/06/the-truth-about-our-norm-core/.

3. Bond and Smith, “Culture and Conformity”; and Natalie Frier et al., “The Effects of Group Conformity Based on Sex,” Celebrating Scholarship and Creativity Day, Paper 83 (2016), http://digitalcommons.csbsju.edu/elce_cscday/83.

4. Tim Harford, “Trump, Brexit and How Politics Loses the Capacity to Shock,” Financial Times, November 16, 2018, https://www.ft.com/content/b730c95c-e82e-11e8-8a85-04b8afea6ea3.

5. Caroline Criado Perez, Invisible Women (London: Chatto & Windus, 2019); the interview was broadcast on BBC Radio 4 on May 17, 2019, and is available on the More or Less website: https://www.bbc.co.uk/programmes/m00050rd.

6. Peter Hofland, “Reversal of Fortune,” Onco’Zine, November 30, 2013, https://oncozine.com/reversal-of-fortune-how-a-vilified-drug-became-a-life-saving-agent-in-the-war-against-cancer/.

7. R. Dmitrovic, A. R. Kunselman, and R. S. Legro, “Sildenafil Citrate in the Treatment of Pain in Primary Dysmenorrhea: A Randomized Controlled Trial,” Human Reproduction 28, no. 11 (November 2013): 2958–65, DOI: 10.1093/humrep/det324, Epub August 6, 2013, https://www.ncbi.nlm.nih.gov/pubmed/23925396?dopt=Abstract.

8. BBC, “Supermarket Stockpiling, A-Level Results and Covid-19 Gender Disparity,” More or Less podcast, March 31, 2020, https://www.bbc.co.uk/sounds/play/m000h7st.

9. Mayra Buvinic and Ruth Levine, “Closing the Gender Data Gap,” Significance, April 8, 2016, and Charlotte McDonald, “Is There a Sexist Data Crisis?,” BBC News, May 18, 2016, https://www.bbc.co.uk/news/magazine-36314061.

10. Shelly Lundberg, Robert Pollak, and Terence J. Wales, “Do Husbands and Wives Pool Their Resources? Evidence from the United Kingdom Child Benefit,” Journal of Human Resources 32, no. 3 (1997), 463–80.

11. Mayra Buvinic and Ruth Levine, “Closing the Gender Data Gap.”

12. Suzannah Brecknell, “Interview: Full Fact’s Will Moy on Lobbyist ‘Nonsense,’ Official Corrections and Why We Know More about Golf Than Crime Stats,” Civil Service World, May 5, 2016, https://www.civilserviceworld.com/articles/interview/interview-full-fact%E2%80%99s-will-moy-lobbyist-%E2%80%9Cnonsense%E2%80%9D-official-corrections-and-why.

13. Maurice C. Bryson, “The Literary Digest Poll: Making of a Statistical Myth,” American Statistician 30, no. 4 (1976), 184–85, DOI: 10.1080/00031305.1976.10479173; and Peverill Squire, “Why the 1936 Literary Digest Poll Failed,” Public Opinion Quarterly 52, no. 1 (1988), 125–33, JSTOR, www.jstor.org/stable/2749114.

14. P. Whiteley, “Why Did the Polls Get It Wrong in the 2015 General Election? Evaluating the Inquiry into Pre-Election Polls,” Political Quarterly 87 (2016), 437–42, DOI: 10.1111/1467-923X.12274.

15. John Curtice, “Revealed: Why the Polls Got It So Wrong in the British General Election,” The Conversation, January 14, 2016, https://theconversation.com/revealed-why-the-polls-got-it-so-wrong-in-the-british-general-election-53138.

16. Nate Cohn, “A 2016 Review: Why Key State Polls Were Wrong about Trump,” New York Times, May 31, 2017, https://www.nytimes.com/2017/05/31/upshot/a-2016-review-why-key-state-polls-were-wrong-about-trump.html; and Andrew Mercer, Claudia Deane, and Kyley McGeeney, “Why 2016 Election Polls Missed Their Mark,” Pew Research Fact Tank blog, November 9, 2015, http://www.pewresearch.org/fact-tank/2016/11/09/why-2016-election-polls-missed-their-mark/.

17. “2011 Census Statistics for England and Wales: March 2011 QMI,” Office for National Statistics, last updated May 16, 2013, https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/methodologies/2011censusstatisticsforenglandandwalesmarch2011qmi.

18. Author interview with Viktor Mayer-Schönberger, March 2014.

19. “Social Media Factsheet,” Pew Research Center, research conducted January 2018, https://www.pewinternet.org/fact-sheet/social-media/.

20. Kate Crawford, “The Hidden Biases in Big Data,” Harvard Business Review, April 1, 2013, https://hbr.org/2013/04/the-hidden-biases-in-big-data.

21. Leon Kelion, “Coronavirus: Covid-19 Detecting Apps Face Teething Problems,” BBC News, April 8, 2020, https://www.bbc.co.uk/news/technology-52215290.

22. Kate Crawford, “Artificial Intelligence’s White Guy Problem,” New York Times, June 25, 2016, https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html.

Rule Seven: Demand Transparency When the Computer Says No

1. Jeremy Ginsberg et al., “Detecting Influenza Epidemics Using Search Engine Query Data,” Nature 457, no. 7232 (February 19, 2009), 1012–14, DOI: 10.1038/nature07634.

2. Parts of this chapter are closely based on my article “Big Data: Are We Making a Big Mistake?,” Financial Times, March 28, 2014, https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0. I interviewed David Hand, Kaiser Fung, Viktor Mayer-Schönberger, David Spiegelhalter, and Patrick Wolfe in early 2014 for the piece.

3. David Lazer and Ryan Kennedy, “What We Can Learn from the Epic Failure of Google Flu Trends,” Wired, October 1, 2015, https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/; and Declan Butler, “What Google Flu Got Wrong,” Nature, February 13, 2013, https://www.nature.com/news/when-google-got-flu-wrong-1.12413.

4. Google Flu Trends and Google Dengue Trends data pages, updated August 2014, https://www.google.org/flutrends/about/ [inactive].

5. D. Lazer et al., “The Parable of Google Flu: Traps in Big Data Analysis,” Science 343, no. 6176 (March 2014), 1203–5.

6. Samantha Cook et al., “Assessing Google Flu Trends Performance in the United States during the 2009 Influenza Virus A (H1N1) Pandemic,” PLoS ONE 6, no. 8: e23610 (2011), https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0023610.

7. Janelle Shane, You Look Like a Thing and I Love You (New York: Little, Brown, 2019).

8. For comprehensive reporting of this affair, see “The Cambridge Analytica Files,” on the Observer/Guardian website, https://www.theguardian.com/news/series/cambridge-analytica-files.

9. Charles Duhigg, “How Companies Learn Your Secrets,” New York Times Magazine, February 19, 2012, https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html.

10. Hannah Fry, Hello World: Being Human in the Age of Computers (London: W. W. Norton, 2018).

11. Cathy O’Neil, Weapons of Math Destruction (London: Allen Lane, 2016).

12. Freakonomics, radio episode 268: “Bad Medicine, Part 1,” August 16, 2017, http://freakonomics.com/podcast/bad-medicine-part-1-story-rebroadcast/.

13. P. A. Mackowiak, S. S. Wasserman, and M. M. Levine, “A Critical Appraisal of 98.6°F, the Upper Limit of the Normal Body Temperature, and Other Legacies of Carl Reinhold August Wunderlich,” JAMA 268, no. 12 (1992), 1578–80, DOI: 10.1001/jama.1992.03490120092034.

14. Jeffrey Dastin, “Amazon Scraps Secret AI Recruiting Tool That Showed Bias against Women,” Reuters, October 10, 2018, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G.

15. Gerd Gigerenzer and Stephanie Kurzenhaeuser, “Fast and Frugal Heuristics in Medical Decision Making,” in Roger Bibace et al., Science and Medicine in Dialogue: Thinking through Particulars and Universals (Westport, CT: Praeger, 2005), 3–15.

16. Paul Meehl, Clinical vs. Statistical Prediction (Minneapolis: University of Minnesota Press, 1954).

17. Fry, Hello World.

18. Mandeep K. Dhami and Peter Ayton, “Bailing and Jailing the Fast and Frugal Way,” Journal of Behavioral Decision Making 14, no. 2 (2001).

19. The original draft of this chapter, like most of the articles I cite, used words such as “commit further crimes” and “reoffend.” But as Cathy O’Neil pointed out to me, what is usually being predicted is being arrested again. Not everyone who is arrested has committed a crime, and not everyone who commits a crime is arrested. I (and my sources) should have been more careful, per my advice under rule three.

20. Jon Kleinberg et al., “Human Decisions and Machine Predictions,” Quarterly Journal of Economics 133, no. 1 (February 2018), 237–93, https://doi.org/10.1093/qje/qjx032; see also Cass R. Sunstein, “Algorithms, Correcting Biases,” working paper, December 12, 2018.

21. David Jackson and Gary Marx, “Data Mining Program Designed to Predict Child Abuse Proves Unreliable, DCFS Says,” Chicago Tribune, December 6, 2017; and Dan Hurley, “Can an Algorithm Tell When Kids Are in Danger?,” New York Times Magazine, January 2, 2018, https://www.nytimes.com/2018/01/02/magazine/can-an-algorithm-tell-when-kids-are-in-danger.html.

22. Hurley, “Can an Algorithm.”

23. Andrew Gelman, “Flaws in Stupid Horrible Algorithm Revealed Because It Made Numerical Predictions,” Statistical Modeling, Causal Inference, and Social Science (blog), July 3, 2018, https://statmodeling.stat.columbia.edu/2018/07/03/flaws-stupid-horrible-algorithm-revealed-made-numerical-predictions/.

24. Sabine Hossenfelder, “Blaise Pascal, Florin Périer, and the Puy de Dôme Experiment,” BackRe(Action) (blog), November 21, 2007, http://backreaction.blogspot.com/2007/11/blaise-pascal-florin-p-and-puy-de-d.html; and David Wootton, The Invention of Science: A New History of the Scientific Revolution (London: Allen Lane, 2015), chap. 8.

25. See, for example, Louis Trenchard More, “Boyle as Alchemist,” Journal of the History of Ideas 2, no. 1 (January 1941), 61–76; and Adam Mann, “The Strange, Secret History of Isaac Newton’s Papers,” Q&A with Sarah Dry, Wired, May 14, 2014, https://www.wired.com/2014/05/newton-papers-q-and-a/.

26. Wootton, The Invention of Science, 340.

27. James Burke, Connections (Boston: Little, Brown, 1978; repr. 1995), 74.

28. Wootton, The Invention of Science, 357.

29. Jeff Larson et al., “How We Analyzed the COMPAS Recidivism Algorithm,” ProPublica, May 23, 2016, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.

30. Sam Corbett-Davies et al., “Algorithmic Decision Making and the Cost of Fairness,” Cornell University, arXiv:1701.08230; and Sam Corbett-Davies et al., “A Computer Program Used for Bail and Sentencing Decisions Was Labeled Biased against Blacks: It’s Actually Not That Clear,” Washington Post, October 17, 2016, https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/17/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas/.

31. Ed Yong, “A Popular Algorithm Is No Better at Predicting Crimes Than Random People,” The Atlantic, January 17, 2018, https://www.theatlantic.com/technology/archive/2018/01/equivant-compas-algorithm/550646/.

32. Yong, “A Popular Algorithm.”

33. Julia Dressel and Hany Farid, “The Accuracy, Fairness and Limits of Predicting Recidivism,” Science Advances 4, no. 1 (January 17, 2018), DOI: 10.1126/sciadv.aao5580.

34. Onora O’Neill’s “Reith Lectures: A Question of Trust” (BBC Radio 4, 2002, http://www.bbc.co.uk/radio4/reith2002/) and TED Talk “What We Don’t Understand about Trust” (September 2013, https://www.ted.com/speakers/onora_o_neill) are both well worth listening to. Themes of intelligent openness are explored in depth in the Royal Society report Science as an Open Enterprise (2012), of which O’Neill was an author. In his book The Art of Statistics (New York: Basic Books, 2019), David Spiegelhalter shows how O’Neill’s principles can be applied to evaluating algorithms.

35. Author email interview with Cathy O’Neil, August 29, 2019.

36. Jack Nicas, “How YouTube Drives Viewers to the Internet’s Darkest Corners,” Wall Street Journal, February 7, 2018, https://www.wsj.com/articles/how-youtube-drives-viewers-to-the-internets-darkest-corners-1518020478; and Zeynep Tufekci, “YouTube, the Great Radicalizer,” New York Times, March 10, 2018, https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html. But see in contrast Mark Ledwich and Anna Zaitsev, “Algorithmic Extremism: Examining YouTube’s Rabbit Hole of Radicalization,” Cornell University, arXiv:1912.11211, https://arxiv.org/abs/1912.11211.

37. Ryan Singal, “Netflix Spilled Your Brokeback Mountain Secret, Lawsuit Claims,” Wired, December 17, 2009, https://www.wired.com/2009/12/netflix-privacy-lawsuit/; and Blake Hallinan and Ted Striphas, “Recommended for You: The Netflix Prize and the Production of Algorithmic Culture,” New Media and Society, 2016, https://journals.sagepub.com/doi/pdf/10.1177/1461444814538646.

Rule Eight: Don’t Take Statistical Bedrock for Granted

1. This is a translation of a Danish TV interview, discussed in Peter Vinthagen Simpson, “Hans Rosling: ‘You Can’t Trust the Media,’” The Local, September 5, 2015, https://www.thelocal.se/20150905/hans-rosling-you-cant-trust-the-media.

2. Laura Smith, “In 1974, a Stripper Known as the ‘Tidal Basin Bombshell’ Took Down the Most Powerful Man in Washington,” Timeline, September 18, 2017, https://timeline.com/wilbur-mills-tidal-basin-3c29a8b47ad1; Stephen Green and Margot Hornblower, “Mills Admits Being Present during Tidal Basin Scuffle,” Washington Post, October 11, 1974.

3. “The Stripper and the Congressman: Fanne Foxe’s Story,” Rialto Report Podcast, episode 82.

4. Alice M. Rivlin, “The 40th Anniversary of the Congressional Budget Office,” Brookings: On the Record, March 2, 2015, https://www.brookings.edu/on-the-record/40th-anniversary-of-the-congressional-budget-office/.

5. Philip Joyce, “The Congressional Budget Office at Middle Age,” Hutchins Center at Brookings, Working Paper #9, February 17, 2015.

6. Quoted in Nancy D. Kates, Starting from Scratch: Alice Rivlin and the Congressional Budget Office (Cambridge, MA: John F. Kennedy School of Government, Harvard University, 1989).

7. Elaine Povich, “Alice Rivlin, Budget Maestro Who ‘Helped Save Washington’ in Fiscal Crisis, Dies at 88,” Washington Post, May 14, 2019, https://www.washingtonpost.com/local/obituaries/alice-rivlin-budget-maestro-who-helped-save-washington-in-fiscal-crisis-dies-at-88/2019/05/14/c141c996-0ff9-11e7-ab07-07d9f521f6b5_story.html.

8. Andrew Prokop, “The Congressional Budget Office, Explained,” Vox, June 26, 2017, https://www.vox.com/policy-and-politics/2017/3/13/14860856/congressional-budget-office-cbo-explained.

9. John Frendreis and Raymond Tatalovich, “Accuracy and Bias in Macroeconomic Forecasting by the Administration, the CBO, and the Federal Reserve Board,” Polity 32, no. 4 (2000), 623–32, DOI: 10.2307/3235295; Holly Battelle, CBO’s Economic Forecasting Record (Washington, DC: Congressional Budget Office, 2010); Committee for a Responsible Federal Budget, “Hindsight Is 2020: A Look Back at CBO’s Economic Forecasting,” January 2013, https://www.crfb.org/blogs/hindsight-2020-look-back-cbos-economic-forecasting.

10. Office for Budget Responsibility, Forecast Evaluation Report 2019, December 2019, https://obr.uk/docs/dlm_uploads/Forecast_evaluation_report_December_2019-1.pdf.

11. Malcolm Bull, “Can the Poor Think?,” London Review of Books 41, no. 13 (July 4, 2019).

12. Bourree Lam, “After a Good Jobs Report, Trump Now Believes Economic Data,” The Atlantic, March 10, 2017, https://www.theatlantic.com/business/archive/2017/03/trump-spicer-jobs-report/519273/.

13. Esther King, “Germany Records Lowest Crime Rate since 1992,” Politico, May 8, 2017, https://www.politico.eu/article/germany-crime-rate-lowest-since-1992/.

14. For discussion and the full Trump tweets, see Matthew Yglesias, “Trump Just Tweeted That ‘Crime in Germany Is Way Up.’ It’s Actually at Its Lowest Level Since 1992,” Vox, June 18, 2018; and Christopher F. Schuetze and Michael Wolgelenter, “Fact Check: Trump’s False and Misleading Claims about Germany’s Crime and Immigration,” New York Times, June 18, 2018.

15. Diane Coyle, GDP: A Brief but Affectionate History (Princeton, NJ: Princeton University Press, 2014), 3–4.

16. European Commission, Report on Greek Government Deficit and Debt Statistics, January 2010, https://ec.europa.eu/eurostat/documents/4187653/6404656/COM_2010_report_greek/c8523cfa-d3c1-4954-8ea1-64bb11e59b3a.

17. Beat Balzli, “Greek Debt Crisis: How Goldman Sachs Helped Greece to Mask Its True Debt,” Der Spiegel, February 8, 2010, https://www.spiegel.de/international/europe/greek-debt-crisis-how-goldman-sachs-helped-greece-to-mask-its-true-debt-a-676634.html.

18. The International Statistical Institute has a chronological account of the sorry tale, last updated by G. O’Hanlon and H. Snorrason, July 2018: “Court Proceedings against Andreas Georgiou,” https://isi-web.org/images/news/2018-07_Court-proceedings-against-Andreas-Georgiou.pdf.

19. “Commendation of Andreas Georgiou,” press release, International Statistical Association, September 18, 2018, https://www.isi-web.org/images/2018/Press%20release%20Commendation%20for%20Andreas%20Georgiou%20Aug%202018.pdf.

20. R. Langkjær-Bain, “Trials of a Statistician,” Significance 14 (2017), 14–19, DOI: 10.1111/j.1740-9713.2017.01052.x; “An Augean Stable,” The Economist, February 13, 2016, https://www.economist.com/the-americas/2016/02/13/an-augean-stable; “The Price of Cooking the Books,” The Economist, February 25, 2012, https://www.economist.com/the-americas/2012/02/25/the-price-of-cooking-the-books.

21. Langkjær-Bain, “Trials of a Statistician.”

22. Author interview with Denise Lievesley, July 2, 2018.

23. “Tanzania Law Punishing Critics of Statistics ‘Deeply Concerning’: World Bank,” Reuters, October 3, 2018, https://www.reuters.com/article/us-tanzania-worldbank/tanzania-law-punishing-critics-of-statistics-deeply-concerning-world-bank-idUSKCN1MD17P.

24. Amy Kamzin, “Dodgy Data Makes It Hard to Judge Modi’s Job Promises,” Financial Times, October 8, 2018, https://www.ft.com/content/1a008ebe-cad4-11e8-9fe5-24ad351828ab.

25. Steven Chase and Tavia Grant, “Statistics Canada Chief Falls on Sword over Census,” Globe and Mail, July 21, 2010, https://www.theglobeandmail.com/news/politics/statistics-canada-chief-falls-on-sword-over-census/article1320915/.

26. Langkjær-Bain, “Trials of a Statistician.”

27. Nicole Acevedo, “Puerto Rico Faces Lawsuits over Hurricane Death Count Data,” NBC News, June 1, 2018; and Joshua Barajas, “Hurricane Maria’s Official Death Toll Is 46 Times Higher Than It Was Almost a Year Ago. Here’s Why,” PBS NewsHour, August 30, 2018, https://www.pbs.org/newshour/nation/hurricane-marias-official-death-toll-is-46-times-higher-than-it-was-almost-a-year-ago-heres-why.

28. Office for National Statistics, 2011 Census Benefits Evaluation Report, last updated October 9, 2017, https://www.ons.gov.uk/census/2011census/2011censusbenefits/2011censusbenefitsevaluationreport#unquantified-benefits; Ian Cope, “The Value of Census Statistics,” Office for National Statistics, https://www.ukdataservice.ac.uk/media/455474/cope.pdf. http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/.

29. Carl Bakker, Valuing the Census (Wellington: Statistics New Zealand, 2014), https://www.stats.govt.nz/assets/Research/Valuing-the-Census/valuing-the-census.pdf.

30. Mónica I. Feliú-Mójer, “Why Is Puerto Rico Dismantling Its Institute of Statistics?,” Scientific American: Voices (blog), February 1, 2018.

31. Testimony, CBO’s Appropriation Request for Fiscal Year 2020, Subcommittee on the Legislative Branch, Committee on Appropriations, U.S. House of Representatives, CBO director Keith Hill testimony, February 26, 2019, https://www.cbo.gov/publication/54965.

32. Ellen Hughes-Cromwick and Julia Coronado, “The Value of US Government Data to US Business Decisions,” Journal of Economic Perspectives 33, no. 1 (2019), 131–46, DOI: 10.1257/jep.33.1.131.

33. Milton and Rose Friedman, Two Lucky People (Chicago: University of Chicago Press, 1998), quoted in Neil Monnery, “Hong Kong’s Postwar Transformation Shows How Fewer Data Can Sometimes Boost Growth,” blog post, London School of Economics and Political Science, June 30, 2017, https://blogs.lse.ac.uk/businessreview/2017/06/30/hong-kongs-postwar-transformation-shows-how-fewer-data-can-sometimes-boost-growth/.

34. James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (New Haven: Yale University Press, 1998).

35. Perry Link, “China: From Famine to Oslo,” New York Review of Books, January 13, 2011.

36. For a discussion of the death toll under Stalin, see Timothy Snyder, “Hitler vs. Stalin: Who Killed More?,” New York Review of Books, March 10, 2011—a more sensitively written piece than the title suggests. For more on the 1937 census, see Daniel Sandford, “In Moscow, History Is Everywhere,” BBC News, November 2, 2012; and Catherine Merridale, “The 1937 Census and the Limits of Stalinist Rule,” Historical Journal 39, no. 1 (1996); and “Called to Account,” The Economist, September 3, 2016, https://www.economist.com/finance-and-economics/2016/09/03/called-to-account.

37. Catherine Merridale, “The 1937 Census and the Limits of Stalinist Rule.”

38. Adam Tooze, Statistics and the German State, 1900–1945 (Cambridge, UK: Cambridge University Press, 2001), 257.

39. Author interview with Denise Lievesley, March 11, 2019.

40. Hetan Shah, “How to Save Statistics from the Threat of Populism,” Financial Times, October 21, 2018, https://www.ft.com/content/ca491f18-d383-11e8-9a3c-5d5eac8f1ab4.

41. Nicholas Eberstadt et al., “‘In Order That They Might Rest Their Arguments on Fact’: The Vital Role of Government-Collected Data,” AEI/Hamilton Project report, March 2017.

42. For more on the review, see G. Hoinville and T. M. F. Smith, “The Rayner Review of Government Statistical Services,” Journal of the Royal Statistical Society, Series A (General) 145, no. 2 (1982), 195–207, DOI: 10.2307/2981534; and John Kay, “A Better Way to Restore Faith in Official Statistics,” blog post, July 25, 2006, https://www.johnkay.com/2006/07/25/a-better-way-to-restore-faith-in-official-statistics/.

43. Ellen Hughes-Cromwick and Julia Coronado, “The Value of U.S. Government Data to U.S. Business Decisions.”

44. Jackie Mansky, “W. E. B. Du Bois’ Visionary Infographics Come Together for the First Time in Full Color,” Smithsonian, November 15, 2018, https://www.smithsonianmag.com/history/first-time-together-and-color-book-displays-web-du-bois-visionary-infographics-180970826/; and Mona Chalabi, “WEB Du Bois: Retracing His Attempt to Challenge Racism with Data,” Guardian, February 14, 2017, https://www.theguardian.com/world/2017/feb/14/web-du-bois-racism-data-paris-african-americans-jobs.

45. Eric J. Evans, Thatcher and Thatcherism (London: Psychology Press, 2004), 30.

46. Ian Simpson, Public Confidence in Official Statistics—2016 (London: NatCen Social Research, 2017), https://natcen.ac.uk/media/1361381/natcen_public-confidence-in-official-statistics_web_v2.pdf.

47. The Cabinet Office, Review of Pre-Release Access to Official Statistics, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/62084/pre-release-stats.pdf.

48. Mike Bird, “Lucky, Good or Tipped Off? The Curious Case of Government Data and the Pound,” Wall Street Journal, April 26, 2017; and “New Data Suggest U.K. Government Figures Are Getting Released Early,” Wall Street Journal, March 13, 2017.

Rule Nine: Remember That Misinformation Can Be Beautiful, Too

1. For more information about the life and statistical contribution of Florence Nightingale, see Mark Bostridge, Florence Nightingale: The Woman and Her Legend (New York: Penguin, 2009); Lynn McDonald, ed., The Collected Works of Florence Nightingale (Waterloo, ON: Wilfrid Laurier University Press, 2009–10), and “Florence Nightingale: Passionate Statistician,” Journal of Holistic Nursing 28, no. 1 (March 2010); Hugh Small, “Did Nightingale’s ‘Rose Diagram’ Save Millions of Lives?,” seminar paper, Royal Statistical Society, October 7, 2010; I. Bernard Cohen, “Florence Nightingale,” Scientific American 250, no. 3 (1984), 128–37, JSTOR, www.jstor.org/stable/24969329; Eileen Magnello, “Florence Nightingale: A Victorian Statistician,” Mathematics in School, May 2010, and “The Statistical Thinking and Ideas of Florence Nightingale and Victorian Politicians,” Radical Statistics, issue 102.

2. Draft from John Sutherland (presumed on behalf of Florence Nightingale) to William Farr, March 1861.

3. These quotes from Nightingale are in Marion Diamond and Mervyn Stone, “Nightingale on Quetelet,” Journal of the Royal Statistical Society 1 (1981), 66–79.

4. Alberto Cairo, The Functional Art (San Francisco: Peachpit Press, 2013).

5. Robert Venturi, Denise Scott Brown, and Steven Izenour, Learning from Las Vegas: The Forgotten Symbolism of Architectural Form (Cambridge, MA: MIT Press, 1977); see also Kurt Kohlstedt, “Lessons from Sin City: The Architecture of ‘Ducks’ Versus ‘Decorated Sheds,’” 99% Invisible.org, September 26, 2016, https://99percentinvisible.org/article/lessons-sin-city-architecture-ducks-versus-decorated-sheds/; and Edward Tufte, The Visual Display of Quantitative Information (Cheshire, CT: Graphics Press, 1983, 2001), 106–21.

6. Scott Bateman et al., “Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts,” Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI, Atlanta, 2010.

7. Linda Rodriguez McRobbie, “When the British Wanted to Camouflage Their Warships, They Made Them Dazzle,” Smithsonian, April 7, 2016, https://www.smithsonianmag.com/history/when-british-wanted-camouflage-their-warships-they-made-them-dazzle-180958657/.

8. David McCandless, Debtris US, Information Is Beautiful, video, December 30, 2010, https://www.youtube.com/watch?v=K7Pahd2X-eE.

9. David McCandless and Stephanie Starling, “The Billion-Pound-O-Gram,” Information Is Beautiful, data visualization, updated November 2019, https://informationisbeautiful.net/visualizations/the-billion-pound-o-gram.

10. Brian Brettschneider, “Lessons from Posting a Fake Map,” Forbes.com, November 23, 2018, https://www.forbes.com/sites/brianbrettschneider/2018/11/23/lessons-from-posting-a-fake-map/#5138b31959ec.

11. Florence Nightingale, “Notes on the Health of the British Army,” in McDonald, The Collected Works of Florence Nightingale, vol. 14, 37.

12. See McDonald, The Collected Works of Florence Nightingale, vol. 14, 551–52.

13. Letter from Florence Nightingale to Sidney Herbert, August 19, 1857.

14. Alberto Cairo, How Charts Lie (New York: W. W. Norton, 2019), 47.

15. William Cleveland, The Elements of Graphing Data (Monterey, CA: Wadsworth, 1994); Gene Zelazny, Say It with Charts (New York: McGraw-Hill, 1985); Naomi Robbins, Creating More Effective Graphs (Hoboken, NJ: Wiley, 2005).

16. Edward Tufte, Envisioning Information (Cheshire, CT: Graphics Press, 1990).

17. Larry Buchanan, “Idea of the Week: Inequality and New York’s Subway,” The New Yorker, April 15, 2013, https://www.newyorker.com/news/news-desk/idea-of-the-week-inequality-and-new-yorks-subway.

18. Simon Scarr, “Iraq’s Bloody Toll,” South China Morning Post, December 17, 2011, https://www.scmp.com/infographics/article/1284683/iraqs-bloody-toll.

19. Andy Cotgreave, “Lies, Damned Lies and Statistics,” InfoWorld, June 28, 2016, https://www.infoworld.com/article/3088166/why-how-to-lie-with-statistics-did-us-a-disservice.html.

20. Letter from William Farr to Florence Nightingale, November 24, 1863, quoted in John M. Eyler, Victorian Social Medicine: The Ideas and Methods of William Farr (London: Johns Hopkins University Press, 1979), 175.

21. Science Museum, “Florence Nightingale: The Pioneer Statistician,” December 10, 2018, https://www.sciencemuseum.org.uk/objects-and-stories/florence-nightingale-pioneer-statistician.

Rule Ten: Keep an Open Mind

1. Leon Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails (New York: Harper Torchbooks, 1956).

2. Walter A. Friedman, Fortune Tellers: The Story of America’s First Economic Forecasters (Princeton, NJ: Princeton University Press, 2013); Sylvia Nasar, Grand Pursuit (London: Fourth Estate, 2011).

3. Friedman, Fortune Tellers.

4. Irving Fisher, How to Live, 21st ed. (New York: Funk & Wagnalls, 1946).

5. Mark Thornton, The Economics of Prohibition (Salt Lake City: University of Utah Press, 1991).

6. Esther Ingliss-Arkell, “Did a Case of Scientific Misconduct Win the Nobel Prize for Physics?,” Gizmodo, April 22, 2014, https://io9.gizmodo.com/did-a-case-of-scientific-misconduct-win-the-nobel-prize-1565949589.

7. Richard Feynman, “Cargo Cult Science,” transcript of speech at Caltech, 1974, http://calteches.library.caltech.edu/51/2/CargoCult.htm.

8. M. Henrion and B. Fischhoff, “Assessing Uncertainty in Physical Constants,” American Journal of Physics 54 (1986), 791–98, https://doi.org/10.1119/1.14447.

9. Author interview with Jonas Olofsson, January 22, 2020.

10. T. C. Brock and J. L. Balloun, “Behavioral Receptivity to Dissonant Information,” Journal of Personality and Social Psychology 6, no. 4, pt. 1 (1967), 413–28, https://doi.org/10.1037/h0021225.

11. Baruch Fischhoff and Ruth Beyth, “‘I Knew It Would Happen’: Remembered Probabilities of Once-Future Things,” Organizational Behavior & Human Performance 13, no. 1 (1975), 1–16, https://doi.org/10.1016/0030-5073(75)90002-1.

12. Philip Tetlock, Expert Political Judgment (Princeton, NJ: Princeton University Press, 2005); Philip Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction (New York: Crown, 2015), 184.

13. Welton Chang et al., “Developing Expert Political Judgment: The Impact of Training and Practice on Judgmental Accuracy in Geopolitical Forecasting Tournaments,” Judgment and Decision Making 11, no. 5 (September 2016), 509–26.

14. Philip Tetlock and Dan Gardner, Superforecasting, 127.

15. Sylvia Nasar, Grand Pursuit (London: Fourth Estate, 2011); John Wasik, Keynes’s Way to Wealth (New York: McGraw Hill, 2013).

16. Anne Emberton, “Keynes and the Degas Sale,” History Today 46, no. 1 (January 1996); Jason Zweig, “When Keynes Played Art Buyer,” Wall Street Journal, March 30, 2018; “The Curious Tale of the Economist and the Cezanne [sic] in the Hedge,” BBC News, May 3, 2014, https://www.bbc.co.uk/news/magazine-27226104.

17. David Chambers and Elroy Dimson, “Retrospectives: John Maynard Keynes, Investment Innovator,” Journal of Economic Perspectives 27, no. 3 (2013), 213–28, DOI: 10.1257/jep.27.3.213.

18. M. Deutsch and H. B. Gerard, “A Study of Normative and Informational Social Influences Upon Individual Judgment,” Journal of Abnormal and Social Psychology 51, no. 3 (1955), 629–36, https://doi.org/10.1037/h0046408.

19. Philip Tetlock, “Here’s a robust psychological effect that does not wilt under replication scrutiny,” Twitter, January 6, 2020, 10:09 a.m., https://twitter.com/PTetlock/status/1214202229156016128.

20. Nasar, Grand Pursuit, 314.

21. Friedman, Fortune Tellers.

The Golden Rule: Be Curious

1. Orson Welles, remarks to students at the University of California, Los Angeles, 1941.

2. Onora O’Neill, Reith Lectures 2002: A Question of Trust, Lecture 4: “Trust and Transparency,” http://downloads.bbc.co.uk/rmhttp/radio4/transcripts/20020427_reith.pdf.

3. Dan M. Kahan et al., “‘They Saw a Protest’: Cognitive Illiberalism and the Speech-Conduct Distinction,” February 5, 2011, Cultural Cognition Project Working Paper no. 63; Stanford Law Review 64 (2012); Temple University Legal Studies Research Paper, no. 2011-17, available at https://ssrn.com/abstract=1755706.

4. Dan M. Kahan, “Why Smart People Are Vulnerable to Putting Tribe before Truth,” Scientific American: Observations (blog), December 3, 2018, https://blogs.scientificamerican.com/observations/why-smart-people-are-vulnerable-to-putting-tribe-before-truth/; Brian Resnick, “There May Be an Antidote to Politically Motivated Reasoning. And It’s Wonderfully Simple,” Vox, February 7, 2017, https://www.vox.com/science-and-health/2017/2/1/14392290/partisan-bias-dan-kahan-curiosity; Dan. M. Kahan et al., “Science Curiosity and Political Information Processing,” Political Psychology 38 (2017), 179–99, DOI: 10.1111/pops.12396.

5. Author interview with Dan Kahan, November 24, 2017.

6. J. Kaplan, S. Gimbel, and S. Harris, “Neural Correlates of Maintaining One’s Political Beliefs in the Face of Counterevidence,” Scientific Reports 6, art. no. 39589 (2016), https://doi.org/10.1038/srep39589.

7. G. Loewenstein, “The Psychology of Curiosity: A Review and Reinterpretation,” Psychological Bulletin 116, no. 1 (1994), 75–98, https://doi.org/10.1037/0033-2909.116.1.75.

8. L. Rozenblit and F. Keil, “The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth,” Cognitive Science 26 (2002), 521–62, DOI: 10.1207/s15516709cog2605_1.

9. Philip M. Fernbach et al., “Political Extremism Is Supported by an Illusion of Understanding,” Psychological Science 24, no. 6 (2013), 939–46, https://doi.org/10.1177/0956797612464058.

10. Steven Sloman and Philip M. Fernbach, “Asked to Explain, We Become Less Partisan,” New York Times, October 21, 2012.

11. Michael F. Dahlstrom, “Storytelling in Science,” Proceedings of the National Academy of Sciences of the United States of America 111, supp. 4 (2002), 13614–20, DOI: 10.1073/pnas.1320645111.

12. Bruce W. Hardy et al., “Stephen Colbert’s Civics Lesson: How Colbert Super PAC Taught Viewers about Campaign Finance,” Mass Communication and Society 17, no. 3 (2014), 329–53, DOI: 10.1080/15205436.2014.891138.

13. “The Planet Money T-Shirt,” Planet Money podcast, special series, December 11, 2013, https://www.npr.org/series/262481306/planet-money-t-shirt-project-series?t=1580750014093.

14. Economics: The Profession and the Public, seminar held at the Treasury, London, May 5, 2017.

15. “The Cure for Boredom Is Curiosity. There Is No Cure for Curiosity,” Quote Investigator, updated September 26, 2016, https://quoteinvestigator.com/2015/11/01/cure/.

16. “Why is this lying bastard lying to me?” This sentiment courtesy of the renowned British reporter Louis Herren.

CREDITS

Chapter epigraphs

1: Umberto Eco, Serendipities: Language and Lunacy. London: Orion, 1998.

2: The Empire Strikes Back (1980); also known as Star Wars: Episode V; screenplay by Leigh Brackett and Lawrence Kasdan.

3: Muhammad Yunus, interviewed by Steven Covey, “Stephen Covey Interviews Grameen Bank Founder Muhammad Yunus,” SocialBusinessPedia, last updated November 21, 2016, http://socialbusinesspedia.com/wiki/details/248.

4: Douglas Adams, The Hitchhiker’s Guide to the Galaxy. London: Pan Books, 1979.

5: Terry Pratchett, Reaper Man. London: Victor Gollancz, 1991.

6: Alan Moore, Watchmen. New York: DC Comics, 1986.

7: Anna Powell-Smith, MissingNumbers.org.

8: 2001: A Space Odyssey (1968), screenplay by Stanley Kubrick and Arthur C. Clarke.

9: Translation of a Danish TV interview, discussed in Peter Vinthagen Simpson, “Hans Rosling: ‘You Can’t Trust the Media,’” The Local, September 5, 2015, https://www.thelocal.se/20150905/hans-rosling-you-cant-trust-the-media.

10: Michael Blastland, personal correspondence, May 13, 2013.

11: Leon Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails. New York: Harper Torchbooks, 1956.

12: Orson Welles, remarks to students at the University of California, Los Angeles, 1941.

ABCDEFGHIJKLMNOPQRSTUVWXYZ

INDEX

The page numbers in this index refer to the printed version of this book. The link provided will take you to the beginning of that print page. You may need to scroll forward from that location to find the corresponding reference on your e-reader.

Page numbers in italics denote images and captions.

accessibility of data, 81, 132–34, 179–80, 183, 205–11

accountability, 180, 181

Active Lives Survey, 143

actively open-minded thinkers, 254–55

ad hominem attacks, 39

Adams, Douglas, 65

Adapt (Harford), 123

addiction, 5, 222

administrative data, 148–50. See also official statistics

affirmative action, 33–34

age data

and childhood mortality trends, 66–67, 91

and criminal justice data, 178

and defining “children,” 70n

and divorce rates, 253n

and evaluation of statistical claims, 133

and incidence of strokes, 98

and misuse of statistical methods, 117–19

and “priming” research, 121

and scale of numerical comparisons, 93–94

and self-harm statistics, 75

and smoking research, 4–5

and teen pregnancy, 55

and vaccination research, 53–54

aggressive behavior, defining, 70–71

AIDS, 28

alchemy, 171, 173–75, 173n, 179, 207

algorithms

alchemy analogy, 175

and consumer data, 159–64

criminal justice applications, 176–79

and excessive credulity of data, 164–67

and found data, 149, 151, 154

and Google Flu Trends, 153–57

vs. human judgment, 167–71

pattern-recognizing, 183

and proliferation of big data, 157–59

and systematic bias, 166

and teacher evaluations, 163–64

trustworthiness of, 179–82

See also big data

Allegory of Faith, The (Vermeer), 29–30

Allied Art Commission, 21

alternative sanctions, 176–79

altimeters, 172–73

Amazon, 148, 175, 181

American Economic Association, 186n

American Statistical Association, 194

Anderson, Chris, 156

Angwin, Julia, 176

anorexia, 75

antidepressants, 126

anti-vaccination sentiment, 53–54

Apple, 175

Argentina, 194–95, 211

Army Medical Board (UK), 215

art forgeries, 19–23, 29–32, 42–45

Asch, Solomon, 135–38, 141–42, 260

assessable decisions, 180

astrology, 124

Atkinson, Tony, 83

atomic weapons, 90, 249–50

Attenborough, David, 277–78

authority, deference to, 138–39

autism, 53–54

automation, 128

Avogadro’s number, 246

Babbage, Charles, 219

Babcock, Linda, 27

Babson, Roger, 263

backfire effect, 129

Bad Science (Goldacre), 2

bail recommendations, 158, 169, 180

Bakelite, 32

Bank for International Settlements, 100–101

Bank of England, 256, 258

Bannon, Steve, 13

bar charts, 228, 234, 235

Bargh, John, 121, 122

barometric pressure, 172–73

Barrack Hospital, Scutari (Istanbul), 213–14, 220, 225, 233, 235

base rates, 253–54, 253n

Battistella, Annabelle (Fanne Foxe), 185–86, 212

battlefield awareness, 58–59

Baumeister, Roy, 121, 122

BBC, 10, 276

behavioral data, 69–70

behavioral economics, 25, 27, 41, 96, 271. See also choice research

Bell, Vanessa, 256–57

Bem, Daryl, 111, 113–14, 119–23

benefits of statistical analysis, 9

Berti, Gasparo, 172

Bevacqua, Graciela, 194–95, 212

Beyth, Ruth, 248–49, 251, 254

biases

biased assimilation, 35–36

confirmation bias, 33

current offense bias, 169

and motivated reasoning, 27–29, 32–36, 38, 131, 268

negativity bias, 95–99

non-response bias, 146–47

novelty bias, 95–99, 113, 114, 122

optimism bias, 96

and power of doubt, 13

publication bias, 113–16, 118–23, 125–27

racial bias in criminal justice, 176–79

in sampling, 135–38, 142–45, 147–51

selection bias, 2, 245–46

survivorship bias, 109–10, 112–13, 122–26

systematic bias in algorithms, 166

and value of statistical knowledge, 17

big data

and certification of researchers, 182

and criminal justice, 176–79

and excessive credulity in data, 164–67

and found data, 149, 151, 152, 154

and Google Flu Trends, 153–57

historical perspective on, 171–75

influence in today’s world, 183

limitations and misuse of, 159–63, 170–71

proliferation of, 157–59

and teacher evaluations, 163–64

See also algorithms

Big Data (Cukier and Mayer-Schönberger), 148, 157

“Big Duck” graphics, 216–18, 217, 229–30

Big Issue, The, 226n

“Billion Pound-O-Gram, The” (infographic), 223

billionaires, 78–80

binge drinking, 75

Bird, Sheila, 68

bird’s-eye view of data, 61–64, 203, 221, 265

BizzFit, 108

Black Swan, The (Taleb), 101

Blastland, Michael, 10, 68, 93

blogs, 76

Bloomberg TV, 89

body count metrics, 58

Boijmans Museum, 20

Boon, Gerard, 19, 30–31

border wall debate, 93–94

Borges, Jorge Luis, 118

Boyle, Robert, 172–75

brain physiology, 270

Bredius, Abraham, 19–23, 29–32, 35, 43–45, 78, 242, 262

Bretton Woods conference, 262

Brettschneider, Brian, 224

Brexit, 71, 277

British Army, 213–14, 220–21

British Election Study, 145–46

British Medical Journal, 6, 67

British Treasury, 256–57

Broward County Sheriff’s Office, 176

Brown, Derren, 115

Brown, Zack “Danger,” 108

Buchanan, Larry, 229, 232

budget deficits, 188, 192–93, 195

Buffett, Warren, 259

Bureau of Economic Analysis, 190, 205

Bureau of Labor Statistics, 190, 205, 212

business-cycle forecasting, 258–59

business writing, 123–24

Butoyi, Imelda, 62–63

Cairo, Alberto, 227

Cambridge Analytica, 158

Cambridge University, 162. See also King’s College, Cambridge

Cameron, David, 146

camouflage, 218–19

campaign finance, 274–75

Campbell, Donald T., 59

Campbell Collaboration, 134

Canada, 196–97, 209, 211

cancer diagnoses and research, 3–6, 24, 97, 241, 248, 279

Capital in the Twenty-First Century (Piketty), 83

capital punishment, 35–36

Caravaggio, Michelangelo, 30–31

carbon emissions, 272–73

Carter, Jimmy, 188

casualty statistics, 58, 231–37, 234, 235

categorization of statistical data, 68–70

causation vs. correlation, 15, 64, 156–57, 275

Census Bureau, 190, 205

census data, 147–48, 196–99, 205–6

Centers for Disease Control and Prevention (CDC), 153, 154, 155

Centre for Evidence-Based Medicine, 128

certification of researchers, 182

Chalmers, Iain, 133

Chambers, David, 259

charts, 215–16, 218n, 224, 227–29, 232–36, 232, 234, 235. See also visualization of data

cheating, 164

cherry-picking data, 245–46. See also selection bias

Chicago Tribune, 170–71

child abuse and neglect, 170–71

child benefits, 142

child safety, 170–71

childbirth data, 55

childhood vaccination, 53, 99

China, 59–61, 92

choice research, 105–8, 110–14, 123–24

Christ at Emmaus (forged painting), 19–22, 23, 30–32, 43

Christ in the House of Martha and Mary (Vermeer), 29–30

Christianity Is Evil (pamphlet), 248

cigarette smoking, 3–6, 14–15, 39, 52, 279

climate science and data

and motivated reasoning, 34–35, 36–39, 268–70

and novelty bias in reporting, 100

and power of doubt, 13

and scale of news stories, 90

and visualization of data, 224

clinical research, 57, 120, 128, 139–40, 180–81. See also randomized clinical trials

Clinical vs. Statistical Prediction (Meehl), 167

Clinton, Bill, 188–89

Clinton, Hillary, 147

CNN, 158

Cochrane, Archie, 132–33

Cochrane Collaboration, 132–34

Cochrane Library, 133–134

codes of ethics, 180

cognitive reflection test, 41–42

Colbert, Stephen, 274–75, 274n

Colbert Report, The, 274–75, 274n

commute times, 47–49

comparability of data

and forecasting, 252, 254

and income inequality measures, 82

and infant mortality measures, 66–67

and official statistics, 189

and scale of data, 93–95

and value of imagery, 62–63

and visualization of data, 221–23, 228, 230–31, 235

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), 176–79

complex choices, 105–8

composite measures, 91–92

confidentiality, 208

confirmation bias, 33

conformity, 135–38

Congressional Budget Office (CBO), 186–89, 199, 204, 212

Conservative Party (UK), 146–47

consumer data, 159–64, 175–76

contactless payment data, 49

context of data, 88. See also perspective on statistical data

convergence of estimates, 246

conversion of units, 165n

Corbett-Davies, Sam, 177

Corbyn, Jeremy, 14

coronavirus pandemic (COVID-19)

and biases in journalism, 99, 102

and evaluation of statistical claims, 26, 29, 68, 68n

and exponential growth, 41n

and gender disparities, 140

and Nightingale Hospital, 213

and sanitary practices, 226n

and selection bias in data, 109–10

statistical challenges of, 6–11, 120n

and vaccination trends, 99

correlation vs. causation, 15, 64, 156–57, 275

corruption, 59, 209–10

cost-benefit analyses, 198, 199

Cotgreave, Andy, 232

COVID-19. See coronavirus pandemic (COVID-19)

Cowperthwaite, John, 200–201, 203, 204

Crawford, Kate, 150

credibility of data, 192, 195

Credit Suisse, 80, 80n

credulity in data, 164–67

Criado Perez, Caroline, 139

crime and criminal justice

and alternative sanctions, 176–79

and bail recommendations, 158, 169, 180

and capital punishment, 35–36

and COMPAS system, 176–79

and human judgment vs. algorithms, 168–71

and missing data issues, 143

and murder rate data, 55, 87–89, 88n

Crime Survey of England and Wales, 143

Crimean War, 213–14, 220, 226, 233–37

Cuddy, Amy, 121, 122

Cukier, Kenneth, 157

cultural influences, 37, 136–37, 201

curiosity, 265–79, 276n

currency speculation, 255, 257–58

current offense bias, 169

curse of knowledge, 71–72

cyberspace, 149

cynicism and skepticism about statistics, 8–10, 13, 14, 266

Daily Mail, 133

dark data, 146

data recording practices, 66–67

dazzle camouflage, 218–19

De Meyer, Kris, 38–39

De Waarheid, 44

death penalty, 35–36

death rates, 214–15, 236. See also casualty statistics

debt

and derivative markets, 100

and forecasting failures, 261–62

and Keynes’s economic management, 255

and net wealth measures, 79

and official statistics, 192–95

and Trump anecdote, 79n

unhelpful comparisons, 95n

and value of statistical data, 10

and visualization of data, 222–23

Debtris (infographic), 222–23, 235

definition of terms, 87–88

Degas, Edgar, 256

Democratic Party, 34, 269, 270

demographic data, 198

denialism, 25, 28, 36, 38–39, 40

Descartes, René, 172

Deutsch, Morton, 260

development, 142–43

diabetes, 56

Dilnot, Andrew, 10, 93

Dimson, Elroy, 259

distance measures, 94

divorce rates, 96–97, 253–54, 253n

Dobelli, Rolf, 101

Doll, Richard, 3–6, 12, 15, 225, 279

Dollar Street, 62–64

domestic violence, 143

double standards, 40

doubt, fostering, 12–14, 15, 35–36, 40, 46. See also cynicism and skepticism about statistics

Dow Jones Industrial Average, 243

Draeger’s supermarket, 106

Dressel, Julia, 178

drugs and drug trials, 61, 126, 129–30, 139–40

Du Bois, W. E. B., 206

Duhigg, Charles, 159, 161–62, 163

Eco, Umberto, 1

economic data and statistics

and budget deficits, 188, 192–93, 195

and business-cycle forecasting, 258–59

and choice experiment, 106

debt data, 10, 79, 79n, 95n, 100, 192–95, 222–23, 255, 261–62

and forecasting, 250

and Greek debt crisis, 192–95, 211

gross domestic product data, 94

growth measures, 200–201

income measures, 141

labor statistics, 11, 68–69, 141, 189n, 207, 218n

and perspective on data, 59–62

and poverty, 59–60, 79, 91, 98, 99–100

wealth and income inequality, 62–64, 76–84, 85, 229–31

Economist, The, 89

education levels, 34–35, 38, 163–64

Edwards, Kari, 12–13, 33

elections, 55, 144–47, 164–65

Elliott, Andrew, 94

ELSTAT (Greek statistical agency), 192–93, 195

emotional responses to data

and cognitive reflection, 38–42

and curiosity, 265

and data visualizations, 225, 232, 237

and denialism, 25, 28, 36, 38–39, 40

and errors in judgment, 78

and Fisher’s failings, 242

and influence of expertise, 32–36

and interpretation of statistics, 22–27

manipulation of emotions, 19–23, 29–32, 42–46

and motivated reasoning, 27–29, 32–36, 38, 131, 268

and open-mindedness, 248–49

and personal experience, 48–49

and self-deception, 23–27, 262–63

and social consequences of beliefs, 36–38

Empire Strikes Back, The (film), 19

employment/unemployment data, 11, 68–69, 189n, 207, 218n

Energy Information Administration, 190, 205

England, 67. See also United Kingdom

Envisioning Information (Tufte), 229

epidemiology, 8, 29, 120n

Equivant, 176

erectile dysfunction drugs, 140

esoteric knowledge, 175

estimation, 49. See also forecasting

ethnicity data, 150

eugenics, 241

European Commission, 192

European Union, 192, 193, 277

Eurostat, 192, 193–94, 212

evolution, 38

experimental method, 173

Expert Political Judgment (Tetlock), 250

expertise

and forecasting failures, 249–53

and motivated reasoning, 32–36, 268

and political polarization, 272–73

and self-deception, 262–63

exponential growth, 41n

Facebook, 26, 56, 132n, 150, 158, 175, 180, 182, 224

face-to-face surveys, 145–46

Factfulness (Rönnlund, Rosling, and Rosling), 96

fake news, 13–14, 41, 44

false positives, 161

Farid, Hany, 178

Farr, Mary, 226

Farr, William, 214, 214n, 221, 226, 233–34

fast statistics, 56, 57, 60–61, 63–64

Federal Bureau of Investigation (FBI), 72n

Federal Reserve, 186n, 190

“Feeling the Future” (Bem), 111–12

Feller, Avi, 177

Fernández de Kirchner, Cristina, 194

Fernbach, Philip, 272

Festinger, Leon, 239

Feynman, Richard, 245

50/10 ratio, 82, 91

filtering information, 247

financial bubbles, 243–44

financial crises, 71, 83–84, 124, 192–95, 211. See also Great Depression

financial incentives, 27–28, 28n

financial markets, 255

financial reporting, 89, 100–101

Financial Times, 89, 100, 101, 138–39, 142, 196, 214n

Finland, 66, 92

First World War, 255–56

Fischoff, Baruch, 248–49, 251, 254

Fisher, Irving, 239–44, 249, 252, 255, 257–58, 259–63

501(c)(4) groups, 275

flat tax, 272–73

Fletcher, Harvey, 245

fluke findings, 118, 122

folic acid, 159

food security, 63

forecasting

and base rates, 253–54, 253n

business-cycle forecasting, 258–59

election predictions, 164–65

Fisher and Keynes compared, 258–63

and non-representative samples, 144–47

and selective memory, 248–51

superforecasters, 252–55

forgeries, 19–23, 29–32, 42–45

found data, 149, 151, 152, 154

Foursquare, 150

Foxe, Fanne (Annabelle Battistella), 185–86, 212

fracking, 268

Frank, Anne, 43

Franklin, Benjamin, 32

Frederick, Shane, 41–42

free speech, 196

Friedman, Milton, 200, 204

Frisch, Ragnar, 239

Fry, Hannah, 160, 167–68

Fuentes, Ricardo, 76

Full Fact, 143

Fung, Kaiser, 161–62

Galileo Galilei, 16

Gallup, George, 144, 145

Gallup polls, 34

Galtung, Johan, 89

gambling, 262

Gapminder, 62

garden of forking paths, 118, 121–22, 127

gas law, 172

Gelman, Andrew, 118, 127, 171

gender issues, 75, 138–42, 150, 152

geopolitics, 7, 249–52

Georgiou, Andreas, 192–95, 212

Gerard, Harold, 260

germ theory, 225

Gini coefficient, 83n, 92–93

Girl with a Pearl Earring (Vermeer), 21

global development, 62

global financial crisis (2009), 193

Global Wealth Report, 80, 80n

Goel, Sharad, 177

Goldacre, Ben, 2, 123, 128

Goldin, Rebecca, 70

Goldman Sachs, 193

Goldstein, Jacob, 115

Good Judgment Project, 252

Goodhart, Charles, 59

Google, 175, 182

Google Flu Trends, 153–57, 165

Gorbachev, Mikhail, 250

Göring, Hermann, 22, 44

Gould, Stephen Jay, 24

Gove, Michael, 277n

Grant, Duncan, 257

graphics. See infographics; visualization of data

Great Depression, 243, 261

Great Leap Forward, 202

Greek debt crisis, 192–95, 211

Greek Ministry of Finance, 193

Greek Prosecutor of Economic Crimes, 193

Greenpeace, 42

Greifeneder, Rainer, 107

gross domestic product measures, 94

gross vs. net measures, 222

Guangdong, China, 60

Guardian, 67, 74, 77, 101

gun control, 33–34, 268

gun deaths, 72–74, 72n

Gun Violence Archive, 72n

habits fostering statistical analysis, 25

Hand, David, 146

HARKing (Hypothesizing After Results Known), 118

Harper, Sean, 196–97

Hawking, Stephen, 277–78

Hayek, Friedrich, 59

Hazard, Margaret, 240

headlines, 229

health and medical data

and clinical trials, 4n, 53, 61, 125–26, 129–30, 133, 139–40, 180

and correlation vs. causation, 14–15

and emotion’s impact in data interpretation, 24–27

and epidemiology, 8, 29, 120n

and Fisher, 240–41

and gun deaths, 72–74, 72n

HIV/AIDS data, 36

and informed consent, 181

medical records, 220–21

and motivated reasoning, 28–29

and optimism bias, 96–98

and publication bias, 125–26

and sanitation advocacy, 225–26, 233–37

and scale of news reporting, 91

and smoking research, 3–6, 96, 100, 248, 279

and trust in statistics, 52–53

and vaccinations, 53–54, 99

and war casualties, 231–37, 234, 235

health care reform, 188–89

heart attacks, 6

hedge funds, 255, 257

height distribution data, 93, 228

Hello World (Fry), 160, 167–68

Herbert, Sidney, 220, 227, 236

herpes virus, 25

Hill, Austin Bradford, 3–6, 4n, 12, 15, 225, 279

Hitchhiker’s Guide to the Galaxy, The (Adams), 65

Hitler, Adolf, 4, 42–43, 201

HIV/AIDS data, 36

Holmes, Nigel, 217

Holocaust, 43

homicides, 55, 72–73, 87–89, 88n

Hong Kong, 60, 200–201, 203

Horse Guards, 215

hot sauce paradigm, 70

House of Commons, 146

House Ways and Means Committee, 186

household incomes, 141

How Charts Lie (Cairo), 227

How to Lie with Statistics (Huff), 2–3, 6, 15, 120

How to Live (Fisher), 241

Huff, Darrell, 2–3, 6, 8–10, 15, 120, 266, 278

human evolution, 38

human immunodeficiency virus (HIV), 28, 36, 38

human judgment vs. algorithms, 167–71

humility, 259

Hurricane Sandy, 150

hurricanes, 150, 197–98

hygiene standards, 225–26, 226n. See also sanitation advocacy

hypotheses, 255

Hypothesizing After Results Known (HARKing), 118

ideologies, 267–69

Illinois, 170–71

illusion of explanatory depth, 272

images as data, 62–64

immigrants, 55, 71

immigration policy, 191

IMPACT algorithm, 163

impartiality, 139, 188, 190

In Search of Excellence (Peters and Waterman), 123

incentives, 115

income measures, 141. See also wealth and income inequality

income statistics, 59–60, 91–93

incomplete information, 246

independence of official statistics, 188–91, 192–200

Independent, 77–78

independent statistical agencies. See official statistics

Index Number Institute, 242

Index Visible, 242, 244

infant and childhood mortality, 65–67, 91, 99

inflation, 189n, 194–95, 218n, 257

influenza, 153–57

infographics, 215, 219, 221–25, 229–31. See also visualization of data

information gap theory, 271

Information Is Beautiful (McCandless), 222

informed consent, 181

Instagram, 150

integrity of data, 233–37

Intelligence Advanced Research Projects Activity, 251–52

intelligence community, 251

International Committee of Medical Journal Editors, 127

International Monetary Fund (IMF), 185, 192, 193, 262, 277n

International Statistical Association, 194

International Statistical Congress, 221

internet, 132, 148. See also social media

Internet Institute, 148

Internet Movie Database, 182

Inuits, 136

investing, 124–26, 242–43, 255–60, 262

Invisible Women (Criado Perez), 139

Ioannidis, John, 7, 120–22, 120n, 130

iPhones, 150–51

Ipsos Mori, 55

Iran sanctions, 273

Iraq War (begun 2003) , 222, 231–33

deaths (2003–2011), 231–33, 232

Irving Fisher’s Business Page, 242

Is That a Big Number? (Elliott), 94

Iyengar, Sheena, 105, 107

jam experiment (more vs. fewer choices), 105–8, 110–11, 123–24

Jamieson, Kathleen Hall, 275

Johnson, Richard, 168, 169–70

Journal of Personality and Social Psychology, 111–14, 120

journalism, 41. See also news media

Kahan, Dan, 267–71, 275

Kahneman, Daniel, 41, 56, 61n, 96, 121–22, 130, 254

Keil, Frank, 272

Kestenbaum, David, 115

Keynes, John Maynard, 255–59, 260–61, 262

Kickended.com, 109–12, 122, 127

Kickstarter, 108–9, 110, 112, 122

King’s College, Cambridge, 255, 257–58, 259

Kirchner, Néstor, 194

Knowledge Illusion, The (Fernbach and Sloman), 272

Kosara, Robert, 227

Kunda, Ziva, 24

Kurov, Alexander, 209

Kvitkin, Olimpiy, 202

labor statistics, 11, 68–69, 141, 189n, 207, 218n

laissez-faire economics, 200–201, 203

land ownership, 201

landmark numbers, 94–95

law and legal statistics, 66

Leave Means Leave, 71

LePan, Douglas, 255

Lepper, Mark, 35, 105, 107

leveraged investments, 243–44, 259–60

Lewin, Kurt, 260–61

Lewis, Martyn, 99, 99n

Lewis, Michael, 13

libertarian ideology, 203

Lievesley, Denise, 195–96

life expectancy, 214, 236

Life Extension Institute, 241

lifespan data, 93

Limeburner, Shannon, 108

LinkedIn, 150

literacy rates, 203–4

Literary Digest, 143–44, 164

lobbyists, 13, 273

Lodge, Milton, 33, 38

Loewenstein, George, 27, 271

London, 87–88, 167–68

London Underground, 47, 50–51

Lord, Charles, 35

Lorusso, Silvio, 109, 111

Lovelace, Ada, 219

Lundberg, Shelly, 142

lung cancer, 3–6, 52, 248. See also smoking research

Mackowiak, Philip, 165–66

mad cow disease, 12

Madison, James, 204

Malkiel, Burton, 125

Mao Zedong, 201, 248

maps, 224–25

Marks & Spencer, 205

Martineau, Harriet, 236

mass shootings, 72–73, 72n

mass surveillance, 202

maternity services, 65

“Math Is Racist” (CNN), 158, 166–67

mathematical risk models, 71

May, Theresa, 14

Mayer-Schönberger, Viktor, 148, 157

Mayraz, Guy, 27

McCandless, David, 222, 223

McDonald, Lynn, 214n

McGregor, David, 108

McMaster, H. R., 58

McNamara, Robert, 58

measles, mumps, and rubella (MMR) vaccine, 53–54

measurement methods and choices, 68, 117–18, 165n

media reporting. See news media

median income measures, 82

medical data. See health and medical data

Medicare, 199

Meehl, Paul, 167

Mellers, Barbara, 252

memes, 42

memory

and criminal sentencing, 168

and data visualization, 217–18

and forecasting failures, 248–49, 251, 254

and Milgram experiment, 138

and “precognition” study, 111

and scale of numerical comparisons, 95

and value of imagery, 64

Merkel, Angela, 191

Messy (Harford), 57–58, 203

Metropolitan Police (London), 199

microcredit, 61

Microsoft, 175

Milgram experiment, 138–39

Milkmaid, The (Vermeer), 21, 31

Millikan, Robert, 244–47, 260

Mills, Wilbur, 186

mindfulness meditation, 134

miscarriages, 66

misconceptions of statistics, 55–56

misinformation, 3, 26, 41, 223. See also visualization of data

misleading claims, 107

missing data, 135–52

and gender issues, 138–42

limits of combating, 151–52

and sample bias, 135–38, 142–51

MissingNumbers.org, 135

mixed-gender groups, 138

Modi, Narendra, 196

moiré patterns, 223

Molière, 32

monocultures, 137

Moore, Alan, 105

Moore, Don, 252

More or Less (radio show), 10, 11, 47, 68, 71, 93, 213, 266–67, 276

mortality rates, 65–67

Mosteller, Frederick, 15n

motivated reasoning, 27–29, 32–39, 131, 268

Moy, Will, 143

Mullainathan, Sendhil, 169

Muller, Jerry Z., 57

multiple-occupancy housing, 148

murder rates, 55, 87–89, 88n

“N = All” assumptions, 150, 152, 155

naive realism, 54–55

Naked Statistics (Wheelan), 2

Nasar, Sylvia, 262

National Aeronautics and Space Administration (NASA), 217

National Gallery, London, 256

National Health Service (NHS), 74, 159

National Safety Council, 72

national statistics. See official statistics

natural disasters, 197–98

Nature, 153, 155

Nature News, 155

Nazi collaborators, 42, 45

Nazi Germany, 202

negativity bias, 95–99

Nelson, Leif, 119

net wealth, 78–79, 79n

Netflix, 148, 149, 182

Netherlands, 43

neuroscience, 270

New Orleans, Louisiana, 197–98

New York City, 87–88, 169

New Yorker, 229

New York Times, 2, 240, 261

New York Times Magazine, 159

New Zealand, 199

news media

biases in, 95–99

and context of statistics, 56

and data visualizations, 229, 232

and Fisher, 242

and headline wording, 229

perspective of news stories, 99–102

scale of news stories, 89–91

Newton, Isaac, 173, 174–75

Nightingale, Florence, 213–16, 214n, 219–21, 225–27, 226n, 233–37, 234

Nightingale Hospital, 213

Nixon, Richard, 187, 188, 248

Nobel Prizes, 58–59, 61, 130, 200, 239, 244

non-response bias, 146–47

Nosek, Brian, 114, 116, 120, 122, 130–31

novel representations of statistics, 95n

novelty bias, 95–99, 113, 114, 122

NPR, 276

nuclear power, 268

nuclear weapons, 90, 249–50

numeracy, 268

Nyhan, Brendan, 129

objective truth, 37

Office for Budget Responsibility (OBR), 189

Office for National Statistics, 75, 200, 207, 212, 253n

Office of Management and Budget (OMB), 186n

official statistics, 185–212

benefits of, 211–12

and Congressional Budget Office, 186–89, 199, 204

and governmental incompetence, 200–204

importance of political independence, 188–91, 192–200

public availability of, 204–11

oil drop experiment, 245

O’Neil, Cathy, 158, 163, 164, 180

O’Neill, Onora, 179–80, 267

open-mindedness, 239–63

and failure of expertise, 249–51

and Fisher’s failings, 239–44, 259–63

human failings and biases, 244–49

and Keynes’s successes, 255–62

and superforecasting, 254–55

opinion polls

and election forecasts, 147, 165

and missing data problems, 149

and motivated reasoning, 34

and naive realism, 55

and non-representative samples, 144–47

and van Meegeren’s popularity, 45

optimism, 96, 262

ostrich effect, 25

overconfidence, 96, 262

“Oxfam: 85 Richest People as Wealthy as Poorest Half of the World,” 76–78, 80–81, 80n

Oxfam International, 76–77, 80, 98, 107

Oxford University, 27, 29, 128, 148

Paris Exposition (1900), 206

Parliament (UK), 236

particle accelerators, 173n

partisanship, 34–36

Pascal, Blaise, 172, 173

patterns in data, 154, 155–56, 227

payment data, 49

Pebble smartwatch, 108

peer review, 111–12, 189n

pensions, 80, 80n

personal experience, 47–64

and limitation of statistics, 57–59

and perspective on data, 59–64

and public transportation usage, 47–51

and trust in statistics, 52–56

and wealth inequality, 62–64

perspective on statistical data

and context of data, 102–3

and evaluation of credibility, 131

and evaluation of statistical claims, 64

and income inequality measures, 91–93

and news reporting, 89–91, 95–103

and scale issues, 89–91, 93–95, 102–3

and time horizon of news reporting, 99–102

worm’s-eye and bird’s-eye views, 61–62, 63–64, 265

persuasive power of data, 237. See also open-mindedness; visualization of data

Peters, Tom, 123

Phillips curve, 218n

photos as data, 62–64

physics research, 244–47

pie charts, 224, 228, 233–34

Pierson, Emma, 177

Piketty, Thomas, 83

Pinker, Steven, 71–72, 96–97

placebo, 53, 125–26, 130

Planck’s constant, 246

Planet Money, 276

plausibility of data, 49

pointillés, 30–31, 44

polar area diagrams, 233–36, 234

polemics, 231

policy proposals, 70–71, 84–85. See also official statistics

political action committees (PACs), 274

political expertise, 33, 33n

political forecasting, 249–53. See also elections

political influence, 13, 33–34, 192–97

political partisanship, 34–35

political polarization, 267–70, 272–73, 277

population data, 94, 202

positivity bias, 247–48

post-hoc theorizing, 117–18

pothole-detection apps, 165

poverty statistics, 59–60, 79, 91, 98, 99–100

Powell-Smith, Anna, 135

practice and memory, 111–12

Pratchett, Terry, 87

“precognition” study, 111–12, 113–14, 116, 121, 123, 128–29

preconceptions, 33, 245–46

prejudice, 167

premature enumeration, 65–85

and gun death data, 72–74, 72n

and infant mortality statistics, 56–67

and self-harm data, 73–76

and understanding statistical claims, 67–72

and wealth inequality, 76–84

preregistration of research projects, 127–28

pre-release access to data, 208–11

Presence (Cuddy), 121

priming, 121

PRIS (Puerto Rican Institute of Statistics), 197–200

prisoner data, 55

privacy issues, 181–82

private pensions, 80

Proctor, Robert, 13

Prohibition, 242

propaganda, 12

ProPublica, 176–79

proxy measures, 59

psychology research

and choice experiment, 105–7

and conformity studies, 135–38

and political affiliations, 268–69

and “precognition” study, 111–14

and reproducibility crisis, 112–16, 120–22, 128–29, 130–31

public health. See health and medical data

public opinion, 149, 220

public transportation, 47–49

publication bias, 113–16, 118–23, 125–27

publicity, 107

Puerto Rico, 197–98, 200

Puy de Dôme, France, 172

Quetelet, Adolphe, 219

racial data, 176–79, 206

Random Walk down Wall Street, A (Malkiel), 125

randomized clinical trials, 4n, 53, 125–26, 133, 180

randomness, 123–24

Rapid Safety Feedback, 170–71

Rayner, Derek, 205–8

Reaper Man (Pratchett), 87

recessions, 11

recommendation engines, 181

record-keeping practices, 220–21

refugees, 191

Reifler, Jason, 129

Reischauer, Robert, 187

Reiter, Jonathan, 108

reliability of data, 233–37

religious authority, 16

religious beliefs, 247–48

Remington Rand, 244

replication/reproducibility studies and problems, 107, 112–16, 120–22, 129–31

Republican Party, 34, 189n, 269, 270

résumé-sorting algorithms, 166

ridership data, 49–51

Riecken, Henry, 239

risk models, 71

Rivlin, Alice, 186–87, 188, 212

Robinson, Nicholas, 168, 169–70

Roman Catholic Church, 16

Rönnlund, Anna Rosling, 62, 63

Roosevelt, Franklin Delano, 143–44

rose diagrams, 215–16, 233–36, 234

Roser, Max, 89, 96

Rosling, Hans, 63, 185

Ross, Lee, 35

Royal Naval Reserve, 218

Royal Society, 13

Royal Statistical Society, 194, 214, 219, 233

Rozenblit, Leonid, 272

Ruge, Mari, 89

sampling techniques, 135–38, 142–51, 155

Samuelson, Paul, 239

sanitation advocacy, 225–26, 233–37

Santos, Alexander, 198

Say It with Charts (Zelazny), 228

scale of statistical data, 92, 93–95, 103

Scarr, Simon, 231–32

Schachter, Stanley, 239

Scheibehenne, Benjamin, 106, 111, 114, 120–21

Scientific American, 102

scientific curiosity, 268–69

scientific literacy, 34–35

scientific method, 173

Scott, James C., 201, 203

Scott Brown, Denise, 217

screen-use studies, 117–18

Scutari (Üsküdar, Istanbul) barracks hospital, 213–14, 220, 225, 233, 235

search algorithms, 156–57

Second World War, 4, 262

secrecy, 174–75

Seehofer, Horst, 191

Seeing Like a State (Scott), 201, 203

selection bias, 2, 245–46. See also survivorship bias

self-awareness, 40

self-harm data, 73–76, 85

self-reported data, 2

senses and perception, 247

sex scandals, 185–86

sexual activity data, 93

Shane, Janelle, 156

Shared Services Canada, 197

Sharot, Tali, 96

Sheikh, Munir, 197

Shenzhen, China, 60

sick building syndrome, 12

significance measures, 117, 129–30

sildenafil, 140

Simmons, Joseph, 119

Simon, John, 226, 233

Simonsohn, Uri, 119

Singh, Sameer, 156

skepticism, 54, 69, 183, 197, 205

Sloman, Steven, 272

slow statistics, 56, 57, 59–60, 63–64

Small, Hugh, 214, 234

Smith, Edward, 12–13, 33

Smith, Lucy, 65

Smith, Wayne, 197

smoking research, 3–6, 96, 100, 248, 279

social consequences of beliefs, 36–38

social media

and Cambridge Analytica scandal, 158

and early access to official statistics, 208

and evaluation of credibility, 132n

and fake news, 41

and misuse of statistical methods, 118

and visualization of data, 224

social pressure, 38, 45–46

socioeconomic status, 59, 68–69, 150–51

Somerfeld, Erik, 37–38

Soviet Union, 202

Spicer, Sean, 191

Spiegelhalter, David, 162–63

Stalin, Joseph, 73–74, 201–2

Starbucks, 106

state pensions, 80n

statistical methodology

categorization of statistical data, 68–70

and causation vs. correlation, 15, 64, 156–57, 275

and cherry-picking data, 245–46

and cognitive reflection test, 41–42

and credibility of data, 192, 195

credulity in data, 164–67

cynicism and skepticism about statistics, 8–10, 13, 14, 266

definition of terms, 87–88

and fast vs. slow statistics, 56, 57, 60–61, 63–64

habits fostering statistical analysis, 25

Hypothesizing After Results Known (HARKing), 118

and mathematical risk models, 71

measurement methods and choices, 68, 117–18, 165n

and misconceptions of statistics, 55–56

misuse of, 117–19, 120

and “N = All” assumptions, 150, 152, 155

and novel representations of statistics, 95n

and peer review, 111–12, 189n

and premature enumeration, 65–85

and publication bias, 113–16, 118–23, 125–27

and reproducibility crisis, 107, 112–16, 120–22, 129–31

and sampling bias, 135–38, 142–45, 147–51

and scale of statistical data, 92, 93–95, 103

and significance measures, 117, 129–30

and theory-free analysis, 156–57

understanding statistical claims, 67–72

varieties of statistical questions, 11

and “winter detector” problem, 156

See also algorithms; big data; missing data

Statistical Science, 2

Statistics and the German State (Tooze), 202

Statistics Canada, 196–97, 212

stem cell research, 140

Stereotypical Daydream, 109, 114

stock market crash (1929), 243–44, 258, 260–61

“Storks Deliver Babies” (Matthews), 1

Street Bump, 150

Stroke Association (UK), 97–98

stroke data, 10, 97–98

student achievement data, 163–64

Subacchi, Paola, 192

subways, New York City, 229–31, 230

suicides, 73–75

Summer Olympics (2012), 143

Sunstein, Cass, 169

Super PACs, 275

superforecasters, 252–55, 260

surveillance, 202

surveys, 28, 49, 50, 143, 164–65. See also face-to-face surveys; telephone surveys

survivorship bias, 109–10, 112–13, 122–26

Sustainable Development Goals, 142–43

Sutherland, John, 214n

Swarthmore College, 138

Sweden, 209

Taber, Charles, 33, 38

Taft, William Howard, 241

Taleb, Nassim, 101, 123

Tanzania, 196

Target, 159–63, 175–76, 181

tax policy, 188–89, 192, 201

teacher evaluations, 163–64

TED Talks, 106

Teekeningen 1 (van Meegeren), 42–43, 44

teen pregnancy, 55, 55n

telephone surveys, 145–46

television, 54–55

Temne people, 136–37

temperature measures, 165–66, 165n

terrorism, 55, 88, 96

Tesco, 160

test performance data, 163–64

Tetlock, Philip, 249–52, 254–55, 260–61

Tett, Gillian, 100–101

Thaler, Richard, 130

thalidomide, 139

Thatcher, Margaret, 205

theory-free analysis, 156–57

“They Saw a Protest” (Kahan et al.), 267–68, 268n

Thinking, Fast and Slow (Kahneman), 41, 56, 61n, 121

Thomson Reuters, 231–32

Tierney, John, 121

Time, 217

Times (London), 213, 215, 220, 225

tobacco use, 3–6, 13–15, 35, 52, 241, 248, 279

Tooze, Adam, 202

Torricelli, Evangelista, 172, 173

totalitarian leadership, 190, 201–2

transparency of data, 127, 176, 181, 207

Transport for London (TfL), 49–50

trends in data, 16, 227

Trials (journal), 128

tribalism, 268

Trudeau, Justin, 197

Trump, Donald

and border wall cost, 93–94

and Cambridge Analytica scandal, 158

and coronavirus pandemic, 7

debt anecdote, 79n

dismissal of official statistics, 190–91, 196, 207

and early access to official statistics, 207–8, 210

and election polling, 147

and “fake news” claims, 13–14

and impact of fact-checking, 129

and social media use, 42

trust in data, 164–67, 179, 267

tuberculosis, 3–4, 4n, 240–41

Tufte, Edward, 217, 223, 229

Tversky, Amos, 97

Twitter, 149, 150, 208, 224, 277

2001: A Space Odyssey (film), 153

Tyranny of Metrics, The (Muller), 57

Uganda, 141

understandability of data, 180

understanding statistical claims, 67–72

unemployment rates, 11, 189n, 207, 218n

UNESCO, 203–4

unintended consequences, 58

United Kingdom

death rate data, 236

demographic data, 94

early access to official statistics, 208

income inequality measures, 82–83, 92–93

reform of statistical system, 207

stroke incidence in, 98

Universal Credit program, 142

value of official statistics, 198

United Nations, 142–43, 185

UNESCO, 203–4

United States

and demographic data, 94

and early access to official statistics, 207–11

and election predictions, 147

and income inequality measures, 92

and infant mortality measures, 66

See also entries beginning with US

units of analysis, 141

Universal Credit, 142

unskilled workers, 71

US Air Force, 110

US Congress, 186–87, 204. See also Congressional Budget Office (CBO)

US Department of Agriculture, 190

US Food and Drug Administration, 126, 180–81

U.S. News & World Report, 58

US Senate, 14–15

usability of data, 180

vaccinations, 53–54, 99

validity testing, 116–17

van Meegeren, Han, 21–23, 30–32, 42–46, 242, 262

vanity, 2

varieties of statistical questions, 11

Venturi, Robert, 217

Vermeer, Johannes, 19–22, 23, 29–32, 43, 45

Versailles Peace Conference, 257

Viagra, 140

Victoria, Queen, 215, 226

video games, 69–70

Vietnam War, 58, 115

viral spread of information, 26, 105, 111, 224–25

visualization of data, 213–37

historical uses of, 216–19

and issue advocacy, 227–33

and Nightingale’s influence, 213–16, 219–21, 225–27, 233–37

poor and misleading uses of, 221–25

and sanitation advocacy, 225–26, 233–37

and war casualty statistics, 214–16, 219–21

vitamin supplements, 132–33

wait-time data, 58

Wald, Abraham, 110

Wales, 67

Wall Street crash (1929), 243–44, 258, 260–61

War Department (UK), 215

Washington, DC, 163–64, 186–87n

Watchmen (Moore), 105

Waterman, Robert, 123

wealth and income inequality, 62–64, 76–84, 85, 229–31

Weapons of Math Destruction (O’Neil), 157, 163, 180

webcam data, 151

WebMD, 133

weight distribution data, 228

“WEIRD” subjects (Western, Educated, and from Industrialized Rich Democracies), 136

Welles, Orson, 265, 274

Wheelan, Charles, 2

When Prophecy Fails (Festinger, Riecken, and Schachter), 239

“Why Most Published Research Findings Are False” (Ioannidis), 120, 130

Wi-Fi networks, 49

Wilkinson, Normal, 218

Willpower (Baumeister and Tierney), 121

windfalls, 27

“winter detector” problem, 156

Wired, 155, 156

wishful thinking, 27, 29, 36, 40, 45

Woman Reading a Letter (Vermeer), 21

Woman Taken in Adultery, The (Vermeer), 22

Woolf, Virginia, 257

Wootton, David, 173, 175

World Bank, 91, 99–100, 196, 203, 262

World Health Organization (WHO), 7

World Inequality Database, 83

worm’s-eye view of data, 61–62, 63–64, 265

Wunderlich, Carl, 165–66, 165n

Yale University, 2, 138–39, 267

Yangshuo, China, 60

yoga, 133–34

Yong, Ed, 178

You Look Like a Thing and I Love You (Shane), 156

YouTube, 181

Yunus, Muhammad, 47, 61

Zelazny, Gene, 228

ABCDEFGHIJKLMNOPQRSTUVWXYZ

ABOUT THE AUTHOR

Tim Harford is an award-winning columnist, broadcaster, and economist. He is the author of Messy, Fifty Inventions That Shaped the Modern Economy, and the million-selling The Undercover Economist, and is the host of the Cautionary Tales podcast. He is an honorary fellow of the Royal Statistical Society, and in 2019 he was awarded an OBE for services to improving economic understanding.

What’s next on

your reading list?

Discover your next

great read!

Get personalized book picks and up-to-date news about this author.

Sign up now.

*She also became president of the American Economic Association, deputy at the Federal Reserve, and head of the Office of Management and Budget. And she “helped save Washington,” as a later head of the CBO put it, by helping guide the city out of its own local fiscal crisis in the 1990s. Given this range of high-level positions, one colleague called her a “decathlete” of public service.

*For example, a peer-reviewed study by two academics published in 2000 found that Republican administrations tended to produce forecasts that were too worried about high inflation, while Democratic administration forecasts were too pessimistic about unemployment. CBO forecasts showed neither bias and were more accurate overall.

*I’ll spare you my efforts to define these physical constants. For our purposes what matters is that they are hard to measure precisely, and that in each attempt to improve the accuracy, those measurements seem to have been systematically swayed by previous attempts.

*It’s naughty of me to call this a “straightforward” statistic. In the UK, according to the Office for National Statistics (Statistical Release, November 29, 2019), 22 percent of 1965 marriages had ended in divorce by 1985. That figure has risen over time: 38 percent of 1995 marriages had ended in divorce by 2015. There is now evidence that the divorce rate is falling again—but it is obviously too early to say how many recent marriages will last twenty years. Evidently it’s a matter of judgment—and the available data—as to which base rate you think is relevant. All UK marriages? All recent marriages? All marriages between people of a certain age or education level? It’s not straightforward at all, if I am honest. But it is better to try to find a relevant base rate and reason from there than to pull numbers out of your brain without any context.

*Then there is the question of what a COVID-19 death is: some who succumb were already terminally ill; some, indeed, died with the virus but not of it. With that in mind, 1,500 deaths is perhaps an overstatement after all.

*And by “children,” do we mean five-year-olds? Ten-year-olds? Sixteen-year-olds?

*Even the definition of “mass shooting” is slippery. The FBI keeps a record of incidents of mass murder, but their definition includes only attacks in a public place, which leaves out many drug-related incidents, as well as domestic homicides. An alternative count, maintained by the Gun Violence Archive, includes such incidents. That makes a big difference to the total count—but either way, the number of people killed in mass shooting incidents is a small fraction of the total number of gun deaths.

*And there’s an oft-repeated anecdote about Donald Trump, years before he became president. He was heavily indebted after some failing real estate deals and, pointing to a homeless person, told his young daughter, “See that bum? He has a billion dollars more than me.” I’ve no idea if the story is true, but the financial logic is sound.

*I’ve used the 2018 Global Wealth Report. The 2013 version—the foundation of the original “85 Richest People” headlines—offers slightly different numbers, but the big picture has changed only slowly.

*Credit Suisse did not include the entitlement to a state pension in its calculations. That matters, because state pensions are very valuable to those who have them. It’s unclear whether counting state pensions as assets would increase measured inequality (since many of the poorest people lack them) or reduce measured inequality (since a state pension represents a substantial asset for the poorer people in richer countries). I’m guessing things would look less unequal if state pensions were included, but it is just a guess. I might be quite wrong. Around the world, a third of older people have no pension of any kind.

*Another popular measure of inequality—one we’ll encounter in the next chapter—is the Gini coefficient. This measure was telling the same story of falling inequality in the wake of the crisis.

*In both cases it’s conceivable that people were swayed less by the modest financial incentive and more by the emotional power of the role they were being asked to adopt. Either way, taking a particular perspective on the situation proved to be a strong influence on the decisions they made.

*Political expertise in this experiment was measured by asking people questions about the workings of the US government. For example, how many congressional votes are needed to override a presidential veto?

*I’m quite sure that I’m guilty, too. I just can’t see exactly how.

*The answers: five cents, and forty-seven days. Perhaps the second question is less of a stumbling block than it once was. The lily patch is growing exponentially, and we have all received a hard lesson from the coronavirus in what exponential growth looks like.

*A follow-up question might also be worth asking: Why does it make me feel that way?

*Although if it is a recipe, it is a recipe written by a particularly pedantic chef. Most recipes leave room for common sense, but if an algorithm is to be interpreted by a computer, the steps must be tightly specified.

*The problem was exacerbated by a conversion of units. Wunderlich’s original measurements were made in centigrade, and his results concluded that the typical body temperature was a range around 37°C—implicitly, given that degree of precision, a range of up to a degree centigrade, somewhere above 36.5°C and below 37.5°C. But when Wunderlich’s articles in German were translated into English, reaching a larger audience, the temperature was converted from centigrade to Fahrenheit and became 98.6°F—inviting physicians to assume that the temperature had been measured to one tenth of a degree Fahrenheit rather than one degree centigrade. The implied precision was almost twenty times greater—but all that had actually changed was a conversion between two temperature scales.

*A particle accelerator will turn base metals into gold, although not cheaply. In 1980, researchers bombarded the faintly lead-like metal bismuth and created a few atoms of gold. The cost was a less-than-economical rate of one quadrillion dollars an ounce. We have yet to discover an elixir of eternal life.

*How could multiple experts on Nightingale get this detail wrong? It’s unclear who first made the error, but once made, it spread. I’ve repeated the mistake myself in an article for the Financial Times. My first inkling that there was a problem was when I consulted a biography of the far-less-famous William Farr, which says the letter was written to Farr, not by him. I contacted the wonderful archivists at the British Library and discovered that the letter in question is an unsigned draft; the final version is lost. The draft is in the handwriting of Dr. John Sutherland, a close collaborator of Nightingale who would often draft for her and may have been taking dictation. It was definitely intended for Farr. Even if it was not dictated by Nightingale, it likely reflected her views closely. Professor Lynn McDonald, the editor of a multivolume collection of Nightingale’s work, explained to me, “She, perhaps, wrote her own version and sent it, but evidently such letter has disappeared. [Sutherland and Nightingale] saw eye to eye—they are her views, AND his” (McDonald, email correspondence, May 31, 2019).

*Admittedly, sometimes the underlying data, when plotted in a straightforward fashion, do suggest a picture emerging from the dots on the graph, like a Rorschach test. For example, if you plot a chart of unemployment versus inflation in Japan—what economists call the “Phillips curve”—you may notice something. As a 2006 economics paper observed, “Japan’s Phillips curve looks like Japan.”

*The Big Issue put Florence Nightingale on its cover in March 2020. “Hail the Hand-Washing Queen: How Florence Nightingale Is Helping Us Fight Coronavirus,” the magazine trumpeted. But it takes more than handwashing to power a revolution in public health: it takes statistical detective work. The virus reminded us to wash our hands, but more important, it taught us that fighting a pandemic requires information, as quick and as complete as possible. Florence Nightingale understood that nearly two hundred years ago. I’d rather remember her not as a handwashing queen, but as a data detective.

*Joking. Don’t use a pie chart.

*In an act of sweet revenge, Hill later showed how to cure tuberculosis, in what is generally accepted as the first rigorously randomized clinical trial.

*This aphorism is popular among statisticians. I often see it attributed to the great statistician Frederick Mosteller, but haven’t been able to confirm the provenance.

*The study is titled “‘They Saw a Protest,’” echoing a classic psychology paper from 1954, “They Saw a Game,” which found similarly biased perceptions when rival fans watched a film of a bad-tempered game of football.

*Trolls, populists, manufacturers of outrage, and other professional controversialists will, of course, try to frame debates in ways that crush curiosity and reinforce preconceptions. But curious and open-minded folk would be wise to take the opposite tack in the way we express ourselves.

*I was once a guest on The Colbert Report. Stephen was a gracious host. In the green room, as himself, he explained the basic idea of the show to me: “I’ll be in character, and my character is an idiot.” Then later, after getting into character, “I’m going to tear you apart, Harford!”

*And if people are not curious, they will not learn until their curiosity can somehow be sparked into life. Remember the TV producers who were making a program about why inequality had risen, but apparently weren’t curious enough to check whether it had?

*When pro-Brexit campaigner Michael Gove said this, he was referring specifically to experts based at international organizations such as the International Monetary Fund. The statement, however, took on a life of its own.

*This is a reminder of how useful it is to stop and think. There is no advanced mathematics required to realize that the 20 percent figure simply cannot be squared with our everyday experience. In some countries, people say they believe that 50 percent of teenage girls give birth each year, which would imply young women typically enter adulthood with three children of their own.

*Admirers of Daniel Kahneman and his book Thinking, Fast and Slow may recognize here what he calls “system 1” and “system 2.”

*In 2019, for example, London saw 149 murders—the highest number for a decade. There has been a rise since 2016. UK media tend to present this rise as apocalyptic; with context, it looks less worrying but is clearly a move in the wrong direction. A temporary blip, or a reversal of the long decline in murder rates? “Only time will tell” is a cliché; it’s true, though.

*If Mexico paid for the wall, the cost per person would be almost $200, since the population of Mexico is smaller. If.

*Less illuminating is the habit of writing something along the lines of “If the US national debt was a pile of dollar bills, it would stretch all the way to space / to the moon / to the sun.” Some journalists seem to think this is a great way to put a big number into context. Is it? Generally I find myself stupider at the end of such sentences. Do you know how many dollar bills there are in a pile a yard high? (About eight thousand. I had to look it up, of course. Anyone would.) Space is generally regarded as being more than 60 miles above us, the moon is nearly 238,900 miles away, and the sun 93.6 million miles away—so a pile that stretches to the sun is a lot bigger than one that stretches to space. By my calculations, the US national debt would be a pile of dollar bills reaching to the moon six times. Happy now? I find it much clearer to note that it is about $70,000 per US citizen.

*It was also understandable. Lewis was the author of books such as Cats in the News and Dogs in the News.

*In our basketball analogy, that’s like the organizer noticing that she can get a ten-point gap if she counts the field goals and ignores the free throws. Outrageous in that setting, but in a scientific context it makes perfect sense to explore different approaches. Perfect sense—but a statistical trap if not done carefully.

*Ioannidis, you may recall, is also the epidemiologist who warned of a “once-in-a-century evidence fiasco” in March 2020, as countries around the world found themselves having to respond to the coronavirus pandemic armed with very patchy data.

*Or try this. After reading an article or a Facebook post describing some cool finding, just ask yourself how you’d explain it to a friend. Do you know what the researchers did, and why, whether the research was a shock or exactly what experts would have expected? If your explanation is along the lines of “Some boffins discovered that blueberries give you cancer,” maybe you didn’t read a good piece of journalism.