[The name comes from the following old joke scientists love to tell: Late at night, a police officer finds a drunk man crawling around on his hands and knees under a streetlight. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street. Then why are you looking over here? the befuddled officer asks. Because the light’s better here, explains the drunk man.”] This is an excerpt from an article in the July/August 2010 issue of Discover magazine, written by David H. Freedman.

Researchers tend to look for answers where the looking is good, rather than where the answers are likely to be hiding. The result: a lot of dubious science. A bolt of excitement ran through the field of cardiology in the early 1980s when anti-arrhythmia drugs burst onto the scene. Researches knew that heart-attack victims with steady heartbeats had the best odds of survival, so a medication that could tamp down irregularities seemed like a no-brainer. The drugs became the stand of care for heart-attack patients and were soon smoothing out heartbeats in intensive care wards across the United States. But in the early 1990s, cardiologists realized that the drugs were also doing something else: killing about 56,000 heart-attack patients a year. Yes, hearts were beating more regularly on the drugs than off, but their owners were on average, one-third as likely to pull through. Cardiologists had been so focused on immediately measurable arrhythmias that they had overlooked the longer-term but far more important variable of death. Physicists have a good excuse for huddling under the streetlight when they are pushing at the limits of human understanding. But the effect also vexes medical research, where you might think great patient data is there for the tabulating. The story of the anti-arrhythmia drugs only hints at the extent of the problem. In 2005, John Ioannidis of the University of Ioannina in Greece examined the 45 most prominent studies published since 1990 in the top medical journals and found that about one-third of them were ultimately refuted. I have spent the past three years examining why expert pronouncements so often turn out to be exaggerated, misleading, or flat-out wrong. There are several very good reasons why that happens, and one of them is that scientists are not as good at making trustworthy measurements as we give them credit for. […] A bigger obstacle to reliable research, though, is that scientists often simply cannot get at the things they need to measure. By far the most familiar and vexing consequences of the streetlight effect show up in those ever-shifting medical findings. Take this straight-forward and critical question: Can vitamin D supplements lower the risk of breast, colon, and other cancers? Yes, by as much as 75%, several well-publicized studies have concluded over the past decade. No, not at all, several other equally qualified well-publicized have concluded. The streetlight can also derail study results if scientists do not look at the right subjects. Patient recruitment is an enormous problem in many medical studies, and researchers often end up paying for the participation of students, poor people, drug abusers, the homeless, illegal immigrants, and others who many not adequately represent the population in terms of health or lifestyle. Yet another problem is that much of what scientists think they know about human health comes from animals studies. Unfortunately, three-quarters of the drugs that prove safe and effective in animals end up failing in early human trials, sometimes spectacularly. In 2006 the experiment leukemia drug TGN 1412 was given six volunteer human patients. All six of them quickly fell seriously ill with multiple organ trauma, even though the stuff had worked well on rabbits and monkeys at doses up to 500 times as large. Mice in particular let researchers era all sorts of exceptionally clean measurements without complaint. Yet it is a well-documented fact hat mouse research often translates poorly to human results. Yes, using mice in early studies can spare human test subjects from harm, which most people would argue justifies the frequently misleading findings. But mice are also used all the time to obtain easy measurements in harmless lifestyle and behavioral studies. The proposition remains dubious even if the mice are genetically engineered to be more “like” humans in some way. How seriously do you what to take the advice of a much-hyped 2008 Boston University study declaring that weight lifting can burn more fat than cardio exercise, when the conclusions were based entirely on sedentary mice genetically engineered to have bizarrely large muscles?