Recent studies suggest that the majority of recent studies are false, according a December article by Jonah Lehrer published in The New Yorker. In “The Truth Wears Off,” he points to a frighteningly disparate set of accepted scientific findings that are now becoming uncomfortably difficult to replicate, including research on the benefits of widely-prescribed anti-depressants, or on what birds like to find in their mates.
Though much of his piece is spent holding up some sensationalist examples which are calculated to unground our faith in the scientific method — and we know that enough odd coincidences exist in the world that a journalist might find a few freak studies — Lehrer has also interviewed respected scientists.
One of them is John Ioannidas, whose “Why Most Published Research Findings are False,” cited hundreds of times since 2005, includes lines like “It can be proven that most claimed research findings are false,” and a statistical argument about how, if 10 teams independently researched 100,000 genetic factors to determine which ones cause a predisposition towards schizophrenia, then any one factor identified by any one team would be about as likely to be a true link as if the team had picked it out of a hat.
To make his claims, Ioannidas points to numerous flaws in the way research is currently conducted. We all know that bias can do mighty things in the way of seeing what we want, even in the most fair-minded and honest of us, and that journals — unless there is a new theory to be debunked — generally prefer findings over “we looked, but we didn’t see anything. Yet there are other thorn-covered obstacles which we see less often while they turn us back from truth.
Towering and yet forgotten among these problems is that scientists are looking for too many things at once, so some of them are statistically certain to turn up with a false positive. Even worse, if they don’t find anything they were looking for, they might search after anything they can publish, any statistically significant correlation related to something in their field.
Great discoveries have indeed come from this. But by the very way statistics are measured, if I take two random groups of 1,000 people, giving one of them a new drug and the other a placebo, there is a 5 percent chance that I will find a statistically significant difference between the two groups in the number of blondes.
Another difficulty lies in the tiny effects that are often searched for with today’s acute tools for measuring. Lehrer’s example is the study of risk factors for disease. Again, due to the way statistics are gathered, if most factors based on diet or genes change your odds by less than 5 percent, “then genetic or nutritional epidemiology would be largely utopian endeavors.”
All of our medicine and physics students already know these dangers, of course, and they will now march gloriously into their fields, brains overflowing with healthy skepticism, ready to rigorously apply the remedies that Ioannidas prescribes, so there is no use telling them.
This is, rather, something that needs to be known more generally, so that when some insipid magazine cites outrageous studies, like eggs will kill you, or that men and women think entirely alike, you’ll know it’s probably nonsense, and either examine the literature yourself, or ignore it.
Not to say scientific research is totally pointless — they’ll get the truth in the end — but there’s no point being upset about last month’s tenure-seeking tome by somebody in Women’s and Gender Studies. Instead, realize what many others have said before: Science is human.
— Gerard Keiser, linguistics and classical languages junior
The Oklahoma Daily is pleased to provide you the opportunity to share your thoughts about this article. We encourage lively debate on the issues of the day, but we ask you refrain from using profanity or other offensive speech, engaging in personal attacks or name-calling, posting advertising, or straying from the topic at hand. To comment, you must be a registered user of OUDaily.com. Thanks for taking the time to offer your thoughts.
You must be logged in to leave a comment. Log in | Register
brandeis 1 year, 3 months ago
So what you're saying here isn't that the method is flawed, but that some research methods used within the actual scientific method structure are flawed due to things like bias and lying.
My god this is ground breaking.
blandon 1 year, 3 months ago
You are only citing examples from medical science and are generalizing this to all science including more rigorous and exact science like physics. You also ignore that some of these 'medical' studies are driven by capitalistic incentives such as research funding from private companies who use these 'studies' to sell their products.
robertwm 1 year, 3 months ago
As Pascal, the founder of statistics said, we are "thinking reeds", limited in knowledge, yet with eternity in the heart inside us. But sciences have grain, and knowledge and each science's epistemology differs; from chemistry to astronomy to human nutrician. Drug studies require 0.05 significance because we want to protect people, but researchers know we only know we can be wrong 5% time.
We need to protect students with statistical literacy. Social sciencies are fun but many feel they may not be science. "Pluto's republic" peter medarwar's book discusses it.
robertwm 1 year, 3 months ago
Leher's examples are mostly pschiatry or behavioral science. I can't speak for the ecological ones. mostly what you are talkilng is statistical illiteracy. Its believable that most behavioral scienctist are statistically illiterate. Reproducing chemical synthesis and other types sciene is not as problematic and largely not done by those who are illiterate. Yes shotgun mass-testing of genomes for association is a problem. medicine is important becasue human life is important, not because we are sure of things. Read comroe: "Retrospectroscope: Insights into Medical Discovery" about critical thinking and the usefulness of medical research.