Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
motherjones.com/kevin-drum/...
The authors collected every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012. Then they displayed them on a scatterplot.
Prior to 2000, researchers could do just about anything they wanted. All they had to do was run the study, collect the data, and then look to see if they could pull something positive out of it. And they did! Out of 22 studies, 13 showed significant benefits. That’s 59 percent of all studies. Pretty good!
Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report.
And guess what? Out of 21 studies, only two showed significant benefits. That’s 10 percent of all studies. Ugh. And one of the studies even demonstrated harm, something that had never happened before 2000
In the research biz, we called this "p hacking" -- combing through the data to find statistically significant differences. I avoided it on my job by requiring a paper beforehand spelling out the model being tested, and what difference will be considered meaningful and not just statistically significant.
There's also another effect that is more recent - NIH now requires that ALL listed clinical trials post results. Previously, authors of clinical trials that had null or negative findings rarely got published. This was because of a combination of two factors: (1) researchers and their sponsors did not want to publicize failures, and (2) journals did not want to feature studies that were not practice changing.