Why You Shouldn’t Trust Every Scientific Study You See

[cesanon]

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

motherjones.com/kevin-drum/...

The authors collected every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012. Then they displayed them on a scatterplot.

Prior to 2000, researchers could do just about anything they wanted. All they had to do was run the study, collect the data, and then look to see if they could pull something positive out of it. And they did! Out of 22 studies, 13 showed significant benefits. That’s 59 percent of all studies. Pretty good!

Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report.

And guess what? Out of 21 studies, only two showed significant benefits. That’s 10 percent of all studies. Ugh. And one of the studies even demonstrated harm, something that had never happened before 2000

[Tall_Allen]

In the research biz, we called this "p hacking" -- combing through the data to find statistically significant differences. I avoided it on my job by requiring a paper beforehand spelling out the model being tested, and what difference will be considered meaningful and not just statistically significant.

There's also another effect that is more recent - NIH now requires that ALL listed clinical trials post results. Previously, authors of clinical trials that had null or negative findings rarely got published. This was because of a combination of two factors: (1) researchers and their sponsors did not want to publicize failures, and (2) journals did not want to feature studies that were not practice changing.

[tango65]

This is an interesting study indicating that this problem is very real.

medscape.com/viewarticle/90...

[cesanon]

"Then Trust in GOD!" Nalakrats

Yes. Of course. Foolish me.

[Shooter1]

With what I have, most of the studies say I should be dead....won't listen to them and planning life for at least another 10 years.

[herb1]

Sorry, but I think we're trying to compare apples and avocados. Clinical studies as done/required today are fine. A goal is set, criteria that need to be met are set and researchers get money to study them. But even there I hope the researchers look beyond the target to see if anything unusual (and good) is happening. THEN such observations need to be subjected to clinical trials. And, in the other case where researchers just collect a bunch of data and see if anything unusual and good is present, is also good, but only as a beginning. Years ago, on one of these now defunct sites, others and I even offered personal results as a first stage indicators to be studied and, maybe, pursued.