Lancet study on carbs: a critique - Healthy Eating

Healthy Eating

61,446 members8,179 posts

Lancet study on carbs: a critique

TheAwfulToad profile image
7 Replies

This is quite long, so rather than clutter up Zest's thread (or the other one discussing the newspaper editorial) I thought I'd better create a separate one, simply because lot of people might find my comments too boring to respond to :)

First off (and I'm not pretending this is an academic observation) Walter Willett is a extremely influential figure in the US Establishment and has been for decades; he was one of the original authors of the low-saturated-fat orthodoxy, and although he's since moderated his views considerably in light of emerging evidence, he makes no secret of his disapproval of low-carb diets. There is no deliberate fraud here, as far as I can tell, but I have no doubt whatsoever that without his name on the paper, the journal editors would have thrown this paper straight in the bin, or possibly passed it around the office for entertainment value.

Here are the scientific criticisms. The first six are real doozies that would get this a D- if it were an undergraduate handing it in as a term paper:

1) There is no hypothesis (that is, no scientifically-testable statement) apart from a rather vague query concerning plant- and animal-sourced fats. At best, it's an evidence-gathering exercise, and the authors go too far in drawing definite conclusions. This is a cardinal sin in science, known as 'drawing the target around the bullethole' (see point 7).

2) The selection of the study cohort is not random. ARIC publish the cohort characteristics here: www2.cscc.unc.edu/aric/syst.... The subjects appear to have been selected (back in 1987) because they had dangerously unhealthy lifestyles; mostly smokers, for example, with pre-existing medical conditions. That is, they're not representative of the general public.

This should have been stated clearly in the paper.

3) Combining the data from this unique cohort with others that have completely different characteristics (as they've done in the accompanying meta-analysis) is just downright confusing. It's impossible to tell if it adds anything to the picture or if it introduces a statistical error.

4) Despite stating that the primary outcome was all-cause mortality (which is a very sensible choice of endpoint) no figures are given for mortality apart from the total (6283 over 25 years). No data is presented showing association of death with carbohydrate intake or any other factor. Since animal fat is supposedly associated with heart disease, it would have also been pertinent to describe the cause of death in each recorded case.

This is such a 'WTF?' problem I was just scratching my head in disbelief and confusion. Not having any data (or keeping it hidden) means your study is worthless; no third party can assess the validity of your conclusions.

5) The description of the statistical analysis is maddeningly vague ... especially since we can't even see the deaths information. However, what they appear to have done is this: they discarded all their data and built a mathematical model for how long they expected the still-alive participants to survive. The models are based upon ... well, it's hard to tell. They hint that it's a combination of observed ARIC deaths and previous study results, but they're very coy about the exact method. They then plotted the results of this model on a graph, and used a curve-smoothing technique to, um, show some smooth curves.

Someone else might have a different take on this. I honestly can't figure out what they've done.

I won't go into the theoretical issues of curve-fitting, but suffice to say that, when you do this, you must have some idea of the underlying process. If you don't, you can't correctly decide upon the mathematical form of the fitted curve (linear, quadratic, exponential, etc). A cubic-spline interpolation doesn't represent a process at all; it's just a generic method of improving cosmetic appearance. It's doubly bizarre in this case because they're not even examining a process, but a set of models ... which, from the shape of the curve, appear to be second-order quadratic.

6) The graph (Figure 1) is just laughable. As noted, it is simply a plot of their mathematical model. There is no actual experimental data on it (which would appear as scattered dots). The lower end of the curve is extrapolated all the way down to 20% carbs, even though their experimental data only goes down to 30-something.

7) The Interpretation is almost a word-for-word copy of the Harvard dietary recommendations, which again might have something to do with Willett's involvement. "Look, we found exactly what we expected to find!".

8) As another poster already commented, they've broken everything down into carbohydrates, fats, and proteins (as Nutritionists always do without thinking), but without retaining potentially-pertinent information about the sources or form of those nutrients. The only exception they've made is in differentiating plant and animal fats; in other words, they've decided that whatever risk differences might exist must be due to this factor and no other.

9) None of the subjects reported eating a low-carb high-fat diet (presumably it didn't even appear on the questionnaire) and none of them were eating even close to the accepted LCHF limit (<10% carbs, 70%+ fat). The lowest quintile represented a nominal 37% carbs, which for an average adult is 800kCal (200g). That's somewhere between two and four times the amount eaten on a low-carb diet. The study therefore tells us nothing concrete about LCHF.

I would love to see one of the study authors follow up on this here; in particular, I'd like to know what happened to the death stats. I suppose the chances of that happening are pretty slim.

Written by
TheAwfulToad profile image
TheAwfulToad
To view profiles and participate in discussions please or .
Read more about...
7 Replies
TheAwfulToad profile image
TheAwfulToad

I was completely floored by the fact that the study doesn't actually show any mortality data. I was scrolling up and down thinking I'd missed it, tucked away in a little table somewhere. Or maybe a link to the raw data online. But no. The most important bit of information, the stated purpose of the paper - look, the word 'mortality' is in the title! - is nowhere to be seen. I've never seen anything like it, not even in university tutorials where exceptionally bad papers were wheeled out for us to practice on.

As you said, if you can see the data, presented clearly, then you can determine if something is believable or not, however unexpected or strange the result might be.

I really can't understand why nobody has done a proper study on low-carb. There are hundreds of millions of funding $ sloshing around, and it seems that not one researcher has hit upon the idea of (say) tracking down a few hundred of Robert Atkins's original clients from the late 70s and comparing them against an age/gender/race-matched group of high-carb dieters. There are all sorts of possible variations on this theme. We'd only need three or four good-quality studies to settle the safety question once and for all.

I can't help wondering if the Establishment don't want the question answered definitively, because they suspect the answer will not make them look good.

Anyway, I've just fired off an email to the contact address to ask if the raw data is available for public inspection. I'm not really expecting a reply, but you never know.

Fran182716 profile image
Fran182716Prediabetic

Thankyou TheAwfulToad i had a quick read through the article before work yesterday and was hoping someone would point out its flaws in a much more coherent way than I could. Unfortunately it's headline grabbing which is dangerous as the general public will not be able to critically analyse.

TheAwfulToad profile image
TheAwfulToad in reply to Fran182716

Yeah, it really is sad that the general public will just accept this uncritically, even though the headline is completely inaccurate (can I say 'untruthful'?).

Zest profile image
Zest

Hi TheAwfulToad

I shall look forward to reading your post later.

Zest :-)

Cooper27 profile image
Cooper27Administrator

Very well broken down :)

I saw the headline, but didn't really bother to read the articles. I don't think I'll bother now either!

Most people I've found on LCHF are doing it because they have health problems (mostly thyroid). The diet allows them to lose weight when they struggled before - and given excess weight leads to early death as well, this study is just suggesting they have a catch 22 on their hands (early death or early death!) If that is the case, then pick whichever one gives you the better quality of life (probably the one that includes weight loss).

TheAwfulToad profile image
TheAwfulToad in reply to Cooper27

I'd still recommend you read it, if for no other reason than to hone your BS meter and to judge for yourself whether my remarks are justified.

Sometimes bad science can teach us as much about the state of the world as good science.

>> this study is just suggesting they have a catch 22 on their hands (early death or early death!)

This one crops up again and again, and it always makes me laugh. What people eat now is absolutely catastrophic. 25% of the NHS budget is now spent on obesity, heart disease, diabetes, and related chronic conditions - which rather suggests that most of the population is suffering from diet-related problems. And yet the mumbling about low-carb just won't go away, as if the status quo were just fine. It clearly isn't.

G1nny profile image
G1nny

Thank you for posting. I like my low carb diet and am sticking with it!

You may also like...

Low (ish) carbing is for me.

I went low carb back in Feb this year. I'm now 3 st lighter and now on maintenance. I'm finding it

The Truth About... Carbs BBC1

properly metabolizes complex carbs vs simple carbs? Meaning that good complex carbs, by and large,...

NHS the truth about carbs.

https://www.hsph.harvard.edu/nutritionsource/carbohydrates/carbohydrates-and-blood-sugar/ The truth...

Diabetes and High Carb diet

APPROVES, which talks about the benefits of a high carb, high fibre, plant-based diet on diabetes...

eating too many carbs

calories and notices how many carbs I eat. I don't necessarily eat bad carbs, since I mostly have...