Amazing peer-reviewed AI bots that predict premature births were too good to be true: Flawed testing bumped accuracy from 50% to 90%+

[helvella]

We hear a lot about Artificial Intelligence, including within medicine. But we need to be very cautious about accepting any claims.

If someone puts forward any form of AI for thyroid issues, and I am sure someone will (if they haven't already), we absolutely must ensure that it has been very thoroughly tested before a single treatment choice is made (or changed).

We know how complex and subtle thyroid issues can be. In time, we might see AI outperforming doctors but even if they do so, it is vital to question the system and flag up any issues found or questions as to validity. Not just now, but forever.

Amazing peer-reviewed AI bots that predict premature births were too good to be true: Flawed testing bumped accuracy from 50% to 90%+

'These models should not go into clinical practice at all,' academic tells El Reg

By Katyanna Quach 24 Jan 2020 at 06:03

A surprising number of peer-reviewed premature-birth-predicting machine-learning systems are nowhere near as accurate as first thought, according to a new study.

Gilles Vandewiele, a PhD student at Ghent University in Belgium, and his colleagues discovered the shortcomings while investigating how well artificial intelligence can predict premature births using non-invasive electrohysterography (EHG) readings. By premature, we mean before 37 weeks into a pregnancy, and by EHG, we mean the electrical activity in uterine muscles.

They identified an EHG data set on Physionet that was used to train premature-birth-predicting software in 24 published studies. After analyzing each one, they determined 11 of the papers mixed their training and testing data, which led to wildly incorrect accuracy scores.

Rest of story here (may require free registration - afraid I do not know):

theregister.co.uk/2020/01/2...

/2020/01/24/ai_models_data/

[Sybilla14]

Sadly, I think that you may not be too wrong about an AI bot for thyroid dysfunction. At least Hashimoto lends itself to AI diagnosis and maintenance perfectly as it already follows a very mechanistic approach. All you need is a list of a handful of stereotypical symptoms and a TSH range as not a lot of thinking is involved. The patients know that this is nonsense and that the disease is complex to treat but not the medical profession, including endocrinology. I think the Hashimoto bot would be very accurate at replicating doctors!

[Treepie]

GIGO

[Angel_of_the_North]

Well, since AI is just an algorithm, I can't see that it would be any different from insisting that TSH must be over 10 before treatment. You could also add OR TSH must be over range AND FT4 under range, and also OR (TSH is normal OR low) AND FT4 is low = central hypo. If anything we'd be worse off as there'd be no leeway for a human to override it. Since the general consensus seems to be that it's silly fat women trying to blame their symptoms on thyroid instead of lifestyle, I can't see that the AI would "learn" anything extra except "tell silly fat woman to go on diet and take more exercise and take up a hobby to put meaning into her life". Sorry, but I'm unimpressed by technology - it ain't magic

[helvella Administrator Thyroid UK in reply to Angel_of_the_North]

The medical establishment seems to consider excessive testing for thyroid (e.g. TSH, FT4 and FT3) to be a "bad thing". (Never understood any reason for this other than the cost of the tests and, possibly, inappropriate interpretation.)

My take is the opposite. In order for any AI system to stand a chance of being properly trained, extremely frequent testing is required. For example, some research has done blood draws at 15 minute intervals to investigate the absorption of levothyroxine. Some research has looked at hourly sampling to understand circadian patterns. Other research has done daily tests to see how things change over time.

I'll argue that, at the very least, the training of AI systems requires this extremely fine grain information. Otherwise we will see current practicie replicated blindly by AI.