Algorithm used to predict sepsis in hundreds of US hospitals isn’t as good as maker claims — study

An algorithm used by hundreds of US hospitals to predict whether or not patients with infections have contracted sepsis is less accurate than its maker claims, according to a published study. Sepsis is a leading cause of death in American hospitals. So Epic Systems, a major healthcare software provider whose products are used …

[Anonymous Coward]

Billing != Action

Plus: They prescribe antibiotics just-in-case it's sepsis or other infection..... false positive on the billing code Billed items require consent, and that is a non clinical decision of a person or insurer...false negatives on the billing code

Seems they're chasing the wrong metric entirely. There won't be many objective useful features to match that against, billing code is *not* one of them.

The metric they should be tracing should be "what decision in hindsight would the doctor make if he had the foresight information?" i.e. what tool could I make that would give the doctor the information NOW, that he will have LATER, so he can make the correct decision now.

Diagnostic-tools not MBA-data-wank tools.

So you could data mine what you have and try to match it to billing codes, i.e. chase the model as it is today and try to replace the clinician in that scenario, to make a slightly less successful model minus the clinician. That's what they're doing here. It's a sales pitch to the accounting-monkeys. "Our algo is better than doctors, so who needs to doctor....spend that money on our crap instead"

Or, you could go look for blood tests, or tissue tests or antibody tests, or something, that indicate the person will likely be susceptible to sepsis.

Or you could even try to find the infection vectors and eliminate those, nip the problem in the bud.

It's a Hancock vs Hippocrates situation.

The first one of these is cost saving dressed up as medicine, the latter two are medicinal.

[Pascal Monett]

Re: Billing != Action

I totally agree with you, but to the MBA, billing codes do not require human intervention. The software can scan them in an instant and make what it considers the appropriate decision.

All the rest of the stuff requires grabbing data from a plethora of places, maybe needing OCR to try and decipher the Egyptian hieroglyphics doctors specialise in, and then giving it some hard thought - something software is not very good at.

So they went the easy path, and produced something that is marginally better than flipping a coin.

[ThatOne]

Re: Billing != Action

> Diagnostic-tools not MBA-data-wank tools.

Came here to say the same thing: Since when are billing codes reliable symptoms, given you have to be diagnosed, so you can be treated, so you can be billed for that treatment?...

Actually this can only predict sepsis infections which have already been successfully diagnosed!

[Little Mouse]

"only right in about 63 per cent of cases"

Ouch.

That figure is barely better than a totally random 50/50 guess.

I wonder how well it compares to a truly educated guess made by a doctor or nurse with knowledge of the patient & their particular circumstances?

[Jon 37]

"They did not take into account the analysis and required tuning that needs to occur prior to real-world deployment to get optimal results."

Convenient how that claim can be used to cast doubt on any study the makers don't like.

Also interesting to hear that the model "requires tuning". One would think that the human body is pretty much the same all over America, so tuning wouldn't be necessary. However, if I was ever to start a company selling snake oil, I'd use this line - it lets me bill for the software and get in there billing for integration and "tuning" without having to deliver anything useful, and if there are problems reported I can go do more "tuning to fix them", and who knows with enough "tuning" I may even be able to come up with something that mostly works.

[ThatOne]

> the human body is pretty much the same all over America

Yes, but this algorithm has nothing to do with human bodies, it relies solely on accounting, and that can have local differences.

That been said, you're of course right that the "tuning" excuse is a convenient one, letting them accumulate some profits before somebody attempts to wash the emperor's new clothes.

[Jon 37]

> it relies solely on accounting, and that can have local differences.

True. But the US has standardised medical billing codes, so it's just about how they are used. And most hospitals will want to bill the maximum justifiable amount, although they may have varying success with that.

[T 7]

“A better approach for the software would be to use a model that analyses healthcare symptoms defined by health agencies, such as the US Centers for Disease Control and Prevention, rather than just relying on billing codes, it would seem.”

Amazing at it seems, as a consultant in the NHS I make diagnoses all the time without needing billing codes.

In fact I spent a long time answering exam questions about sepsis without once resorting to billing codes.

All this data is stored within our Cerner behemoth. If only we knew how to get it out clinicians might not need AI / startups and lots of money.

[A Non e-mouse]

If only we knew how to get it out clinicians might not need AI / startups and lots of money

There's your problem. From a Bean Counter's view, Clinicians are terrible. They have to be paid to work, have to have paid time off, sometimes get ill themselves, cost a fortune to train, occasionally get things wrong and in parts of the world with employee rights are difficult to get rid of. AI is a simple fee, which is predicable, easy to fire (i.e. turn off) if you don't like it and people blindly believe because "It's computers"

[Swarthy]

Having done some development in the medical field

Having done some development in the medical field I can attest that "ICD10 codes" is akin to "blockchain" in the buzzword scale.

[Anonymous Coward]

Epic fail

Please send to Matt Hancock because data saves lives. Great idea in principle which I'm in favour of, it's the execution and governance that worries me.

[ThatOne]

Artificial Medicine - Real Patients?

Artificial intelligence algorithms will only be 100% efficient when hospitals eventually replace their human clients with AIs!

Till then, most AI systems will not be any more efficient than the average intern. A lot cheaper though, so watch your lifestyle, avoid at all cost having your life depend on some totally inadequate decision tree construction, made by a programmer knowing nothing of the matter under the supervision of some 3rd rate physician (they usually are 3rd rate. Good ones tend to have their hands full...).

[Anonymous Coward]

So what they *really* developed is a model that tells them when the insurance company is likely to be *billed* for sepsis treatments, not the patient actually contracting it.

Garbage in, garbage out.

[gobaskof]

But now if they are selling it to insurers so they might need a blockchain to make it sound fancy.

Garbage in, garbage stays in forever

[Anonymous Coward]

I know a small amount about EHRs.... this 'study' has its own flaws

So. I worked in IT for a major integrated healthcare company that not only uses EPIC software, they had a hand in the early stages of it. Here's what you need to know. (DIsclaimer: I have little direct knowledge of what is inside EPIC, and hospital procedures in general... but being around those that did for 10 years lets you pick up SOME knowledge.)

What follows is my semi-informed conjecture of what this is all about.

This almost certainly appears to be an attempt to data-mine that information which is present in the EPIC database, in an attempt to predict issues. There is nothing wrong with that, in fact it is admirable! If you have existing data, and you can use it to predict a coming crisis, you should do all you can to do that prediction.

My ex-employer does things of this nature all the time... having a massive database of procedures performed and outcomes, and a membership of well over 10 Million members, allows them to perform clinical studies and create automated watchdogs looking for things about to go wrong. Data from EPIC is fed into massive (anonymized) clinical databases and mined using massive farms of analytical servers (e.g. Teradata and the like).

Here's the thing: this approach only works if you have data, and only with the data you have collected. It was not designed (and would not have been accepted by hospitals) to have ADDITIONAL DATA collected in order to work better. All it can do is work with whatever is ALREADY being collected... which by and large is what procedures have been performed and recorded (which, for most hospitals drives the billing... hence 'billing data').

It is not clear to me what level of integration with the various Clintech systems (heart rate monitors and the other things that do bedside beeping) exists with any particular EPIC system, and/or to what extent that varies from hospital to hospital. EPIC implementations are as 'standard' as any other software package; environment and local process matters.

So: given all the above...

This 'study' is based on ONE hospital, and one hospital only. That hospital is a University hospital, so right there it is different from the majority of hospitals. We have no idea how detailed their billing records are, so we have no idea whether they are representative of the total universe of EPIC-using hospitals. We have no idea how detailed their recording of procedures performed are. This needs some peer review.

[calonddraig]

NEWS to me

Why not just appropriately roll out something like NEWS2 which is an algorithm for predicting sepsis risk that is so easy even humans can do it...

[redpawn]

I'm almost as good as an X-ray machine

I predict the presence of broken bones by looking for people wearing casts. Those without, probably don't have broken bones. I have a better than 90% accuracy record and I'm willing to sell the algorithm!

[Aussie Doc]

Wow

"They did not take into account the analysis and required tuning that needs to occur prior to real-world deployment to get optimal results."

Is that EPIC-speak for "You're holding it wrong"?

Correct me if I'm wrong but their 'real-world' (by EPIC's own definition) is only dealing with billing data not actual clinical data ¯\_(ツ)_/¯

They know more -------------------------------------------------------------------------------------------^^