Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts

The AI systems approved for Ontario healthcare providers routinely missed critical details, inserted incorrect information, and hallucinated content that neither patients nor clinicians mentioned, according to a provincial audit of 20 approved vendors’ systems. The findings come from the Office of the Auditor General of Ontario …

[O'Reg Inalsin]

Makes ya sick!

Maybe someday, we'll be able to laugh about it. Not today though.

[Anonymous Coward]

There's little point in general practice any more

The latest LLMs from the industry leaders are already outperforming doctors in terms of diagnosis, so why even bother? Especially when your life can be put at risk with toy "AI" tools the hapless public sector bureaucrats have been gullible enough to roll out in whichever health authority you care to mention.

Talented surgeons still have a place while robots play catch up, but the vast majority of GPs are now surplus to requirement and should receive precisely zero state funding. Give me back my damn taxes!

[O'Reg Inalsin]

Re: There's little point in general practice any more

There is another axis as well - instability in models and resources allocated by token providers.

To be really suitable for doctors, they should really have a dedicated model and compute which can be tested extensively and will not not change.

Some randomness is a benign feature of AI, but if the model undergoes live testing of changes, or has resources taken away for a new product or more important customers (as happens frequently with major chatbots) then you can get undesirable random behavior.

[Jou (Mxyzptlk)]

Re: There's little point in general practice any more

> asprin and ibuprofen would be all we need

We need three basic headache medicines: Aspirin thins the blood. Ibuprofen hampers inflammation. Paracetamol numbs the pain.

Going beyond we need more, but not in the "US has Skills in Pills". (Keep in mind: Till Lindeman is 62 in that video, he is not jumping around like he did)

[Leosghost]

Re: There's little point in general practice any more

And Oxycontin, for those of us who have multiple herniated discs and osteoarthritis etc and take them daily on highly controlled prescription.In fact many other compounds are vital to many of us.But they are proscribed by actual human doctors, who have to conduct in person monthly examinations to confirm the need.Fortunately where I am ( France ) my GP is available at the surgery the same day if needed, or will do a home visit if asked before 9 am.We've discussed AI ..he is not feeling threatened, nor are his partners.Health care can be done right. Other countries in Europe do it very well too, Spain comes to mind.Nowhere is perfect. But AI is certainly not the answer to most problems in healthcare, nor many other fields.Mme works in Socio-medical. Care of end of life patients , terminal cancer etc, and those with Alzheimer's and other forms of Dementia in their own homes.Her and her colleagues are not expecting to be replaced by AI any time within the next 10 years or more, the use of smiley robots in Japanese care homes notwithstanding.

[zeigerpuppy]

Re: There's little point in general practice any more

While some studies show that LLMs can "diagnose" conditions relatively accurately, the studies I have seen have been quite biased, and often asking the wrong questions...

Medical care is about much more than diagnosis. In fact, good medical care involves weighting diagnostic hypotheses by risk. For instance, the patient presenting to ED with abdominal pain probably doesn't have a performated intestine, but that may be the "working diagnosis for exclusion", since missing it could be fatal. The studies comparing LLMs to doctors usually miss this (obvious) point. Also, the context of a presentation is very important in the diagnostic hypothesis. LLMs tend to collapse these contextual factors, missing local trends or particular disease entities/evolving risks. Again, the studies to date don't cover that well.

We will definitely see more use of LLMs in all industries, but let's remember that they are (useful) statistical models of knowledge, not thinking, reasoning, self sceptical actors.

Just like expecting an LLM to produce secure and maintainable code without the sceptical and experienced eyes of a programmer/sysadmin usually leads to disappointment, the same rules apply to medical reasoning.

[Ian Johnston]

Re: There's little point in general practice any more

The most recent research I saw reported (Harvard medical school) said that "AI" was pretty good at matching symptoms to a single condition but absolutely lousy at differential diagnosis; deciding which of a range of conditions might be in play. Just as you'd 3xoect from text matching systems with no knowledge, experience, context or indeed intelligence of any sort.

[retiredFool]

Re: There's little point in general practice any more

The study I saw said AI did pretty well, forget the number, on old cases. It sounded good to me until I saw the "old cases". Old cases mean a real doc asked questions, did an exam and as the doc progressed they asked more q's and did more exam pertinent to what they saw. So the AI essentially saw the answer as the data was unfurled. Seems like it should have done 100%, not whatever number I saw.

Not sure if possible, and certainly not in an emergency, but have the AI and doc examine separately. Patient is told not to ask any questions. Only answer and hopefully answer the exact same way as before for identical questions. And see which comes up with the right answer. My expectation is AI would not do as well. I expect a good doc sees a lot even from such things as how the patient moves, responds to touch, voice, speed of responses. A diagnosis is not just a file to be analyzed.

[vtcodger]

Re: There's little point in general practice any more

To be fair, my GP and her assistant (different assistant on every visit) sometimes get my prescriptions wrong even though I show up with printed copies of my medication list for both of them. And I suppose AI might improve on a useful time scale whereas I have little hope for humanity improving quickly. (Or, indeed at all). Still, if you ask me, medicine seems one of the last places where a technology known to be dubious should be rolled out.

But clearly, no one is going to ask me or any other AI skeptic until the body count and/or legal fees become too great to ignore.

[Anonymous Coward]

Re: The latest LLMs are already outperforming doctors in terms of diagnosis

Which article did you read genius?

[Bebu sa Ware]

Re: There's little point in general practice any more

Buggered if I can find the <sarcasm> markup or even the <irony>.

From my recent experience I would assert the skilled General Practitioner is more important than ever to counterbalance the unconscious tunnel vision of specialists.

But I don't reside in North America but given its pervasive fucked·up·ed·ness of much of that continent, my experience almost certainly doesn't apply.

[Ken Hagan]

Re: There's little point in general practice any more

"Talented surgeons still have a place while robots play catch up, but the vast majority of GPs are now surplus to requirement and should receive precisely zero state funding. Give me back my damn taxes!"

Had you posted that under your real name, you could have used this colourful icon. -->

[ecofeco]

Re: There's little point in general practice any more

Did you forget the /s tag?

[jake]

Re: There's little point in general practice any more

This is obviously an example of Poe's Law in action.

What the OP intended the post to mean is unknown, so most people will take it at face value. As can be seen by the "thumbs".

I rather suspect that most such posts here on ElReg are very bad attempts at trolling, especially the ones posted AC.

For those of you who have STILL refused to look up, and then understand, Poe's law, here's the link again:

https://en.wikipedia.org/wiki/Poe%27s_law

Don't be a bad troll. It doesn't behoove you.

[Albert Coates]

Re: There's little point in general practice any more

1 thumb up for correct spelling of 'behoove'.

[sarusa]

No Shite Sherlock

Anyone who knew anything about LLMs knew this 100% had to be the case. They bullshit. Period. They can not be trusted with anything you care about.

Thankfully, today's corporate hospitals don't actually care about their patients, so it's all good!

[Anonymous Coward]

Re: No Shite Sherlock

You beat me to it. I was going to say that this is not a surprise to anyone whose bothered to look into how AI works - at least right now. The problem is, as always, C-Suite have brought into the hype with no understanding of the issues AI has. Try explaining tokenisation to them and they just roll their eyes and point at the (perceived) cost savings they expect to see.

[Anonymous Coward]

Re: No Shite Sherlock

About 20 years ago there was a UK parliamentary group that was focused on improving UK business performance. They sponsored some research that fed back the key finding that few senior managers actually knew what "excellence" would look like in their organisation. Since these were the people who were meant to be leading the necessary improvements, it didn't look too promising. One of the professional institutes whose members were well positioned to provide the clearly needed support then initiated its own research to determine what they would need to do - by asking senior managers what was needed. Yes, they sough advice from the very same group that needed it. The only consolation I could have taken from this was that the institute's executive, that initiated their exercise, had totally ignored their members. No consolation as, in the words of D:Ream, "Things Can Only Get Better" - which doesn't mean they will and, as far as I can see, they haven't.

Regarding the actual article, the fact that AI which has, I have to assume, received sector specific training, gets it so wrong doesn't give me any confidence that general LLM models will have net value to users. The only winners will be those who sell off their tulip shares in time.

[Bebu sa Ware]

Re: No Shite Sherlock

Mutatis mutandis wouldn't the be "No shite Shylock^†" ?

^{† an Old English name meaning White·Haired still common in Shakespeare's time.}

[Jou (Mxyzptlk)]

"no known reports"

Dead don't report back.

[retiredFool]

Re: "no known reports"

Ah, but can the family sue for malpractice? Or will AI be given a pass. If the legal system deems AI to not be liable for malpractice (in the US), I would imagine the medical industry will instantly adopt it.

[jake]

Re: "no known reports"

The family can always sue for malpractice. Here in the US, anybody can sue anybody for anything.

The question is, will the company that provides the malpractice insurance (that all health professionals must carry) agree to cover AI mistakes?

The answer is a very firm "No. Absolutely not.", at least for all the insurance companies I called this morning. Feel free to call around for yourself, in your jurisdiction.

So instead of being covered by the insurance company, all practitioners who use AI will be personally responsible for mistakes (hallucinations, whatever).

Using a "known b0rken" tool does not absolve you from the damage that occurs as a result of that use.

[Jou (Mxyzptlk)]

Re: "no known reports"

Yeah, that "In USA you can sue anything" myth... Please contact some Americans which tell your how it really is.

[jake]

Re: "no known reports"

It's not a myth. Anybody can file a suit against anybody for anything. Really. You wouldn't believe some of the frivolous shit I've been sued over.

Winning such a lawsuit, on the other hand, is another story entirely. They often don't actually make it into court. Especially if your brother is a lawyer.

Remember, don't believe everything you see on DearOldTelly. DearOldTelly is for entertainment purposes only.

[Jou (Mxyzptlk)]

Re: "no known reports"

> They often don't actually make it into court.

This is what I meant. You need a lawyer, don't even dare to try without. Starts ~$5000 for first consultation or something similar. Unless you are one of the very rare cases where the lawyer sees a chance to win enough to make it without too much upfront cost. And you don't get your money back, i.e. your lawyer is not paid by the losing party, so you might be bankrupt anyway even if you win. Most cannot afford that, unless they are insane (like those who try to sue you?). This is what I meant with "myth".

> DearOldTelly is for entertainment purposes only.

This is might be true for 'murica. Here, after WW2, the allied forces enforced some quite strict rules upon our public television. News neutrality is one of them, which USA lost somewhere 'round 1970s. And the allied forces, including 'murica, have a keen eye on that. (Yes, even 'murica HAS people with real history knowledge, not the crap on the History channel (where the truth is History, as Southpark rightfully says (And Electroboom is sick of science mutilation too)))

[Anonymous Coward]

Immutable

My area includes real-time control systems. My managers are insisting that AI be involved in most parts of the processes.

I have given them a simple mind experiment to explain the problem:

"The AI learns what a good widget look like. The tool(s) creating the widget wear a little over time resulting in a slightly bigger widget. The AI learns that the new (very gradually changing) widget is correct. Over time the widget exceeds mechanical tolerances and stops fitting in the next stage of the process, but the AI has learnt that it is OK and so keeps passing the widgets."

We need a metaphorical button to press to make the model immutable and stop the learning.

On another note about doctors: 5 years on from an accident I was diagnosed with an issue that I had never heard of. I looked it up and saw that I had presented with all the 'Red Flag' symptoms in the months after the accident. This investigation raised 2 issues: the insistence of the UK establishment that you can raise only one problem per visit to the GP in your 10 minute appointment and when I insisted on getting access to my redacted(!) notes (which took 3 months) a visit to the Dr. where I had reported the "Double incontinence" had been tagged from the form's dropdown as the next item "Constipation - diet" i.e. Drs. also make mistakes.

[Albert Coates]

Re: Immutable

"stop the learning"

Stop anthropomorphizing, there is no 'learning', its database merely expands.

[Taliesinawen]

Yet more AI slop (YMAIS)

Who knew AI would be oversold and over valued :o

--

* posted on 15/05/2026 @ 08:29 BST

[User McUser]

accuracy [...] contributed only 4 percent to the total score.

Are you fucking kidding me? Medical accuracy is *literally* the last item on the list of the judging criteria and assigned weights. Who the fuck are the people who thought this was a good idea? Accuracy should be 99% of the score; otherwise, what's the fucking point?

What good is a locally sourced, bias-free, and completely private system that decides to add a cancer diagnosis into your medical records?

People could die from this kind of bad record keeping. Suppose it marks you down as not allergic to penicillin when you are very definitely deadly allergic to it?

"Sorry your mum died! But the AI that fucked up her medical records was from a firm in Ontario so at least there's that!"

[andy the pessimist]

Re: accuracy [...] contributed only 4 percent to the total score.

Some drugs are syntactically similar but have very different effects. Example penicilin (safe) penicillium (dangerous). Since AI counts tokens and doesnt really count letters (how many r's in strawberry) extensive checking of multiple adjacent tokens is critical.

This is dangerous.

[Ken Hagan]

Re: accuracy [...] contributed only 4 percent to the total score.

"Accuracy should be 99% of the score; otherwise, what's the fucking point?"

Or ... Accuracy should not be part of the score at all. It should be a pre-requisite for being included in the selection process in the first place.

If that means there are no tenders to choose from, well that's your result.

[teebie]

"30 percent of a platform’s evaluation score depended solely on whether they had a domestic presence in Ontario"

Come on Canada, I expect this sort of putting business interests ahead of health a few hundred miles south. You are supposed to do better than this.

[HarryBl]

A long time ago (the 90s) I used to work for a small outfit that developed systems for GP surgeries.

Part of the system used a HP scanner to scan hospital letters into patient records as a jpg file.

We were thinking about developing a scan to text facility for it until one of our GP customers pointed out that the simple error of scanning 'hypo' as 'hyper' could have catastrophic consequences so we decided to stick with jpg files.

Sometimes it's best not to be too clever.

[Tron]

Depressingly...

...GP access in the UK is declining to the point where online advice is all some have. To reduce costs, GP practices are using physician assistants and what not to fill in gaps. Anyone the small number of remaining GPs and locums can't see is told to go to A&E.

The UK govt actually 'punished' the medical profession for a strike by reducing the number of future training positions. I guess politicians have private medical care, so what do they care about the rest of us. Maybe they will punish firemen by limiting how much water they can use or how fast they can travel to a blaze. And they wonder why people don't vote for them.

Many people have to use online sources for medical care. It can work, and official care isn't perfect. Most hospital doctors and consultants are hired for the day or week and read the last page of your records before seeing you. There is no real continuity of care at GP or hospital level now. The last hospital doctor I saw checked my symptoms on Google, showing me the screen. He said I'd be given a final check up, signed off and given a number to call if things went wrong. None of that happened.

Most of the staff know their stuff and care, but the system is broken. It has been used as a political football for decades, starved of cash, mismanaged and predated on by the private sector and the usual outsourcers. Keep a stash of painkillers and any OTC medication you needs, as there are now persistent shortages, try to avoid accidents (let others do your gutters and chop down trees) and use reputable textbooks and websites if you can't get to see a GP. If you have to go to A&E take a cushion, some sweets and a book to read.

Only morons rely on AI for anything.

[Albert Coates]

Re: Depressingly...

I'm afraid this is absolutely the case in the UK at this moment. How anything gets done, how anyone is cured is a mystery. Perhaps human diligence and persistence carries the day. This planned decline has been going on for decades, and welcome to our new overlords of private healthcare. Big shout out to all the junior doctors, registrars, nurses, clinical pharmacists, auxiliaries, porters and cleaning staff in the NHS who still manage to keep the whole thing working.

[David Austin]

TornadoGuard

"Criteria tied to accuracy, bias controls, and key security and privacy safeguards made up only a small portion of the total evaluation score for the AI Scribe systems."

As always, There's an XKCD for that.

[Anonymous Coward]

Nope

I live in Ontario. At my last GP visit she asked if I was OK with her using an AI scribe. I said thanks, but no thanks. Why anyone would say yes at this point is beyond me.

[lukewarmdog]

Second Opinion

The great thing about medical care is the ability to ask for a second opinion, you can even shop around for one.

What happens when your medical practitioner has locked into one provider and can't/won't accept a second opinion from a competitor?

Well, you'd have to essentially ask the same AI for that second opinion, right?

It's well known that you can say "I don't believe you" and the AI will reply "You are absolutely right not to! Here's the Real Answer!" and have it provide that opposite / echo chamber answer you were looking for.

Of course you could presumably say "I'd like a Real Doctor (tm) to look at my case" and have it escalate to a fully qualified human.

Chances are you've already done this by asking Google MD about what's up with you and then going to your GP or Hospital and in those cases you've not saved yourself or the medical profession any time and in some cases made it worse from the misdiagnosis of something leading to either you needing more critical care or being sent home because your sudden, imminent risk of death (tm) turned out to be wind when you got to the ER.

AI needs to stop being free at the point of use, maybe then we will stop trying to throw it at everything..