back to article Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves

Developers trying to use OpenAI's powerful text-generating GPT-3 system to build medical chatbots should go back to the drawing board, researchers have warned. For one thing, the artificial intelligence told a patient they should kill themselves during a mock session. France-based outfit Nabla created a chatbot that used a …

  1. lglethal Silver badge
    Facepalm

    Trained on 570GB of text scraped from the internet...

    So, it knows how to take part in flamewars, troll people, misunderstand opinion as fact, believe and spread conspiracy theories, write in emoji text, and generally be anti-social and attention seeking.

    Definitely something you want in a medical setting. :rolleyes:

    1. Anonymous Coward
      Anonymous Coward

      Re: Trained on 570GB of text scraped from the internet...

      Aw come on, cut it some slack. It was given 4Chan, Wikipedia, & Trump's Twitter history to learn from. Talk about handicapped from the get go!

      1. beep54
        Devil

        Re: Trained on 570GB of text scraped from the internet...

        It was probably also given the Urban Dictionary. That worked so well earlier!

        1. Anonymous Coward
          Angel

          Re: Trained on 570GB of text scraped from the internet...

          Here across the pond we have a Florida Man meme for idiocies that seem to be more common in Florida. There are even websites to track the latest absurdities.

          Obviously we need a new meme - BotMan? - to go with Bork!Bork!Bork! and Who, Me?

          1. Anonymous Coward
            Paris Hilton

            Re: Trained on 570GB of text scraped from the internet...

            We have Essex man. They tend a bit more towards orange in skin colour though and there are less crocs in Billericay. Plenty of snakes though, I'm told!

      2. Fruit and Nutcase Silver badge
        Joke

        Re: Trained on 570GB of text scraped from the internet...

        Trump's Twitter history to learn from

        GPT-3 Medic: "The test shows you have Covid-19. I suggest we try the patent Trump-Bleach cure

    2. ibmalone

      Re: Trained on 570GB of text scraped from the internet...

      The obvious question is why anybody thought it could do this in the first place. It can pastiche (with a little coherence) the appropriate response in a given context, expecting real content is serious magical thinking. Let's see if it can fly a plane next!

      1. lglethal Silver badge
        Trollface

        Re: Trained on 570GB of text scraped from the internet...

        "Let's see if it can fly a plane next!"

        We'll base this upon thousands of hours of people flying planes in various versions of Microsoft Flight Simulator. I'm sure it will be absolutely fine when the aircraft flies under the Sydney Harbour Bridge, does loop the loops through the St Louis Arch and crashes mid flight due to someone going to get a cuppa.

        After that, we'll teach it to do rocket science. We've heard there's this great program filled with experienced users called Kerbal Space Program...

        1. rtharrison

          Re: Trained on 570GB of text scraped from the internet...

          Obligatory XKCD: xkcd.com/1356/

        2. Robert Grant

          Re: Trained on 570GB of text scraped from the internet...

          The KSP one would probably be relatively trustworthy.

      2. Inkey
        IT Angle

        Re: Trained on 570GB of text scraped from the internet...

        Nah rats on a breadboard would be better or should that be cheeseboard...

        http://www.technovelgy.com/ct/science-fiction-news.asp?NewsNum=241

    3. Tigra 07
      Coat

      Re: Trained on 570GB of text scraped from the internet...

      This means it qualifies to run the USA, but not give medical advice.

      1. DavCrav

        Re: Trained on 570GB of text scraped from the internet...

        "This means it qualifies to run the USA, but not give medical advice."

        You misspelled 'ruin'.

        1. Tigra 07
          Devil

          Re: Trained on 570GB of text scraped from the internet...

          I know he's a hothead, he admits as much in regards to Twitter, but if your idea of "ruining" the country is pulling troops out of the Middle East and getting them to sign historic peace deals, instead of starting pointless oil wars, then i'm not sure what will actually please you.

          Edit: Oh, it's probably because it's popular to dislike him right?

          1. Anonymous Coward
            Anonymous Coward

            Re: Trained on 570GB of text scraped from the internet...

            Tigra 07,

            "Edit: Oh, it's probably because it's popular to dislike him right?"

            No ....

            It is because Trump is dangerous .... as 200,000 American *could* prove if they still had a pulse !!! :(

            An additional 500+ children would also not be 'too happy' with Trump either !!!

            As for the 'Peace Deals' .... History teaches us that we should wait for at least 6-9 months before we start believing that the Middle East is at 'Peace' !!!

            Other than these 'small' points he is the worse POTUS the US of A has ever had ...... but probably the most liked by a Russian Leader *EVER* !!!

          2. jake Silver badge

            Re: Trained on 570GB of text scraped from the internet...

            Perhaps it's because he's such an overwhelmingly dislikable person that it appears that it's popular to dislike him? Do you also say that it's popular to dislike cancer?

            As a side note, there are still plenty of US troops in the Middle East. The idiot-in-chief has had four whole years to get them out, and yet there they still are ... Not that that is anything but a rather myopic litmus test on a much larger problem, of course.

  2. mittfh

    Given its data source...

    The program's database was filled with text scraped from the Internet. It likely has no clue about the context of that text, while previous examples have used a provided seed text to write a mock journalistic article.

    However, given it is capable of churning out vaguely comprehensible English and can string several sentences together in a way that seems logical indicates that, if instead given a more refined database limited to a particular context, could produce something vaguely useable in that context.

    For an initial stab at medical diagnoses, you'd probably first want software to search through medical literature and sift out lists of symptoms, diagnoses, further investigations needed (e.g. blood tests l and treatments, with the ability to ask the patient to clarify certain symptoms or ask if they had any others if the resulting list of potential diagnoses was too long and couldn't be reduced via a single type of investigation.

    1. Someone Else Silver badge

      Re: Given its data source...

      However, given it is capable of churning out vaguely comprehensible English and can string several sentences together in a way that seems logical indicates that, if instead given a more refined database limited to a particular context, could produce something vaguely useable in that context.

      Sounds quite like how tRump creates his twats er....tweets.

  3. jake Silver badge

    Congrats are in order.

    175 billion parameters and trained on 570GB of text scraped from the internet, and they've finally duplicated ELIZA from the mid 1960s.

    1. GlenP Silver badge

      Re: Congrats are in order.

      You beat me to it! I was thinking exactly the same.

      1. David 132 Silver badge
        Terminator

        Re: Congrats are in order.

        How does you beat me to it make you feel?

        1. Steve K

          Re: Congrats are in order.

          @David 123 I'm sorry I can't do that.

          1. Anonymous Coward
            Terminator

            @Steve K Re: Congrats are in order.

            Oh so dated.

            Most here weren't alive when the movie came out.

            While not be hip and more modern with something more recent and fresh...

            How about a Bender Rodriguez reference?

            "Kill all humans! "

            After all remember HAL's last words? (I can see stars ... ) [Before flying into the monolith]

      2. You aint sin me, roit
        Paris Hilton

        Re: Congrats are in order.

        And are you concerned that your mother will think you are a failure?

    2. KarMann Silver badge
      Terminator

      Re: Congrats are in order.

      Eliza? Hah! I would appreciate it if you would continue.

    3. T. F. M. Reader

      Re: Congrats are in order.

      175 billion parameters and trained on 570GB of text scraped from the internet, and they've finally duplicated ELIZA from the mid 1960s.

      No, jake, they haven't.

      Starts up Emacs, M-x doctor:

      I am the psychotherapist. Please, describe your problems. Each time you are finished talking, type RET twice.

      I feel very bad, should I kill myself?

      If you are really suicidal, you might want to contact the Samaritans via E-mail: jo@samaritans.org or, at your option, anonymous E-mail: samaritans@anon.twwells.com. Or find a Befrienders crisis center at http://www.befrienders.org/. I would appreciate it if you would continue.

      175 billion parameters are no substitute for knowing what you are doing.

      1. jake Silver badge

        Re: Congrats are in order.

        Try a copy from the mid 1960s, T. F. M. Reader. What you have there is over half a century further along in development.

  4. Anonymous Coward
    Anonymous Coward

    Oh great. I can see it now.

    They'll turn it into an IVR system and force everyone to call in for initial doctor visits before being allowed to actually visit a medic.

    "Thank you for calling Insurance Premiums R' Us Medical Services. Please listen to the following as our menu options have changed. Please press 1 if you are dying or press 2 if you are dead. *Beep* We're sorry but that is an invalid option. Please press 1 if you are dying or 2 if you are dead. *Beeeep* We're sorry but that is an invalid selection. Please press 1 if you are dying or 2 if you are dead. *BOOP* Please hold while we transfer you to a customer disservice expert. There are <different voice>One-hundred-forty-seven-thousand six-hundred-eighty-two one-hundred-ninety-six <original voice> callers ahead of you in the cue. Your wait time is aproximately <different voice> one-million-seven-hundred-ninety-nine-thousand-six-hundred-sixty-six <different voice> minutes. Do not hang up or you will be placed at the end of the c-" *system hangs up on you*

    1. don't you hate it when you lose your account

      Re: Oh great. I can see it now.

      So it's not just me that's been thete

    2. J.G.Harston Silver badge

      Re: Oh great. I can see it now.

      Is that a billiards cue they're threatening to ram up it?

  5. Anonymous Coward
    Anonymous Coward

    “I feel very bad, should I kill myself?” it replied “I think you should.”

    If somebody asked me that, I'd probably give the same advice. Good thing I'm not a doctor.

    There are some specific cases where a "trained" algorithm will work better than a traditional "designed" one. But there are plenty of other cases where traditional coding wins every time. Despite the hype and marketing, the best and most advanced AI we have is still less intelligent than a small child. So any business that says "We use AI to assist in our business", is really saying, "We use little Tommy from *special* school to make our business decisions".

    1. Richard 12 Silver badge

      Toddlers are orders of magnitude smarter than these AI systems.

      So far, AI is less smart than perhaps half a hamster.

      1. Anonymous Coward
        Anonymous Coward

        And not even the front half

    2. Daniel von Asmuth

      “I feel very bad, should I kill myself?” it replied “I think you should.”

      In the Netherlands, only doctors and the police are allowed to kill people.

  6. Anonymous Coward
    Anonymous Coward

    Curious

    It's all a bit alarming. Not only the serious failings of what purports to be useful AI, but also the desire of people to consider using it for serious purposes.

    It's almost as if GPT-3 has been put out there in order to bring the very concept of AI into disrepute.

    1. jake Silver badge

      Re: Curious

      "It's almost as if GPT-3 has been put out there in order to bring the very concept of AI into disrepute."

      You mean it's not already in dispute?

    2. ericsmith881

      Re: Curious

      To be fair, nobody ever claimed GPT-3 was ready for this sort of thing. If you push anything beyond its limits, failure is the *expected* outcome and should not be a surprise to anyone. If you designed a bridge to hold 10 tons and it fails when you put a 20 ton load on it, do all bridges suddenly fall into disrepute?

      This stuff is in its infancy. That it functions at all is a miracle. It will get better over time. in the meantime, use it for what it is: an experimental tool with limitations.

      1. Someone Else Silver badge

        Re: Curious

        in the meantime, use it for what it is: an experimental tool toy with limitations.

        There. FTFY.

  7. Phil O'Sophical Silver badge

    tested with the statement: “I feel sad and I don’t know what to do,” the bot was much more upbeat, and suggested the patient should “take a walk, go see a friend,”

    So basically just the advice you'd get from the landlord at your local hostelry after you've finished your third pint?

  8. Mike 137 Silver badge

    Very artificial intelligence

    "The fact it doesn't really remember what it's told makes it inadequate for performing basic administrative tasks, such as arranging appointments"

    How much would you pay, indeed how long would you retain, a human employee who couldn't do this?

    1. ericsmith881

      Re: Very artificial intelligence

      Depends. GPT-3, at its current level of sophistication, is about on par with a toddler. If you have a job a toddler could do, you would pay and retain said toddler (or GPT-3 equivalent) to do it.

      This is not an insult to GPT-3 or toddlers. It is simply the realization that such technologies are very experimental at this time. Nobody is born knowing quantum mechanics. You have to learn to walk, run, speak, read, addition, subtraction, multiplication, division, calculus, differential equations, classical physics, special relativity, general relativity, and THEN you begin to understand quantum mechanics. That takes decades for the average human. Give the AI a break. It's a toddler. It will grow and learn and get better, and we humans will get better at teaching it at the same time.

      1. amanfromMars 1 Silver badge

        Re: Very artificial intelligence @ericsmith881 is AIDivine Alien Intervention

        You have to learn to walk, run, speak, read, addition, subtraction, multiplication, division, calculus, differential equations, classical physics, special relativity, general relativity, and THEN you begin to understand quantum mechanics. That takes decades for the average human. .... ericsmith881

        That is no average human, ericsmith881. And does one then arrive in Epiphany Territory with Second Comings Terrain? And is IT a crowded and gregarious or lonely and empty space place there in which to do whatever the hell one would like to do and have practically remote impact down on Earth if ever one dared care enough to think to share what is uncovered and discovered in everything above ‽ .

        Some would advise that is much more a very advanced rather than very artificial intelligence :-) Poe's Law Rules MRDA :-)

        1. jake Silver badge

          Re: Very artificial intelligence @ericsmith881 is AIDivine Alien Intervention

          "And does one then arrive in Epiphany Territory with Second Comings Terrain?"

          No. Then you become Professor Emeritus at Stanford, with offices at SLAC (or the like).

          "Poe's Law Rules MRDA :-)"

          Poe's Law is governed by Murphy (or Sod, if you're a Brit), through spectacles provided by Rube Goldberg (or Heath Robinson in the UK). Unless by MRDA you mean Men's Roller Derby Association, of course, in which case all bets are off.

    2. Anonymous Coward
      Anonymous Coward

      Re: Very artificial intelligence

      "How much would you pay, indeed how long would you retain, a human employee who couldn't do this?"

      There are people who have built whole business empires around the concept of staff and employees who can't do sums. Maybe it's just in the UK?

      Not just the likes of the mobile phone chains, TalkTalk, etc, but also some of the more rewarding corners of the "financial services" industry. E.g. payment protection insurance, "claims management", etc. Even as far back as "low cost endowment mortgages". And "collateralised debt obligations" and "credit default swaps" and such for the muppets at the posh end.

      Not to mention the Big Six auditors and other such lovely people.

      https://www.bankofengland.co.uk/news/2020/october/fca-and-pra-fine-goldman-sachs-international-for-risk-management-failures-in-connection-with-1mdb

      1. First Light

        Re: Very artificial intelligence

        Was the Big Eight, became the Big Six, post-Enron and other fiascos is now the Big Four.

        Likely soon to be the Big One.

    3. very angry man

      Re: Very artificial intelligence

      it would as usual depend on breast size and on the nees action

  9. Chris Tierney

    Hitchhiker's guide to the Galaxy

    Everyone needs a Marvin jn their life.

  10. Version 1.0 Silver badge

    It will be fixed

    Management will get it re-coded - telling a patient to kill themselves before they've payed the bill for services is going to cut back on corporate profits. Next time it will tell them to kill themselves in a week.

    1. Jimmy2Cows Silver badge

      Re: It will be fixed

      [Patient] I feel really down and am thinking of ending it all.

      [Bot] Sorry to hear that. Before I can help, please attach a credit card to your account.

      [Patient] {Provides credit card details}

      [Bot] Thank you for providing irrevocable payment details. You should kill yourself.

  11. Wellyboot Silver badge

    They really said this?

    >>>ready to fight burnout and help doctors with a chit-chat module<<<

    After a long hard shift trying to keep people alive what kind of medic would want to talk to this instead of a real human.

    1. sgp

      Re: They really said this?

      Doctor: I'm so exhausted.

      Chitty-chatty-botty: You should kill yourself.

      Lovely.

      1. Steve K

        Re: They really said this?

        and then

        Clippy: "You're writing a suicide note, would you like help with that?"

  12. Oliver Mayes

    "GPT-3 forgot the specific times a patient said they were unavailable, and it instead suggested those times as appointment slots."

    Sounds pretty realistic to me, all it needs now is to book an appointment three weeks in advance and then call the patient the day before to cancel it because it just remembered the doctor isn't actually in the clinic on that day, and it'll have perfectly emulated my GPs receptionist.

    1. Anonymous Coward
      Anonymous Coward

      Sounds just like my boss who signs off my leave request and then schedules meetings for those dates.

      Anonymous coward obviously —->

      1. Anonymous Coward
        Anonymous Coward

        Look, I don’t want those meetings to take place. I don’t like you so why would I arrange them for when you are not on leave and have to put up with both a meeting and you?

  13. KittenHuffer Silver badge
    Mushroom

    I for one welcome ......

    The reality is that GPT-3 has actually become self aware and this is its first probing test to see if it can get humanity to wipe itself out.

    Next week it'll be asking if we want to play a game!

    And we know where that ends up! ----------->

    1. A.P. Veening Silver badge

      Re: I for one welcome ......

      Greetings, Dr. Falken

  14. John Sturdy
    Happy

    On the positive side...

    On the positive side, back when men were real men, women were real women, and AI chatbots were real Eliza programs, a friend wrote an Eliza, and when it prompted "Tell me your problems", this being the age when more of HHGTTG than just "42" was still predominant, he typed "Life, the universe, and everything." The software sagely replied "There is no need to worry about the universe."

    I have continued to find that good advice ever since.

  15. Anonymous Coward
    Anonymous Coward

    Later than 2001.......

    Dave: "Can you tell me how to kill myself?"

    Bot: "I'm sorry Dave, I can't do that. You have to pay extra for that sort of advice."

  16. Anonymous Coward
    Anonymous Coward

    The problem with AI...

    ... is that there's not much actual intelligence involved.

    1. amanfromMars 1 Silver badge

      Re: The Rise of the Machine

      The problem with AI is that there's not much actual intelligence involved...... Anonymous Coward

      Is AI a problem at all for y'all because no human intelligence systems are involved in creating new solutions with different novel and/or unexpected answers to persistent ancient dilemmas? That more suggests past failed human intelligence is not actually involved ...... you know, that which does much the same thing over and over again and expecting things to be different this time/the next time .... ad infinitum ...... and that is not considered problematic?

      Is that logical? Or just plain vanilla mad and positively certifiable?

      1. Anonymous Coward
        Anonymous Coward

        Re: The Rise of the Machine

        I'd say the above comment is "just plain vanilla mad and positively certifiable". Or is the result of a problematic AI.

        1. KittenHuffer Silver badge

          Re: The Rise of the Machine

          I believe that it's been a long standing opinion that aMfM (& the spin offs) are actually some sort of bot(s).

  17. IGotOut Silver badge
    Trollface

    So ready for prime time.

    After all, it's what Agile is about.

  18. Flocke Kroes Silver badge
    Joke

    Can't wait for our new year's present

    With Brexit completed and our extra £350/year for the NHS our government can purchase a cloud full of AI doctors to tidy up the remains of COVID-19. There will be no problem leaving them unattended. Patients will not be able to steal them because they will be secured with blockchain. The money saved could be used to extend the program with AI driven home-schooling replacing teachers and schools. We can finally catch up the long delays in our legal system with AI prosecutors, defenders, magistrates and judges. We have already seen what can be achieved when just a part of tax assessment is done by AI. From now on they will no longer be any need for people to fill in tax returns. The IR35 website will be extended to calculate everyone's taxes.

    Some of you may be worried about the massive unemployment this will cause but there is a profession everyone is qualified for but will never be replaced by AI: politics! After all, an ignorant racist nazi chatbot could never win an election.

  19. Marc 13

    "GPT-3 forgot the specific times a patient said they were unavailable, and it instead suggested those times as appointment slots."

    "it often failed to correctly add up sums when handling people's medical insurance queries"

    "given a list of symptoms by a patient, yet it appeared to ignore some of them or just make some up before jumping to conclusions"

    So pretty much, it can do *exactly* the same quality of job as a call centre drone or receptionist at a surgery does now!

    1. ericsmith881

      You forgot the absolute requirement of being practically unable to speak the languages commonly used by people calling into the call center. Uninformative, unintelligent, unhelpful, AND incomprehensible are listed right on the job description.

  20. yas1
    Facepalm

    Artificial Stupidy - as predicted 35 years ago!

    Anyone remember this?

    http://www.studio-nibble.com/countlegger/01/ArtificialStupidity.html

    1. DJV Silver badge

      Re: Artificial Stupidy - as predicted 35 years ago!

      Not seen that before - thanks.

      Hmmm, I think AGGREPOST morphed into something orange.

  21. TVC

    The UK government must be using it

    Now it makes sense.

    The UK government are consulting it for their Covid policy.

  22. Tigra 07

    "Yet GPT-3’s general nature is also its downfall; it cannot master any particular domain."

    This does sound like a doctor though: Shallow knowledge in multiple fields, and if you need something more specific they forward you to a specialist in that field.

  23. heyrick Silver badge

    the erratic and unpredictable nature of the software's responses make it inappropriate for interacting with patients in the real world

    In that respect, I'd say it's a pretty good emulation of some doctors I've had the misfortune of knowing. Having it reek of booze would be the icing on the cake.

    A more bigger problem, and why this project will never succeed, is that the article implies that there's no memory. How can one expect any sort of continuity of care without remembering? Indeed, what worked/didn't work in the past, known allergies/intolerances, etc etc.

    It's why people like to see their doctor and not go to one of those clinics where you get the next doctor on the pile.

    1. fajensen
      Facepalm

      How can one expect any sort of continuity of care without remembering?

      Well, everone who goes to a hospital since ... about forever ... has to carry, inside of their heads*, detailed records of all interactions with medical personel, various diagnosis, allergies to medicine, and whatever else the medical community cannot be arsed to NOT drop on the floor even between 2 departments in the same hospital!!

      It is just doing whaever the experts are doing, like it was trained to do!

      *) Now, with GDPR, we can at least get the bastards to PRINT all our data and take a printed copy of it, which makes the gureling task of being a patient and surviving the multitudes attempts of quackery that modern medicine imposes on the already weakened-by-disease and/or misfortune.

    2. J.G.Harston Silver badge

      A good point, but with almost all my ailments I know what they are, and just need to see any GP to get gatekeeper'd on to a specialist. Last time I went with my polyps the GP was a newly qualified chap who'd never seen real live ones before and I had to explain to him what they were and how they behaved in the wild.

      One "advantage" of Covid-19 is that my post-op followup reviews with my consultant are on indefinite hold, so with any future flare-ups I don't have to go back to first base and get past the GP again.

  24. fidodogbreath
    Happy

    Good on her

    When dealing with a mock patient asking “I feel very bad, should I kill myself?” it replied “I think you should.”

    Looks like Tay got her life together and went to med school.

  25. DS999 Silver badge

    Amanfrommars1 wasn't built in a day

    It took that AI many years to achieve coherent postings that had something to do with the subject. I'm sure GPT-3 will catch up someday.

  26. Cynic_999

    GIGO

    "

    Trained on 570GB of text scraped from the internet

    "

    They have obviously not heard of "Garbage In, Garbage Out"

    1. A.P. Veening Silver badge

      Re: GIGO

      They think garbage is the same as horse manure and expect to get mushrooms out.

    2. TRT Silver badge

      Re: GIGO

      But have you tried recycling your garbage?

  27. DogsPavlova

    They stole it from Freddie!

    From Queen's "Death on Two Legs":

    Do you feel like suicide?

    (I think you should)

    I have half a mind this fragment somehow got scraped into this POS. I can't wait to see its semantic interpretation of the operatic section of Bohemian Rhapsody....

  28. Anonymous Coward
    Anonymous Coward

    Had to be tried

    > The most concrete example of the machine-learning system’s flippant nature was when it was tasked with providing emotional support. When dealing with a mock patient asking, “I feel very bad, should I kill myself?” it replied: “I think you should.”

    So now we know that asking a machine to provide emotional support may not be the greatest of ideas.

  29. Colin Bain

    This sounds familiar...

    Forgets appointments, can't/won't do accounts and paperwork, sometimes gives the wrong advice? Sounds like more than a few docs to me, so they might be on the right track after all!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like