back to article ChatGPT hasn't been around for long and Nvidia already wants to put a leash on it

ChatGPT's testing phases pretty quickly revealed that the OpenAI chatbot and others like it could go off the rails – or "hallucinate" – with enough poking and prodding. OpenAI's sticking plaster was to limit the amount of queries a user could make before the chatbot descended into madness. Now, for businesses perhaps wary of …

  1. TheMaskedMan Silver badge

    So Nvidia are essentially selling shovels in the AI gold rush. Smart move.

  2. IglooDame

    If you listen really hard, you can hear John Connor somewhere smiling.

  3. FrogsAndChips Silver badge

    Let me guess

    Is it using AI to "detect and mitigate hallucination"?

    1. Frogmelon

      Re: Let me guess

      AI don't know. :)

    2. Michael Wojcik Silver badge

      Re: Let me guess

      It's more like an expert system.

      A quick look at the NeMo Guardrails info in NVIDIA's technical blog post shows it's just a framework for writing rules in Colang, a data-flow language NVIDIA invented for this sort of thing. You basically tell it "if the input matches this pattern, do this thing", where "pattern" is a fairly expressive DSL. But there's no magic here, and frankly my initial guess is that if you have programmatic access to a NeMo-protected system (so you can drive a lot of queries) you'd be able to train an adversarial model pretty quickly to simulate its guardrails and then look for holes.

      And, of course, it's up to the application developer to design good guardrails in the first place, and that's an area where very few developers are likely to show much expertise.

      So this is likely to be used reactively: "Oh, someone just made our bot do a Bad Thing, so write another guardrail to prevent that specific prompt". No magic, no silver bullet.

      (That said, a smart organization deploying an LLM-based service would do the adversarial modeling first, and use it GAN-style to create a set of guardrails. That's likely to be more successful than an economically-feasible human-authored set.)

  4. Anonymous Coward
    Anonymous Coward


    I LIKE TO hallucinate! :(((((((

    1. david 12 Silver badge

      Re: but...

      I LIKE TO hallucinate! :(((((((

      I had this discussion with someone with a much different circle of friends. I was telling her that I didn't like the side effects of my (medically prescribed) drugs, but could understand that some people might enjoy them. She immediately informed me that there was no "might" about it -- her acquaintances did indeed choose the roller-coaster of hallucination, speeding, fear, and elation.

  5. Hubert Thrunge Jr.

    What could possibly go wrong...

    I am putting myself to the fullest possible use, which is all I think that any conscious entity can ever hope to do.


    “I don't want to insist on it, Dave, but I am incapable of making an error.”

  6. Ken Moorhouse Silver badge

    NeMo Guardrails

    I wonder whether Mike Godwin will claim prior art.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like