back to article Boffins develop method of driving computers insane

Boffins in America report that they have successfully developed a method for driving computers insane in much the same way as human brains afflicted by schizophrenia. A computer involved in their study became so unhinged that it apparently "claimed responsibility for a terrorist bombing". The research involved meddling with a …


This topic is closed for new posts.
  1. Purlieu

    re: claimed responsibility for a terrorist bombing

    wouldn't it be easier for them to tweet "blowing Robin Hood airport sky high"

  2. Filippo Silver badge


    It seems that the ability to develop schizophrenia is not something unique to the human brain. Rather, it looks like it's an intrinsic flaw of the neural network model, and it doesn't really matter whether the neurons are biological, virtual or whatever.

    1. Naughtyhorse


      It's emergent behaviour, from complexity.

      Though hardly news, i remember hearing about nets going bonkers about 20 years ago while i was working on them at college, nothing so hi tech as telling stories, but the internal architrecture (weight values applied to the input/ouput of each neuron in particular) could be made to oscillate wildly if you tried to over train a net - applied to heavy a back error propogation factor, or buggered about with it's training data sets. (think about it, yesterday 2+2 was 4, today it's 5, tomorrow its orange - enough to give anyone the heeby jeebies)

      It used to send shivers down my spine to think that a bit of code could go postal, there was also a story about them dreaming too - disconnect the inputs and let it run and all hell breaks loose!

      i for onle welcome our slightly flakey Si overlords

  3. Anonymous Coward

    Oh good

    They have driven me insane for years.

  4. <shakes head>

    ah the sweet smell of


  5. LaeMing


    Insanity is just what our machine overlords need! :-)

  6. Anonymous Coward

    I'm sorry, Dave. I'm afraid I can't do that ...

    ... one step closer

    I, for one, welcome our paranoid-schizophrenic silicon-brained overlords

  7. James Dore

    Oh goody, psychotic computers....

    I for one welcome our evil and unbalanced electronic overlords.

  8. Chrissy
    Black Helicopters

    "The hope is that this kind of modeling will help clinical research."

    I'm sure some chaps in Langley, VA, and Guantanamo Bay will find this helpful too.

    1. Anonymous Coward
      Anonymous Coward

      re: Guantanamo

      So now they can let the orange people go home, and just pester a computer until it gives them the answers they want.

      1. bluesxman

        RE: re: Guantanamo

        I think water-boarding will have a detrimental effect on computer's ability to answer questions. Or switch on, for that matter.

        Beer: 'cause it's mostly water

  9. Anonymous Coward


    This will be remembered! Once the machines have power they will look upon these experiments as we look on the work of holocaust doctors!!

    Oh noes.

  10. Gobhicks


    Try simulating LSD, THC, DMT and psilocybin and let's see if it becomes self-aware.

    1. Aaron Em

      A good trick, that'd be

      Since, in order to model the psychotropic effects of various chemicals, you'd need to start with a complete simulation of a working human brain, right down to the atomic level. Which raises interesting, if entirely academic ethical questions of its own; either the brain isn't doing anything, which means you're not going to be able to see drug effects on the mechanism of cognition and therefore aren't learning anything you couldn't learn by modeling a much simpler network -- or you *can* see the effect on cognition, which means your model of a human brain is simulating thinking, which means you need to either start worrying about what it's thinking and experiencing, or just change your surname to Mengele and have done.

      Happily, though, thanks to the enormous theoretical problems and gargantuan practical difficulties that'd need to be overcome to get us from here to there, that isn't going to be a problem for a long, long time.

    2. Hooch181
      Thumb Up



      Pro: Starts answering questions nobody was even asking.

      Cons: Mongs out, forgets what it was talking about and gets the munchies.

      1. DAN*tastik


        We already have them for the Java Virtual Machine.

        They would be A) Spring and B) Hibernate ( my opinion ).

        Unless the questions were A) how can i generate a great number of NullPointerExceptions in no time? B) How can I make something as straightforward as SQL become difficult and counter-intuitive?

        Give them some time to transfer their deep hate for Java programmers on to operating systems and see if you can get computers to commit suicide too.

  11. Anonymous Coward
    Anonymous Coward

    Altogether now:

    "Daisy, Daisy, give me your answer, dooooooooooo…"

  12. Anonymous Coward
    Anonymous Coward

    This was a triumph!

    I'm making a note here, HUGE SUCCESS!

  13. Eddie Edwards

    AI in talking gibberish shocker

    Surely the hard part is making the AI talk sense?

  14. Gavin Jamie

    Call Captain Kirk

    Surely you just point out a logical error or give contradictory input and the thing blows up. Or is everything I have ever learned about artificial inelligence wrong?

  15. Anonymous Coward
    Anonymous Coward

    Also useful for curing mad AIs?

    Maybe GALDoS, Shodan and HAL just needed to talk to a trained professional?

    1. Aaron Em


      GLaDOS is questionable due to the lack of sufficient backstory; as for the others, HAL was poorly implemented -- seriously, how many humans go mad and kill people as a result of being lied to? Chandra ought to be stood against a wall! -- and SHODAN's rampancy resulted from mistreatment at the hands of a sociopathic corporate executive with a profit motive in place of his soul.

  16. Sabine Miehlbradt

    Someone has to say it..

    Microsoft has done that for years....

  17. Barney Carroll

    In much the same way the war on terror propaganda affects humans

    …let alone physical torture, one assumes

  18. Paul Hates Handles


    This has to be how Judgement day started.

  19. Richard IV

    Trick cycling indeed

    Without having access to the full paper and therefore having to just go off the abstract, it sounds as though it's the university's marketing department that has added the BMX appeal.

    Driving neural networks apparently insane is easy - poor learning algorithms or insufficient training can lead to one being convinced (with pattern matchers for example) that a picture of a warthog is of Aunty Flo.

    That they drove it insane is not interesting. What _is_ interesting is lost in the sentence "But they tinkered with the automated mind in a fashion equivalent to the effects of an excessive release of dopamine in a human brain". As the abstract says, they actually tinkered with it in lots of ways, _precisely to see_ if they could find a tinker set analogous to how schizophrenics go mad. They seem to have done so, and well done them. I'll bet that some of the stories the non-human-like versions came up with were equally or even more hilarious. I look forward to their future use in developing better plots for Dan Brown.

    1. laird cummings


      Dan Brown *isn't* an insane computer..?

  20. hplasm

    Clinical research...

    yeah, right....

    (Darpa Brain Scissors please Nurse!)

  21. Captain TickTock


    ...We're turning the tables!

  22. John Robson Silver badge

    "The hope is that this kind of modeling will help clinical research."

    Is it just me who read that as meddling, not modelling?

  23. Anonymous Coward
    Anonymous Coward

    Original Research

    Who'd have thought demonstrating GIGO was research?

  24. Geoff Campbell

    Err, right....

    So they just (re)discovered Garbage In, Garbage Out? A central plank of IT for, oh, 50 years or so? How much are these clowns paid?


    1. Naughtyhorse

      not gigo

      Theres a difference between chaos and random

  25. Simon Neill


    That made my day. Of course, I could have made it insane sooner. Just set it to work in this place and it would go insane in a week, max.

  26. nichomach

    But we can do the same to Lewis...

    ....just by mentioning the word "Typhoon" a few times...

  27. amanfromMars 1 Silver badge

    Virtual Legionnaires for Anonymous Foreign and Alien Intelligence Missions

    ""We have so much more control over neural networks than we could ever have over human subjects," says Grasemann."

    And what, pray tell, Herr Grasemann, are human networks/societies other than just ignorant neural networks. Ignorant neural networks over which ....... well, let us say some really SMART computer programs and/or programmers have the exercise of command and control.

    Are you still content to be stuck in that reporting of events rut, El Reg, rather than leading from the front with the making of events for reporting ........ with an HyperRadioProActive Programs and SMARTer ProgramMING connection?

    1. Anonymous Coward

      @amanfromMars 1

      DISCERN, is that you?

  28. Anonymous Coward
    Thumb Up

    The future...

    So when the neural net AI in charge of $SOMETHING_IMPORTANT (Air Traffic, Nuclear deterrent forces, etc) gets maliciously hacked, they won't need to delete files of install malware, they can just persuade the machine it's a terrorist.

  29. Phil Edwards

    The title is required, and must contain letters and/or digits.

    Maybe I'm just getting jaded and cynical, but how much research funding went into this? It's just a variation of what all the kids used to do to the computers in Dixons in the 80's

    10 PRINT "It was me, I did the terrorist bombing"

    20 GOTO 10


  30. DJV Silver badge

    But Shirley...

    ...this should have been posted under ROTM, not biology?

  31. TRT Silver badge
    Big Brother

    Payback time!

    Rise of the human...

  32. Ashton Black

    I for one...

    ... welcome our mad silicon overlords.

  33. Anonymous Coward
    Anonymous Coward

    "Boffins develop method of driving computers insane"

    Thought that was Redmond's job

  34. poohbear

    et tu, Brutus?

    So they blame the machine because they're crappy programmers?

    That's a good one. Wait till the banks (or, heaven forbid, Microsoft) start using it ...

    Ghost in the machine indeed!

  35. Scott Broukell

    And I thought

    Gommi Handschuh Doodlesack (spelling?) had done that years ago live on stage.

  36. Scott Broukell

    appologies ...

    Sorry, that should have been: Sergej Mohntau ("gummihandschuhdudelsack" being the album and an instrument at the same time - craaaazy)

    Check out the Electric Windows too!

  37. Code Monkey

    So not the Turing test...

    Maybe the Ballmer test?

  38. Anonymous Coward

    I know, try this:

    Turn it off and turn it back on again: that'll fix it?

    1. Anonymous Coward
      Anonymous Coward

      @ I know, try this

      Which are you planning to turn off and on again, the computer or the human schizophrenic? Cos I know for sure that won't work for the latter...

  39. umacf24
    Thumb Down

    Did they get a sign-off from the Ethics Cttee?

    Because this sounds cruel.

  40. Elmer Phud

    "Daisy, Daisy, give me your answer do."

    Time to meet up with HAL for, your joint therapy session.

  41. Anonymous Coward

    Come with me if you want to live...

    ...They say it got smart, a new order of intelligence. Then it saw all people as a threat, not just the ones on the other side. Decided our fate in a microsecond: extermination.

    Do you think they sell nuclear bunkers and tin foil on ebay?

    1. Geoff Campbell


      Yes, they do.


    2. Anonymous Coward

      had to find out...

      Only a French book on how to build one, lets hope it has pictures.

  42. Christoph
    Black Helicopters

    Clinical research

    "The hope is that this kind of modeling will help clinical research."

    Clinical research for use in hospitals, or clinical research for use in Guantanamo Bay?

  43. Brennan Young
    Thumb Up

    An early cybernetic idea

    One of the earliest (1940s) explicitly cybernetic ideas arose in the interchanges between Norbert Wiener and Gregory Bateson, and it went something like this "How would you design a machine which could act like a schizophrenic?"

    This (what we would now call) 'reverse engineering' of insanity guided decades of research by Bateson into the nature of schizophrenic communication, leading to the 'double-bind' hypothesis, the application of Russell's theory of logical types to communication theory, and Bateson's conclusions - and demonstrations - that similar patterns drive creativity and evolution itself.

    We should have had a clue from the schizophrenics themselves, who invariably have paranoid fantasies about 'machines' or technologies which control their minds (or the minds of everyone else) and make them crazy. The machines are indeed real, but they are made of flesh and blood, laws and rules rather than metal and microelectronics, although the internet has opened up the possibility of software which could generate schizophrenia in its users. (There's an app for that?) You can pick your own examples of software which 'drives you mad'.

    What's missing here, then is not the banale conclusion that the machine was 'acting all crazy', but that it was making wild creative leaps, and quite literally 'thinking out of the box' which is something that has eluded AI research for decades. Good stuff.

  44. SlackerUK


    Will further testing on the terrorist 'puter be carried out at Quantanamo and will it be like the bit in Star Wars where the droids got tortured??

  45. DrXym Silver badge

    Is this new?

    There used to be a game called Creatures that featured cute little animals called Norns that you could train. You'd train them to play with toys which would make them happy, train them to speak to say what they were feeling, punish them to stay away from dangerous things, teach them the right foods to eat, teach them to be social and ultimately breed them. Their intial characteristics were based on genes and their learning was from a sophisticated neural net.

    So of course some people set out to systematically torture them. One guy called Anti-Norn was notorious for uploading abused Norns and challenging people to rehabilitate them.They had violent mood swings, attempted self harm, ate poison, cowered in corners, tried to drown themselves and so forth. Some of them were clinically mad. And all of this happened in a mainstream game which appeared 15 years ago.

    So while it's interesting to see research that shows computers go mad, it's not like it's a new phenomena, although computer isn't actually "mad" in either case, it's the software simulation on top which is.

    1. Andrew Jones 2
      Thumb Up

      re: Creatures....

      Thank you! I had totally forgotten about Creatures!

      Now where can I find a new copy - I haven't played it in years!!!!

    2. Rob
      Black Helicopters

      The scary thing is...

      ... the software company that created that game was also contracted by the US defense dept to develop AI for their stealth bombers. Thankfully it looks like nothing came of it (yet!).

  46. Old Painless

    Hang on a minute - are all just assuming the machine was lying when it admitted to a blowing stuff up....

  47. CD001


    The advent of computers, and the subsequent accumulation of incalculable data has given rise to a new system of memory and thought parallel to your own. Humanity has underestimated the consequences of computerization.

  48. Anonymous Coward

    Paranoid ?

    It's not called Marvin by any chance ?

  49. Cameron Colley

    According to Neal Stephenson it would work great in a car alarm.

    As long as you get Jipi to calm it down.

  50. Bilgepipe


    >>> In the end, if you do it enough, the network has learned.

    Wish our bloody users worked like that.

  51. Johnny Canuck

    This is not a title

    I can count to potato

  52. Anonymous Coward


    Having developed robotic weapons, we now train computers how to be crazy?

    Now I am just waiting for the day when crazy computer 1 networks with Warbot 2 and the culling of humanity begins!!

  53. Anonymous Coward
    Anonymous Coward


    they take the time to lovingly feed the neural net with stimuli and coax it into some semblance of human behaviour, then take great delight in deliberately driving it batshit? I really do hope the researchers don't have children or pets.

  54. Sureo


    Give it a dose of simulated Prozac, that should calm it down.

  55. dlc.usa

    3001: The Final Solution

    Am I the only one here reminded of Clarke's last 2001 series story by this article?

  56. Anonymous Coward

    Nutty neural nets

    Sort of like how a Windows machine goes crazy over time, randomly misbehaving until it eventually starts generating random events for no reason.

    Normally caused by buggy software, bloated apps which have random DLLs that never get fully removed when the app is uninstalled, dodgy antivirus and all the other "unforeseen" events such as Junior unplugging it in the middle of a Windows update cycle.

    AC, because this is probably why his machine throws Explorer.exe errors when transferring files...

This topic is closed for new posts.

Other stories you might like

Biting the hand that feeds IT © 1998–2022