back to article Creep learning: How to tamper with neural nets to turn them against us

Bit boffins from universities in China and the US have devised a way to tamper with deep learning models so they produce misleading results. In a paper posted to pre-print service ArXiv, "PoTrojan: powerful neural-level trojan designs in deep learning models," authors Minhui Zou, Yang Shi, Chengliang Wang, Fangyu Li, WenZhan …

  1. Harry Kiri

    There's a lack of understanding

    About neural networks, what they are, what / how they 'learn', what they do and what the limits of their performance are. I really thought this was all understood 25 years ago, but a new generation has re-sprung nnets into magic and 're-discovering' basic properties.

    And while we're at it 'researchers' should also understand how the front-end features contribute to the overall classifier performance too.

    1. Anonymous Coward
      Anonymous Coward

      Re: There's a lack of understanding

      Yes, we knew a lot about neural networks back then but the problem was that we didn't have the marketing wonks to put the spin and buzzwords on it. Now we do have them and the MBAs so it has to be learned all over again.

  2. TRT Silver badge
  3. Cuddles Silver badge

    Not really news

    If someone purchases a black-box system with no knowledge of how it was produced or how it works, the person who created that system could have sabotaged it in some way. The more complex a system is, and the more inputs and outputs it's expected to handle, the more difficult it is for the user or other third parties to test. That has nothing to do with machine learning, it's true for literally everything.

  4. Gordon 10 Silver badge

    Researchers discover how to hack an ML model

    ... by having write access to the files that make it up. Popes and woods come to mind.

    I wonder if they would have got the research grant if they phrased it as I did?

    So in summary - you need write access to either the memory or disk where the model code is stored. In fact lets call it the "Application". You need write access to the Application space - and possibly the OS space on a good implementation.

    Lets take a IRL example - say the iPhoneX

    I have full write access to the OS level Application partition where the FaceID ML model is stored. I have reverse engineered the FaceID ML model enough to understand how to tweak it.

    Do I

    a) Only allow my ugly mug to unlock the phone.

    b) Rampage through the whole OS looking for a much higher value target.

    1. JeffyPoooh

      Re: Researchers discover how to hack an ML model

      Actually, recently on this El Reg, it was described how the "hackers" can imperceptibly modify the input stimulus to cause the AI output to be slammed to the pegs (producing idiotic output). Examples were given of the image recognition case (chrome toaster and banana), as well as audio.

      So the hackers only need to borrow a system, or buy an example from eBay. Then experiment with it endlessly (i.e. about two weeks) until they uncover a built-in weakness. Then they use this to attack in the wild against the stock system, unmodified. They don't need to touch or modify the actual system to be attacked in the wild. And their example is only "examined" and "tested", not modified.

      [Mythical-example-for-illustration-purposes Alert] So they walk up to the bank vault at 3:00am holding large spinning pink and yellow lollipop distraction disks over their heads, the AI gets into an unforeseen state, an output goes High, and the bank vault immediately opens for the bandits.

      That specific example is unlikely, but it's unlikely multiplied by infinite possibilities. So this sort of nonsensical attack will occur instantly (about a month) with the seemingly-inevitable widespread naive deployment of AI with "hidden" (undefined) layers. It's my opinion that this is all crystal clear, so it mystifies me how the "AI Boffins" or their Press Office can't see this coming.

  5. Ian.G

    The backprop network is hungry...

    Keep feeding me data and I'll tell you the answer is - with a high degree of certainty - maybe/maybe not.

    I remember my 5 neuron backprop network running on 486 DX2/66 back in the last century. You train it with weather data for a few hours hoping the CPU doesn't overheat and it could accurately predict there would be 'some weather' on a given day. You forget to save your training data and have to do it all over again - doh!

    With all this compute power we can have a vastly more accurate maybe now. Think of all those neurons in the fluffy clouds...

    1. DCFusor

      Re: The backprop network is hungry...

      Quantity is necessary but not sufficient for quality.

      Having the correct metrics as inputs might matter along with the rest as well.

  6. ITS Retired

    So, if I understand things correctly...

    This is another step towards machines thinking just like humans.

    1. Ian.G

      Re: So, if I understand things correctly...

      You could argue that computers have been more 'intelligent' than quite a lot of humans for quite some time. Computers don't appear to be compelled to re-invent the wheel - having said that - you could probably build some pretty cool AI (ok - lets just call it an algorithm) that forks npm projects and makes subtle changes and creates new components from them. Create some industry buzz and off we go !

      What can you make with 475,000 building blocks (and climbing) ? probably a total mess.

      1. JeffyPoooh

        Re: So, if I understand things correctly...

        Ian suggested that "computers have been more 'intelligent'..."

        The human brain contains a myriad of 'hardware' co-processors. The human brain is clearly NOT just a big silly neural net to be trained. It's got plenty of odd structures, producing instincts, emotion, empathy, and some common sense.

        If you installed legs onto IBM's Watson and took it outside, it would immediately run away and be run over by a bus. Or it would wander over a cliff while explaining the formulas and relationships involved with gravity. It wouldn't last ten minutes.

        Famously, "A.I. is hard." Where "hard" means 'very nearly impossible'.

        Strong A.I. is Really Hard.

        And 'Strong A.I. Outdoors' is going to be alternately hilarious and deadly.

        1. Ian.G

          Re: So, if I understand things correctly...


          To 'mimic' what a human does in a repetitive physical task with some defined scope for correction is one thing. To 'understand' why the task is being performed in the first place and the implications of that task is a totally different thing. Can the algorithm truly 'feel' dissatisfied and fatigued from carrying out the task and ponder why it's being done in the first place, show instinct and make the task more pleasurable? or simple stop from the futility of it all?

          Myna birds can 'mimic' human words - but its near impossible to say if they 'understand' the meaning of a word (or what a word is) and what context it should be squawked in. This is more of a task/reward scenario - squawk what sounds like a word to make the human happy and the outcome is a tasty treat. When you train a neural network should it demand a treat?

          Can an AI Chess program/algorithm that repeatedly beats a human player (it's ultimate goal) sense despair from the human moves being made, show instinct and empathise with the human and occasionally let them win for a better shared experience? Sure - an algorithm can track win counts and target a threshold of balance but there is no feeling or emotion involved - just pure logic.

          Can AI exhibit a 'sense' of morality (to be fair a lot of humans struggle with this one) in a situation? The moral dilemma of crashing an autonomous vehicle with 3 occupants to spare a 30 car pileup resulting in a higher death toll? are the occupants more important given they are the vehicle owners?

          In the meantime warehouse staff are being taught to behave more like robots which is somwhat ironic.

          If Watson hits the streets with legs - its time to get to the panic room LOL.

  7. The Man Who Fell To Earth Silver badge


    It's not like this hasn't happened before.

  8. JeffyPoooh

    "...multiple hidden layers..."

    "...multiple hidden layers where mathematical computations occur..."

    The technology concepts of Google Deep Dream more or less makes those "hidden" layers more or less visible. It's not off the shelf, but the concept of bringing those hidden layers to the surface is quite obvious.

    Deep Dream is visual, but the same concept could be applied to any application.

    We went over this obvious connection a couple years ago, when Deep Dream was in the press. There's really no excuse for "hidden" layers today, and certainly not for any safety critical systems.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021