How to hide a backdoor in AI software – such as a bank app depositing checks or a security cam checking faces

Boffins in China and the US have developed a technique to hide a backdoor in a machine-learning model so it only appears when the model is compressed for deployment on a mobile device. Yulong Tian and Fengyuan Xu, from Nanjing University, and Fnu Suya and David Evans, from University of Virginia, describe their approach to ML …

[Long John Silver]

Would you trust a stranger to make sensitive decisions now delegated to AI?

'AI' is a 'black box'. What goes on inside and how or why particular outputs arise is pretty impenetrable to persons designing and to those training neural networks. Designers at the level of coding would be little more able to give specific meaning to a given set of weights than anyone else.

Bear in mind, it is the weights assigned by 'learning' that have closest analogy to what human software coders do. The weights are the program. Code specifying an untrained neural network and transaction protocols among 'neurones' is better regarded as equivalent to background firmware; few ordinary programmers need delve into firmware code.

Present day AI has become established as an heuristic tool of value in circumstances involving assessing and classifying complicated patterns within data submitted to the AI. However, current AIs offer no insight into how they arrive at results. They can (supposedly) reliably draw inferences and make prediction, each within their realm of operation, but they cannot explain underlying 'reasoning'. That would necessitate a higher order of functioning wherein not only incoming data is processed but there is an introspective mechanism for examining some currently assigned weights as if they too are data; this loosely called sentience.

[amanfromMars 1]

Re: Would you trust a stranger to make sensitive decisions now delegated to AI?

'AI' is a 'black box'. What goes on inside and how or why particular outputs arise is pretty impenetrable to persons designing and to those training neural networks. ..... Long John Silver

Oh? And there I was thinking the exact opposite in order that things can continue to work and bamboozle, just as "AI black boxes" plan it, and expect it to be so.

Is it wise to trust a stranger to make any sensitive decision previously delegated or made by anyone or anything else, anywhere else? It is impossible not to realise such is a crazy gamble unless one knows what the stranger can do, of course. That then transforms it immediately into a surefire bet and worthy guaranteed future derivative option for marketing and capitalising.

However, current AIs offer no insight into how they arrive at results.

Would you spill the beans and cast the pearls of the secrets of crazy fantastically rewarding magic before swine, LJS, or realise there's certainly no earthly need, for such only invites madness out to play to harry and fail to hinder Surreal Sterling Stellar Stirling Progress.

[Pascal Monett]

"consider a bank that is building a mobile app to do things like process check deposits"

Any bank that uses today's pseudo-AI to process my money is a bank I am going to leave.

Numbers are not subject to interpretation. OCR is good enough for recognizing what's written on a check, and when the code gets the numbers, there is no fuzzy logic needed, nor any AI required to know what to do. It's done already. No bank is going to replace what works with something that might go wrong.

That is a stupid example.

[jmch]

Re: "consider a bank that is building a mobile app to do things like process check deposits"

"No bank is going to replace what works with something that might go wrong."

I wouldn't be so sure. They wouldn't replace an OCR that works with an AI that probably works almost all the time, in the first place because the OCR is cheaper. But say it wanted to replace something semi-sunbjective, such as whether to accept a loan application based on a number of different criteria known about loan and loanee. Currently that's done by (relatively expensive) humans, so if bans see savings potential they'll go there

[Anonymous Coward]

Re: "consider a bank that is building a mobile app to do things like process check deposits"

If your check amounts are printed, OCR works for the front but not for the back where you endorse it. If they are handwritten, OCR can only pick up the preprinted part of the check and some form of AI is needed to process the handwriting.

[amanfromMars 1]

Prime Directive Raison d'Être

No bank is going to replace what works with something that might go wrong. ..... Pascal Monett

Maybe so, Pascal M, however banks and/or AI are not simply replacing something which works whenever possibly trying out something new and capable of generating greater interest and delivering the holy grail ...... colossal profit/massive practical reward for virtually nothing extra added.

Systems normally just love that and those able to successfully stably provide that kind of heavenly magic without suffering its sinister and self-destructive, spell-binding charms.

[katrinab]

I assumed that they would send the image to the server and do the processing there?

[Anonymous Coward]

2nd Best Engineer

Wow, working away and just watched Wallander on Netflix, very similar theme as to the trigger.