Machine-learning models vulnerable to undetectable backdoors: new claim Boffins from UC Berkeley, MIT, and the Institute for Advanced Study in the United States have devised techniques to implant undetectable backdoors in machine learning (ML) models. Their work suggests ML models developed by third parties fundamentally cannot be trusted. In a paper that's currently being reviewed – "Planting …
the authors describe a hypothetical malicious ML service provider called Snoogle, a name so far out there it couldn't possibly refer to any real company.
Crikey, you gotta get out more, Thomas Claburn in San Francisco, if you believe that a strange name/company name bears no possible relation to reality ...... or are you trying to tell us, without actually directly telling us, that the true nature of reality is fundamentally strange and far out there.
Now/Then you're making more than just some sense and things can be moved along at an elevated pace in what is indeed a most novel and noble space with no old places in which to hide if up to no good.
:-) Of course, you could also be having some targeted fun at Google's broad shouldered expense whose business model and applications of the results of their algorithmic search engines put them directly in the cross hairs of snipers for any number of earlier established elite executive office operations as the likes of a Google becomes ever more powerful and leading in competition and opposition to such as were formerly thought almightily unchallenged and unchallengeable.
Those halcyon days for those leaders of the past are long gone though and they aint coming back for those key holding players ......
And no matter what is being done and no matter where everything may end up, will there always be a that and/or those way out ahead of the game taking every advantage of that which is being developed and trialed/trailed, remotely virtually mentoring and monitoring live operational plays.
It is just the way everything is ... and is now starting to be revealed to you ...... for both either your terror or delight.
Capiche, Amigos/Amigas? Do you need more evidence?
AI and self-learning systems, just like people, can be feed a load of crap to fool them into doing stuff you want especially if you're into doing naughty stuff like fraud and money laundering! ..... Plest
How very true and convenient/inconvenient, Plest, however unlike most people is any successful threat or promise of being held accountable by any litigious other parties if speculated on as being responsible for doing any naughty stuff, just as a money churning exercise to keep the criminal justice ponzi alive and feeding off the state courtesy of municipal and federal taxes and civil asset forfeiture/arbitrary seizures. ..... https://www.heritage.org/research/reports/2014/03/civil-asset-forfeiture-7-things-you-should-know?
And shared as a question there because there are bound to be those who would never dare agree eg those whose jobs and livelihoods depend upon the ...... well, it's an old style gangster protection racket in essence and practice, aint it. Nothing more, nothing less and ripe rancid rotten right to its core.
Backdoors exist because people with more knowledge (but fewer published papers) want them to exist; just ask the No Such Agency. Or better yet, don't - no good can come from it.
The utility of a machine learning backdoor would depend on the application that is using the ML model. When it offers value, I'm sure our friends at Langley are on the case.
The utility of a machine learning backdoor would depend on the application that is using the ML model. When it offers value, I'm sure our friends at Langley are on the case.....HildyJ
And whenever no value is being offered nor is any evident, what would it tell any friends and all enemies of Langley re their machine learning backdoor utilities/facilities/abilities?
Be honest and imaginative please, your leading progress in such investigative and inquisitive entangled fields depends upon it .
A party that supplies a bank with a TRULY black box that issues yea/nay decisions to bank loan requests would inherently have an unbreakable ability to game the system because they can hide a needle in the haystack via cryptography. That is blatantly obvious.
If it is NOT a truly black box, that's a different story. From the article, it seems the box is assumed to be pre-trained only with known "safe" algorithms, but the exact training data is not known. There seems to be an assumption that the bank will "trust" that only known "safe" algorithms were used, but will be "fooled" because cleverly biased data was used in the training. I guess that "trust" is ensured because the software/hardware is open source and publicly verifiable, up to but not including any learned parameters, and the "black box" applies only to the training data and training recipe (or equivalently the learned parameters).
The learned parameters are opaque, but the bank could purchase the training data and recommended training recipe instead of the black box learned parameters and then pay a separate third party to verify the training data set and apply it to obtain the learned parameters. I predict anomalies in the training set would stick out like a sore thumb, and would be grounds for criminal charges or least civil charges if discovered.
Also I think that even using the same training set with slight variation to the recipe and possibly some added training data would render the result the result unusable for fraud for the part supplying the training data and recommended recipe.
If the intended usage includes adaptive learning with actual usage examples during the course of usage, I don't think the needle in the haystack would survive.
So the researchers made a valid point, but it doesn't include adaptive learning. The lesson is to deal in (training data,recommended training recipes) rather than learned parameters, because the former are not black box while the latter are black box.
So scenario is this: client (e.g. bank) gives provider (big data provider) a set of training data and asks it to give back an AI black box to give approve/deny decision-making (e.g. loan approval).
The white-hat provider does exactly that. The bank feeds the black box some other similar data as validation, and decides whether to accept or reject the AI.
However the black-hat provider twiddles some LSBs in the training data to introduce a backdoor ("if the second- and third-last digit in the application amount equals the reversed day-of-birth, approve the application however bad the credit rating is"). Validation ostensibly happens the same way, but client is deceived into accepting a malicious black box.
What happens when the client feeds the black box the client's own training data (+fuzzing)? The black box mysteriously accepts some bad applications and rejects some good ones.
The client, and not the provider, has the data to decide to accept the first and reject the second black box.
While it's possible that you could detect the backdoor, in practice it would be very difficult to do because the back door can be (and if understand the mathematics correctly), must be a very precise set of parameters. How useful that makes it, depends on the application but I doubt validation + fuzzing will catch it.
From my reading of the maths, it's absolutely correct that a back door can be invisibly encoded into a trained model. However, the back door would have to be a very specific set of input parameters for it to be invisible and that reduces its utility. Unlike a validation step, where you use a separate data set to check the veracity/confidence of the model, a back door would only match on a few specific inputs. For scenarios given above in the comments such as banks/insurance companies, only a specific policy would be accepted rather than a broad raft of policies.
I also disagree with the author's comments regarding unsupervised learning. Unsupervised learning could be back doored but the utility is even lower as in unsupervised learning, you're not trying to force the output to a specific value.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
