Machine learning models leak personal info if training data is compromised

Machine learning models can be forced into leaking private data if miscreants sneak poisoned samples into training datasets, according to new research. A team from Google, the National University of Singapore, Yale-NUS College, and Oregon State University demonstrated it was possible to extract credit card details from a …

[Filippo]

The thing is, NLP models are the ultimate black boxes. We don't have any way to apply binding constraints on a model's behavior. if a model has been trained on something, you have no way to prove with mathemathical certainty that it won't reproduce it eventually. And being "almost quite sure" that your model won't tell everyone my private data is, simply, not good enough.

The best way to defend against this kind of attacks is to not train models on personally identifiable information. At all. Like, any model that has any PII in its training set should be legally considered a derivative of that PII, with all that entails.

This makes training them harder? Tough luck.

[Anonymous Coward]

Spot on

This hits the nail on the head - a model is, by design, meant to represent the training data. So if there is PII then the model has it in - by design (and should be beholden to relevant legislation covering these areas).

[register536]

this is precisely what Differential Privacy provides though -> robust mathematical probabilities that the unique datapoints cannot be distinguished

[Cuddles]

I wonder

How many training sets are in common use that haven't already been poisoned? There seems to be a new article every week or so on how easy it is to screw with ML training data, and it seems an awful lot of people use a limited number of shared public datasets because actually gathering your own data is both difficult and often illegal. What are the chances that no TLAs or criminals (but I repeat myself) have put two and two together? Even if it's difficult to poison a dataset in advance without knowing the precise result you're going to want in the future, you'd have to assume they've done it even if only as practice for future operations. Any dataset lacking a full chain of custody for every element has to be assumed to be riddled with backdoors. Even if they're not implemented in a way that's useful for a malicious actor, how can you trust any results when any random query could hit some hidden trigger?