Re: Not at all clear how to decide what is biased
If this was reflected in training sets and this in turn was reflected in the output of an AI trained using these sets would the result be 'biased'?
This. The problem everyone is trying to pretend it does not exist.
The real world is a messy place. All kinds of things are correlated with all kinds of other things -- for reasons good, bad and random. Often the reason is simple: History took a specific course. You don't have the entire multiverse available for training.
The more inputs and outputs, the more odd correlations the classifier will find. And as you say there isn't really any way to sort them out. We would need some automated way, perhaps some sort of AI? Right...
We are always getting headlines about sex and race. But that's just the tip of an iceberg.