Re: "No one really understands why machine-learning code is so brittle"
There was the nice paper that showed that an AI system hyped as successfully interpreting chest X-rays was, for example, identifying 'fluid on the lung' by spotting the chest drains that doctors put in before sending the patient to X-ray when they diagnose fluid on the lung.
At the end of the day hyped AI is way too often saying 'It's black not white' on the basis of a 50.1% vs 49.9% probability that it is black, rather than having the honesty to say 'Dunno guv'