Re: "No one really understands why machine-learning code is so brittle"
I think the premise above in false. Everyone who worked on this for a little will understand pretty well why machine learning models are so brittle. They are nothing else but heuristics trained to recognize a particular correlation. The meaning of heuristic is pretty clear. The correlation is also a hint although more subtle - the fact that something "looks like" a bottle only means that there is a strong correlation between a particular set of pixels and a "bottle" label, nothing else (in particular, it does not mean that the pixels actually show a bottle).
How that correlation was arrived at? By some heuristics. How does that heuristics work, actually? Whoa, back off, we just threw lots of data at it and some got stuck. How did we arrive at the situation where ML is treated as an oracle? A friend pointed at this recently http://backreaction.blogspot.com/2019/03/merchants-of-hype.html
Biting the hand that feeds IT
