So, how does this play with "non-algorithmic" stuff like neural networks? The decisions are embodied in data structures that are not immediately clear to interpret, even to experts, even if they were open to scrutiny.
An interesting discussion, for sure, but also not sure what exactly is to be done about it. I guess being aware of the potential for problems is a very good start.
As far as say the "wrong hiring rule engine", one would hope that customer firms eventually wise up that they are not being advised in the most optimal manner for their needs and shop elsewhere. Though, having seen firms rely on the oddest schemes for hiring advice (handwriting analysis is/was a biggy in France), I wouldn't hold my breath.