Fooled by 'black boxes'?
I shall draw an analogy between a simple statistical analysis tool and current incarnations of so-called 'artificial intelligence' (AI). Each exemplar can be, indeed commonly is, used without understanding its internal mechanism.
Multiple linear regression (MLR), together with variants on the theme, is a powerful tool for aiding analysis of numerically representable data. The data may consist of what happens to have been collected (e.g. for a routine information system), or may be assembled with specific intent to elucidate a research question in the context of a designed study.
In essence, multiple linear regression seeks 'structure' or 'specific patterns' among data; this not in a nebulous sense, but rather as imposed by the analyst. Some structure is predetermined by the manner in which the data were collected: a designed study may have sampled individual 'subjects' by fixed quotas from strata e.g. people according to sex. Predetermined data structures acknowledge the possibility of relationships/similarities within groups (defined by some characteristic e.g. sex) that may differ between groupings, and should this be ignored may lead to erroneous conclusions.
Imposed atop the inherent, and/or anticipated, structure, mentioned above, is speculative structure and relationships among data arising from study hypotheses. Added to the mix is consideration of measurement errors and random fluctuations; these might give the impression of non-existent (in the wider world from which the data were sampled) relationships or may fail to convincingly detect true relationships of interest; the latter often the consequence of inadequate study sample size.
AI 'training data' bears analogy to the dataset for MLR. Each is drawn from a large potential pool. MLR analysis consists of multiple passes through the data using a straightforward to understand computational procedure; each step ideally determined by the analyst; a veil is pulled over the rampant misuse of statistical packages (worthy of themselves) by people with tenuous grasp of statistical analysis. Analogy continues regarding pitfalls from drawing inferences from a model extrapolated beyond the range of the data from which it was derived.
Bear in mind, some instances of MLR are pragmatic/empirical in use. They may give reliable predictions of their outcome variable for a host of combinations of the 'independent' (predictive) variables so long as the 'dependent' (predicted) variable and the independent variables are within the set of MLR 'training' data. Yet, note that statistical associations present in the finally adopted (most parsimonious) model cannot be assumed to denote cause and effect. No insight is offered into mechanisms. Yet, MLR and AI can be starting points for planned investigations seeking to reliably answer specific questions.
Analogy fails when AI models have open-ended training sets. Also, the discipline for selecting training sets, for choosing extensions to sets, and for evaluating success of training is, putting it kindly, in its infancy.
The internal working of an AI is 'black box' to everybody. That includes people at the forefront of developing the technology. They presumably have strong insight into what happens in specific instances of simulated 'neural nets' (and similar) when fiddled around with at small scale. They can tweak 'learning' (memory?) features and look for improvements (the how?, we shall skirt around). However, a scaled-up AI put to serious tasks is truly a black box to everyone. Returning to MLR, every step of every procedure, even when applied to huge sets of data, is open to scrutiny by a statistician or mathematician.
Of course MLR and AI differ in conception, ambition, and in computation. I'm not setting them up in competition, but do suggest there are epistemological issues common to each; in the context of MLR these are easily understood. AI, at least for the present, is mired in handwaving, misleading analogy to human neural function, metaphysical speculation, and unjustified notions that AI models can transcend their initial training data, then arrive at inferences from traceable information which are justifiable by chains of logic, and that this betokens 'understanding'.