Ensemble neural networks are still separate models
While the architecture of Pathways is extraordinary and to be applauded, I think there is a leap taken here:
> Pathways will enable a single AI system to generalize across thousands or millions of tasks, to understand different types of data, and to do so with remarkable efficiency – advancing us from the era of single-purpose models that merely recognize patterns to one in which more general-purpose intelligent systems reflect a deeper understanding of our world and can adapt to new needs
That's not quite true. In ensemble neural networks, you have separate models each trained to be an expert in a single thing. You have a language model, an image recognition model, etc, each are called an "expert" and thus an ensemble of experts. When an input is presented to the trained network, it picks the expert that is most appropriate and shows the input to that one. That network then performs its analysis and gives an output.
From the outside, it's sort of general purpose because you only need one system running to do different things but it's not really general purpose from an AI perspective because inferred knowledge is trapped in each of the expert silos. You can tell that this is the case because truly general purpose AI has the problem of encoding inputs from different domains - how do you represent an image in numbers in the same way that text is so that knowledge can be inferred? That's a *very* hard problem that this paper doesn't address, so I'd draw the line at calling it general purpose.
For current deep learning, this is very cool indeed.