HPC teams already use heterogenous hardware mixing x86 with GPUs because x86 hardly shines at parallel vector work.
Well, it depends on the workload. Xeon Phi is quite a big beast, and we'll suited to some workloads. As ever, it depends.
GPUs are problematic for some workloads. Their downfall is latency; they're (still) all about loading up some data, doing a lot of math very quickly, and then unloading the results. For some problems this is less than ideal. Machines like RIKEN's K is very impressive because they did so much to reduce data sharing latency in the machine, which gave it an unparalleled peak:mean performance ratio.
ARM chips already come with optional hardware acceleration packages, throw in FPGAs and GPUs and, at the right price*, the HPC crowd will be drooling.
Drooling, but facing a massive code rewrite!
If you look at some the biggest HPC installs it's obvious that purchase price is not that important.
That's mostly because the chips they use are the same (more or less) as gamers / server farms use.
It costs Intel around about $6billion to do a step in their design, and it's about the same for everyone else doing circuits that complex and fast (be it GPU, Ethernet switch, whatever). If Intel stops bothering, or if NVidia give up because we're playing games on phones instead of PCs or consoles, the HPC community would have to bear the cost themselves. The cost is enormous.
The only reason NVidia engaged with the HPC community in the first place was a reduction in PC sales.
Biting the hand that feeds IT
