I remember the time when different CPU and systems architecture were good for different applications and type of workloads.
If you wanted good performance on the SGI NUMA machine, just use have of the CPU's and make sure you ran one process on each 2-CPU part. If you wanted it to perform really bad, then make sure you ran more processes than you had CPUs, and after a while all processes where running against remote (slow) memory. On a Sun Starfire (E10000) you had a very small slowdown when overloading the system.
Different CPU architectures where good for different applications and of course correct use was critical. I remember one HPC app that ran almost 2x faster on SPARC by changing from standard 8k pages to 4M large pages. And to get to the point, I do think that the default page size is a bigger problem than there being some old instructions laying around unused in the silicon. My first PC had 384kB of RAM, and then a 4kB page size was quite reasonable. But using that same 4kB page size when you have TB of memory - that is a lot of overhead as TLB entries need to repeatedly be loaded from memory. I also have the feeling that many applications are no longer well optimized.
I do hope that we in the future will see more different CPU architectures in HPC. Someone claimed that RISC-V has a lot of MIPS heritage and that might not be that bad - at least that heritage is less of a burden than what X86 has.
