Re: So some fast processors + biggish RAM + bigger chunk NVM with quick sharing by others.
"The idea was that it wouldn't have RAM or storage, just enormous quantities of NVRAM operating in both roles. "
So single level storage. Fine if you can get the price/bit of NVM down to DRAM at the same access speed. As for "large number of processors" SP2's of the late 90's were running up to about 64 K processors and IIRC 100's of 1000s of processors are certainly known today.
As for "be specialized hardware cores" aren't all processors SOC's and so (to a certain extent) "specialized"? If you mean specialized to individual companies servers then that would make each block of machines sold unique to their customer.
"Persistent" storage was available 40 years ago.
It's called "core."
NVM as the main memory for a processor node is somewhat bold. So how does it compare to flash?