Scale up vs. scale out
Scale out exists not because you want to have more storage. It's because storage array controllers and SANs are too slow to meet the needs of high density servers. Storage has become some a major bottleneck that it's no longer possible to populate modern blades and actually expect to have even mediocre performance of VMs because it's like running a spinning disk on a laptop. It's just horrible.
Local storage scaled out is far better. So internal tiered storage works pretty good. You get capacity and performance in a single package. It doesn't scale up as well as a storage array... unless you buy more blades. Instead, it's pretty damn good for trying to make sure that your brand new 96 core blade isn't sitting at 25% CPU usage because all machines are waiting on external storage to catch up.
Scale-out in a SAN environment is just plain stupid. Even with the fancy attempts by some companies to centralize NVMe which is performed using PCIe bridge ASICs, the problem is that you'd need to have dedicated storage centralized for each blade to make use of that bandwidth. Additionally, NVMe is quite slow. NVMe generally only uses 4 PCIe lanes. Using local storage, I can use 32 PCIe lanes which is a small but noticeable improvement.
Scale up is still quite useful. Slow and steady wins the race. Cabinets that specialize in storing a few petabytes are always welcome. You really wouldn't want to use them for anything you might need to read, but an array that can provide data security would be nice. So, maybe Netapp should be focusing on scaling up instead of out. Cluster mode was kinda of a bust, it's just too slow. 8 controllers with a few gigs of bandwidth each don't even scratch the surface of what's needed in a modern DC.
Biting the hand that feeds IT
