as mr wonderful says
Take violin out behind the barn and shoot it
A new phase of disruption is hitting the performance data storage array market, giving new, old, startup and struggling all-flash array vendors a shot at making it big by using NVMe flash drives and NVMe Fabric-class connectivity to provide direct-attached SSD performance from external arrays. The charge is being led by Dell- …
Yes, NVMe is replacing SATA, new gen with reduced write IOPs will be priced at 30-40cents/GB, not the few $ they used to just couple of years ago
BUT having a SAN, any SAN, regardless if it using SCSI (iSCSI/FCP) or NVMeF is not so much the future, just watch AWS or Google, the future is in object, files, and new scale-out databases
pure got it, introduced an All Flash object, DSSD has K/V
next week watch iguazio announcement, a preview, its 2.5M 4KB file, DB, or Object ops/sec per node and under 100us latency, by integrating the DB engine with NVMe & NV-RAM in a way that bypass the entire stack, its architecture is so efficient that it is priced significantly lower than any of the vendors today even AWS.
the layered approach of building databases over file over block/SAN and dedup/compression/caching at the block layer is good for legacy stuff, but than you end up with All flash NAS that can hardly scratch the 200K IOPs and databases that do 20K ops/sec, why compress data in 4KB sectors when i can do it in the DB layer and search encoded/compressed data vs be forced to decompress data pollute memory w garbage and scan the entire data ?
yep, right, it is hard to make this mental change from managing sectors to managing data, thats why no enterprise vendor can go after AWS or Google, they are stuck thinking infrastructure layers, not apps or services.
And those legacy type systems will be around for a very long time yet - even today such systems are still being built as new. The last time I spoke with DBAs about MySQL compression for example the basic suggestion was don't use it. The last time we attempted to use it in a test environment it blew up badly.
You may like new funky DBs that have weird names and are optimized for special edge cases, but the bulk of the world will continue working off of things like Oracle, MySQL, MS SQL and other "legacy" platforms with apps that won't support things like you are talking about for the next decade or more (in many cases they will get along fine without them).
Been working closely with devs on "new" web apps over the past 13 years(across several companies all sort of in SaaS in one way or another) and they still have problems grasping basic things like "single points of failure", or "take the whole app/site down while we make a small schema change". Or let's have microservices, where in many cases there is so many inter dependencies that you are just increasing your points of failure rather than reducing them. Or how about "HEY there are two cpus in this server let's run with more threads to better utilize them, but no the code is too highly serialized and doesn't scale like that..".
I saw one recent app update to a 3rd party component deliver literally an 82% drop in performance(worst case). Can't get the CPU usage above 30% (10 CPUs), vs before could get to 93%. Vendor who makes the software doesn't seem too bothered by the results, or at least not enough to even prioritize a simple investigation into the cause when it's so easy to reproduce.
I remember another time, at another company, in house app - new version comes out - 66% drop in performance due to new serialization in the code. They squeezed an extra 15-20% out of it again(eventually) but admitted that was as fast as it was going to get.
The more I do stuff(and talk with others that do it) the more I realize this is more the norm rather than the exception.
yes you are right, lets look at Oracle, MS/My SQL
they dont use Block, Oracle Exadata is using RDS over IB and Smart Scans
MS SQL is best using their SMB3 distributed file system over RDMA
My SQL is extremely primitive and slow, doing full scan on every query, no column compression, i suggest you look at AWS fork of it (Aurora) does 6x performance and linear scaling, and is using Object as back-end NOT block, or Google internal Spanner SQL DB using yet another distributed file/object
i agree that many of the open NoSQL DBs suc.., extremely serialized, eventually consistent, no security .. many came from university grads, but new ones are coming and they all understand that block doesn't allow scale (distributed locks, journals, ..) and cannot be really shared
specifically if you look at what we did @iguazio, our multi-model DB engine is faster than the fastest block storage around (2.5M ops/sec on 1 server @100us w 99% percentile), our AWS compatible Web APIs deliver 800K HTTP req/sec w a single client/server, its way faster than your fastest ALL FLASH NAS and at fraction of the cost, only way to do it is if you cut through the stack and keep CPUs & mem 100% busy, not by using Linux poor page cache or CPU locks or blocking threads all around the OS stack.
yes, legacy wont go away anytime soon, but it will shrink
not because IT dinosaurs want it to, simply because new companies, developers, and Biz owners understand Cloud and PaaS brings agility and efficiency, and they want to build modern digital apps like ebay, uber, and Netflix, no one wants to maintain Exchange, they rather use Office 365 and do more interesting things w their life.
NVMe is at the moment the only way to use the potential of flash storage (it's clear that electronic even behind a "rotational" interface like SAS is faster that mechanics, but for what? to stretch EOL of storage systems?). However filling up servers with NVMe quickly shows the limitations of the current server architectures. So why should one try to even distribute flash storage over a network that is always by a magnitude slower than the PCIe bus? - useless game. Consequently solutions will become server, respectively application centric. Currently the only answer reflecting this is VSAN. Sorry, good night storage industry. To keep quiet about will not very long help to survive. It's just a question of time until mature customers will leave the old iron behind - extreme quickly when taking into account all the additional benefits of VSAN - an isolated storage never can compete with.