Behold this golden era of storage startups. It's coming to an end We are living in one of the most fascinating storage times with a great and rewarding war of storage access latency, but the major gains have already been won – and the scope for future advances is narrowing. A previous storage revolution concerned space reclamation with deduplication, compression and thin provisioning …
And that means the technology is maturing. That's supposed to be a Good Thing (TM). It means that standards are easier to put in place, which in turn allows for more ubiquitous usage of said tech.
Let's put it this way : when gravity modulators can be had for a dime a dozen, it will be sad for all those rockets we won't need anymore, but we'll be regretting them in space.
The elephant in the room here is backups. And DR. None of these adequately solve those issues in a useful way. I wish they did, but that's why they'ret still making LTO# drives, for those who need to move large amounts of data around.
It has gotten cheaper, but doing DR over 100mb/s links when you have 1Tb of changes a day and 60-80ms of latency... not going to happen. Maybe, just maybe, if you have a 1Gbit/sec link you could do that. Assuming you can afford the price.
So how do all these hyperconverged people handle an outage of the entire cluster? Or a fire? Or a flood? It's all well and good until you need to recover it, then you're up the creek. So I think there's plenty of spaces for innovation to happen in the storage arena for sure.
John
For DR there's a couple ways to do it; all require a 'warm' or 'hot' site style of DR implementation and replication software installed at either the storage array or hypervisor level. Some applications are able to do site resiliency (Exchange and Active Directory being one I'm personally most familiar with) which is more toward the 'hot site' end of the spectrum.
Done right, it can be done efficiently across a 100 MB link.
For data archival and data corruption issues, I've still been a champion of tape for it's simplicity and durability. (I can 'accidentally' drop an LTO tape off my desk and still ready it afterwards- can't do that with spinning rust, and an SSD in that size is still too expensive for holding backup data.)
I'll echo the sentiments of some of the other comments, however: Companies tend to hang on to every bloody bit of data they make, regardless of if it's still relevant. At some point, you have to let it go.
So how do all these hyperconverged people handle an outage of the entire cluster?
Can only speak for VMware's VSAN but...
1. Fault Domains (Split cluster across cabinets or data center pods).
2. Stretched Cluster (VMware HA fires off VM's on remote site)
3. vSphere native Replication with compression and SRM to orchistrate the failover.
4. One of the 100x other products that does virutalization backups or replication...
I'd argue Physical disasters (That are limited in local scope) can be handled by HCI and replication systems. I'm not arguing HCI is the solution to everything and some idea's for DR are kinda silly (Push backups to AWS seems to be a weird one that people think can quickly be turned into something useful).
1. storage will no longer be addressed as blocks -- it will be addressed as a map, much like memory. this means the entire existing storage access paradigm must change.
2. storage must become informed by the application(s). this is not "putting the workload closer to the storage" a la nutanix. It is being able to model the workload in real-time and adjusting the access dynamically.
The result of these 2 capabilities will enable new forms of applications, especially in the emerging area of deep learning which every business application will rely upon.
Right, it is enough to look at the public cloud trend, block storage adds complexity, and doesn't solve the fundamental app perf problems, the new front is data APIs and management
We have Mega IOPs storage, with Databases doing 10k iops and 10s of ms latency
The next wave is collapsing the stack, serving high level APIs at the speeds of NVMe and NVRAM by stripping out redundant layers, chatter and serialization, moving data search and vector processing to the storage, securing/monitoring data not LUNs, managing it all as a PaaS
That is a much harder problem to solve, fewer startups and some cloud giants, but stay tuned for iguazio announcement & demo
Yaron
I'm going to pitch what's probably going to be a very unpopular sentiment, especially to people in the storage field, but...
I think the next truly significant advance in storage would be companies finally realizing they don't need to keep so much damn data around in the first place, and to stop wasting so much money on procuring the storage gear to hold it.
I used to think that as data grew, more and better tools would be developed to help mine it. But that never happened. So now, the haystacks keep growing, but the needles in them get harder and harder to find. If you can't make meaningful business decisions from all that data, decisions that directly result in increased revenue or decreased costs, then what's the point? It would be better to stop spending so much on storage, stop keeping so much useless data around, and invest that money in other areas of the business.
Back in the 90's, I worked with an IT manager who refused to implement quota controls on his users' home directories because he said it was cheaper to buy more storage than manage it. I always thought that was stupid and fiscally irresponsible. I can't help but feel even more so in today's world where companies spend millions/billions on storage that, to me, is little more than a money pit.
I second that. I work for a software defined storage company, and the single biggest problem we face is scalability. Controlled lab environments with small numbers of systems and disks work great, but once you scale up to hundreds of systems you begin to see the inefficiencies of the code, and seemingly trivial I/O tasks suddenly become major bottlenecks.
Every time we solve a problem related to scalability, another client pops up seeking to push the boundaries further. And what for? Do we really need all that data?
I look at Microsoft's $26 billion purchase of LinkedIn as a sign that the big data market has gone insane, and is going to collapse, hard.
Well, I agree and we have seen this "too cheap to manage" argument in other areas. Remember when the proponents of nuclear power for generating electricity said it would be too cheap to meter? How many people actually got free electricity generated by a nuclear reactor? Today, people think cloud storage will essentially be free and you will have as much as you can ever possibly want. Somehow I just don't think it will work out that way. There is always a cost involved in storing and preserving data. The question that needs to be answered is how much of the existing stored data will have any social, economic, scientific or cultural value in 10-20 years? The answer is probably only a small fraction of it. The mountains of data being generated as a result of people tapping their fingers on their smartphones will have a pretty short half-life before it is not worth the storage it is occupying in some cloud bit barn. Data storage is not an infinite resource. People will eventually need to determine what data is worth keeping and for how long.
"Back in the 90's, I worked with an IT manager who refused to implement quota controls on his users' home directories because he said it was cheaper to buy more storage than manage it."
You may be right, but I agree with the IT Manager.
I used to implement quota's too, but in practice people are going to hit their quota and call IT that they can't save their (important) document. Then either a whole debate is going to happen or the quota will just get extended. Often people don't understand file sizes, so a lot of people will start deleting many small files just to hit the quota again real soon. Having to explain everything multiple times a week isn't much fun and a real waste of time.
If there are budget issues then it is better to bill the costs to the individual departments and have them sort it out. But that raises the issue that people don't understand why enterprise storage is so insanely expensive since the huge disk in their home computer is so cheap. Also every department has different requirements so they might feel they don't need the expensive storage system you selected for them. So then you need to get very detailed, and even though you might be right everyone may still be unhappy about it and make your job just a bit more miserable in subtile ways.
So in the end it's often best to just buy more storage and pass on the whole debate. Nonetheless, the arguments you bring forward do come up in my mind every now and then but I quickly supress them when I think of the effort it'll be.
I'm not saying you should just let everyhing grow out of control if it clearly makes no sense at all, but it makes sense to pick your battles carefully or just educate the few people that are responsible for the most amount storage growth.
Total. Agreement.
I fought with several departments at $company for _years_ to get them to prune stuff down. I finally gave up, primarily because the last network manager bought an absolute shedload of storage which gave us some breathing room. (one of the very few good things the git did during his tenancy)
The problem with quotas is they tend not to be thought through. Someone sets a value for the amount that people's quota should be, and that's the extent of it.
Here's an example. I once worked for IBM and they had quotas set in Lotus Notes. Once you hit that quota you could no longer send emails until you reduced your mail file below that quota. So if you went on holiday, you would sometimes find that people had been sending you huge attachments, so all of those emails which genuinely needed your response on your return couldn't be responded to. You could apply for an increased quota, but that would take days for approval and would inevitably be rejected. So you'd spend hours, or days, wading through your email, removing attachments from less important mails, before you could respond to the important ones. Great way of doing business.
Another thing they never imposed was a limit to signatures. Some people have signatures packed with pointless images, sometimes into the MB range. On their own, not that significant, but when that person contributes dozens of emails to a conversation, where the useful text is a small percentage of what's stored, it can get annoying. And the email tools (Lotus Fucking Notes, of course) had no intelligence whatsoever and couldn't strip out the crap. Not that I've ever found anything else that can.
Anyway, if you want to enforce quotas, you need to do it properly. I've yet to see that happen.
An era started when we chose to network block storage by putting Fibre Channel under the host's SCSI software stack. So SANs and Fibre Channel disk arrays came to be. iSCSI chose to extend this to Ethernet.
As flash storage became cost effective, and found the legacy SCSI stack too slow, we got NVMe over Fabric. Right now these are more extensions of local disks (think networked SAS storage) than they are storage systems in the model of Fibre Channel. Will be interesting to see how this plays out, particularly if Omni-Path takes hold in the data center.
Looking further out in time, the whole nonvolatile memory in the server DIMM slots as storage thing is still taking shape, on multiple fronts. I'll leave device technology, where I have no expertise, to the device technologists noting that they're all chasing the tens of billions of US$/year which the "brass ring" of becoming the DRAM replacement would bring.
On the product front, best and highest use of this technology requires repartitioning work at the basic boundary between hardware and software, with the current partitioning of products between "server", "SAN", and "storage system" as collateral damage. This is where flash was 14 years ago when the first flash-array vendor came to us and presciently told us how flash would take over storage...about 10 years ahead of when it did. I think there will be a similar race, with the usual wasteful VC funding of me-too startups starting perhaps 5 years from now, after the core technology foundation is better laid. Will be interesting to see how much ends up open source, how much is driven by the big Internet players, and how much follows the traditional commercial model.
So yes, venture investment in the earlier era ought to ebb and be very focused now. But there will be another wave.
Oh, and the storage business evolves glacially. We'll still be finding new customers for today's era products a decade from now, and still selling them two decades from now. An "abrupt" shift in storage buying patterns takes a decade.
I'd agree that the time for innovation in storage technologies which drive latency improvements is a reducing return. Where all the storage innovation is going now is how to manage Petabytes of data, geographic dispersion, hybrid storage management and analytics. The storage innovation focus is moving to what we can do with the data (software led) - not the storage media improvements (hardware led).
Hi Chris, you definitely make good points here, however I would not agree with the conclusion.
Let's group advances of technology in two large buckets:
1. New research, which overtime enters the "real world" as new technologies and products. A good example is SSDs, which as you say were invented in 1980 but never got wide adoption till recent years (mostly because they needed to align with the second point);
2. Changing business demands, which are addressed by various technologies. Good example here is data deduplication - amounts of data were soaring in corporate environments, with many files being same or with minor changes (think Word, Excel, etc files) and thus deduplication was a smart technology to solve this.
What you focus on is the first group: technologies, which are conceived and wait till the time is right for them to "happen". I think many of the things happening now in IT will actually create a new wave of innovation. For example - Hyperconverged (HC) is here because companies need simplicity. This is the value prop. Companies do not need HC by itself. Similarly SDS is a solution to storage problems/needs not addressed by traditional SAN solutions.
It is this problem-solution fit, which creates cycles of innovation. With the proliferation of new technologies, new needs and use cases are appearing (think IoT, self-driving cars, AI, space exploration, etc.) which will introduce new set of issues. And this will drive a huge new wave of innovation to solve this issues.
We should all be excited of what's coming. And get some popcorn - the show will be an interesting one! :)
Boyan @ StorPool Storage
www.storpool.com
Just came across this great piece and it makes a good point in the discussion on so many levels:
http://www.collaborativefund.com/blog/when-you-change-the-world-and-no-one-notices/
- maybe in the next 5 years we'll see so many products, based on knowledge, which we had for the past 30 years;
- Likely some of the things that are already discussed and ridiculed will actually become huge shortly.
And a point I missed - if we're close to exhausting the innovation potential of the current generation of technologies - we'll just move to a new paradigm - maybe the Microsoft DNA-like storage (https://www.microsoft.com/en-us/research/project/dna-storage/) or something else. But just as we replaced the "how we build computers" from Charles Babbage's concept of a mechanical machine, to "vacuum tubes" to "transistors" to "integrated circuits", so will storage reinvent itself.
Boyan @ StorPool Storage
www.storpool.com
To come full-circle: Storage is more than just speeds and feeds. The next battleground will be around scalability and flexibility under a single architecture. High-performance storage is important but the ability to have different types of storage for different applications and uses is also a critical dimension too, and central to making your storage environment cost effective.
Scale-out is also critical across different use cases, particularly for larger organisations that need secure, virtualised IT services. Again a cohesive compute and storage architecture is the next battle ground beyond pure performance.
Paul Silver, Vice President EMEA, Tegile Systems
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
