Where will storage go over the next 15 years? We rub our crystal ball

Let’s fly up to 20,000 feet and survey the storage landscape from there, and then stay at that height until 2030. What will we see? Our starting view is utterly basic and pretty reasonable. Most of the storage equipment is based on disk and most of it is on-premises. Some of the data is being stored in flash and some of the …

[Platypus]

Interesting piece, Chris. If you don't mind, I'll try to add on a bit based on my perspective as a developer in this area.

Traditional big-box on-premise storage vendors also face another pair of closely related threats: open source and roll-your-own. The relationship between something like Isilon and something like Gluster (which I work on) is obvious, so I won't dwell on it. The relationship between something like Isilon and something like AWS is also obvious: more AWS usage means less Isilon sales. The relationship between Gluster and AWS, or any of several similar things on either side, is more nuanced. Sometimes people abandon their own open-source scale-out storage in favor of AWS services. Sometimes they deploy that same software within EC2. It's both a threat *and* an opportunity.

That brings us to roll-your-own. If you were to look under the covers at Amazon's storage offerings, I'm sure they'd look an awful lot like what's out there in open source. Ditto for Google. Ditto for Facebook. And Twitter, and LinkedIn, and so on. The fact is that the techniques for doing a lot of this are now pretty well known. Many of those techniques were developed are refined at the aforementioned companies, each of which has rolled their own not once but several times to address various needs and tradeoffs. I've seen a public presentation from GoDaddy - not generally regarded as a company in the vanguard of storage research - about their own home-grown object store. I know of many more that I can't talk about. Perhaps the biggest threat to both traditional storage vendors and someone like me (or my employer) is not some one new product or project but the general idea that scale-out storage software can be assembled rather than developed. That doesn't mean there'll be no place for people who know this stuff and can assemble those parts into a smoothly functioning stack, but we'll be providing less of a product and more of a service. As in so many other areas, increasing levels of automation might put customization in the hands of more than the elite.

"Good morning, madam. What kind of storage system would you like me to build for you today?"

[Anonymous Coward]

That is a very good point. We have built our own, bought some "professional" NAS kit, and are going back to roll-your-own because its cheaper AND we get control of what we can (and can't, of course) do with it.

The glory days of NetApp, etc, demanding huge extra licence fees for features like snapshots, CIFS support, etc, are going, if not gone. Sure they know a lot and you get the promise of a well designed system, but the high profits of days gone by are over and that is a big factor.

As far as "cloud" is concerned, we are a sceptical lot. We have no massive dynamic demands so we see it as a costly way of renting space that can be turned off at a moment's notice. Not for us!

Sure we are looking at co-locating our storage and servers to take advantage of infrastructure savings (cooling, UPS, security, etc) but in that model we still own the storage hardware, we have sovereignty over our data.

[Youvegottobe Joking]

What about support?

How large do you need to get before someone asks the question "how are we supporting our <FreeNAS or similar> storage environment?" and the answer is "John", and the next questions are "John the desktop guy???, isn't he leaving next week?"

Cue storage system going titsup and frantic forum searching for a fix.

There is a point where a company cannot rely on roll your own.

[Paul Crawford]

Re: What about support?

You have a valid point, that someone has to support it.

Sadly, often the paid-for support is only a little better than what a popular (e.g. FreeNAS) forum has. I'm guessing you can also get paid support for open source solutions like FreeNAS, so its not an either-or option.

Can we have a straw poll on which major storage vendors really provide good support?

[bitpushr]

Snapshots have never been a paid feature from NetApp. They've been free since 1994...

Nice try, though.

[Paul Crawford]

Re: Snapshots have never been a paid feature from NetApp

What about accessing the snapshot'd data?

[bitpushr]

Re: Snapshots have never been a paid feature from NetApp

The same.

[Gannettt]

I think you're mixing up two phrases: To rub the magic lamp" and "look into a crystal ball". Sorry, had to get it out of my system!

[Numen]

And the other areas?

With the impact of cloud on storage products, there's the related impact on on-premise servers and server networking, and the follow-on effects to systems and reseller staff. (Personal/workstation device type networking will still be important!)

Sounds like a major dislocation for folks working today, especially those starting: many of their jobs may not be needed over a relatively short horizon A look at that would be a very interesting article.

[Grikath]

nice try, but no biscuit.

The article fails to address the current kerfuffle about data privacy and the laws/rights/guarantees ( or lack thereof in the case of the US) that pertain to them in different legislative power blocks. This will segment the world. Pandora's Box has been opened, and pretending the lid never opened will not work.

As Platypus says, the technology itself is becoming a commodity instead of a luxury, and it is quite possible to "roll your own". Whether or not that would be cost-effective for a single company depends heavily on what shenanigans regarding small print the Big Ones will try to get away with, how much arm and leg they're going to charge for their services, and how well the open source community can develop feasible "plug in" alternatives suitable for busineses.

Another thing the article fails to take into account is the very simple fact that most data stays, very much, local and often doesn't even leave the building. With SSD's measured in 100's of GB's and spinning rust in TB's. Currently. With 3D setups and other innovations using current technology developing at a moderate pace you're looking at TB's on a stick soon, and then you're not even looking at the current lab-stage developments.

And Cloud = Connection. No connection: no data access, no VM, nothing.. It's the big Achilles' Heel of the whole thing, and it will never go away. Most of the worlds' infrastructure can't handle the World Going Cloud now, and I can't see it happening in 15 years. Not unless there's a huge growth in data transfer capacity, and a serious drop in the cost of data transfer. And there's no telling how much capacity the entertainment and advertising(!) industry will hog.

All this will result in small "local" setups being able to compete a hell of a lot better than you estimate in the article against the Big Guns, especially with the Big Guns being US based, and there being a paper gnats' chance in hell the Agencies are going to relinguish power there within the next 5-10 years, giving many startups the chance to offer the same services locally, under local regulations and certifications, at locally guaranteed connectivity. And they will win.

The picture you scetch may hold true for major multinationals, but they are the "1%".. The majority or SME's and local subsidiaries will either keep things in-house, or find a local solution that guarantees stability.

[Anonymous Coward]

a couple easy predictions

1) The smaller tier of on-premises suppliers will consolidate faster than predicted, as venture financing in this sector (high in the last 10y) dries up, and consolidation ensues.

2) The smaller and mid-tier of public cloud providers will grow in visibility. This is consistent with a number of comments above. More of this business is local than is widely perceived. Some of these providers will do better than others, and a few will go public.

3) Cisco will make an acquisition to get into data storage. It will not be NetApp.

[Platypus]

Re: a couple easy predictions

"Cisco will make an acquisition to get into data storage."

I'd be very surprised if they only made one. They'll probably mess up a couple before they get one to drive any real revenue.

[Anonymous Coward]

Re: a couple easy predictions

Instead of acquisition, I bet Cisco will do their own.

Storage isn't hard things to do, especially today.

You see tons of startups to do those today.

Previously, they are scared to break up with EMC.

Now, no such worries since VmWare acquired Nicira

and Dell bought EMC.

[CheesyTheClown]

They already have storage

They make 52 drive bay servers with substantially more bandwidth than anyone else has. All they need is the software and if they actually stop screwing around long enough to make a Microsoft solution work or an OpenStack solution work, they could ship finished solutions as finished racks. They're too damn busy trying to be everything to everyone that they may never be anything you I am body.

[Anonymous Coward]

Oracle started promoting shared environments recently and I think data-storage should transcend "having a box" soon, for many purposes.

We do not all need to have a powergenerator to use elctricity and we do nto all need to own hardware to store data. A cloudOS for productivity software that stores on multile "cheap-as-chips" disks is haard to beat. Arguments that "on-premise" hardware iss still necessary are slowly being eroded by law-firms using box.net etc.

I do not even see data-sovereignty scares like the current safe-harbour scares stop storage professionalization.

We'll see.

[Paul Crawford]

Electricity is a basic commodity, it has no real unique characteristics. Just 230V +/- 10%, 50Hz, (mostly) sine wave here in Europe.

My data is unique which is why it is so much more valuable. Do I trust others to look after it? No!

Sure, I might use a cloud provider to store an encrypted backup, but then if they bugger me around I still have the original, and they don't have access to whore me from advertiser (or TLA) to advertiser. Going cloudy might suit small businesses that have no tech support and limited requirements (say just email & dropbox share) but if you have big demands the cost of the "cloud", and the bandwidth needed to work with, it becomes uneconomical even before we get to data sovereignty.

[Roj Blake]

"My data is unique which is why it is so much more valuable. Do I trust others to look after it? No!"

Do you have a plethora of security accreditations?

Are you made aware of security vulnerabilities before they are announced to the public?

Do you have a dedicated cyber-security department?

Is your IT monitored 24/7/365?

Is your IT in a facility that's essentially as hard to get into as a bank?

If not, the question shouldn't be if you trust others to look after your data, but whether you trust yourself.

Your money is also pretty valuable to you. I bet you trust a bank to look after it. A *good* cloud host is to your data what a *good* bank is to your money.

[Open]

Yes and no

How many banks do you know that have been hacked? How many have owned up to it when they have found it and not months later? Why do you not know because knowone in the banking industry is so stupid to tell you your money is not safe witht them. So the same is true of your data and a cloud provider. Their businesses are built on security, if thats in question they go out of business.

I like the idea of doing it yourself with a consultancy who knows the pitfalls. All those ex -solutions customer engineers might be having the last laugh.

Seperate your internal and external networks. If you can't see it you cant hack it. Have a data management business plan that keeps important stuff private and public stuff avalible.

Or just encrypt it all at rest and in flight. Use all that spare compute power to make it hard. You can't hack what you can't see. Oh and remember ILM or you'll loose your data anyway.

[Paul Crawford]

@Roj Blake

Do I put my data on public-facing networks?

Am I subject to USA data snooping laws? Would I know if I was subpoenaed?

If I run out of short-term cash will I delete my own data?

Also money is, like AC power, or water, etc, interchangeable. The numeric value of my balance is not something that would be of special advantage for industrial espionage.

[CheesyTheClown]

It's obvious

Dell will have their own, Cisco will have their own, HP will have their own and Lenevo will have their own. When you buy virtualization, you'll buy a prebuilt virtualization solution from a vendor and it will come preinstalled with storage, networking, compute and management. You'll install prebuilt application containers from a self service portal and if you want to move to another vendor, you'll just migrate.

Piecing together data centers is as stupid as companies piecing together their own PCs. You'll buy a rack/mainframe and be done with it. Cisco should just buy nutanix and be done with it.

[leon clarke]

15 year timescale

Over that timescale you have to assume some sort of technology like X-Point or some other phase-change memory will be developed. When that happens, its liable to be much more rapidly disruptive than flash was. I suspect that flash will look like a briefly-forgotten intermediate technology between disc and phase change. And things will get very hyper-converged very quickly - it's the only way to make use of the speed of phase change.

[Steven Jones]

What this all seems to ignore is where the processing is relative to the data. Even given unlimited bandwidth, physics dictates that there is a minimum latency defined by the speed of light and distance it has to travel (over fibre it's about 1ms round-trip time for every 100km). So if you have an application that requires low latency access to its data, then that application is going to have to be in the cloud too.

Of course that's no impediment to applications where latency is not critical to performance, or for backups, archives and so on, but it is to a lot of transactional applications. For that you need the entire package of at least the latency-sensitive parts of you application "in the cloud" and very close to your data. Not much point in those flash drives with latency measured in 100s (or even 10s) of microseconds if you app is sitting on a few ms of network latency.

So I have my doubts that a pure commoditised cloud storage system is going to suit a large proportion of applications.

[Paul Crawford]

The only solution to the latency issue is to have you processing "machine" on the cloud-provider's infrastructure. At that point you surrender any security as that machine would necessarily have the key(s) to decrypt your data.

Otherwise you can use cloud for secure storage so long as it is encrypted at your end using a key not known to the storage provider, which is a good options for some situations (e.g. off-site backup).

[brunob]

Availability

I think that there is consensus that the big players all need a better offering. They all need to be able to offer 'cloud' as that is the fashion, so CIOs don't want to be shown up at the golf club by not being into cloud.

The problems identified by others are also real; security, latency, availability. Whatever the actual security of a cloud storage provider there will always be a perception (sometimes shared by auditors) that it might be less secure than on-site. It is the same logic that makes people put cash under the matress rather than in a bank.

The network connection from your data centre to the cloud storage will break from time to time; arguably that also means that your users likely cannot access your servers anyhow, so you will have made a plan for that. As noted by others, for some applications five-nines availability is not required so cost will be the main factor.

What a smart cloud provider may be able to do for you more cheaply than you can do yourself is make your data available to another of your data centres, so you just fail over your application. But if you were doing that, wouldn't you put the application into the cloud also?

I'd suggest that the big vendors will blur the distinction between storage - virtual SAN using in-server disks, on-site flash for the small % of data that is 'cached' in your data centre, and cloud for everything else. They just need to find a way to put one badge on all of it, rather than coming at you with lots of product names and even company names.

[DeliberatusFreeman]

Lord

you improperly titled the article.

The last word in it should have been plural, so as to live 'up' to Reg's evil sense of humor.

[Anonymous Coward]

This is the end... ???

I love this line at the very end of the article - "Welcome to the first day of the rest of your life." I wonder how true this is. If this was 1991 and I am a mainframe expert, I still just might have a career. If this was 2001 and I'm still a Novell Netware expert, I'm probably looking for a new job. Will monolithic block based storage really die or will it become a niche product segment? I think I better become an expert in this new wave of technology --- and become one quickly.

[hoola]

And that exponential growth of data

The elephant in the room is the insatiable growth of data. It has to go somewhere and the more there is the greater the problem. Cloud is all well and good (ignoring all the privacy issues) until you need move multiple petabytes because your service provider has just imposed unacceptable terms and conditions.

Azure I think gives you 90 days to get the data out if the account is closed. It is simply not possible with larger quantities because the bandwidth is not available. This of course is the actual end game, total lock-in with maybe 2 or 3 suppliers in a virtual monopoly. Economies of scale will play a part but at the end of the day it is just going to be some sort of object storage under the fancy name. There is going to be a finite limit on the time it can be run without generating profit. At the moment the race is to get your data into the cloud with the hope that it is then locked in. At some point in the timescale we are looking at here cost models will have to change. Extracting value from data is going to be the goal of businesses, whether that is storing it for someone or analytics.