Kryder's law craps out: Race to uber-cheap storage is voer Kryder's Law says disk storage will keep on getting cheaper. Yet it is increasingly looking like it won't, which has baleful implications for cloud storage and archiving. Like Moore's Law, Kryder's Law is (a) not a law, merely a wonderfully apt observation for a prolonged but not eternal period of time, and (b) borne of an …
...between a cost of zero and a cost of bugger-all? I'm using Amazon S3 and Glacier to store several months worth of daily backups of about 100 live customer sites - about 300GB at any one time. Monthly cost? about $4. That's a cost I can live with for ever. The VAT on the micro server instance that actually does the backups is more than that!
Yes, local storage would be a bit cheaper, but the nightly bandwidth costs would bankrupt me.
Obviously the price won't get to zero for infinite storage, but some of the cloud providers are getting as near as makes no difference.
Of course, you wouldn't trust them with anything important!
I think you're missing two points:
1. The amount of data will continue to grow exponentially, but the cost will not decrease as fast so total storage costs will increase. That may not be happening to your customers data at the moment, but in general it's an industry-wide trend.
2. Even storing today's amount of data at today's prices is untenable in the long term. Today's prices are a loss-leader and the business model of these cloud companies is based on the fact that they will start to make money someday based on ever-decreasing supply costs.
Fill a 2.5"/3.5" form factor with these, and watch disks, NAND, and tape die.
Actually, just before the Thailand floodings there were 4 HDD vendors and HDD prices fell rapidly, competition was fierce. But the fourth vendor got bought, so the remaining three vendors formed an oligopoly. Slightly after the Thailand floodings occured, and the three vendors declared HDD shortage and prices skyrocketed (2-3x increase) and never actually came down to the old rapid development pace. Competition disappeared.
But fact is, during that flooding year, all three vendors shipped MORE disks than ever, and they posted record profits. There never were a disk shortage. It was all a lie. That is the reason we stayed at 2TB disks for several years, and no HDD development occured. With the old pace, we should have seen disks double in capacity fast, just as before the floodings. Instead of staying at 2TB disks, we should rapidly have seen 3TB, 4TB, etc. Today we would have seen 12-14TB or so, and 6TB would probably would have been standard. But because of the HDD oligopoly the pace crawled to a halt, why would the vendors spend money on R&D when they could stay at expensive 2TB disks?
Then came SSDs and the pace quickened a bit again. So lately (as 1TB SSDs have been announced) we have seen 8TB HDDs. Thanks to the SSD competition 4TB disks are not considered large enough, so they quickly released 6 and 8TB disks as of late. When SSDs reach 2-3TB, only then we will see 12-14TB disks. Not before.
Here is a series of articles, uncovering the fake HDD crisis and the oligopoly:
http://news.softpedia.com/news/Enjoying-Market-Manipulation-Western-Digital-Posts-Huge-Profits-Again-Part-4-283225.shtml
The US stock market HAS risen forever: http://stockcharts.com/freecharts/historical/djia1900.html
Except the DJIA is a nearly meaningless metric, since it's an arbitrary and changing basket of stocks. And that graph doesn't appear to be in constant dollars (but then it's not clear whether the "keep rising forever" claim is meant to refer to constant terms, or just to the market tracking inflation, or something else).
That 50 million increase took place over 41 years, a rate of improvement fantastically faster than Moore's Law, which, the article said involves, "doubling the power and memory of computer semiconductors every 18 months".
41 years is just over 27 periods of 18 months. 2^27 is over 134 million, so a nearly three times faster rate of improvement than Kryder's Law (as you'd expect with a doubling every 18 months vs every 2 years).
The doubling rate for total data generation appears to be faster than the doubling rate in total storage. So we're actually forgetting more, faster, than ever before.
Only under a specific and rather peculiar definition of "forgetting". Most of the data being "generated" is actually either low-entropy data that contains little additional information beyond what is stored, or it's the result of measuring signals that weren't being measured before (or calculations upon such measurements, which are equivalent for our purposes here). Discarding the former only means "forgetting" information at a rate much lower than the actual data rate; and the latter means "forgetting" things we wouldn't have known under other conditions, so there's no net increase in "forgetting" in a pragmatic sense.
Basically, the "forgetting more" claim is an abstract epistemological one. And as epistemological problems go, it's not one of the important ones.
Point missed - after even 5 years, under the previous rate of storage capacity improvement, the drive was well and truly obsolete, in capacity and also likely in terms of throughput. With a slowing rate of capacity increase a drive won't hit that obsolescence wall as quickly, so might, just might, have a longer effective service life.
I've seen a lot of changes in my time in IT - the DEC RK05, IBM 3330 for instance - and there's no way you'd want to be running something like that any more. If I dig deep into my spares pile I'll find some low capacity IDE drives (the old MFM interface stuff went west a long, long time back, along with the SCSI-1 and SCSI-2's), but I recall fondly the 320MB full height 5.25" SCSI server drives I used to run in my old dialup BBS days. Enough current draw to make the lights dim when you powered the box up (not really, but compared to todays stuff, they sucked huge amounts of power for what now seems to be bugger all capacity).
Going forward, we'll want our 3/4/5/6TB drives to last longer. My current server box is running 2TB drives, and most of the drives have 4-5 years of spin time already, with no urgency to replace them any time soon.
Well actually, the service life (or warranty.. how long it is under warranty - take your pick) is 5 years. The reason manufacturers don't go beyond that is they can't. Spinning parts and the failure rate greatly increases. Plenty in the GooglePlex that speaks to this. These guys think that 50% of their drives will still be running in year 6:
https://www.backblaze.com/blog/how-long-do-disk-drives-last/
"Going forward, we'll want our 3/4/5/6TB drives to last longer. My current server box is running 2TB drives, and most of the drives have 4-5 years of spin time already, with no urgency to replace them any time soon."
Maybe because you don't understand the risk? Everything you have RAID6? Because at some point UBE/URE may bite you in the ass and a RAID5 rebuild will go belly-up.
But I think the point is that drive makers haven't tried to engineer their drives for longer service lives since, given the business of the time, a five-year-old drive would've been rendered obsolete not just due to wear but due to becoming too small a capacity. If longer-lived drives become demanded, there will be an incentive for the manufacturers to try harder and come up with drives with say 7-year working lives in some shape or form.
IOW, they didn't have a good reason to build a seven-year drive. Now they do.
> IOW, they didn't have a good reason to build a seven-year drive.
Well.. this is like the 100 miles per gallon carburetor, it doesn't exist but fun to speculate none-the-less. I'd speculate that if vendor A were to deliver a 7 year warrantied hard drive, they would have captured a large segment of the marketplace (other factors being equal). The reason to go beyond 5 years is there and always has been.
But note your own words: "other factors being equal". The problem in the past has been the increasing amount of data to be stored has historically made existing drives too small by the time five years came up (there's your "other factor"). I still remember the time around 1990 when 200MB was considered pretty big. By around 2000, it was 20GB, then 200GB, and now we're routinely doing multiple terabytes. Thing is, that pace as noted has started to slow. Now capacity is harder to boost up (which I hate since longer-term bulk storage stinks on the consumer end), much as the GHz wall was hit. So now, much as CPUs have moved to multiple cores, drives need to move in another direction: in this case, longevity. Maybe the longevity won't necessarily some from spinning rust but in improvements in SSD tech; just saying that if the customers want longer-lasting drives, someone will deliver it somehow and the give-and-take of price will then ensue.
PS. Maybe the 100mpg carburetor doesn't exist, but people started demanding more efficient cars because of scary-high petrol prices, and manufacturers eventually started delivering. Now, gas/electric hybrids are becoming more and more commonplace.
> But note your own words: "other factors being equal".
The only thing I was trying to do there is "apple to apples", in other words a 7200 RPM 2 TB versus the same. 15K 2.5 900 GB SAS versus same, etc. So some twit didn't try to run off in a direction "of what about this and that" which invariably happens.
Thing is, you still forgot to notice that other factors doesn't necessarily include such specifics as drive speed and bus but also other things such as the sheer amount of data that people and firms want to store. This created an externality that put a ceiling on the service life of a drive independent of mechanical reliability.
As far as I can tell, the ceiling on the service life is mechanical. Everything I read, everything I've been able to google. If you have some hidden knowledge that shows otherwise, please share I'm much interested and would have me re-work my presentation B^).. heh. If there were a 7 year warrantied hard drive, the big players would use it in the 1000+ drive frames and perhaps extend their bundled warranties out further and make their long term costs less, ROI better , etc. There are so many reasons a much longer warrantied drive makes sense, but it isn't here yet - if ever. I believe it is a hard physics/mechanical problem. Again Charles 9, if you can show us otherwise, much appreciated.
But what happens when all those 1000+ drives fill up because the Big Data just got TOO Big Data? That's what I'm pointing at. Big Data is growing faster than the drives would normally be able to keep pace. Even if you left room to slot in more drives (and the power costs this would entail), it would just keep growing until it reaches Brobdingnagian proportions and you reach the point where swapping out for bigger drives becomes more economical and continuing to grow your data center.
This is not a failure of the observation / law, but a cessation of any interest in continuing progress and elimination of the core research group.
Several years ago Seagate very suddenly eliminated its research group in Pittsburgh, allowing the 50 or so top people in this area to wander off into the sunset. This group was the one doing the primary research, being used by *everyone* in the business. Only one or two of them even stayed in the business, and since then no serious research is being done. If you look at the curves of capacity, performance, etc., they began to level off at that time.
Source: I got this from someone who was there. According to my source, it was in fact done so callously, without any attempt to support these finely-tuned people, that several of them were emotionally ruined and no longer able to even function in a research capacity.
One result of this is that the capacity and price/performance of SSDs will surpass HDs sooner, probably in late 2015. Therefore, one might argue that Seagate simply saw the handwriting on the wall and decided to stop spending money on the research that wouldn't give them any additional value. Or one might argue that, since the market is essentially a monopoly/duopoly, there was no incentive to continue to try to outdo the competition.
* "everyone" means Western Digital and Seagate. Between them they have well over 90% of the entire hard drive market.
If you look at the curves of capacity, performance, etc., they began to level off at that time.
Wouldn't that actually mean that the research group started getting no further resultsf a few years earlier? Performance indicators will "level off" once the last ideas have rolled through the production pipeline, which may take some time.
Perhaps. It's like with CPUs. They were probably running into a hard limitation set simply by the size of the electron, which is fixed and sets a size floor. If storing data requires an absolute minimum of electrons to work, then you flat-out can't get any smaller, just as CPUs aren't likely to shrunk much further due to physics properties that kick in at those sizes. It's like trying to cram a baker's dozen into an egg carton: something's gonna break.
So it's the end of the line. Now everyone's going to be scratching their heads and wondering, "What now?"
I find the 2009 Kryder prediction surprising, given that he must have been aware of the limits to PMR and the predicted costs of future technologies at the time.
As for the IDC data/storage growth projections, I'm sure they will turn out to be wildly wrong. The reality is that applications and data grow to use the available processor / storage capacity (until we run out of ideas for worthwhile new applications). If the rate at which storage becomes more affordable is slowing, then applications that would utilise larger amounts of storage will not be implemented.
Similarly the recent slow down in the advance of CPU processor speed has not resulted in a crisis of CPU power not keeping up with application demands. Developers write new applications or new features to utilise the available CPU power.
ergo YouTube will probably decide not to let users upload unlimited amounts of 4K or 8K videos that are full of duplicates, fake movies, people filming themselves mumbling on for hours about their dull opinions and not bothering to edit their footage at all.
This. I think there might be an amendment needed to your economic model, Chris. I don not believe the assumption that data demand is largely independent of the price of storage is as robust as you make out. If we instead assume that demand is a function of the price of storage then we can also see how the demand curve meets the supply curve somewhere around a very low (if not zero) cost.
As a consumer, I choose to limit myself to using only free cloud storage - 5, 15, 25GB, whatever. I do not plan on ever paying for storage (hostage to fortune alert).
Flash has skimmed the cream off the hard drive market. Without a low volume high profit market that drives R&D, drive makers are just left competing with each other in terms of $/GB. Everyone knows when a market goes full commodity mode, R&D is the first thing cut.
Maybe I'm wrong, but hard drives quit doubling in density right around the time that SSDs went mainstream.
yep Cube storage will save the day. Its actually astonishing that we have mechanical items that are spinning around at 7200/10000rpm and we expect/demand that they stay good for 5 years. that's almost space craft levels of reliability. how many R(pm) over the life of a single drive? 19 beellion revolutions, almost as much as the mgmt at HP, or Abraham Lincoln in his grave?
Cube storage has been around in some form since the 1990's (I once saw it on an episode of Beyond 2000). Trouble is, they always run into problems: destructive reading, alignment, and so on. There's also the matter that crystals aren't the stablest forms of matter on our planet (I kid you not; diamonds are NOT forever—give it enough time and they'll turn back into graphite).
Tape's good for long-term low-activity storage in the enterprise, but it's way too expensive for the consumer market (used to be Travan devices were at least within reach, but tape's now a strictly enterprise affair). And if all your data is high-activity (say a place like Google where content is ever-changing even in the most remote areas), you'll need something else.
"To reach 20TB by 2020, the 500GB/platter drives will have to increase areal density 44 times in six years. It isn't going to happen."
Huh? to reach 20TB by 2020, areal density would need to double a little more than 5 times in the next 6 years. (.5TB / 1TB / 2TB / 4TB / 8TB /16TB), not 44 times. All right, this is not gonna happen either.
More likely that we'll 4 or 5 platter (or maybe 6 platter) 3.5" HDD's at 40TB by 2020, which only requires 2 generations of capacity doubling, as vendors put the HAMR down.
Larry @ NetApp
"To reach 20TB by 2020, the 500GB/platter drives will have to increase areal density 44 times in six years. It isn't going to happen."
Huh? to reach 20TB by 2020, areal density would need to double a little more than 5 times in the next 6 years.
Yes, 44 is 2a little more than 5. (It's a bit less than 25.5, actually, but why quibble.) He didn't write "double ... 44 times". I admit the phrasing is confusing, but it hardly seems a stretch to interpret it as "increase by a factor of 44".
That said, I'm not sure where the 44 comes from. 20*1024/500 looks like ~41 to me.
If the missing storage hardware investment is going into solid state storage, this will bring forward the point at which the marginal cost of flash approaches that of the Winchester disk.
However, we know there isn't enough flash foundry capacity to cater for storage growth, even if total Winchester disk capacity continues to increase. This puts a brake on Winchester -> flash substitution.
If the marginal price of storage rises or stops dropping, there will be an incentive to raise investment. That investment is likely to go to solid state tech. But building a foundry is like building a ship or an office building - a colossal bet on future prices. The dynamics are exactly like those of shipping or commercial property, which lurch from periods of echoing overcapacity to periods of frantic building. Half the projects in those businesses go bust, and no-one knows which half until it's too late.
In semiconductors, the flat panel debacle still haunts manufacturers. Building flat panel factories to cater for seemingly insatiable demand turned former industrial giants into shells. LG, Samsung, Sharp and Sony were all humbled by flat screen bets. Between 2004 and 2010 the industry lost $13 billion, according to the Economist (2012). That is chicken feed in comparison to what awaits poorly judged solid state storage investments.
Will we have a Mexican stand-off of non-building? Or there will have to be consolidation or co-operation on semiconductor fab mega-projects that don't lead to overcapacity because they are cartels?
If manufacturers compete on megaprojects, some of them will be fatally injured, which leads to de facto concentration, and more manufacturer pricing power.
Long term price falls are still possible, with current dynamics, but not in the most likely scenarios.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
