How NOT to evaluate hard disk reliability: Backblaze vs world+dog A few months ago, Brian Beach, a distinguished engineer at cloud backup joint Backblaze, published a set of study-like blog postings relating to his firm's experiences with hard drive lifespan in its 25,000+ spindle environment. The blogs garnered quite a bit of interest due to the subject matter, and provocative titles like: …
..burn half as long?
When selling Host Independent Disk Subsystems into a certain part of the UK public sector, we always filled the slots up with "Enterprise" class SATA disks.
That gave the end users and our good selves peace of mind, but those disks remained powered on and rotating at 7200RPM 24/7/365.
The failure rate was very low, I cannot remember the stats, but I do recall for every 24 bay system we flogged, the end user purchased two spare disks "just in case" and the reported failure rates were less than single digit.
Our biggest installation comprised of over half a Petabyte of storage comprising of 500Gb, 1Tb and 2Tb disks, depending on the age of the disk array and the biggest disk capacity available at the time.
We wouldn't have dreamt of putting "consumer disks" into that environment for a few reasons.
The biggest stigma was the fact they were "desktop drives" and not recommended for heavy duty use by the manufacturers (Seagate or Hitachi)
Secondly was the warranty, the Enterprise disks had a better warranty and a "disk retention policy" where the end user would only have to return the lid of the failed disk to get a warranty replacement unit, thus never risking sensitive data leaving the site, even if the odds of any data recovery were very low.
Thirdly, the Disk Subsystem manufacturers had a "compatibility list" of 'certified hard disks" and this always comprised of the Enterprise class disks, desktop platters never, ever featured and to get support (when needed) we, as integrators adhered to that list religiously
Lastly, the enterprise disks cost more, so from an integrator perspective, by following the vendors' recommended configuration, there was a little extra money to be made.
The notable differences between Desktop and Enterprise disks was the cache size with the Enterprise offering double the cache size of the desktop equivalents, other than that, they were all 7.2K RPM disks.
This article comes as no surprise to me but the statistics make for interesting reading. However, most integrators will continue stuffing Enterprise disks into those RAID arrays that allow us to use disks bought from distribution and we will adhere to the tested disk list because we just have to do things right.
Aye, I gotta say, 1 spare on the shelf for every 12 deployed spindles has done us well so far. Usually have several left over by the time the warantee is up. Enterprise-class narline drives seem to be absolute tanks these days. Mind you, managing a few thousand spindles is far cry from cloud scale.
Still, I often wonder about things like vibration, humidity, etc. Alberta is very dry. Most of my deployed disks are not exactly in noisy rooms are where there are sudden noise level changes. How does the environment of a "proper" cloudy datacenter change things?
There's just so much I don't know about the factors that affect spinning disk life. From what I can tell it seems to be pretty much a crapshoot...but that doesn't seem to stop everyone from creating a magical algorithm to figure out which are "better".
While my spindle count is orders of magnitude lower than what Backblaze used, my findings are broadly similar. I got a bunch of drives from Seagate, WD, Samsung and HGST for various systems, and made sure the redundant peers at different sites always had drives from different manufacturers in them to hedge my bets and avoid deeply inconvenient need to restore from backups. Since then the warranty on various desktop drives has been reduced from 5 years to as little as one or two. All disks have been used 24/7, and since then I hot some 2.5" Toshiba drives as well in the mix.
Over the past 6 years, the failure rate within warranty on Seagate drives has been approximately 200%. A nontrivial fraction of Seagates were on their 3rd or 4th warranty replacement before the warranty (5 years back then) was up. All of the drives in question are 1TB in size, but models vary due to replacements.
By contrast, I have had 1 WD start to get dodgy enough to replace (spurious pending/reallocated sectors). I have had two Samsung's fail with massive media failure and discovered some interesting firmware bugs in WD and Samsung models that make me not trust them to deal with their own defect detection and management (Seagates are more trustworthy in this department).
That is over the past 6 years.
In comparison, I have not yet had a single HGST disk failure with similar number of drives deployed since 2002.
All drives of all makes I am referring to are desktop drives.
Now, you can argue about sample size and statistical significance all you like - I openly admit my total sample size is in the low hundreds vs. many thousands for Backblaze, but the findings are undeniably similar.
Also they are much more important for home consumers where disk failures are much more inconvenient due to slow or patchy backups.
So yes, caveat emptor, but don't reject the best study available due to DUF.
With a few thousand spindles deployed I haven't seen what you see. Seagate enterprise disks don't seem to go boom that often on me. WD Raptor 300GB drives were the bane of my existence for 6 years whereas WD Blacks seem to be made out of neverdie. Hitachi stuff...actually...I can't remember the last time I RMAed a Hitachi drive.
*shrug*
We need bigger sample sizes. We need to isolate for conditions like noise, humidity, vibration-isolating case design, etc.
... isn't what I'd call "mainstream" for Backblaze. I see their drives, in enclosures that required shutting the whole thing down to replace a single drive, as "jam jars". You fill 'em up once with lots and lots of writes, then occasionally you ask pretty please for some of the data back. Meanwhile you truck on to filling up the next "jam jar". As the data on the drives age, the read requests reduce, to the point where they can probably be powered down, waiting for a stray read request that needs something from a dormant spindle.
This isn't "classic" OLTP or datacentre duty for a spindle. Apples and oranges ... please.
Some "science" on the duty cycle per spindle of the drive comparison would better inform us the of the job these drives were doing. You put a layer of write buffering flash in front of these boxes to further level out the writes to become almost synchronous to each spindle, and you change the drive head step pattern again. Heck ... we'll have the stuff on tape soon [SMILE].
I'm no expert but...
On a cursory read-through it does seem to me that they're not normalising (or whatever you want to call it, I did say I'm not an expert) their data before doing an analysis of it. This is bound to throw the numbers off somewhat.
Also, what others said above me, the duty cycle and usage hasn't really been taken into consideration.
~W
I think that may be a moot point in real world context.
Yes, Enterprise drives might have a longer life in the duty cycled described. But does that matter if for the most part drives aren't used in what we think of as a classical HPC environment? If 90% of real world applications look more like Backblaze than a classic HPC does the duty cycle actually matter to the purchaser of the hard drives?
And I don't really accept the argument about the small sample size. An additional 2 spindles not failing sounds small, but when you're supposed to be talking six sigmas or better on your engineering quality, that's a huge number. Put in another context, in order to get to the 4.2 failure rate given 17 hard drives failed, you'd need to add another 103 drives to the mix before you'd expect to achieve an experimental 4.2 failure rate if the true failure rate is 4.2. (4.2463). That is, I'm assuming the true rate is 4.2, and we only see the true rate of failure over the next 103 drives without any variation.
"The difference between a 4.2 per cent and 4.6 per cent annual failure rates on 368 drive-years worth of service is only 1.5 spindles."
Yup, if you want to compare things: Do It Like You Learned in Stats101. Even my geography students (1st years) should be able to come up with that point... (let's see, the exam is tonight...), although here the statistics are a tad more fancy, with stuff beind dscrete and all... Do not forget to communicate the uncertainties.
I forgot how the discussion was in the comments here, so maybe somebody saw this coming.
Meh, I'm stuck with what the Uni's hardware people buy anyway.
although here the statistics are a tad more fancy
They're not though. Hard drive failure rates follow the common "bathtub" curve - you get a comparatively high rate of infant mortality as the drives that weren't quite right to begin with fail. Then you get a long steady period of fairly low failure rates (the "bottom" of the bath) before it rises again as the sound drives wear out.
They admit their figures are for younger enterprise drives than consumer units so both sets have gone through the early purple patch, but the consumer units have had longer to travel along the "bottom" to get their failure rates (in failures/year terms) down. I actually find the fact the two figures are so close after this gross distortion to be reassuring that it is worth spending the money on the pricier drives.
Sadly, at the consumer level deciding which drive to buy it's all 50/50, your drive either fails or it doesn't. Regardless of type, size, manufacturer etc. a hard drive failure is much more than a mechanical thing that's busted. And data backups won't safe you from downtime while you wait for the replacement drive to arrive in the post or for your system re-install.
Spinning disks will be gone within a few short years anyhow so the focus should be on which SSD to buy now to future-proof you for the life of your next box.
Don't see that happening, TBH. I don't see any 4TB SSDs for sale, and for long term storage, platters are where we are for now. SSDs aren't the save-all they are cracked up to be either, with - as far as I remember - a mere 5-year working life. I have an ancient IBM XT with an MFM drive in it that still works perfectly, and old 20 - 25 year old 40MB IDE drives that still work fine. I don't use them as such, I use them to show the kids at work just how far computing has come since I were a lad. Then there's the <insert your currency>/GB cost comparison between the 2 technologies. I think there's too many factors. SSD for the OS, by all means but for storage, at least for me and untill the tech changes, I'll use hard disks every time.
1) the margin of error is bigger than the difference in quality wrt consumer vs enterprise disks (which actually is very, very, bad for the enterprise disks -- even with 600 data points the enterprise disks are indistinguishable in quality from the consumer disks!)
2) Seagate's drives WERE significantly worse than the other manufacturers. That the bulk of the problem was two models which Seagate itself apparently has stated were junk doesn't change that.
Also, for what very little anecdotal evidence is worth, I too have bad experiences with Seagate.
At 1.5% failure rate of drives you replace, you are looking at 1.5% of those 4 million dollars in replacement cost, plus the $5000 for the guy replacing them. That's 65000 Dollars in total. If you want that back to 1% you are looking at 43333 in cost. The difference between those two scenarios is a bit over 21666 dollars in money you would save, if the purchase price of the drives was identical. On 30000 drives, you're looking at about 70 dollar cents more for the more reliable drives each to break even. Yes, they don't do warranty claims so each failing drive is a write off, they didn't put that in their cost model.
However, even at these numbers, this sort of maths doesn't fly. First, you need to have a guy available 24/7 to replace drives, or you need to invest in a lot more redundancy if you want them to only work office hours. That means investing probably double in the hardware, or you need four guys on shift duty, costing you 40000 dollar per month, if you take the 5000 for 2 weeks of one person's example given. You can't afford to lose data, so you need to keep on top of replacing drives, even if you have multiple copies of your customers data.
Second of all, you need to either invest well ahead in ample stock of drives, or take your chances on the market and buy whatever is available when you need new storage. Both options take a hit on the amount of money you are spending since either you have a cash flow problem and are buying devaluing drives without using them, or you're paying whatever price market demands at low numbers and whatever drive is available.
No nice large purchase discounts and having to buy more expensive drives if the cheapest ones are out of stock can easily set you back over 10% in purchase price and you have no full control over reliability of your systems. Buying large and not using devaluing drives will also cost you money since you can't invest it somewhere else and prices are dropping at a steady rate. No matter what you do, you're spending or saving way more money on stock control and operating costs than you would be on reliability of the individual drives. So yes, at this scale buying cheap and replacing more drives is the smartest option, but the calculations involved are a lot less extreme than 5000 vs. 4 million dollars as they suggest.
The drive replacement cost assumes zero cost for the new drive? If they are under warranty, then the cost of packing and shipping 600 drives must surely be significant. If the drives are consumer and used outside the terms of the limited warranty, then the cost of buying 600 replacement drives is pretty significant. Backblaze have talked in the past about using drives removes from USB caddies, which sounds like an interesting return form to fill in. Just doesn't add up to me.
I haven't read through the warranty on consumer drives, but I assume they specify in some way that they aren't rated for 24x7x365 use. If you use them that way, and it fails, you probably have to lie to make a warranty claim.
I agree about the packing/shipping cost. They have way underestimated the cost of replacing a drive, as well as counting nothing for the non-zero risk that two drive failures within a short amount of time can be fatal if using mirroring or single parity RAID. If that happens to them once, the cost will outweigh all the savings they will ever get by buying cheap drives.
Irrespective of the "science" of the analysis, I fully agree that Hitachi drives are reliable, and fail rarely. I agree that WD are middle of the road and generally offer good reliability vs cost, and I also agree Seagate are overall terrible.
I've had far more Seagate drives fail and especially the stupidly thin 3.5" ones (which were also unreliable when branded Maxtor, whom I believe Seagate acquired).
In *my* experience, in *our* facilities, plus those of customers, i'd say the analysis is spot on and I'd also agree "enterprise" drives don't fail less, they just perform better the rest of the time.
$10K a month for an employee whose job is just to swap disks out? We're hardly talking about highly trained or qualified personnel here, so I think they may be paying above the market rate.
Or to put it another way if these guys think the guy why changes disks is paid $120K a year then I won't trust any of their other numbers either.
"Or to put it another way if these guys think the guy why changes disks is paid $120K a year then I won't trust any of their other numbers either."
I think you will find that the original salary figure was deliberately over the top to illustrate that the labour costs of replacing failed disks were negligible even if you hypthetically used an over-paid, over-qualified senior support technician in a high labour cost location. And I can't speak for the author, but where I sit around 40% of an employee's costs to the business are not their direct salary, but annual leave, sick leave, overheads, payroll taxes, support & administration costs.
"Replacing one drive takes about 15 minutes of work. If we have 30,000 drives and 2 percent fail, it takes 150 hours to replace those. In other words, one employee for one month of 8 hour days. Getting the failure rate down to 1 percent means you save 2 weeks of employee salary - maybe $5,000 total? The 30,000 drives costs you $4m."
But what about the cost of any outage between the drive failing and your engineer getting it back and restoring the data?
Or the data you lose because they were written after the last backup?
Or the reputational damage if the failing drive takes out a customer-facing system?
Momentary performance hit? To rebuild a 4TB drive in an array that sod's law says will be almost fully allocated when it goes bang? That's going to be a long moment... although I'd be more concerned about a second drive failing before the array's rebuilt, personally. You can make yer systems robust, but you've got to pay for it *somewhere*.
In this case? Yes, a moment or two should do it.
They are a scale out storage provider, they aren't running plain old RAID1. If a disk dies then the redundant copies of the files or blocks from that disk will be collected from the dozens of disks they are spread over and written to dozens of other disks.
They included the provision that their setup is planned so that they expect failures. That would seem to entail even the double or triple simultaneous drive failures most of us would never see except in cases of significant manufacturing defect or onsite catastrophe (data center caught fire, got flooded).
Whilst I'm not convinced by his calculations, his findings do actually tie in with mine.
As with most long in the tooth IT guys, there are things which we suffered back in the dark ages which have stuck with us for life. For me it was Seagate drive death... We're talking 400meg drives here! (Yes, that long ago). Unfortunately I was a field engineer at the time, and covered so many miles just to replaced dead Seagate drives that I still wouldn't use them for anything more than a doorstop! (Actually I lie... I took them apart and used the platters to hang from the plum tree to stop the pigeons eating all the buds).
When my data storage needs at home expanded to the point I really needed to think about redundancy, I opted for "special" raid edition drives, only to have two of them die within 12 hours of each other when I was away from home. They were 6 months old. Don't worry, I also had backups.
I replaced those drives with the next model up (went to 500gig from 400gig, but the same brand, and still specific raid drives), upgraded the power supply in the server that contained them, put it on a better, mains cleaning, inline UPS, and increased the air-flow.
This time the first drive failed after 4 months!
I then decided to start again. Dumped all the drives, working or not, and bought a load of cheap Fujitsu 1TB drives, plus a couple of spares... Never needed the spares. They were still running perfectly reliably when I replaced them with 2TB Fujitsu consumer drives... And that's where things are today. The 1TB drives have ended up being used in all sorts of machines, and I've never had a single one give the slightest hint of failure. The 2TB are still whizzing along in the RAID, the uptime of which is now over a year.
All the spares have been pinched for other things (really shouldn't do that!), such is the confidence they have given me.
Meanwhile, in the work environment, I've seen more specific raid drives die in under a year than I can remember... Some things don't change.
Should Seagate be penalized for a bad model of disks? After suffering through some bad models of disks myself, I say "YES!" A bad model happens to every manufacturer and they deserve some shame when it happens. The losses to downtime, swapping disks, restoring backups, and sending the drive in for a warranty replacement is huge. To make matters worse, the replacements are brand new drives with exactly the same problem. Any drive manufacturer wanting top reviews will need to test their hardware more and halt production sooner when there's a high failure rate. Otherwise, they'll have to suffer periods of "Never buy XXXXX" for a while like everybody else.
Unfortunately it's not about a single unreliable model - it's about multiple consecutive unreliable generations. Backblaze figures actually excluded some Seagate models because the failure rate was so ridiculously high that nobody would have taken the results seriously.
The other problem is that most manufacturers are having bad models and even lines increasingly frequently. WD green drives with their aggressive spin-down leading to vastly premature spindle motor and bearing failures are such an example, not to mention firmware bugs that lead to gems such as pending sectors that are either not found by extended self-tests (WD, Seagate) or are located at sector numbers that exceed the LBA limit without HPA enabled (Samsung).
It is strange how we over analyse this but peoples posts in the comments are the same: Seagate is really junk and Hitachi lasts nearly forever.
I have found the same, I have drives (IBM/Hitachi) which ran in a museum for >10 years 24/7 (yes, desktop) which I replaced because I got scared - how could they live so long? I have had ES2 last a few months on very light loads. I move a LOT of seagates into the film industry as consumables and find them pretty unreliable.
My guess is an accountant came up with the idea that if seagate's drives fail more often with a shorter warrantee they make more profit! Very clever I must say but I have no trust in seagate. Their support is also shocking whereas I find Hitachi's second to none. This is 15 years of findings summarised, but as it is not a formal press release and just someone's real world findings it will gather no credence.
RMA's are fine with Seagate, as you have found because we need to use it often BUT support. As in
"I have having the following errors / issues, please work with me to find and solve them". Seagate - "get stuffed / must we RMA the drive?" vs Hitachi having real engineers on skype/email/tel to help anytime for about a decade. To me there is no comparison in having access to real people who will help.
First, I've no intention of debating the granularity Brian Beach’s (Backblaze's) statistics; whilst it's vitally important I found stats to be the driest and most boring of my academic subjects. However, putting numbers into perspective, one should remember that election opinion polls with typically as few as a 1000 samples yield an accuracy of a few percent—which ought to be sufficient samples to deduce a trend in HD failures (that's if one is not comparing apples with oranges and Brian Beach is doing a bit of that). ...And things get iffy with only 300/400 samples.
Nevertheless, Beach's stats reflect my own experience; his failure figures almost mirror mine except his percentage of failures is typically about half mine as I'd expect (I have far fewer drives and they're not in a controlled environment of a data centre).
1. I have boat-anchor boxes full of dead Seagate HDDs, they by far top the unreliability stakes and have not been reliable as expected since the 7200.11 BIOS fiasco of some years back. I now actively avoid Seagate drives whenever possible. (BTW, I've been using Seagate products from days way back when the company was called Shugart and made the 5MB ST-506 HDD.)
As with Beach, I also found the Seagate drives to be initially OK then 'soft'-fail later on—not so much infant mortality but rather in the middle of the bathtub curve. I also found a peculiar fault with many large Seagate drives of 500GB and over. Pull drives which have failed or are failing out of service then clean/reformat them with DBAN or similar utilities then stress-test continually for a week or more with high I/O and often it was impossible to get them to fail! However, put them back in service, especially in environments where they were likely to encounter a half million or more files and they'd fail within a day! (I've only ever seen this strange phenomenon on Seagate drives—where the number of stored files is more indicative of failure than sustained I/O etc.—with huge numbers of files they'd fail even if only at 50-60% storage capacity).
2. Next is WD, I found the red drives to be very reliable, but as Beach states WD's green eco drives were an unmitigated disaster (I wouldn't even put them in a hand-me-down kid's PC—same with the Seagate equivalents). Generally, on a per capita basis, in our environment, Seagate failures would outstrip the WDs by 3:1.
3. Although the sample is small, Samsung proved to be reliable; we've had almost no infant mortality or bathtub-bottom failures although some have failed without warning after 3 to 4 years.
4. Hitachi HDDs are the clear winners by far. Although we have not that many per capita (~25%), they've been by far the most reliable. In fact, there's no Hitachi's in our boat-anchor boxes.
5. I'd also agree with Brian Beach’s assessment over SMART, I've seen many a 'dead' Seagate drive with essentially perfect SMART figures. Except for initial testing and bench maintenance, SMART is essentially worthless (as the mechanisms that are causing the failures are (seemingly) not monitored by SMART).
Irrespective of his method of assessment, Brian Beach’s (Backblaze's) statistics are in keeping with what I've found. The key issue is what is the most reliable drives, and for once Brian Beach’s figures have given us more than just hearsay.
In the past, we've had documents such as Google's Failure Trends in a Large Disk Drive Population but they're only of academic interest when essential facts such as brands/manufacturers and HDD types have been obfuscated or redacted from the documents. This is why Brian Beach's data, albeit having shortcomings, is so important.
Statistics aside, what bothered me most about the BackBlaze articles was that my favorite drive model, the Western Digital Black series, was not even mentioned — because they were considered too expensive. How many other perfectly fine drives were completely left out of the discussion? So while the proffered statistics are interesting, I don't think they are anything that one can draw conclusions from, except maybe to avoid Seagate drives.
And the idea of extrapolating BackBlaze's tech and financial situations to the average consumer just doesn't seem right.
I'd like to point out that Newman seems to misunderstand "hard error rate" as meaning the drive has permanently failed. The actual definition of hard error rate is an error that can not be recovered by a retry. But most of the time, if you write that block with new data, it will hold the new data without any problems.
MTBF is the spec one should use to predict when the drive will become useless but it tends to be a computed value based on guidelines in MIL-HDBK-217F and I would not put a lot of faith in those values.
Dan Olds, Gabriel Consulting, didn't like the headline, so he's told us how clever he really is: he can actually read the posting as well as reading the headline.
Well, I read the posting too, and all the clever points Dan Olds makes are pretty obvious to anyone with reasonable reading comprehension skills. Which makes Dan Olds look not very clever at all.
Not anonymous, because I figure he put his name to it, so I can't do any less.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
