Backup a sec – is hard drive reliability improving? Annual failure rate from Backblaze comes in at its lowest yet Backup specialist Backblaze, which has published hard drive failure statistics quarterly since 2013, says its second-quarter 2020 Annualized Failure Rate (AFR) is the lowest ever recorded at 0.81 per cent overall. Backblaze had 142,630 hard drives in use at the end of June, across four data centres, ranging in capacity from 4 …
Fact is that you're spinning something at 7200rpm or 10,000rpm in a real-world environment and want to consistently write to and read that data from tracks whose sizes are magnitudes thinner than a human hair, in real-time, without delay, in a device that costs less than your year's broadband subscription and which will sit in a computer for years and years, subject to vibration, movement, electrical changes, temperatures etc.
Hard drives are an incredibly difficult thing to make. I'm amazed they've been able to continue this far, to be honest. When you consider SSDs, it doesn't seem as difficult - you want a silicon chip that does what a silicon chip does, and you'd like it to last a long time. But hard drives are immensely more complex and mechanical and subject to the whims of the outside world.
I find it interesting that SSDs tend to fail in completely different ways:
https://www.zdnet.com/article/ssd-reliability-in-the-enterprise/
And the write-life of SSDs is only ever growing. And when you think about that, that's the hard part in SSDs. How do you write to something that will maintain that data for a long period even if you never touch it again? That's why Flash and other tech took decade to be stable. Once it's there, of course you can read it a billion times, but how do you encode that state permanently enough, but where you can change it on will very quickly if necessary?
Hard drives are going to die, and I'm surprised they haven't already. And in the space occupied by an enterprise 3.5" hard drive, you can literally fit dozens, if not hundreds of NVMe-sized chips. I would imagine that in the space of even the largest enterprise hard drive you could easily fit ten to a hundred times the capacity with SSD. With no moving parts.
Now think - you're a hard drive manufacturer struggling to keep up with the technology to produce ever-tinier tracks and heads and cram in more platters. Or you can just fill a hard drive casing with dozens of rows of chips, beat it in capacity, and effectively sell your customer a "RAID" (remember when the I used to mean "inexpensive"?) over all those SSD chips without ever revealing that to the drive interface. One fails, just kick in one of dozens of hot spares and report that in your SMART output. Same capacity. Faster speed. Reliability of individual components made up for in extreme redundancy inside the device itself.
Write endurance in commercial SSD is into the thousands of total drive writes. The survey/paper above, for 1.4 million SSD drives, show that pretty much drives fail in other ways long before they ever hit the limits of write endurance.
I personally think that the large manufacturers should be totally abandoning mechanical hard drives and just ploughing all their money into cheap SSDs. At the enterprise level you can match or exceed everything about hard drive speed, response time, capacity, endurance, life-time and durability. Give it a few more years of R&D and hard drives will be obsolete. SSDs have come further in the last ten years than hard drives have managed in decades.
I was expecting HDDs to start moving to more of a archiving role (which to be fair, I think they have) and to get bigger again since capacity would then be more important than access speed.
So instead of shingling, why don't we just have 5 1/4 and 8 inch HDDs? (or even 10.5 inch platters)
Random guess: larger platters at 10krpm have huge linear velocities on the outer tracks, and are harder to make stiff enough to not wobble which is not good if your drive head sits that low.
(Even waaay back in the late '80s, someone made the comparison that a drive head is a 747 flying at cruising speed at 50ft altitude and a speck of dust is a 12 story appartment building)
Sorry but no, I don't see write life growing. SSDs became much cheaper and the reason is that rewrite capacity was sacrificed (though can be compensated by larger size). That Samsung SSD that I bought in 2015 (EVO 850 Pro, I think) has a higher relative endurance than the new crop (MLC vs. QLC).
A lot of data centers are accumulating data incrementally, which then sits there forever, hardly ever modified (think selfies: ugly, stupid, nobody asked for those, keeps accumulating). For this, SMR hard drives are ideal. They are lamentable for rewrite purposes, but the tradeoff is you gain a huge cheap capacity, and anyway rewrite is not quite necessary anyway in this scenario.
You really don't want an SMR drive for your personal enjoyment in most cases. I heard for years that people had the surprise, completely unannounced. That is a very effective way to kill hard drives: Just scare people by dropping shit on them.
"Hard drives are going to die, and I'm surprised they haven't already."
No, they aren't. This opinion is completely daft when you look at how storage actually works in the real world.
Enterprise-grade SSDs are still 10x the price per GB of HDD. And for some storage needs, all I care about is price per GB, the speed advantages are irrelevant. Thing is, if you actually look at the data growth in the world, the majority of it turns out to be speed-independent and price-dependent, so HDD is the better medium for it. HDD still makes up over 60% of world storage, and will still be about 55% by 2024. Given storage demand doubles every 4 years, this means that HDD space demand is still growing in enormously absolute terms, despite the relative decline in market share. And unless someone manages to reduce SSD production costs by a factor of about 20 (hard, since these costs are actually rising as NAND density increases, and a NAND fab costs 10-20 times as much in startup capital compared to an HDD fab) it's simply never going to compete on price per GB.
SSD is going to take the performance areas of storage from HDD, tho tbh it more or less already has - 15krpm is long dead and 10k is not long for the world. In the 7.2k and slower, HDD is not even really in decline - it's still growing at a healthy rate and will continue to do so for a long time to come.
I had a terrible experence with BB.
Their service from a pricing and technical point of view was absolutely first class - best in the world. I love using them.
Their service from a *billing* and *administrative* point of view was very nearly catastrophic, and very nearly destroyed my backups.
A service isn't just technical. If a customer is blocked from paying by changing you make to your billing system and you're going to delete his account because of it he'll lose his backups just as if you messed up the hard disks.
I had been using them for some time, year or two, and I came to change my payment card.
Turns out they had introduced new constraints on the cards - rules like "card holder address must be in the country of the card", "IP address of whoever is providing a card must be in the country of the card" and so on.
Upshot? back when I joined - and uploaded a metric ton of backup data, took two months - I could pay, and now I couldn't.
After about six weeks of utterly futile non-conversation with BB support, which ended up with them going silent, I had two days to go before my account was deleted and I lost all my backups.
So I faked an address - I'm now the Royal Albert Hall - and the stupid bloody address check they have, which blocked my card and took me to the edge of loosing my account and which took up so much time and effort to try to fix - *ACCEPTED IT*.
So I think they're idiots, when it comes to billing.
I don't dare change now my card with them, and when I get a bit of free time, I need to do the investgiation to find another provider.
That's a huge shame, because in every other way except catastrophic data loss through billing incompetence, they're ideal.
That seems bizarre.
It isn't as if they delivered a 200 inch TV in the middle of a forest to some guy with a beard and a dodgy looking credit card. They're in a position where they can revoke their service if they detect credit card fraud.
I'm thinking it might be their upstream credit card broker that has f..ked up. Plus there seems to be some EU regulation thing going on, because I notice that being a Norwegian living in Sweden means filling out quite many forms whenever I deal with my bank connections (on both sides of the border). Questions like "Are you going to funnel your drug money through this bank account?" and "How often are you going to pay for black labor out of this account?" gets asked every year. (I am assuming it isn't just for me, but I could of course be wrong)
Which isn't an excuse of course. Your situation should have been handled much smoother! :(
Yev from Backblaze here -> That's definitely very strange. I can't speak to why support would go silent, but I can say that we did have to update our billing rules in order to comply with VAT and other tax laws that were going in to effect, so we had to add new checks to verify the card/person were in the same location. It certainly shouldn't have been that hard on you though - if you'd be willing to toss me your ticket number I can take a look and see if I can come up with a suggestion to make it easier for you to update your payment in the future should you get a new card or address!
Sorry, but no - once bitten, twice shy.
I have absolutely -zero- problems with the fact the billing issue happened; I can understand fraud checks, security changes, in fact any change at all, things simply change over time.
With the best will and skill in the world it can well be once you've made changes there will be customers get tripped up by them. -Not- a problem.
But when a customer is telling you he can't pay because of the changes you made to billing, you need to find a way for him to pay. -This did not occur-. It didn't even remotely occur. I recall comments like "we will look into this and we may change it in the future".
Having had that experience, being two days away from losing my backups, but currently having a working solution where I can pay, I am -terrified- of getting support involved. It works now. Support can only make it worse.
(I think Support went silent because they had no solution to the problem. They could hardy write to me directly and tell me my account was going to be deleted and there was nothing they would or could do. I remember when I put in the fake address I received an email from support telling me I'd paid and asking me if I had any other problems!)
Totally understand and you should have at least gotten a "we have no way to fix this, sorry" message from support instead of them going dark - I'll let their managers know so they can do proper training. I'm not sure there could be much done in terms of mitigating that issue in this case (we can't just take a Venmo payment and credit the account for example) - but it's a good for us to know this occurred so we can work with our payment processor and let them know what happened.
I have to doubt the story above because best I can see on Backblaze website there is no form to add an address. Just the zip code and region in my part of Northern Europe. I recently started to use them again after a time period of extreme being broke. That might still not be over because of incompetent politicians in Iceland. But that is outside of this topic.
I also find it strange that you can't provide a case number since most people today use email services that store email for what amount to forever in most cases unless manually deleted from the inbox.
-- I got a title too long error message when I replayed to the above comment. I don't know why that happened since other people have been using that same title without any problems.
> I have to doubt the story above because best I can see on Backblaze website there is no form to add an address. Just the zip code and region in my part of Northern Europe.
It's possible the UI has changed; it is now something like two years since the events described. However, I would still expect that you will need to enter an address somewhere, at some point, for your bank card, for billing. It may be also that what you are seeing is different to what I saw, since in my case, the billing address was different to the address of the issuing bank. You may be travelling down a different UI path.
> I also find it strange that you can't provide a case number since most people today use email services that store email for what amount to forever in most cases unless manually deleted from the inbox.
I also expect the large majority of people keep all email forever. I did so too, many years ago. I do not now, and have not for a long time. It's a matter of privacy. I travel constantly, and so pass borders on a regular basis. At the border, your phone and laptop are wide open. You have no privacy at all. The only way in fact to have privacy is *not to keep private information*. As such, I regularly trim my email, deleting all sent, trash, and I typically archive nothing.
This is I would the emergent behaviour of the costs and benefits of privacy. Loss of privacy incurs a cost entirely born by the individual, with a benefit entirely born by the State. It is not surprising then that privacy has over time consistently been stripped back.
Ditto, I switched from Seagate to WD in the mid 2000s after some bad experience with Seagate. Haven't had a problem with WD, but then I've only bought three pairs of hard drives during that time. I won't ever buy them again since SSDs are getting cheap enough that next time I rev my desktop PC (in a couple more years at least) I'll go SSD only.
However, I'd trust Backblaze's numbers more than such anecdotal evidence, since they have a rather larger sample size.
A lot depends on the model of drive. I had a batch of 3TB Seagate drives that were awful. Each worked for 3-5 years then died - consistently. Out of about 10 drives I had from this range, I think 9 had died by 5 years. And not gracious failures either, they just suddenly popped and took all the data with them to the grave. Still, as the chart shows, Seagate do apparently make other drives that are reliable.
The current WD drives I have (WD Gold) have been solid so far, but I wouldn't buy another WD drive after all their fiasco of quietly sneaking SMR drives into their NAS line-up. Any manufacturer that tries to deceive people isn't worth my business.
Within 4 years of me taking over a site with dozens of brand-new Seagate drives in servers, NAS and desktop, every single RAID array or device with them in had experienced enough complete failures for us to just bin them - sometimes multiple rapid failures, sometimes the rest of the "working" drives pulled and put in other circumstances and failed there too.
In 6 years of running WD (as new replacements for the above, or new systems entirely), I've had a couple of WD Blue failures in desktop clients. That's it.
I know this stuff gets very anecdotal and people get possessive over their favourite brand, but literally I would never chance my data on any of the Seagate drives that are available to me, as a consumer or an IT guy. Sure, I don't do big datacentre stuff, but most IT isn't that and they can make whatever decision they like.
We've seen a very high failure rate with many types of Seagate drives, both regular harddisks and SSHD
For us, WD and HGST are the top drives, with some types of HGST's going beyond 9 years without any failures (Oracle used them in their 7210 ZFS machines, 48 packed tight, in a vertical position , we had 2 with 0 disk failures)
And I have some Maxtor 80GB 10k rpm 320SCSI discs that are still in daily use (or were before lockdown) and are coming up for 15 years old. The computers of which they are a component are powered up for around 7½ hours every day, 7 days a week. Pretty certain there are also some 160GB Maxtor SATA discs of the same vintage still in use too, but in both cases once booted the things are not exactly thrashed :-)
M.
I still have a 1280 Quantum kicking. It's always in use out of the novelty of waiting for it to die... but it won't. I can't find a utility to read the MTBF or hours, but it ain't play'n. It currently sits bare balls out with a USB->ATA adapter, no case, nothing... still kick'n. Sometimes I think i hear it laugh at the vastly larger new comers, it's a bully. Without explanation, the drive label no longer reads anything but "SATA == SISSY".
Since this is an opinionated thread, WD has been the worst over the last 20 years. All problems started when they moved to ATA133, so I'll buy ANYTHING but WD (I gave up after their ~1TB, which was about 20 WD drives past their 40GB ATA133). Samsung has been the best, but since they no longer make desktop platters, I bounce between Seagate and Hitachi (using minimal reviews and price for decisions).
Well my personal stats are four 3TB WD NAS drives that have been functioning flawlessly for over five years now, plus a 2TB WD drive in my main PC that has worked every day since 2013 and is still going strong.
Of course, my main PC also has a 3TB Seagate that has been functioning without a hitch since 2015, so I'm a bit on the fence here.
This post has been deleted by its author
You rate Toshiba as #2. Aren't they very loud? I have a pair of HDWN180 in my closet, and it sounds like there is a gang of dwarves in there mining for gold. They share a wall with the bathroom, so when I take a bath (or reign the porcelain throne) I am tempted to file a noise complaint with the local constabulary.
I'm gonna replace them as soon as I win the lottery.
Unfortunately, these Backblaze reports are not at all objective. The results reflect only their own experience and can't safely be extrapolated to drive reliability in general. This has been pointed out several times since their first report was published, and they don't seem to have improved their method. They therefore smack rather more of PR than of real attempts to inform.
The Backblaze method is subject to too many uncontrolled (and indeed unidentified) variables. The only reliable way to generate universally applicable (but even then only statistically valid) results would be to buy large standard size batches of identical hard drives from multiple vendors and run them side by side to exhaustion under identical activity. This would of course be very expensive and time consuming, but it would yield results that the pubic could rely on with reasonable confidence.
...it wouldn't as the drives used for the test would be no longer available. Average service lives vary enormously from model to model and even from batch to batch for obvious reasons. This explains the anecdotal evidence being very inconsistent. For example I've had multiple failures with HGST drives. What drive to buy? There's no right answer, the only solution is sound backup practice and eternal vigilance.
Unfortunately, these Backblaze reports are not at all objective. The results reflect only their own experience and can't safely be extrapolated to drive reliability in general. This has been pointed out several times since their first report was published, and they don't seem to have improved their method. They therefore smack rather more of PR than of real attempts to inform.
Backblaze make make no claim that there findings should be taken as reflecting anything other than what they are - their results, with their setup, under their conditions. That doesn't make it not objective; perhaps you used the wrong word there. Nor is there any particular reason why Backblaze should change their methodology to try to apply it to use-cases other than own, in fact they'd be nuts to do so. They happen to run a lot of drives, they can produce reliability statistics on those drives, and many people find their results interesting, even if they are a specialised use-case. I'd certainly rather have their results than not.
It may not be solid research, but it's still more significant than personal experience (including mine). Not many companies use that many drives, so it's nice of Backblaze to publish the figures they have.
And they're not saying it's research. They're doing everyone a courtesy.
The report is not a study but a data point provided by a company who uses a lot of drives. It implies drives are can be very reliable on demanding conditions. Conducting such a study would require careful study design, i.e. what use cases are you testing (home/SOHO use, corporate workstation, data center, etc.). If someone who has thousands of drives says they are getting these failure rates it does provide a glimpse into the reliability of drives in real world use.
Many benchmarks are artificial because the testing does resemble real world use. USEPA mileage figures are notoriously wacky (normally high) because the testing does not reflect real driving conditions. But as a comparison number between vehicles they are useful as they give a relative idea of fuel usage.
"A table of cumulative "drive days" since 2013 shows that drives are typically kept in use for several years so the improved failure rate is not just a matter of using new drives."
It probably is, it's just that it's drives that were new several years ago are effectively being compared to ones that were even older. Hopefully the trend will continue as drives that are new today turn out to be even more reliable a few more years down the line.
The tentative conclusion I draw from this is drives have been improving with newer drives being more reliable than older drives. This seems to follow the trend others have seen with drive reliability. However to use this data for drive selection without more research is not wise as it is just one set of data.
The only way you can compare reliability is to place many samples of each drive in identical situations for a period of years. The current method of seeing how many complaints/comments each type of drive gets is flawed as we don't know things like sales. Seagates get lots of complaints but that might be because they sell twice as many? So overall they could be more reliable. IMHO :-)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
