A moment of silence for all the drives that died in the making of this Backblaze report

Cloud storage and backup provider Backblaze has released a report on its hard drive failure rates for 2022 which appears to verify that the age of a drive is a key metric for predicting potential failure. Backblaze regularly releases stats covering the failure rates of all the hard drives under its management, and these …

[Yet Another Anonymous coward]

1% failure rate

Anyone stopped to think how amazing that actually is ?

Given the speed these things are spinning at and the precision the heads need to hit a bit of data the fact that only 1% fail each year is incredible (haven't read the report to see if that includes drives that failed so early they should really have been caught by the manufacturer)

[In Like Flynt]

Re: 1% failure rate

Sure, it's amazing... unless you're in that 1%. At which point the adjective changes from "amazing", to "fuuuuuuuuuuuuh..."

[John Robson]

Re: 1% failure rate

Then it goes to the phase of starting to take backups...

[Timop]

Re: 1% failure rate

This is easy if you have to manufacture only single golden sample.

Try adding variations from part tolerances and assembly process etc. And produce truckloads of drives weekly. Or maybe even daily.

[Potemkine!]

Re: 1% failure rate

I'm not sure this is that amazing.

First, those HDDs aren't that old, 5 years at the maximum. Next, for the mechanical part the physics involved are known enough to be able to make a robust design. The parameter there would be rather the cost, being very reliable is more expensive. The less reliable part is IMHO the electronic part of the HDD. AFAIK, reliability of electronic components is not as much studied than their mechanical counterparts.

[Anonymous Coward]

Re: 1% failure rate

Which part of all the drives that are 10 TB or smaller (bar one model that is 4 3/4 yrs old) already being over 5 yrs old means "those HDDs aren't that old, 5 years at the maximum"? Is 7.5 now magically less than 5?

I got asked a question 10 yrs ago, "what are the 2 types of hard drives?" - the answer wasn't FC/SATA or SAS/NL-SAS, it was drives that have failed, and drives that haven't failed YET.

Mechanical devices fail - and given the cost of hard drives, and the precision that they work at, it is incredibly amazing that the failure rate isn't higher.

Have you ever bought a car, a washing machine, dishwasher, etc, etc?

[Version 1.0]

Re: 1% failure rate

I've got a Windows XP system running with a 10Mb drive, it's still fine after about 24 years now, And there's another 10Mb drive that's still running fine after about 40 years but it hasn't been running full time, I only put the RL02 disks into the drive when I need to access data from back then.

And about 35 years ago I was working with a DEC customer of ours who was having problems with her system failing to boot up on a hard drive that I had installed many years earlier, yes it was "dead" - the disk wasn't spinning. But we talked about it and the next day she'd got it up and running to copy the data onto a new drive ... she solved the problem so quickly. She had unscrewed the top of the hard disk cover off and given her disk a little snip with her finger to get it running - it started spinning and they got everything fixed that day.

[Lost Neutrino]

Re: 1% failure rate

Sounds like the perfect definition for "robust". I wish modern cars were still like that: open hood, whack engine with a hammer, et voilà - vrooom!

[DuncanLarge]

Re: 1% failure rate

How the hell did you fit XP into 10 megabits?

[systemBuilder22]

Re: 1% failure rate

I am pretty sure they screwed up their message and it's not a 10MB drive. Typical 1999 drives were 1GB. I started using the IBM PC XT in 1983 with a 10MB drive. Nobody would be using that 5" drive 17 years later, let alone 40 years later.

[An_Old_Dog]

I'd like to see a report like this for flash drives and SSDs

The good thing about hard drives is they usually give advance clues they're about to fail (check your S.M.A.R.T. logs). The bad thing about flash drives and SSDs is, In my experience, they simply unrecoverably fail once-and-forever, but I'd like to see some volume stats on this.

[alain williams]

Re: I'd like to see a report like this for flash drives and SSDs

Backblaze do provide ssd-drive-stats.

[An_Old_Dog]

Re: I'd like to see a report like this for flash drives and SSDs

Their SSD report shows the number of failures, etc., but not whether or not there were pre-failure indications, which is what I wanted to know about.

[Anonymous Coward]

They(Backblaze) have done them before

But they broke them out into a separate list.

SSD's are tricky because they are so usage dependent though. But there is also a real problem with SSDs keeling over well before their write endurance limit kills them. There were a ton of mundane controller failures out there a few years ago.

[DS999]

Re: They(Backblaze) have done them before

Controller failure is a problem for HDDs as well - that's usually what causes those early failures in the first few months of operation.

It would be interesting if they tracked S.M.A.R.T. stats for remaining life for their SSDs along with failure rates. Storage Review did a long term test a few years ago running SSDs flat out for as long as it took for them to fail. Some exceeded their write life by as much as 3x.

EDIT: I read the article after posting this and it looks like they are doing exactly what I wished for above so it'll be interesting to check back in a few years.

[Sampler]

Re: I'd like to see a report like this for flash drives and SSDs

In an environment like this though, unrecoverably fail isn't really an issue as the data will always exist in multiple copies elsewhere, it's only really an issue for the home users not taught better and badly managed infrastructure that should know better.

[Andy Non]

At least spinning rust hard drives seem more stable nowadays

I remember back in the 80's. There was a tendency for brand new hard drives to fail within the first week or two of use. If they survived beyond that period they tended to last for several years.

[JoeCool]

Re: At least spinning rust hard drives seem more stable nowadays

That's the "bathtub curve" comment

[Andy Non]

Re: At least spinning rust hard drives seem more stable nowadays

Interesting. Not heard the phrase before, but it is quite apt.

https://en.wikipedia.org/wiki/Bathtub_curve

[An_Old_Dog]

1980s Hard Drives

Back then, we would run a testing program overnight on the drives going into the computers we were building. We caught many bad blocks not on the manufacturer's defect list (less than a dozen per drive), and a few outright drive failures, but by-and-large, they passed, and the computers they went into were not brought back to the shop for repair.

We also had a 100% component testing policy, and I caught a shipment of "Hexa" brand multifunction I/O boards from China in which ALL twenty were defective!

[mattaw2001]

Re: 1980s Hard Drives

Hands up all of us who remember the handwritten table of bad blocks on the hard disk!

[Anonymous Coward]

Re: 1980s Hard Drives

Wasn't there a DOS debug command you had to type to setup the drive and type that info in?

[An_Old_Dog]

Re: 1980s Hard Drives

From the Debug prompt ("-") ...

for Western Digital controllers, g=c800:5 [Enter]

for Data Technology Corporation controllers, g=c800:5 [Enter]

for Adaptec controllers, g=c8000:ccc [Enter]

for OMTI controllers ... I've forgotten.

[GraXXoR]

Re: 1980s Hard Drives

We used to call them soak tests, IIRC.

All our machines were run for 24 hours using an automated test suite that marked off all the bad sectors found on the disks and also was good at picking up failures in the SIMMs or CRT.

[simonlb]

Re: 1980s Hard Drives

Not 80's, but early 90's: Samsung SHD3062A 120Mb HDD's suffered from degrading platters so that after less than a year there would be more bad blocks than good blocks.

[alain williams]

How busy are the devices ?

Do we assume that all disks are as busy as the rest of them ? I would have thought that the ones doing more work might fail earlier. I cannot see some sort of I/O count.

They exclude boot devices as presumably they are not that busy.

[DS999]

Re: How busy are the devices ?

I never observed much difference in life between drives that were active 24x7 in DBs and drives that had a more sedate life as lower utilization file servers. There may be some slight correlation there but it wouldn't be worth them selecting for.

The reason they pull out boot drives could be something else, like choosing a smaller size for those or maybe they recycle veteran drives that were used for the main file store and don't want the physical handling required to count against them.

[Richard 12]

Re: How busy are the devices ?

Boot drives have a very different usage pattern to the data drives.

Depending on the OS and setup, a boot drive might be read just the once during boot and then spun down until the next update, spinning all the time but almost entirely read-only, or have near-continuous writes to log files.

They also tend to be much smaller.

Either way, it makes sense to monitor the two separately. Might be clues as to drive types which fare better for each type of workload.

[JMartin]

Re: How busy are the devices ?

> Boot drives have a very different usage pattern to the data drives.

From the link to ssd-drive-stats:

“Boot drives in our environment do much more than boot the storage servers: they also store log files and temporary files produced by the storage server. Each day a boot drive will read, write, and delete files depending on the activity of the storage server itself. In our early storage servers, we used HDDs exclusively for boot drives. We began using SSDs in this capacity in Q4 2018. Since that time, all new storage servers, and any with failed HDD boot drives, have had SSDs installed.”

[Happy_Jack]

Re: How busy are the devices ?

I'd be surprised if many organisations still bought spinning rust drives for boot disks. This is an obvious place to use SSDs given the smaller capacities needed.

[Timop]

Re: How busy are the devices ?

Vibrations might be large risk and it is easy to simulate resonant frequencies nowadays and make sure that nothing starts resonating while full rpm.

Speed changes etc might be the trickier thing. Because you need to set the component frequencies somewhere and definetly not to full rpm frequencies. So there might be some edge cases that wreak havoc inside the drive.

[Kevin McMurtrie]

Re: How busy are the devices ?

Less work can mean more failures. Let us not forget the IBM Deskstar 75GXP. There was a limited number of times that the head could fly over a track before the media flaked off. Idle drives had rapid data loss then head crashes in 1 to 2 years

Less crappy drives seem to spend their idle time scanning for weak blocks to remap. It keeps the head moving and moves data before the error correction bits run out.

Needless to say, I read the Backblaze reports before buying now.

[Graham Cobb]

Thanks to Backblaze

I just wanted to say thanks to Backblaze for doing this and publishing it. I read the reports each time they come out, and I am sure the manufacturers do as well.

Good on them!

[Anonymous Coward]

It's all relative.

A chart showing the average age of each drive model deployed by Backblaze against size shows clearly that the smaller a drive is in capacity, the older it tends to be

We have Seagate drives that are still running that are over 22 years old and run 24/7. To be honest I'm not impressed with those reliability figures for the 16TB drives.

[DJV]

Re: It's all relative.

Having been in IT in one form or another for nearly 40 years, I now refuse to buy Seagate drives due to the number of failures I or the companies I have worked for have experienced.

[Anonymous Coward]

Re: It's all relative.

With all the buyouts over the last few years there are now effectively two HDD suppliers.

Pick your poison, Seagate or WD.

Personally, I've had major issues with WD drives as well.

[eldel]

Re: It's all relative.

Yeah, I've been a Hitachi loyalist for HDDs for years and was really smug when the Backblaze stats seemed to validate that bias. Now that they're owned by WD though I expect them to reduce the QC and thus fail more often.

I did the last refresh 2 years ago so I'm hopeful that by the time they are ready for replacement there will be viable SDD options (home NAS).

[systemBuilder22]

Re: It's all relative.

Hitachi bought the IBM drive division, the inventor of winchester sealed disk drives. So it's no wonder they are head-and-shoulders more reliable than anybody else. They invented the tech and have been building it longer than anybody.

[John Geek]

re SMART data, different vendors use it quite differently. Look at a Seagate drive, and you'll see realistic soft error stats, look at a WD drive and soft errors will be ZERO until the drive dies. this makes doing any predictive analysis via SMART statistics quite meaningless