Western Digital shingled out in lawsuit for sneaking RAID-unfriendly tech into drives for RAID arrays A US law firm is seeking owners of certain shingled Western Digital drives to join a class-action suit against the storage slinger, saying the manufacturer didn't document its use of shingled magnetic recording (SMR) in kit aimed at RAID arrays. Hattis Law today revealed it is looking for plaintiffs in the US who bought the …
As it happens, I *just* unpacked a 4TB WD Red drive that I ordered three days ago. And thanks to Chris Mellor's article, I specified the old 64MB cache version (model WD40EFRX, CMR) rather than the new 256MB cache (SMR) drive.
The model number is identical to the 4TB drive I intend to pair it with in my NAS... but there is no external indication on the label that it is the 64MB cache version, unlike the older drive that explicitly says "64 MB". I suppose hooking it up to a Linux box and running hdparm is in order.
...but I wonder how much the plaintiffs will get. Hm, what? How do I know that they will make a lot of money? Because they will wind up probing the reasons why groups like FreeNAS "highly recommends" WD Red drives to their users:
https://www.ixsystems.com/documentation/freenas/11.3-U3.1/intro.html#hardware-recommendations
I specified WD Reds for a recent small business 4-drive NAS, having had no problems in the past. I now know the business got SMR drives. It's impractical to replace the working drives until failure. However, when that happens, the replacement drives wll not come from WD. WD are now blacklisted, as far as I'm concerned.
What a stupid way to gift future business to a rival.
If one fails and you have to rebuild your RAID, that's the danger time, as these SMR drives are NOT suitable for large data transfer, which is exactly what happens during a RAID rebuild. That's not 'they may be a bit slow' unsuitable, it's 'the drive might seize up and you lose all your data' unsuitable.
"" They're all doing it.""
Yes on consumer line (witch isn't ideal)
but seagate and Toshiba never attempted to drop smr into nas or enterprise hdds (unless specifically marked as SMR for the Enterprise side) they normally make a very specific model that is smr (and most of them are host managed smr)
In the wake of the issues, Seagate and Toshiba also confirmed some undocumented use of shingled magnetic recording in some of their drives.
What a stupid way to gift future business to a rival.
Indeed, all of Western Digital, Seagate, and Toshiba's many, many competitors as hard disk manufacturers must be rejoicing.
I’m not so worried about undocumented use of SMR in Seagate archive drives. I thought everyone knew they were SMR anyway. Barracudas, I’m not so sure about. Who uses mechanical drives in desktops though?
There’s also Hitachi (owned by Western Digital). They’ve been using SMR in the same way.
But the competitors are SSD drives. For anything up to 2TB, it makes no sense to go mechanical now.
Depends on your usage and how important random read access of small blocks is vs large file sequential r/w. Also the quality of SSD. Some are basically the same chip tech as cheap USB sticks but with a different bus interface. How much warning of failure vs mechanical HDD and what backup strategy does the user really have? What is powered down archive life?
That's really too sweeping of a statement.
I think you only get SMART disk failure warnings about 1 time in 3.
When Google published large scale statistics on mechanical hard disk failure rates in their server fleet (enough years ago that I feel old thinking about how long), one of the conclusions they mentioned was that SMART disk monitoring have them almost no false positives but lots of false negatives.
As I remember it, pretty much every time a drive announced an error via SMART, it died soon after. But when a drive failed, only about one third of the time was it preceded by SMART monitoring errors.
Here are some more recent figures from a company that has over 1 exabyte of storage. These links are in order of oldest to newest:
https://www.backblaze.com/blog/hard-drive-smart-stats/
https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/
https://www.backblaze.com/blog/smart-stats-exposed-a-drive-stats-remix/
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2020/
Finally the link to the links for all of the raw data over the last six years and hints, tips, and a few limitations on how this data can be used:
https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2020/
SMART data is useful if you have the tools to understand it. The hardest part is learning all that you need to because it is a data space convoluted by vendor choices and obscured information. I rarely learn anything about it that I don't have to because of this but there are people who do.
"But when a drive failed, only about one third of the time was it preceded by SMART monitoring errors."
Conversely: IF you know what you're looking for, monitoring SMART can tell you a drive is having trouble long before it even issues a SMART warning
Hdd sentinel still requires that they drive actually logs any smart events that's (I use it on every pc it's not perfect at all)
what Google were saying they had drives that showed no signs that they were actually failing via smart they just basically failed without logging anything I smart,, but drives that did show smart of some sort they didn't last too long afterwards
"But the competitors are SSD drives."
There's SSD and there's SSD.
SK Hynix have shipped some utterly shitty NVME drives (less than 140MB write speeds and 300MB/s read speeds) which HP have bundled into their desktops and sold as premium product - but because they refuse to publish specs on what they put in their products they got away with it.
HP Europe response to complaints has boiled down to "We got your money. Tough luck!"
Our response was "You blew any chance of future sales and jeopardised tens of millions of dollars of nationwide future contracts on the supply framework we use. Oh dear, how sad"
** "There’s also Hitachi (owned by Western Digital)" **
Um, no. Not quite.
Hitachi Global Storage Technologies was created by an IBM/Hitachi drive tech business merger in 2003.
WD acquired the business in 2012 and rebranded it as HGST and from that time, it had absolutely nothing to do with Hitachi.
WD does not own Hitachi.
"WD acquired the business in 2012 and rebranded it as HGST and from that time, it had absolutely nothing to do with Hitachi."
More precisely, it was rebranded HGST in 2012 because Chinese regulators wouldn't allow WD to merge the company, citing duopoly concerns.
They relented in 2018, after Toshiba was sold a chunk of WD's 3.5 inch manufacturing business (in the meantime Seagate hoovered up Xratex for their test equipment and dumped the drive arrays)
At the time I commented that the Chinese probably saw large SSDs being on the cusp of being cost competitive with HDDs - they were until a fire at Hynix destroyed a big chunk of flash capacity in late 2018
It's now clear the Chinese duopoly concerns were valid - and farming out stuff to Toshiba didn't "fix" that issue.
In the meantime, SSDs with similar DWPD figures to HDD "NAS" drives (180TB/year = 0.15 D[RW]PD and unlike HDDs, reads on SSDs are "free", so 0.2-0.8 DWPD for things like Micron's ION range is more than comparable) are now about three times the price of HDD "NAS" drives - coming with 5 year warranties instead of 3 year ones and built in power loss protection (Samsung's consumer 860 ranges don't have this, let alone the QVOs. Besides, the QVOs cost more)
I would like to see some proper data on how this affects the performance of the drives in their intended use The only drives in the Red range affected are between 2 and 6 TB. WD have done themselves no favours PR wise but provided the actual RL performance is up to par I really don't care what is under the hood. As an aside I see the WD website now specifies the recording technology used in their drives.
I would also like to see what "harm and damage" clients have experienced. I have a feeling this is mainly driven by lawyers looking to make a buck.
Of course this is about lawyers trying to make a buck. But that doesn't mean there isn't a real problem here. Read the stories over at blocksandfiles.com and do some searching of your own. You will find that most of issues with these NAS drives are when one of the drives in RAID array fails and a rebuild is triggered. Because of the way the SMR tech in the drives works The rebuild fails time and again even with brand new drives received via RMA. That defeats the purpose of having a RAID in the first place and therefore these drives are "not fit for purpose" which legally is a lot different that an Emptor failing to Caveat.
I'm curious as to why the rebuild actually fails.
Although contrary to popular belief drive-managed SMR[*] isn't great for most sequential write patterns, if you stream enough sequential data (100-300MB minimum, I would guess, depending on the drive generation) the ones I've played with will eventually go into a streaming mode where they're as fast as normal drives.
I should go and look at the original reports - I'm going to guess that the rebuild was interrupting its sequential writes, and then wasn't able to handle the multi-second write latencies that were the eventual result.
[* the other SMRs are host-managed - they report errors if you try to perform non-sequential writes. They don't sell those through normal channels, probably because they figure we'd RMA them for working as advertised]
Personally I can't say I have first hand knowledge of why they fail in every case. I have read (see below) that certain versions of the Red firmware may be to blame. I think that only WD could answer that with any certainty at this point and would invite anyone with evidence to the contrary to post a link or two.
As for the what happens when things don't fall over during a rebuild:
https://www.servethehome.com/wd-red-smr-vs-cmr-tested-avoid-red-smr/
They also don't report as SMR. Thus they may fail as software/hardware expect them to perform as CMR and they don't!
It's all about the bait and switch and the hidden fact. All those going "SMR is fine, I don't see the problem" are missing the problem is the consumers were lied to/tricked, and that causes other additional problems later down the line.
Getting a red jumper instead of a blue one has no *physical* harm to the consumer, but does still fall four of sales or trading laws if the customer is tricked or lied to. WD have been very tricky here, and have gained as many shills/Stockholm syndrome supporters as they have lost consumers who wised up to this scam. :(
"I'm curious as to why the rebuild actually fails."
Because the drives have a firmware bug and when given sustained non-sequential writes (ZFS RAID rebuilds aren't sequential. RAID5/RAID6 may not be either) after 40 to 90 minutes they start throwing bogus internal errors "Sector ID not found" - you can see this in "smartctl -x" views of the drive internal extended log) and throw a HARD write error back to the host computer
The only way to stop this happening is to idle the drive after 40 minutes activity for 1-2 hours to allow it to write out its CMR cache area (about 100GB) before you can resume the rebuild.
In my mind, the issue is that they branded them as NAS drives, so people went and put them into RAID arrays. RAID5 is just about the worst possible thing I can think of that you can do with drive-managed SMR.
For that matter, last week I just gave up on a DM-SMR drive I was using for my time machine backup because I was getting annoyed at how slow and noisy the hourly incremental backups were getting. Not gonna ID the vendor because it was a repurposed engineering sample, and the final shipped version might not be quite as crappy...
"I would like to see some proper data on how this affects the performance of the drives in their intended use"
I kicked this off with Chris because the replacement 4TB REDs I got ti replacing aging ones in my home NAS went from 18 hour RAIDZ3 rebuilds on the old drives to failing to complete after 21 DAYS and WD were blowing off complaints.
RAIDZ3 rebuilds were _still_ 18 hours on the old REDs, and on Toshiba N300s + Seagate IronWolfs purchased to crosscheck. The "New" SMR REDs were the same price as the CMR REDs they replaced but more importantly the CMR drives were simply no longer available anyway. The N300s and Ironwolfs were slightly cheaper than the SMR REDs
That's when I started digging, found out about the shingling and realised they had bad firmware - both problems WD were denying but were fairly well documented in tech circles. THEN I found out about the SMR Barracudas and realised "ghost" networking problems my group had been chasing for over a year weren't caused by networking issues - we verified it was down to the SMR drives choking the desktop computers they were installed on.
WD and Seagate were empathically _refusing_ to tell media or customers whether drives were SMR or CMR. It was only when Chris confronted them with retailer lists (skinflint.co.uk) that they relented and admitted it.
As a note, the drives were "tested" previously, so the SMR cache will be full on the Drive Managed SMR drives, as the OS cannot tell it to flush/etc the cache. However, there are many many times you could fill the cache in a few mins and then hit the 9 hour build/write times or just kill performance by asking the drive to do too much in one go! (Unlike a CMR drive, that will continue at an average speed, even though head seeking)
"The use of SMR in WD Red NAS drives was uncovered by The Register's sister site Blocks and Files, whose investigatory work is highlighted in the lawsuit's paperwork."
Uncovered? Don't you mean, "reported on"? The DataHoarder community on Reddit likely discovered it first, and now typical journalistic parasites are trying to take the credit for it.
Plus a ton of "reporting" has been defending WD and how SMR is magic and amazing in all RAIDs and never fails, and any different in performance is only 5% or less and WD is doing everyone a wonderful favour when they submarine these models into the product line.
I don't think they are shills. I just think they are very very broken themselves. :P
The Datahoarder community did document it. Chris made no secret of the discussions in that or that I brought it to him in the first place
Other "journalists" have been less open about it, and there's been a lot of flat out plagarising of Chris's reports without attribution, or crediting him for the story without looking at what he wrote.
A lot of the astroturfing pushing WD smells less of Stockholm Syndrome and more like having the same grammar and language constructs as the WD PR and manglement twats I was dealing with before I went to Chris.
I can only assume the other reporters/forum posters are (not paid shills) working for, friends with resellers/marketers, getting advertisements from, or just have so many WD drives invested emotionally in, that "WD did no wrong" in their minds.
A bit like when Intel/Nvidia/Apple make a mistake, it's still AMDs/Samsungs fault in the hearts of the fans. ;)
Bear in mind we're the ones sticking our neck out on the line when we report things. Any one can post a theory to Reddit, and if it's right, we're thankful they alerted us and the world. If it's wrong, well, who cares, it disappears into the mix.
When we publish something and it's wrong, there are consequences - legally and reputation-rise - which is part of the reason why we try to get everything right first time (the main reason is none of us go to work to spread misinformation, quite the opposite).
Happy to tweak it to say B&F first reported it - but don't call us parasites. We're the ones who have to stand up the rumors, and carry the can if it turns out to be wrong. 50 percent+ of journalism is figuring out if what you've just been told, by someone on the street or someone in marketing, is actually true.
In Chris's case, he managed to get a confession out of WD, one that made it into a lawsuit. That deserves some kudos.
C.
I don't think The Reg was doing anything other that posting one side and the other side of the story. However, a lot of sites, those who do advertising... ahem, I mean product reviews, all strangely back WD as never ever having any ability to actually ever do wrong. ;)
This has to be the best graph ever for everyone who is a professional claiming "It's exactly the same, WD did nothing wrong" and saying all these customers are the ones at fault:
https://www.servethehome.com/wp-content/uploads/2020/05/SMR-RAIDZ-Rebuild-v2.png
(Disclosure: I'm the one who brought this mess to Chris' attention along with various tech mailing lists and notified ixSystems about the F/W bugs. Chris verified the issue, got admissions from WDC & SGT which they'd refused to give to consumers plus a voluntary statement from Toshiba about their use of SMR in consumer products - I'd assumed Tosh had been more ethical in disclosure up to that point)
ixSystems have withdrawn their recommendation for the SMR drives - and they verified my report of FW bugs in RED SMRs.
The reason they gave that recommendation was that WD RED CMR drives have been hellaciously reliable. My old REDs have run 8-9 years 24*7 without missing a beat whilst other drives tended to only last 5-6 years. WD built a reputation and trashed it in a matter of 18 months.
As for the Class Action - this COULD be a submarined way of WD heading off consumer protection "death of one million paper cuts" in US courts. By certifying a class in USA courts it means anyone filing in lower courts (small claims) is likely to find their claim pulled into the class action.
Don't forget: WD, Seagate and Toshiba have been explicitly marketing SMR drives as suitable for home and SOHO use. That means the "Get out of jail free" card that was used against misleading advertising litigation in the past ("these are business devices and businesses must do their homework") is ripped up and thrown away. WD compounded the crime by selling the things explicitly as suitable for RAID use when what they meant was "Mirroring", then gaslighting consumers who complained (as did Seagate) and then issued press statements gaslighting the public (Say kids, Can you say "Exemplary Damages"? I knew you could.)
In this case the class action is targetting WD. I doubt it will be the only one and I'm doubtful consumer protection agencies around the world will sit on their hands - particularly when they look at the combined actions of the three companies
If the rollout of DM-SMR in consumer drives had been "coincidental", then at least one of the makers would have disclosed it on the spec sheets and/or would have released SMRs at a considerably lower price point than their CMR product and/or would have pointed out what the others were doing.
If it looks like a cartel behaviour duck , quacks like a cartel behaviour duck and walks like a cartel behaviour duck, it's highly unlikely to be a coyote.
Icon, because Micron have parked SSD tanks on HDD maker lawns.
You can buy their 5210 ION "cold storage" (meaning 0.2-0.8DWPD, something akin to archival SMR HDDs, not "powered off") SSDs for twice the price of Enterprise HDDs of the same capacity (or about 3 times the price of a RED/IronWolf/N300 - and the quoted endurance of 180TB/year for RED or IronWolf or N300 NAS drives is a lot LESS than 0.2-0.8DWPD)
Not bad for an ENTERPRISE drive that draws ~2W, has power failure protection and a 5 year warranty (the HDDS referenced above are all 3 year warranty)
RIght now on Insight, UK IONs are listed at £308/4TB($380), £580/8TB($680) +tax - and the way SSDs are still falling in price they'll rpobably hit parity with the Enterprise SMR drives by the end of 2020
Over a decade ago I lost nearly all my data to a WD Drive. My fault actually. That back up that was two weeks old, was not really. The 6 month old one was viable.
Then there's the disabling of SMBv1 by Microsoft. Ok, it's insecure, but the NAS manufacturers did not update there firmware, and Amazon et al still sell the old ones, and the Router manufacturers also do the same thing. Yes you can build one (NAS), but buying things off the shelf is no longer really an option. My rescue of the WD data though old was from DVDs I had burned and I have the sense to keep a usb DVD writer around for the truly important stuff with multiple copies made.
Best I can figure is all manufacturers have lost interest and are persuing the sweet subscription stream that is the 'cloud' and making home kit more and more cost prohibitive and offering kit that is no longer reliable.
"Best I can figure is all manufacturers have lost interest and are persuing the sweet subscription stream that is the 'cloud' and making home kit more and more cost prohibitive and offering kit that is no longer reliable."
Exactly. BUT, and this is a big BUT, home/business broadband needs a kick up the arse SERIOUSLY as todays speeds are nowhere near enough when you need to back up terabytes and petabytes of data nightly/weekly.
The "cloud" sounds oh so good until you factor in the total cost in a year, plus the poor speed of internet vs. the cost of hard drives. Hard drives are NOT dead yet.
Not just that, due to the recent take up of remote working and homeschooling, you can't even order FTTC in a lot of areas at the moment, FTTC cabs are at capacity, because Openreach have been 'hot running' their cabinet 'on the cheap' with no excess resilience / contingency.
Openreach are fucking joke of a company and Ofcom need a major overhaul, utterly useless.
FTTC needs to be the legal minimum, not an utterly useless 10Mbps USO, needs to be at least 30Mbps *now*, with a goal of 200Mbps by 2025, even that is the bare minimum.
Yep! I have an old NAS I liberated from the bin at work which I use for backup purposes. Maximum drive size it takes is only 1.5TB (four slots) and write speed is pretty crap due to the age of the thing (roughly 10MB/s maximum - despite having a 1Gb Ethernet connection). Thing is, that's still nearly 10 times my broadband upload speed (roughly 1MB/s), so although syncing 100GB to it takes a few hours, it would take *days* if I were to try and sync an amount like that online.
... although syncing 100GB to it takes a few hours, it would take *days* if I were to try and sync an amount like that online.
It's a corolary of that old saying: "Never underestimate the bandwidth of a truckload of hard drives".
It's easy to overestimate the usefulness of cloud backups when one fails to realize the connectivity bandwith one would have to have to restore from the cloud as fast as one can from a storage array or even from tape.
"Best I can figure is all manufacturers have lost interest "
95+% of volume in HDD sales is to bit barns and it's arguable they should have gotten out of retail years ago.
Instead all three makers have burned a lot of trust and I can't see anyone with any sense buying SSD products from a company that's seen to have been willing to commit fraud (ie: ALL THREE makers). We'd already seen them willing to gouge prices and slash warranties in the wake of 2011 Thai flooding, along with drive reliability _PLUMMET_ - this latest stuff is just the icing on the gilded turd.
SSD makers are more pragmatic:
The _smallest_ Samsung drives you can buy in most ranges are 256 or 512GB for the simple reason that there's no profit in selling something that hits retail channels at less than $80
You can't even _buy_ a sub-2GB USB stick anymore, unless it's someone's refurb or been kicking around at the back of a warehouse for 15+ years
My rescue of the WD data though old was from DVDs I had burned
You are fortunate, but don't rely on them long-term.. For curiosity I looked through some old (10 years +) DVD-Rs of mine and a surprising amount were unreadable - they had an index of the filesystem but the files themselves were unrecoverable.....
The files themselves were unimportant or backed up elsewhere (local NAS + Cloud) at least....
*gloats at being able to restore files off an LTO1 tape once a drive and machine and software were found for it.*
If it's important, back it up to multiple places, on different types of media.
(bring on the downvotes- You know I'm right about tapes!)
As a happy user of XigmaNAS (formerly NAS4Free, related to but not the same as FreeNAS), we are all set for a new "2nd backup or archive box". In other words our NAS needs a backup (along with a cloud backup). Religiously following standard practices of WD Red. Our RAID is just RAID 1, i.e. mirror, on all boxes. We have now WD Reds. And bought some new WD Reds (4 TB). THEN read the Register item. What to do??? It seems like the planned ZFS snapshotting between boxes (or something, need a little more research on the best approach) is going to run into the new shingling problem.
Our use case is likely very common. A very small group of people (a little business, a little family) with individual systems AND a shared file server. Which is backed up itself. And individual systems are also backed up to the backup system. Not exactly a "corner case".
So what does WD do? Ruin the "Red" reputation where it was just the "no brainer" choice. For a "slightly more informed market segment" of "sysadmins" who take some responsibility for their technology. I suspect that a fairly large portion of the buyers of WD Red NEED the CMR technology! And yet WD is making the excuse that SMR is suitable for "families or small business". In fact -- NOT. Because it's easy to image a write-heavy workload in a SMB or family, especially if you have proper backups.
Market Analysis for WD Reds: Segments
1) SAVVY TECHNOLOGISTS / Family and Small Business: WD Red CMR was the "go-to brand". And use case DOES for many have a lot of writes.
2) NAIVE TECHNOLOGISTS / Family and Small Business: Never heard of WD or "Red". Not so many writes.
3) SAVVY TECHNOLOGISTS / Larger SMB: Target now WD Red Pro or higher. Big price jump.
It appears that WD Red marketing has managed to kill a very strong mini-brand. And trust. And in a way that does not make sense. Basically confused #1 and #2 above. I wonder what the extra margin on manufacturing was? Of course if you can get everyone in #1 to redfine themselves as #3, then the margins will certainly be higher. But you have taken a HUGE risk. And a loss of brand trust.
I'm beyond my 15 days at Canada Computers, but I'm going to see about a refund anyway. And what do I replace with? Seems that CMR is required. What brand remains to be determined. The idea of brand is to avoid this sort of extra market discovery. If I have to think about WD Reds, it's no longer a brand. Bravo The Register!
IS it worth the cost and time and effort to sort out? If not, just go for CMR.
If it's worth the effort, you can test, test and test again for this new type of drive behaviour (SMR that does not report as SMR and hopes you can trust it to do things right, and not do garbage collection at the worse time).
At least so far with SSDs they learnt their lesson, and most will be made to have decent garbage collection speeds and controllers that can cope with it (though I've had at least one SSD that died after 15 mins ever single time after a firmware level clearance of the entire drive, so good, it never worked!).
Sometimes you *can* do something and make new technology. But you need to ask if it's worth the hassle, and if there is a use case. Here SMR can be used in some types of NAS, and in others... it might chug.
"I'm beyond my 15 days at Canada Computers"
No you're not - and the reason is simple consumer protection laws.
The drives are not as represented(*) and are unfit for the purpose for which they were marketed.
(*) They are MARKETED for home/SOHO "RAID and NAS use" - which means firstly they can't claim they're a business device being repurposed and secondly that consumer protection laws apply
Resellers are only belatedly becoming aware of the can of worms that WDC, SGT and Toshiba have handed them in terms of product liabilities and the ones I've spoken to are NOT HAPPY about it, however they're the ones who have to carry the can in most countries and will have to recover the losses from the HDD makers. At least one large distie has made a decision to withdraw all SMR product from their catalogs
If I get it right, there are two problems with SMR: One, write operations sometimes take an extraordinarily long time, which is pretty annoying. Two, RAID drives assume that the drive is broken because it takes too long, which isn’t annoying but fatal.
On the list of changes in the last MacOS is one “make large writes to RAID drives more reliable”. They can’t do anything about the performance, but in a software RAID they can stop the drive being flagged as faulty.
Hello Smart People!
I am trying to catch up on all things tech, while dealing with a virus, a hacked email account and identity theft (been a great year for me, lol). This was all while using BD and 2 factor auth and a VPN. Obviously, I have a long way to go!
(Computers should come with a copy of The idiots guide to online survival.)
My apologies if this has already been answered already, but is anyone still selling PMR external hard drives? I just opened my new WD hard drive and read class action suit, googled, and found all of you...so Thank YOU! It will go back.
Any ideas on what I should have purchased? I need a reliable external hard drive and stat.
Many thanks,
S
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
