Re: At DEC's headquarters in Maynard
Going back maybe 7 or 8 years, a friend of mine had a rack half-full of SAN boxes. 3.5" hard discs throughout for bulk storage of research data, so you're looking at maybe 120 spindles.
Friday evening, he gets an alert to his phone - drive failure. No problem, as the hot spare will pick it up and he can arrange the replacement on Monday.
On Monday, no servers are up, and the alarms are kicking off on the SAN everywhere. With no servers responding, of course, it can't alert him again, so he's gone in totally oblivious to the carnage that awaits.
It turns out that the bearings on a drive have failed, and it's just sat there vibrating. All weekend. And vibrating so badly that dozens of the other drives can get the heads to settle enough to read or write. RAID sets with part-commits. Loads of corrupted data. Basically, a single failed spindle took out all of the data on 100+ drives.
He had good backups, but had to beg / borrow drives from anywhere he could get them to get stuff restored.
Biting the hand that feeds IT
