Wanna curb datacenter outages? Try combating burnout with shorter shifts Datacenter outages remain a perennial problem, with human error among the top contributors. Analyst outfit the Uptime Institute suggests the key to curbing these disruptions could be as simple as shortening shifts. Uptime reported that four in ten surveyed datacenter operators experienced a major outage in the past three years …
I did find the 2021 version of this report on google scholar, It's here [pdf] .
p.11 lists a breakdown of what the reason behind the "human error" outages was. only 48% of them was actually an employee making a mistake even being part of the reason. 41% indicate improper procedure was (partially) at fault for the outages. is it even human error if the human follows procedure? yes, sure, but you can hardly blame it on the person making the "mistake".
Assuming this year's report includes this data as well, this would be great to know, as it gives important background on this article.
> .. 41% indicate improper procedure was (partially) at fault for the outages. ..
“In 2020 and 2021, we asked about the primary cause of .. most recent major outage .. The results show that on-site power problems remain the single biggest cause of significant site outages”
"Ooh, deffy-nevvy plonky all your eggy-wegs in the one baskie-waskie"
One of the problems with firefighters and support staff is if they're doing their jobs properly (and the stars also align correctly) then what you end up with is a whole bunch of expensive professionals sitting around doing nothing. In accountantese, "They're redundant". So its really tempting to start paring them back or giving them a lot of other tasks to do so as to get value for money from them. Then the odure hits he air mover.....
Its not just these sorts of jobs as well. Its well known that most people don't understand risk, they think that (for example) paying insurance premiums is a waste of money because "they've never had a claim in twenty years". With firefighting at least there's an accumulated body of knowledge that tells us what kind of coverage is needed for a particular area at a particular time (and there's often laws to back it up). Business isn't so well endowed.
Having local "firefighter" support is also tough if you have strong central controls that keep problems from happening often or the opportunity for anyone local to participate in meaningful work.
It's a similar problem to hiring police officers for school safety. You wonder why you can't have an on-site swat team hero after they have been sitting on their butt talking to kids about their feelings for 20 years.
AKA The Maytag Repairman Syndrome. I spent my entire 50 years in various types of technology hardware support.
No one wants you around when there is nothing to fix. But better be quick when the Merry Go Round stops. Be quick to take that call at 3 AM and drive through snow drifts. Half your time is spent filling reports that justify your existance. My son works in a similar existance that he actually loves because many of his service calls take him to Hawaii and he is a member of every existing travel perk club. Much of his time "off" he is on his laptop doing remote troubleshooting.
It's like the Rudyard Kipling Poem TOMMY:
For it's Tommy this, an' Tommy that, an' "Chuck him out, the brute!"
But it's "Saviour of 'is country," when the guns begin to shoot.
So tech wants to save money on paying employees but they want to buy Boston Dynamics robot dogs ?
A few of those dogs must cost millions, and they cant possibly do anything approaching what a human can do.. so basically thats wasted money.
Its amazing how big tech always wants to screw workers, just because it can when the reasonable option woul dbe get more people and stop wasted money on bullshit like robots.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
