Human error and power glitches to blame for most outages Datacenter outages are less frequent and severe, but human error remains one of the most persistent challenges, with between two-thirds and four-fifths of major wobbles involving some element of meatbag-related cause. According to the latest Annual Outage Analysis report from Uptime Institute, the overall picture is one of …
Was curious, I ran a search but didn't come across any. El reg's article mentioned some facilities on generator power, I saw another article mentioned some telecoms were running on generators, one was operating at 70% capacity apparently(which to me could mean LTE/5G cell sites were down), but not seeing mention of large scale data center failures. If so that is pretty surprising given the sheer range of resiliency options that different data centers provide.
came across this article that mentions a bunch of specific providers saying they had no loss in power as they were able to use UPS/generators without a problem as a result of the large power outage in the country
https://www.datacenterdynamics.com/en/analysis/how-did-iberian-operators-fair-in-the-great-power-cut/
I haven't experienced an unplanned data center power loss for the companies I work for since 2007 (that facility had a bad design(which wasn't widely known) and the company I joined was already there, I moved them out). I have experienced maybe 3-5 facility power outages at the colo where I host my personal equipment for the past 13 years( no outages in past 3 years though). But that is a super old facility(perhaps 25-30 years old) and does not have redundant power. So not a huge deal for my personal gear at that price point.
actual power outages(to me means you have redundant power coming from two or more different UPSs backed by two or more different generators and having both fail simultaneously) at properly built data centers are exceptionally rare(this excludes hyperscale facilities as they are built to a lower standard by design and customers are expected to place data in multiple locations to handle failures) from what I have seen over the last 20 years. Also excludes facilities exclusively using flywheel UPS (which for me is not proper design as a flywheel UPS doesn't last long enough for a human to even try to get involved in the event of an issue such as failure to start the generator or failure to switch to generator).
Yup...can confirm that...I've had more problems caused by UPS kit than UPS has kit helped prevent.
It's the one thing in your comms room that can take everything out if it blows up.
Last UPS meltdown I had took out about 80% of the servers...fried motherboards and PSUs everywhere. Luckily I've always got backups in these situations, but it's an absolute pisser to have to explain a UPS killing itself.
Even if you get your UPS properly serviced once a year, you can still have random battery explosions etc...
Over 25 years, I've had an average of about two UPS failures a year to deal with. Doesn't matter who makes them either. Be it Eaton, Liebert, APC etc etc...it doesn't matter, I've experienced them all blow up at some point or another and it's almost always without warning...sometimes you'll get a few hours warning if you're lucky.
If I was forced to pick a brand of UPS that seems to sometimes survive for insane lengths of time, I'd have to go with APC...it's rare for me to find many UPS units over 5 years old, but when I do, they're usually APC...that isn't necessarily a good thing, because all UPS units are ticking time bombs in my book...but finding a 10 year old UPS in the bottom of a rack is usually pretty scary.
There is a particular model as well that seems to last forever, it's those 2U ones, I think they are APC 1500 or something...there was one particular year they had an amazing run on building units and they just last...similar to the 2008 Western Digital Greens...those bastards seem to go on and on like the Duracell bunny as well...I've come across tons of server (yes servers) filled with those that have been running for over a decade with 0 bad sectors.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
