Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened

A power outage kicked off a fire in web hosting biz WebNX's Ogden data center in Utah on Sunday, knocking the facility offline temporarily and leaving several servers in need of a rebuild. Kevin Brown, Fire Marshal for the US city's Fire Department told The Register in a phone interview that firefighters responded to a call on …

[JassMan]

This would never have happened at a certain broadcaster I used to work for.

The generator was in its own separate building, and tested twice a year. Unfortunately, after 20 odd years of testing, the test failed. There was a book with all engineers responsibilities, their deputies, their managers, all the phone numbers etc. The book contained every procedure, every workaround, every aspect of how to get back to broadcasting within 3 minutes in the event of a supply failure. Except there was one little omission - there was no schedule nor person responsible to ensure that the diesel tank was checked and refilled. The test failed not because of a fire but because of the lack of fire inside each of its cylinders, because there was no fuel.

Still, that is what tests are for - to find that one thing that no one had thought of before.

[Anonymous Coward]

Re: This would never have happened at a certain broadcaster I used to work for.

So, there was a fire because there was no fuel.

Diesel fuel also serves as the lubricant, so the electric starter turned the engine and it overheated due to friction from lack of lubrication, and eventually something ignited?

[jake]

Re: This would never have happened at a certain broadcaster I used to work for.

While two stroke diesels exist, I have never seen one powering a backup generator for a modern data center ... but if one exists, I rather suspect it would have some kind of sump-based lubrication system and a blower of one description or another for proper feeding/scavenging of the cylinder(s).

Besides, in your scenario I rather suspect that the starter and/or associated wiring would melt down (ideally blowing a fuse first) before friction heated the motor to the point of starting a fire.

[Denarius]

Re: This would never have happened at a certain broadcaster I used to work for.

Indeed Jake. Called Detroit Diesels. Designed specifically to go from cold to full load ( V12) in 0.3 seconds. About 3 seconds IMHO. Used a Rootes blower ITIRC. Very thirsty and in my experience, leaked oil worse than a 1950s Triumph on a hot day. But then, they were in a desert. I have found them in offshore fishing boats in West Oz, Naval tugs as well as power stations. I will give them this, they started fast, took a lot of punishment and even when one phase was shorted by idiots along the local grid, not once did the gen set fail.

[Anonymous Coward]

Re: This would never have happened at a certain broadcaster I used to work for.

They are still about. Apparently they are quite popular in certain parts of the US truck modding community, pretty much because they are built like a tank and can take a lot of abuse.

[Paul Crawford]

Re: This would never have happened at a certain broadcaster I used to work for.

Indeed there have been some remarkable two-stroke diesel engines developed, such as this beast:

https://en.wikipedia.org/wiki/Napier_Deltic

In their day an astonishing power to weight ratio, but reliability not on par with modern expectations.

[Potemkine!]

Re: This would never have happened at a certain broadcaster I used to work for.

Napier designed indeed some very initiative motors, as the sleeve-valve H-block 24-cylinder Napier Sabre, which wasn't reliable either.

[Kernel]

Re: This would never have happened at a certain broadcaster I used to work for.

"Indeed there have been some remarkable two-stroke diesel engines developed, such as this beast:

https://en.wikipedia.org/wiki/Napier_Deltic"

Or even one of these:

https://en.wikipedia.org/wiki/W%C3%A4rtsil%C3%A4-Sulzer_RTA96-C

- very reliable engines for which IIRC I've seen a figure of 85% efficiency quoted.

[Anonymous Coward]

Re: Napier Deltic

See also e.g. https://en.wikipedia.org/wiki/British_Railways_DP1

" reliability not on par with modern expectations."

There is a Deltic Preservation Society and several Deltic locomotives (but not the original prototypes, DP1) are still in routine use; highly recommended if you get the opportunity.

Somewhere I think I read that the disappointing reliability of the Deltic locomotive engines was in part due to some unusual materials used in the construction of the engine - something important required specific kinds of lubricating oil, a requirement which wasn't always followed, hence some disappointing failure rates. Can't comment on whether it was true. There was also a requirement for rather tedious manual "priming" of the lubricating oil system, and if that wasn't followed...

[Kernel]

Re: This would never have happened at a certain broadcaster I used to work for.

"While two stroke diesels exist, I have never seen one powering a backup generator for a modern data center ... "

I have, in a major national telco switching point, cell network control site and data centre containing network management systems, the servers that hold the national directory service data and one copy of the telco's billing data - and all three engines are still there and tested on load every month.

In over 25 years the only time I've known of one of these engines to fail was due to a failure of the electric fan motor on the remotely mounted radiator.

There's nothing quite like being close when all three V12s fire up together, after which one settles down to take the load, one runs at the working speed but off load and the third will eventually shut down after 10 minutes or so. For those who are wondering, somewhere on the high side of 22,000 litres every 24 hours.

[jake]

Re: This would never have happened at a certain broadcaster I used to work for.

"In over 25 years"

Your definition of "modern" in this context seems to differ from mine.

Vive la différence, and have a beer :-)

[Andy 97]

Re: This would never have happened at a certain broadcaster I used to work for.

I remember that one.

Also remember the red-faced duty manager in the canteen after.

[Anonymous Coward]

Re: This would never have happened at a certain broadcaster I used to work for.

Many years ago I worked for an organization that ran its own data centers. They had a pair of very large diesel generators in a different building - which provided backup power to the entire campus. Once a month - as a test - grid power was turned off and the backup generators kicked in, then they were failed back. Except one day one of the generators didn't kick in. When the campus was originally designed it was decided to "stripe" power from the generators - every other desk, every other rack in the DC etc. If systems were built that redundant pairs were in alternate racks then the failure of a single generator should be manageable - however somewhere in the 20 years since the campus was built the message was lost - many systems turned out to have redundant capacity fed by a single generator, and one such system was the AD farm.The grid power was quickly restored but systems had already failed by that point. It took a long weekend where everything was turned off, then brought back on in a controlled order before things worked again.

[MachDiamond]

Re: This would never have happened at a certain broadcaster I used to work for.

"however somewhere in the 20 years since the campus was built the message was lost"

The long lost "theory of operation" statements have need of a comeback. I got told off for writing them at an aerospace company, but as the head of the department, I felt they were needed for a better understanding of what the engineer was trying to accomplish and the problems they were trying to solve. A lot of things we did were "one-off" and very unique. The company was growing and at the same time had lots of turn over due to lack of adult supervision. I felt that if everybody had to figure out things from first principle analysis, they wouldn't be there long enough to sort out what they needed to know to be productive. Years after I left, I talked with another engineer that had also left and told me I had been cursed due to somebody that couldn't get their head around a multi-input redundant power supply I had designed and built. I asked if they looked at the ToO doc I wrote and I was told that the documentation system had been "streamlined" to get rid of deadwood and it was quite likely that all of my descriptions were tossed by my replacement who didn't like me with a passion for some reason never specified. Previously, nothing was thrown away, but moved into hierarchs of "Canon" and "Apocrypha" according to the last revision signed off. This way, dead ends would be preserved and previous generations of hardware, circuit boards and software could be identified. Things changed fast and it might be a couple of years down the road when it was discovered that the baby went out with the bathwater or somebody proposes a new approach that had been tried already, but nobody is left around that remembers it was fruitless.

[GruntyMcPugh]

Re: This would never have happened at a certain broadcaster I used to work for.

At our place we did think to get the tank topped up after seven years in the new location, and it proved to be quite hard to arrange, as it had never been done since the place was built and equipped. The generator and tank were in the rear car park, which is quite small and tight to get a tanker into. I never did find out how they did it in the end.

[claimed]

Re: This would never have happened at a certain broadcaster I used to work for.

Buckets? Or, failing that...

[Tom 38]

Re: This would never have happened at a certain broadcaster I used to work for.

When we get heating oil delivered, the chap just unrolls a very long flexible pipe and pumps it in. Surely any problem is fixable with enough flexipipe and pumps?

[MachDiamond]

Re: This would never have happened at a certain broadcaster I used to work for.

"When we get heating oil delivered, the chap just unrolls a very long flexible pipe and pumps it in. Surely any problem is fixable with enough flexipipe and pumps?"

That would be easy, but the problem shifts to red tape. You have to hire a firm that is certified to run the hose, they'd have to file for a permit, have their lawyers check the insurance requirements, blah, blah blah. A government inspector would need to be on hand to check everything and question anything they don't understand, which would be everything since they received a liberal arts degree in something ending in "studies" but scored high on the public employees exam and wrote a very enlightened poem painting the characters exceptionally well. (see "Interesting Times")

[Yet Another Anonymous coward]

Re: This would never have happened at a certain broadcaster I used to work for.

>we did think to get the tank topped up after seven years

A lot of companies got caught by their new green mission statement and switch to bio-diesel.

What's the difference between bio-diesel left in a tank for 5+ years and a veggie burger? Not a lot !

[bombastic bob]

Re: This would never have happened at a certain broadcaster I used to work for.

your standard 5 gallon portable fuel can would do it. No offroad vehicle is complete without one. Just fill up a bunch of them, put them on a cart capable of holding the weight and navi-guessing the terrain, transfer to the diesel generator's tank when you arrive, and rinse/repeat until full.

[Yet Another Anonymous coward]

Re: This would never have happened at a certain broadcaster I used to work for.

There were lots of stories after the hurricane that hit New York.

Generators on roof of office block - for safety reasons you don't want generators in basement - feeling smug when basement floods. Then discovering either diesel tank or pumps for diesel tank are in basement.

Lots of pictures of people carrying 100s of Jerry cans up 20 flights of steps

[MachDiamond]

Re: This would never have happened at a certain broadcaster I used to work for.

"The generator and tank were in the rear car park, which is quite small and tight to get a tanker into."

Some MBA found that buying and taking delivery by the tanker load was more "cost effective". A few smaller tanks mounted on trailers such as the military often uses is the easy solution.

In my early years I was a roadie. One thing we had to check before a tour was if we could get our trucks up to the venue. If a HGV wasn't going to fit, we'd hire two or more smaller trucks and load out into them from the previous gig. This was often the case in large cities and older concert halls. Parking the lorries down the road and rolling all of the gear took too long or cost way more in hiring extra labor (often times union). HGVs also required permits and had to be parked a fair distance away after unloading. The logistics would have been a nightmare. If you want your piano in your 10th floor flat, your first step before making the purchase is to see if you can get it there. Hiring a crane and deleting a window might get the job done, but it would be cheaper to find another flat.

[Anonymous Coward]

Re: This would never have happened at a certain broadcaster I used to work for.

We had something similar to that in a location in rural yorkshire.

There was a powercut for about 10 hours overnight and the diesel generator kicked in.

Power came back on around 6am before everyone came back on.. Unfortunately the email that was supposed to be sent automatically was dutifully sent via a email relay that had been decommissioned, so no-one was the wiser.

Unfortunately, we had another power cut later in the day. The generator ran for about 40 minutes then run out of diesel and conked out. I cant tell you just how much fun that was to deal with.

[MyffyW]

Re: This would never have happened at a certain client I used to work for.

'leccy from the grid failed on Wednesday, diesel generator kicked in and worked tirelessly (fuel having been topped up late Thursday), diesel generator then packed in on Friday night, switch from generator to grid attempted. Massive spike fried the (single) power supplies to each rack. Cue lot of heart ache and recrimination. Oh the joys!

[Keith Oborn]

Re: This would never have happened at a certain broadcaster I used to work for.

A Well Known UK ISP, whose name begins with "Vi" had (still has I think) a dc near Reading. It has resilient power and fibre. But there is only one main road into the business park, which runs over a small river bridge. Guess where all those diverse paths went. Gasman came along with a digger.

I got the 15-minutely Major Service Outage reports:

First report

1: All mains power lost. UPS taking the load

2: UPS fine, but we realised it doesn't run the aircon

3: But it DOES run the coffee machines

4: And it doesn't matter coz it'll only hold for 15 minutes and nothing will melt that quickly

( 15 minutes later)

Generator running and taking load. It DOES run the aircon

(15 minutes)

We don't know how much fuel is in the tank

(15 minutes)

Checked and tank is full.

I am sure a long stick was critical at this stage.

[jake]

Re: This would never have happened at a certain broadcaster I used to work for.

It doesn't mention that it all doesn't matter, because fiber was cut at the same time as the power?

_{There's a song in there, somewhere ...}

[Yet Another Anonymous coward]

Re: This would never have happened at a certain broadcaster I used to work for.

Requirement was that servers were up 99.9999%, connecting to them is an S.E.P.

[Mister Dubious]

Test, test, and test some more!

"... one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure..."

Imagine how bad it might have been had they not tested it!

[Version 1.0]

Re: Test, test, and test some more!

This reminds me of the problem experienced by a hospital in Houston TX. Downtown was flooded and the hospital generators, which had always passed their tests, all failed - closing the hospital down completely.

They were installed in the basement.

[Yet Another Anonymous coward]

Re: Test, test, and test some more!

Texas is a desert so it's a one in a million chance it would ever rain, or have a tropical storm, or a hurricane, or snow - so it's a waste of money

[Anonymous Coward]

Re: Test, test, and test some more!

The same thing happened at the VU Medical Center in Amsterdam. The water main in the street broke, the resulting hole swallowed an entire car, and the generator room flooded.

The result was that the entire hospital lost power and had to be evacuated, they even brought the army in to carry patients through the stairwell.

I guess it was even worse than the short-circuited refrigerator that caused a massive oxygen fueled fire a few years earlier. The ER oxygen main was running behind that fridge.

Videos or it didn't happen:

https://www.youtube.com/watch?v=aYq4pBiZdZM

https://www.youtube.com/watch?v=wO7sZ95zhs8

[MachDiamond]

Re: Test, test, and test some more!

"I guess it was even worse than the short-circuited refrigerator that caused a massive oxygen fueled fire a few years earlier. The ER oxygen main was running behind that fridge."

Richard Feynman talks about his being sent out to go over the plans of a Uranium processing plant to root out storage rooms being back to back and possibly leading to critical masses being formed. He also notes his random question about what he thinks is the symbol for a valve on the plans and what would happen if it stuck. His intention was to verify if he was right in thinking that the symbol was a valve and not a window or something. It turns out it was a valve and it would have been bad if it didn't work correctly.

Industrial facilities have piping exposed so it can be inspected, serviced and so people know it's there. I doubt that many people working at that hospital could interpret engineering drawings, but they'd spot big pipes with "Oxygen" stickers on them.

One flat I had would always have pipes spring a pin hole leak. It was only at night when it was quiet that I'd hear the leak due to being behind drywall and insulation. The first indication was often noticing a bulge in the paint, which would burst if touched, or the ceiling falling down. The big problem was that the leak was likely there for some time and had saturated all of the wood around it leading to mold and questions about the structure. After that, I wouldn't mind a converted space with the internals exposed. I've adopted an industrial look in the house I've purchased. It would take ages to bury all of the wiring inside walls, so I just did a good job of making it neat where it is seen. The new HVAC is going to be lots of fun.

[MachDiamond]

Re: Test, test, and test some more!

"They were installed in the basement."

Reminds me of another incident where the generators were in the basement. Somewhere in Japan, as I recall. I don't think that it ended well for them either.

[Yet Another Anonymous coward]

Re: Test, test, and test some more!

You have to install them in the basement in Japan. if you put them on the roof Godzilla can knock them over

[MachDiamond]

Re: Test, test, and test some more!

" if you put them on the roof Godzilla can knock them over"

that or torch them with his halitosis ray.

[Marcelo Rodrigues]

The weird part isn't the generator fire - shit happens.

The weird part is that a fire on the generator could spread to the servers. They should have a firewall (fnar fnar) between them, with just the electric cables passing through.

And even the cables should pass through a cut point, with material made to expand and cut the cables in case of fire. Better no electricity than some electricity and a lot of fire.

[Anonymous Coward]

Re: The weird part isn't the generator fire - shit happens.

Who would design a DC to have a fire source (generator) within fifty feet of the building?

[Anonymous Coward]

Re: The weird part isn't the generator fire - shit happens.

Perhaps someone who doesn't want any kind of umbilical that could potentially be cut in a disaster scenario? That's the thing with Murphy. Putting them close means nothing to cut, but if the generator itself blows you have a problem, but if you put some space between them, you get other problem when (not if) something happens in the space between them (which you often can't control).

[Wim Ton]

Re: The weird part isn't the generator fire - shit happens.

My brother had a fire on his ship. Unfortunately, the cable from the emergency generator passed through the main machine room that was on fire. In the end he has to use a hand cranked emergency emergency generator.

[khjohansen]

Re: The weird part isn't the generator fire - shit happens.

Sounds like *proper* planning - having a backup for your backup!

(would have been nice to have a none-pedal powered one, of course...)

[Eclectic Man]

Re: The weird part isn't the generator fire - shit happens.

Some of the NASA Space Shuttle systems had 5-fold redundancy, just in case in case in case in case.

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.3162&rep=rep1&type=pdf

[Charles 9]

Re: The weird part isn't the generator fire - shit happens.

Makes me wonder what would be the thought process if the crank had just snapped and resulted in needing a backup backup backup plan...

[Someone Else]

@Charles 9 -- Re: The weird part isn't the generator fire - shit happens.

Stop! Just stop....

[Yet Another Anonymous coward]

Re: @Charles 9 -- The weird part isn't the generator fire - shit happens.

To start the really big marine engines you need a lot of compressed air. When you have 100,000cc / cylinder a starter motor and a 12v battery doesn't cut it.

So the requirement is normally to have enough tanks full to restart the engine several times.

[jake]

Re: @Charles 9 -- The weird part isn't the generator fire - shit happens.

"To start the really big marine engines you need a lot of compressed air."

Or a pony motor.

[Yet Another Anonymous coward]

Re: The weird part isn't the generator fire - shit happens.

>The weird part is that a fire on the generator could spread to the servers

FOE = Fire over Ethernet, it's a new standard

[Anonymous Coward]

Re: The weird part isn't the generator fire - shit happens.

Might be a new standard, but it's been supported by LPT ports for years.

[Yet Another Anonymous coward]

Re: The weird part isn't the generator fire - shit happens.

>Might be a new standard, but it's been supported by LPT ports for years.

But Printer-On-Fire is readonly, you can start a fire by sending it to /dev/lp0

[Korev]

Re: The weird part isn't the generator fire - shit happens.

Nah, Firewire has been around for years...

[KittenHuffer]

Are we sure ....

.... that one of the servers didn't just fire off an HCF instruction?

[diguz]

Re: Are we sure ....

well if we're gonna be pedantic, it was the H part that somehow didn't work, the CF sure did...

[Someone Else]

Re: Are we sure ....

Perhaps a bug in the microcode...

[Anonymous Coward]

Re: Are we sure ....

More of a START and catch fire instruction!

[Anonymous Coward]

Back in the 80's, my father worked on the technical team at a large hospital. They installed an automatic generator to provide backup power to their datacentre and a UPS to bridge the gap while the generator was starting up. This was new and expensive technology at the time. They had a lot of difficulty in sourcing and installing a large enough UPS to do the job. But eventually the install was complete and it was time to test it. It was a nervous moment, when the main power was turned off. Everybody cheered as the UPS kept the systems running. And then, with a mighty roar, the big Rolls-Royce generator started up and delivered 415v instead of 240v and blew up every system in the building.

[ColinPa]

Whoops wrong power

My father was in the Royal Navy (this was 50 years ago). He said a foreign warship came along side the dock, and they plugged in a generator. There were a few problems as the plugs were different shapes, but with British ingenuity they managed to work round it. Unfortunately the generator was 240V AC, these particular circuits on the shop were DC 12V! Whoops.

[khjohansen]

Re: Whoops wrong power

- and this is why plugs & sockets are made in different shapes, of course ;)

[Stuart Castle]

Re: Whoops wrong power

A few years back, two of my friends had an office together. Being technicians, they had a *lot* of equipment. While they did work in a building that had been refitted for the company's needs, no one had thought where to put a technician's office, so they ended up in a room that was large, but other than that, was totally unsuitable for their needs, and had one socket. They had asked for more, but being in a listed building, adding more sockets was a long winded process involving consulting with English Heritage, as they were then, so it didn't happen.

As such, they ended up plugging a 4 way extension cable into this socket, and also running extension cables into neighbouring offices, with daisy chaining (note: I know this is bad). They also had one window, and a lot of very hot equipment. One summer's day, it was particularly hot, so one friend got up, plugged the fan in and went to switch it on. The other, being a wind up merchant, went to pop his bag of crisps.

There was a loud bang at that moment. Not from the crisps, but from the socket that had finally given up under the load and blown.

Not surprisingly, they ended up being moved out of that office into one that was a little better equipped for their needs.

[MachDiamond]

"And then, with a mighty roar, the big Rolls-Royce generator started up and delivered 415v instead of 240v and blew up every system in the building."

Another tale from my roadie days:

We'd always verify the voltage before connecting to the power distro. A whole lot of expensive audio equipment doesn't do well with double (or 4x) the voltage they are set for. The owner of the sound company did the testing and made the connection to the sound system power distribution panel. Replacing the whole sound system on show day wasn't an option. The owner never delegated the task and we never damaged our gear. I didn't ask why that procedure was in place, but I can guess.

[Joe W]

Oooh, look, another one!

Though it seems less severe than in Strasbourg, where the fire spread to several buildings / data centres.

Still, why can fire spread from the generator to the servers?

Why is there no switchover to another datacenter for your customers' websites - I mean, you have an SLA of 100% availability.

What damages will webnx have to pay? Not fulfilling your part of the SLA usually means something (otherwise why the farque do you have that SLA, like at all?).

[mark4155]

Guarantee not worth the paper.....

Well. well. well so WebNX's offer the following:

"WebNX's SLA guarantees 100 per cent uptime and uninterrupted power every month, with account credits of one day per 15 minutes of downtime in each case."

If you are offering 100% uptime (a rather naive and ambitious move) plus UPS power every month (I assume 24/7 as opposed to one day a month) why would you then go to the bother of negating the offer by offering account credits of one day per 15 mins of outage?

Simply doesnt't make sense - unless you know differently....

Toodle Pip!

[Peter2]

Re: Guarantee not worth the paper.....

It's putting your money where your mouth is and meaning that the SLA is something other than an empty promise. Unfortunately though, unless they put an upper limit in there though then unless my math is out though then 24 hours worth of downtime is going to cost them 96 days worth of free credits.

So if they have been out since Sunday then they are offering a free service to all of their customers at this site (is this their only site?) for the next year. Not many companies are well resourced enough to be able to do a total rebuild and then have an entire years worth of revenue sitting around.

[jake]

I helped fix ...

... a diesel generator that might have had a similar issue. The thing ran just fine during a routine heat-cycle the week before, but during a mains power failure a pin-hole leak had developed in a fuel line. On the pressure side. Fortunately a security guard on-site smelled the fuel and hit the Big Red Button within a minute or so. The leak had drenched both the plywood wall of the shed (about four feet away) and the lagging around the hot side of the turbo. Diesel's kind of hard to start burning ... unless it gets hot and has a nice wick.

Even if it had caught fire it would not have affected the computers inside (other than the lack of power) ... the generator shed shared no walls with the data center, and in fact was located about 50 feet away from the main building. Common sense, innit.

[Anonymous Coward]

Re: I helped fix ...

I've always found it amazing (saddening?) the number of people in technical industries who seem to believe that a generator test ,means standing in front of it, pressing the start button, letting it run for 30 seconds, then pressing the stop button and concluding all is well.

It's only when the thing is hot, pressured up, and vibrating away that many faults will become apparent, such as pinholes in high pressure fuel lines.

The only sure way to test is to pull the incoming supply fuses and stand back. Doing this during the day, when you have a full quota of support people around makes a lot of sense and I once got into a very heated discussion with several managers of a large telecomms company about it. They were insistent that the power to the building could not be cut, under any circumstances, which to my mind defeats the object of having or advertising to your clients, backup power. Their compromise was to test the genset on a test load once every few years, but that never tested the failure detect and switchover systems, nor that the UPS were operating or other site system were operating correctly.

One one site I was involved with there was a power control system, run from the mains, which was required to start the generator. There was no battery in the control system. Thus when the mains failed, the control system also failed, and was unable to start the generator. Goodness knows who came up with that idea or how it got implemented!

[Niall Mac Caughey]

Re: I helped fix ...

Many years ago I worked in broadcasting.

Well, OK, many, many, many years ago.

Our studio building was in a rural area and had a diesel genny with a substantial UPS to cover the gap. One day we had a localised power outage while I was on site and everything worked beautifully; the UPS ensured continuous operation and the genny started up perfectly. All was running smoothly and we had lots of diesel, so I hopped in the car and went to our even more remote transmitter to carry out some checks.

There was no interruption in broadcasting because the transmitter still had mains, a backup genny wasn't really practical there, as it consumed the better part of a megawatt at 10kV. The checks went fine and I was congratulating myself on having been well-prepared, until suddenly we lost the studio.

The local record for the 10 miles between the two locations along very narrow, winding country roads was 12 minutes 30 seconds; I made it in 12' 17".

Turns out that the genny was still running, but the UPS wasn't happy with the quality of the power and had never switched from batteries to generator. I had failed to spot this rather critical issue and eventually the UPS shut down when the batteries were exhausted. This oversight on my part resulted in brickbats from the boss instead of the expected kudos.

As you can imagine, my subsequent calls to the electrical contractors and commissioning engineer were not for delicate ears, but it was a useful lesson.

[MachDiamond]

Re: I helped fix ...

"the number of people in technical industries who seem to believe that a generator test ,means standing in front of it, pressing the start button, letting it run for 30 seconds, then pressing the stop button and concluding all is well."

AvE and his sidekick picked up a diesel generator with very low hours that was killed by short "tests". It was never left to run for a bit to work in and splash lube here and there on the insides. I wonder if anybody was checking voltages and doing some load testing. It seems to me that hooking up an EV or 4 might be a good way to load test and somebody(s) would get a free charge out of it as well. For some places, a bi-directional EV charging scheme with company vehicles could mean a pretty big UPS at least part of the time.

[Pascal Monett]

Seems that generators are a bit of danger

Might be time to house them in their own, seperate building.

[Anonymous Coward]

Re: Seems that generators are a bit of danger

I mean who the hell houses their generator in the same building as their servers? It's the kind of rank amateurishness you'd expect of fly-by-night cannabis farmers (I bet the serious guys don't make that mistake).

[Warning: Heresy alert]

I sometimes wonder whether cloud-only is even worth it for anything bigger than SME. What are you actually gaining when the economies-of-scale start diminishing (at least) geometrically and the cloud supplier's incentive to cut corners increases linearly with size?

[Anonymous Coward]

Re: Seems that generators are a bit of danger

We certainly don't have our generators within 50ft of the main suites and DC but we're a disaster recovery house and incidents like this are pored over for details of failure so we can improve and advise clients.

That's not to say we haven't seen clients make some insanely stupid beancounter driven decisions, including the perennial "can't afford to spend a couple of days testing the DR plan" which despite our best efforts, promptly fails when invoked because it's never been updated to take into account the dozen new servers or storage array etc. etc.

[Anonymous Coward]

Re: Seems that generators are a bit of danger

No DRP = Nothing.

[mark4155]

Re: Seems that generators are a bit of danger

For generators in the same building try British Telecom exchange buildings! The noise on testing is unbeleivable.

[Anonymous Coward]

Re: Seems that generators are a bit of danger

I imagine the vibration cannot be good for the servers.

[doublelayer]

Re: Seems that generators are a bit of danger

"I sometimes wonder whether cloud-only is even worth it for anything bigger than SME."

This is not a good example to make that point. The location here is a lot more like what local servers are like than what cloud is like. The large cloud providers operate so many datacenters that they can afford to do lots of expensive things that local or colocated servers don't do for the expense. DR is often one of those things. Large cloud providers also have built-in features for redundancy across datacenters or regions. Meanwhile, a lot of companies will either colocate their servers in a datacenter not unlike the one in this article or will run a facility of their own which isn't much different. This obviously doesn't apply to companies which spend a lot of time and effort on their servers, including really big places or ones with a lot of technical knowledge available, but it does for many others.

Cloud has a lot of downsides. You pay for lots of things you don't have to when running locally, the prices may be higher which you can only figure out by spending time on a boring effort to calculate them, you have another supplier on which you're relying, all that. However, if you're concerned about disasters, it is a lot cheaper to run redundant systems on cloud than to run them yourself unless you have a lot of servers. Now that's not necessarily a reason to go all cloud. You could run redundant systems in colocation datacenters or even run some locally with cloud backup. For a lot of businesses who don't bother with geographic redundancy at the moment, the extra cost of multiple servers to perform one task may not be judged worth it.

[MachDiamond]

Re: Seems that generators are a bit of danger

"it is a lot cheaper to run redundant systems on cloud than to run them yourself unless you have a lot of servers. "

It's a risk calculation. If you go with the biggest cloud companies, the ratio of company sizes puts you at a huge disadvantage. You need them far more than they need you and they know it. Yes, they have esoteric services that smaller companies can't provide, but most of that is marketing tosh vs. the real world. The question is if you really need that last 1% of CYA, just in case or if it's better to have an alternate plan that doesn't drain your wallet on a monthly basis.

If your company is all about data or online services, you should be able to operate/CYA in-house or there is something wrong with the business plan. I see so many companies that have outsourced the vast majority of their operations that it scares me. I certainly don't want to rely on them as they have to rely on so many others in the chain that I don't know about. I would have no way to gauge my risk. If a cloud service is my third tier backup, that can be ok. I'd just get bare bones coverage to limp along until normality is restored. If that cloud service is my core, I don't even have the option of throwing money at the problem to get back up and running. Chances are that I won't be able to contact them to find out what the problem is and what the ETA is on a fix. A big manufacturer can find it cost effective to put parts on a Gulfstream jet at $4,000/hour or more to keep a production line from grinding to a halt. This it what I mean by throwing money at a problem. I can abuse a credit card and buy a whole new computer and restore the data I need from offsite backups to be back up and running in minimum time. If all of my computing is managed through a third party, I may not even know what I'd need to do that.

[Anonymous Coward]

Re: Seems that generators are a bit of danger

Once upon a time gensets tended in be in basements, etc., but over the past few years (c. 20 years) the trend is to fit them in external metal containers. I believe this trend is generally forced by insurance companies (for the obvious reasons beings discussed here today). Similarly, the fuel tank now tends to be in a separate enclosure too.

There are places, also as suggested here, where having things exposed externally is not desirable, but your insurance company would be very interested in knowing it is inside the building.

Having them external also make maintenance and long-term renewal options easier. I know of buildings that have had to have external walls removed (and the building propped up) to remove the genset..

[tip pc]

Re: Seems that generators are a bit of danger

It used to be fashionable for important sites to have power fed from diverse routes from different power stations. That way it would need to be a seriously catastrophic event that required the gen, would also satisfy the housing the main generators away from the building

[Chris G]

Re: Seems that generators are a bit of danger

At the offices of a military contractor, they had a huge gender in a container mounted on a trailer.

One winter the power went off and the Genny fired up flawlessly for about five minutes, a delivery truck pulled into the carpark and sailed straight across the tarmac, nudging a couple of cars out of the way and finally bowling over the genset trailer.

The day was so cold black ice had formed on the carpark.

[Yet Another Anonymous coward]

Re: Seems that generators are a bit of danger

>At the offices of a military contractor, they had a huge gender

Don't ask - don't tell ?

[seven of five]

Re: Seems that generators are a bit of danger

Maybe he stumbled across it...

Sorry, am oot.

[Anonymous Coward]

Narf Narf

" they had a huge gender in a container mounted on a trailer "

[Eclectic Man]

electric cars

I do wonder what happens when the whole world goes to electric power. Electric cars, electric trucks. Research into improving the efficiency of internal combustion engines burning fossil fuel hydrocarbons will be much reduced. So who will design, build or maintain the emergency generators of the future, for when the electric power grid fails due to that inconvenient X rated solar flare (or gets as cold as Texas)?

[Neil Barnes]

Re: electric cars

Well obviously, the emergency generators will be powered by electric motors. And they'll be powered by a nice big stack of batteries...

--> the one with the safety isolator fuse in the pocket, thanks!

[sgp]

Re: electric cars

I don't see the issue, surely after 125+ years the technology is mature enough? I haven't come accross generators that use the latest "clean" technology. On the small petrol generator's side, Honda's OHV engines use decades old designs but are bulletproof. Which should be the primary goal imho.

[Anonymous Coward]

Re: electric cars

The "latest clean technology" is using a small engine with advanced computer controlled fuel injection et al which is used to thrash the engine to within a nanometre of what you can do before it breaks, and as a result they tend to have a short and unreliable service life due to being constantly thrashed to death.

A more traditional engine is massively overspecced, having huge margins for error and in any case even cars only tend to use the maximum performance when doing the 0-70 thing for <10 seconds when joining an A road, after which they use perhaps a third of the maximum available performance. These tend to be pretty reliable.

As SJP points out, a genset is typically more focused on reliability than power to weight ratios etc due to nobody really caring how much a genset weighs since a metre square worth of fluid weighs a metric ton, and an awful lot of places with a serious requirement for their generators tend to (if judged on the size and likely weight of the fuel) probably aren't that interested in shaving away the engine casing and having the castings made of aluminium instead of steel to save an extra couple of kilograms at the expense of service life.

[Anonymous Coward]

Re: electric cars

> I do wonder what happens when the whole world goes to electric power. [...] So who will design, build or maintain the emergency generators of the future

I think we've reached that situation already. Modern large data centres can consume 40MW or more. A 40MW 'generator' is actually a small power station and simply isn't practicable to have sitting around just in case.

The future is not to have backup power at all (beyond redundant supplies into the building) and design systems that are active/active in different data centres around the country, if not globe.

[Anonymous Coward]

Re: electric cars

""The future is not to have backup power at all (beyond redundant supplies into the building) and design systems that are active/active in different data centres around the country, if not globe."

Sadly, that does not work well with hospitals and some other critical power users....

[Someone Else]

Re: electric cars

Modern large data centres can consume 40MW or more. A 40MW 'generator' is actually a small power station and simply isn't practicable to have sitting around just in case.

...or roughly 12 EMD SD70ACe locomotives...

_{Hello, El Reg...do we have another measurement metric here?}

[MachDiamond]

Re: electric cars

"Hello, El Reg...do we have another measurement metric here?"

You could use ergs, but you'd wear out your nought key.

[MachDiamond]

Re: electric cars

"The future is not to have backup power at all"

A small one may be needed so a center can yell for help if mains power goes down. A company may also want to have a lighting circuit operating so techs can reset power switches and such to bring the center back up when leccy is restored. A VOIP phone won't work without power and many DC's are impossible to get a cell signal inside due to screening and concrete. Let's face it, most of the recovery information in those 3 ring binders doesn't get read until something happens. A light to read them by will be important. You also hope that the door security system works so you can get access and not have it default to locked doors/cages. It would be better to have power so people can still use their badges. Nobody is likely to have keys to get to the locker with all of the backup keys in it. The locker itself is probably a CH751, so just ask me for that one. <eg>

[khjohansen]

Lots of places gets as cold as Texas ...

.. as has been repeatedly pointed out. The Big Fail in Texas was due to de-regulation shaving back

emergency measures. With electric cars, we'll have a huge-ish battery to plug into our homes, no?

Anyways, ICE have been optimized for more than a century, I doubt there's still gains to be had.

[Peter2]

Re: Lots of places gets as cold as Texas ...

The big fail in texas was due to the temperatures being desertlike and often above 40 degress celcius. They (correctly imo) didn't massively inflate their energy generation costs by requiring everything to be specced to work in -20c temperatures.

Where they screwed up is not having interlinks in place to draw power from elseware during the couple of days in a decade (or arguably century) that their generation infrastructure is frozen over by Antarctic temperatures.

[doublelayer]

Re: Lots of places gets as cold as Texas ...

Big open spaces are subject to a variety of extreme weather events. You are correct that it gets really hot there, but because it has no shielding from winds, it can turn cold fast. You are implying that this type of weather is completely unexpected in Texas. That's not true. The state receives cold weather and snow with some frequency. Unfortunately for them, they live in a place where you can get very hot and somewhat cold weather in the same place and they have to prepare for that. They could prepare by having generation equipment which could withstand those conditions or they could prepare by importing energy from elsewhere, but they chose to do neither of those things.

[Yet Another Anonymous coward]

Re: Lots of places gets as cold as Texas ...

They also turned off their gas fired plants when the market price of natural gas went above the retail price of the electricity they were selling - daft to supply a vital live-saving infrastructure at a loss.

[MachDiamond]

Re: Lots of places gets as cold as Texas ...

"They also turned off their gas fired plants when the market price of natural gas went above the retail price of the electricity they were selling"

A big problem with electricity in the US is that the companies are publicly traded private entities. I don't want the Government taking over, but there needs to be a separate classification for utilities. Too often they can make more money from investments on the stock/bond market over modernizing their infrastructure. As they are beholden to their shareholders, not customers, this is an issue. When they can afford to sponsor sports stadiums, this is a problem. Advertising is a big waste as well. In the US, you don't have a choice of who supplies the power to your home or gas, or water. Some monopoly has the franchise and that's all there is to it. I don't think I've ever seen a market where there were two cable companies to choose from. I have no problem with these companies being private, but they should be subject to far more scrutiny than other entities. I'd like to know if not sponsoring a racing team or capping management's compensation would lower my monthly bill.

[Brewster's Angle Grinder]

Re: Lots of places gets as cold as Texas ...

"With electric cars, we'll have a huge-ish battery to plug into our homes, no?"

I was about to suggest something similar. Everybody gets to work and plugs in their cars to charge on the understanding that, if there is a power cut, they supply the juice. (You just have to hope the power is restored long before you need to get home...)

Hospitals will always have staff on shift so will always have a supply of backup batteries. (You can imagine calling in staff early because there's a power cut and they need the batteries...)

[doublelayer]

Re: Lots of places gets as cold as Texas ...

A battery that runs a car can provide backup power for some things in a house. For the big hospital machines, no. For the server rooms, no. They are unsuitable because those systems need to stay up reliably, which is not assisted by batteries you don't own whose owners can easily move them. Also, you would need a lot of cars to keep a server room running, and they won't last forever.

Battery backup is certainly an option, but it is currently a very expensive option. Even ignoring that fact, the operators would use batteries they own. I don't think there's reason to worry about difficulties obtaining generators. They're common enough that people will keep making them until there is an economical alternative available.

[Anonymous Coward]

Re: electric cars

Just as critical will be the cost and availability of diesel fuel when there are no road vehicles and industrial heating systems left using it......

[John Brown (no body)]

Re: electric cars

There will still be a few farmers growing crops for bio-diesel. The rest will be whinging at the loss of income stream and wildly trying to diversify...again.

[jake]

Re: electric cars

Actually, farmers grow crops to make money. My cousin in Iowa doesn't give a shit how his corn, beans (soy) and wheat are used ... as long as he gets top dollar for them.

[John Brown (no body)]

Re: electric cars

And he grows the crops most likely to earn him top dollar. If bio-fuel crops are earning more than bread flour from wheat, I assume he'll naturally gravitate to growing more oil producing crops such as Rapeseed. Or is he being sensible and keeping a diverse crop in anticipation of future market fluctuations? As an independent, I'd guess the latter. The big corporate farms will more likely be run by accountants.

[MachDiamond]

Re: electric cars

"Just as critical will be the cost and availability of diesel fuel when there are no road vehicles and industrial heating systems left using it......"

Something to be aware of, but so far down the road <cough> that it's not a problem that needs to be solved immediately.

"Modern agriculture is the use of soil to turn petroleum into food." ~Al Bartlett.

It will be decades before farm machinery is swapped over to some new synthetic fuel. Harvesters that run 24/7 when crops are ready often sit for most of the year so batteries are a huge problem unless they can be shifted over to plowing and seeding tractors. It could be that diesel becomes a byproduct of refining crude oil in the same way that petrol was a byproduct of making kerosene. Battery powered passenger aircraft don't seem to be viable at this point so Jet-A isn't going away. In the US, trains move a large portion of goods and the US is massive. It will be a very long time before all of the tracks can be electrified and even then, service machinery will need to run on liquid fuel (or at least some of it) to be able to service downed electrical lines/equipment.

Even HGVs are not easy to run on batteries. Local shifting, sure, but long OTR hauls aren't cost effective with an ET. A passenger car or small truck can plug in just about anywhere and it's not a huge problem. Trucks need far bigger supplies to keep stops short. A charging station that can accommodate many trucks at one time will need tens of megawatts of provisioning. In the short term, we can expect to see local delivery trucks being transitioned to electric, but big HGVs will take lots of time. Speaking of time, drivers make money when the wheels are turning, not when they are sitting at a charge post. Lorries with tanks on both sides can fill up via dual pumps and spend ~20 minutes or so on a fueling stop. ET's may take hours to top up a 1MWh pack or at least an 80% charge which can be more time efficient.

An argument can also be made for emergency response vehicles, military vehicles, special applications vehicles such as big cranes and construction machinery. Diesel isn't dying anytime soon. The beauty of a properly made diesel engine is they often last for decades if they are maintained and not abused too much.

[MachDiamond]

Re: electric cars

There are still many applications for fossil fuel power generation so it's a long time before it would go away. Many small towns rely on diesel generation for their, very expensive, electricity. Job sites and out of the way temporary locations aren't going to be serviceable with wind/solar if they need more than nominal power. Movie production is another big user of generators. They might only be at a location for a day or two and need more power than what's there.

EV's could be a very good adjunct with 2-way capability. A company with an EV fleet has a big UPS on tap should they need it. A home could be powered for a long time from one EV and a bit of conservation. Even with a backup generator, it's more efficient to run it with a big load part of which is charging an EV that later dribbles the power back out over having the gennie running constantly so the fridge can cycle every so often. One could install dedicated battery systems, but it can be expensive and very redundant if an EV is just sitting there.

[Chairman of the Bored]

A feeling of impending boom

I happened to observe something at my local hospital that gave me a feeling of impending boom: a Caterpillar Mobile Power trailer installed adjacent to a rather massive liquid oxygen tank. The power cables run right next to the tank.

MP vans are typically 500-625kW and have enough fuel inside to run 24+ hours.

The good news is that this whole assembly is roughly 100m from the hospital building proper. But then, given several thousand liters of LOX, I'm not sure that matters.

Reported the problem of course. Not sure if anything happened.

[Eclectic Man]

Re: A feeling of impending boom

At the Royal Berkshire Hospital in Reading I believe I saw a large LOX tank in the grounds, within 10m of the buildings, but I don't know where the emergency generators are, so it might be relatively 'safe'.

At the Royal Institution Christmas Lecture one year, George Porter* (Nobel laureate for his work on high speed chemistry) dunked a cake in liquid oxygen, then said he didn't think it would burn very well as it was wet. He took the longest match I've ever seen from a box and proceeded to light it (from a safe distance). It really did burn exceeding well.

*Nice chap, I got his autograph, but sadly that version of his Christmas Lectures seem to have been lost. If you have a recording, please contact the Royal Institution as they would very much like a copy.

[Chairman of the Bored]

Re: A feeling of impending boom

@Eclectic Man, I'm quite happy that you were able to attend George Porter's lecture and meet the gentleman. That sounds like an outstanding evening. Sadly I do not have a copy of his Christmas lectures, I'm more or a sparky than chemist.

Back when I was in the service, though, we were shown a grainy, B&W video on the "Dangers of LOX". One thing I will never get out out of my head is a quick clip of a man stepping in a puddle of LOX. The oils on his shoes flashed over and I presume he died from horrific burns. It looked like his clothes, skin, everything flashed over when he fell down. Lots of LOX shipboard. My take away was "never mess with that stuff".

[MachDiamond]

Re: A feeling of impending boom

"The oils on his shoes flashed over and I presume he died from horrific burns."

LOx spilled on asphalt can be explosive. Lose a foot explosive. We always moved LOx while on clean concrete or dirt surfaces at the rocket company. It doesn't take too much looking to find a video on YouTube of a diamond burning in a test tube of liquid Oxygen. Granted, diamond is pure carbon, but still....

The Mythbusters once tried to make a rocket motor from a sausage and NOx. They got stymied by the structural strength of sausage as the exhaust velocity would always push the bits of meat out of the motor casing. Sugar can be used as a rocket fuel. Sugar Shot to Space has been working on perfecting a sugar motor for a few years (small hobby group).

LOx is very dangerous.

[Eclectic Man]

Re: A feeling of impending boom

"Reported the problem of course. Not sure if anything happened."

Well, depending on how close you live, you might have heard the 'BANG!' if anything had happened.

[Chairman of the Bored]

Re: A feeling of impending boom

Indeed! Even without the energetic nature of O2, a failure would be spectacular... I was near a 10,000 liter dewar of LN2 that experienced fracture of a large liquid line. All hell broke loose, and I left the area like Usain Bolt

[Flicker]

Flywheels....

The site where I worked 25 years ago in Swindon had two large gensets in a seperate building adjacent to the datacentre, with rapid start obtained via a pair of continuously turning massive, iron flywheels mounted on horizontal axles inside steel cages. Two days after some routine maintenance the bearings on one of the flywheels seized, melted, the flywheel broke free and went straight through the steel cage, through the wall of the building, across the car park, a field and ended up about quarter of a mile away embedded in the trunk of a (formerly...) substantial tree.

We could never work out whether it was luck or design which had the flywheels spinning in that direction - had they turned the other way then it would have quite cheerfully run straight through the main datacentre / processor room taking out at least eight of the fanciest IBM mainframes that money could then buy.

Everyone was always rather careful where they parked their car or walked on that side of the building afterwards...

[Anonymous Coward]

Re: Flywheels....

I came across flywheel UPS's back in the mid 90's too. Usually with a diesel engine attached via a big clutch that could stake up the drive when mains failed,

The technology and size of flywheel and mountings scared the living daylights out of me at the time. I have no idea whether they are still around or popular.

IIRC they were rated in the region of megawatts at the time when conventional UPS and power electronics, batteries, etc., could manage tens or maybe 100kw at most.

[Eclectic Man]

Re: Flywheels....

Many years ago, IBM sold drum storage devices (before the advent of high density hard disks). Well, the big ones weighed about 4 tons. One of them somehow got off its bearings and, being a high speed device, went straight through the external wall and demolished several rather expensive cars in the adjacent car park before coming to rest. Had it been spinning the other way it would have mangled several millions worth of computers and people, and possibly done structural damage to the building.

It was at this point that IBM made the strategic decision that they were no longer in the 'drum storage' business.

[David Roberts]

Re: Flywheels....

I have a vague memory of Motor Alternator sets wher a massive flywheel was spun up by mains power and drove an alternator.

This was meant to smooth out minor variations in the mains supply which would upset the mainframes

Didn't smooth out power dips, though, and I still twitch when the lights flicker.

[Anonymous Coward]

Re: Flywheels....

Some radar sets used to use flywheels that required synchronising with the generator. Get it wrong and the radar wasn't the only thing spinning.

[MachDiamond]

Re: Flywheels....

You would have thought that the flywheels would have been installed below grade inside a concrete silo. The steel cage would just add more shrapnel rather than doing anything useful as far as safety goes.

I'd be less scared of these sorts of things had I not taken a degree in physics. I know, it was a silly thing to do. I was young......

[Anonymous Coward]

The SLA...

... does not cover for Fire.

[Paul Hovnanian]

All things considered ...

... it could have been worse.

[The Oncoming Scorn]

Re: All things considered ...

On that basis, on an oil rig once they conducted a safety test on a fire extinguisher, what came out was ===============>

[Jon Massey]

So what they're saying is

"there are machines that need to be inspected for water damage and may need to be rebuilt."

Don't turn it on, take it apaaaaaaaarrt!

[MachDiamond]

Re: So what they're saying is

Extra kudos for the EEVBlog reference.

Maybe they need to call in Dave. After a bit of a squizz, she'll be right.

[jake]

Re: So what they're saying is

Dave's not here, man.