Give me POWER! Under resourced, overloaded - a moving story

I was the senior systems administrator (in fact I was the only IT person, but not the IT "manager" as that would entail a whole new level of paperwork for the client) and had been tasked with moving our central office to an office one kilometre away. As a joint venture, we were moving into the majority owner's existing office …

[Lysenko]

Inrush current...

...can blow your breakers in unpredictable ways that no amount of UPS upgrading will fix. What you need in cases like this are intelligent power strips with normally open outlet relays configured to reactivate in timed and current monitored sequential groups when power is restored. APC actually make such things. Not the best, but they would have been good enough for this use case.

[Anonymous Coward]

Re: Inrush current...

Look at what industry does. One can automate start-up with a small SCADA system, these days these come built in if one ask the rack vendors to provide one - or - it is a standard option for most of the "Canalis"-type power distribution system (It's an insulated bus-bar with power-pick-up points every meter or so via a plug-in module. Very easy to work with).

I have this discussion right now because I will at some point need to occasionally start / stop about 800 19" racks and there is no way in hell I will go there and do it by hand "to save costs".

[Montreal Sean]

Redundancy when it comes to power

When installing servers, always plug the redundant power supplies into different power strips in the rack.

Each power strip should go to its own upswhich should be plugged into different circuits at the wall.

[Benno]

And another hot tip (learned the hard way...) use the 'startup delay' feature of your servers BIOS. It will give you a staggered startup of the boxen - reducing that dreaded inrush!

(We had 2x 2kVA UPS's running at about 30% load under _normal_ circumstances. But when everything wants to start up at once, one UPS overloaded and went offline and the other was stuck on bypass at 150% load. That's the sort of thing you can't fix remotely...)

[druck]

Have a black start server controlling PDUs, so it can power up the other equipment in a definable order, and keep an eye on the UPS load.

[allthecoolshortnamesweretaken]

PPPPPP.

Disclaimer: PP can PPPP, but may not be enough to counter the effects of PHBs involved.

[Anonymous South African Coward]

Hindsight is the best teaching tool.

[Anonymous Coward]

During one office move I specified a 45A supply for the server room. The CEO insisted it was an unnecessary expense since there were 2 x 13A double sockets in the room and that adds up to 52A.

As much as I tried to explain that both sockets were on the same 30A ring main, he didn't get it until after the move, when we tried to power the systems up and the breakers tripped out.

[Andy A]

Could have been avoided by handing him a three 3kW heaters to test his theory with. Every office seems to have loads of them to keep half the workforce happy.

The breaker would have tripped before he got the chance to try heater number four.

At one place, I had to wire up two dozen desks using long extension leads. I was careful to confiscate every heater and explain that each power socket had a 3A fuse fitted, which would blow if they tried a heater. Responsibility for replacing fuses fell to building maintenance, whose response times were much longer that those of IT. Suffice it to say that they learned quickly.

[FuzzyTheBear]

Another reason ...

Professional tip: never have carpet in a comms room

add to the reasons not to our public enemy 1 : static electricity.

[EarthDog]

OK, some professional tips

1) Read the the labels. Work out the max amperage before you plug the things in.

2) Load balancing is not just for servers, it is also for electrical circuits. Learn how to do it. Don't forget the air conditioning while you are at it. If it means running extension cords until an electrician can come in so be it.

3) Bring server on one at a time. Seriously, there should be a sequence such as networking gear first, then DNS and other critical servers, DB and application servers, etc.

4) Monitoring software! Not just to start spewing emails when the UPS kicks in but also if gear starts to fail or overheat.

Think, plan, then execute.

[seacook]

Re: OK, some professional tips

5) Never connect a UPS supply to an upstream UPS. Each UPS requires its own dedicated circuit.

[Anonymous Coward]

since this was my server rack

Don't worry, in my previous office each rack had it's own duplicated 15A supplies on different power circuits that supplied separate UPS's and from there separate power rails, and this allowed us to lose a circuit and keep most systems online.

Our VM was configured to allow the physical servers to boot, then stagger the starting of the VMs to ensure that there was no inrush.

The problem was that it all worked fine when we started everything up on Friday night and all was fine when I walked out on Saturday afternoon. Alerting failed because the email server went down and the emails were not relayed. i suspect the problem was that the Cisco VM (the parent company's equipment) that was running various servers overloaded the main UPS when staff walked in on Monday morning and started logging into the servers. I have since walked into the comms room to find the UPS madly beeping because the Cisco had overloaded it. (fortunately our rack was no longer connected)

Lesson learnt, always make sure your power load is calculated on peak, not standard running.

[Anonymous Coward]

People don't read BOFH?

They had the same problem of servers trying to power up at the same time, and causing all sorts of havoc with UPS and overloaded circuits.