Reply to post: Re: Back up often

Telecity London data centre outage borks VoIP, websites, AWS...

John Stoffel

Re: Back up often

This is one of those areas where you'd think they would be running tests on the system on a regular basis, with N+1 redundancy. If you don't test, you don't know.

But I've run into stuff like this before where we had a dodgy transfer switch that if you let it sit for a month or two, one of the phases would stick and not flip over the next time you had an outage. But once you tested it... it would be happy as a clam and would switch back and forth no problem. It took me doing monthly tests and metering the panel to finally find and prove the problem. Took over a year to find and solve this issue.

Ever since then... I test and check the voltages on the transfer switch.

Now in this case, if the loads are too high... then I suspect someone goofed and overloaded a singel phase or something so that too much power is being pulled from one leg at a time, which doesn't allow things to come up cleanly. Not a fun situation if you haven't planned for it and know how to shed load (i.e. turn off crap...) as you bring things up to let disks spin up one by one, instead of a huge thundering herd.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon