* Posts by JiminyCricket

1 publicly visible post • joined 28 Jun 2012

Sysadmins: Your best tale of woe wins a PRIZE

JiminyCricket

Many years ago, I worked in 3rd Line Support for a reseller. We had internal IT for our internal systems, but as usual, when there's a problem, "you're a techie - fix it" is usually the way forward.

Arrived at work before 9 o'clock one Monday morning to be greeted by the Sales Director to inform me that no-one could access their email. Popped into the server oom, and found that an entire rack was down. I quickly disvovered a faulty power strip, and as there was already a spare the other side, began moving all plus across to it.

I had to connect the new power strip under the floor of the server room, so called one of the junior members to help with the lifting of tiles, and to make sure no-one fell down the hole when the floor was up. It all went well, and a short time later the rack was booting up nicely, so set off to leave the server room with the junior member to inform everyone that all would be well in the next few minutes.

Now, to leave the server room you had to hit the door release button next to the door. Unbeknown to the junior guy, next to it - under a cover that had to be lifted - was also an emergency shutdown panic button that would kill power to the whole room. Yep, you guessed it - he lifted the cover and hit what he thought was the door release. 20 server racks powering off left the room feeling ever so silent!

Cue spending the next three hours getting all of the systems up from scratch. :(