back to article ‘I crashed a rack full of servers with my butt’

Welcome for the third time to Who, me? The Register’s new column in which readers ‘fess up to messes of their own making. This week, meet “Digby” and his powerful rear. Digby once worked for a business that started life as a division of another company. He arrived just as the new outfit was getting to his feet and found “the …

  1. Tim99 Silver badge

    Just finger trouble

    I was doing some development on a test server and had not saved my work. An associated program would only run if "Turbo Mode" was turned off (this dropped the CPU clock rate down to match an original 8088 chip). Some tower PCs had the turbo button next to their push-button power switch. I pressed the turbo button to start the other program, but hit the power button by mistake, immediately experiencing the well known "How stupid am I?" feeling. I managed to keep the button in, so the power stayed on. Unfortunately, I had used my dominant right hand when I had bent down to touch the switch. Unable to see they keyboard, I managed to find the keys I needed to save my work, and then type in the shutdown command. Then I drank coffee and taped a cardboard flap over the power switch to stop the stupid person doing it again.

    Edit: That is one reason why I really liked the large red power switch at the back of original IBM PCs, it was almost impossible to turn it off by mistake.

    1. Anonymous South African Coward

      Re: Just finger trouble

      Hah, I played around with Smoothwall back then.

      The tower case (which was the Smoothwall) was conveniently next to my work desk.

      Daughter was crawling around all over the place, she saw the tall, white box with blinkenlights, and she did a gefingerpoken at the reset button.

      Daddy was NOT amused :)

      I disconnected the actual reset and power button cables from the motherboard, then set the motherboard's BIOS settings to turn itself on when power was restored.

      Then it was simple - shut it down, it'll power off, and switch off at the wall. When I want to use it again, power on at the wall and the PC'll start up again.

      Little fingers still poked the buttons afterwards, but nothing happened. :)

      1. Anonymous Coward
        Anonymous Coward

        Re: Just finger trouble

        Snap. though at first I wired a separate button then I used wake on lan from an app on the phone.

      2. l8gravely

        Re: Just finger trouble

        I've got you beat on the little fingers department. Way back when, my son was around 2 years old and the wife's computer was in the basement next to my lair of untidyness. Which she did not appreciate at all! Anyway, I had an old PC (gateway? HP?) where the case slid forward to add/remove stuff inside it. I had canibalized it for some parts and left it on the floor *mostly* closed up. Wife is working with the nipper playing under the desk. Then all hell breaks loose because he pushed in the power switch and the flap of spring steel caught his finger and wouldn't let go. Normally he could have pushed it all day until the cows (or I) came home. But the button popped out and he got trapped.

        I got a hysterical call from her and since I was 30+ minutes away and she's not really the techie type, esp under pressure like this when the first born is wailing his head off... I told her to call the local police for help. They came, got the finger out without loss and I got a royal reaming when I showed up just after they had left.

        Consequently, her computer was moved upstairs, and I had to do a major cleanup of my mess.

        Now the kid is as tall as I am... memories.

      3. Soruk

        Re: Just finger trouble

        Too many wires in the study / computer room to allow the mini-monster to toddle in - I've fitted a stair gate across the doorway.

    2. Rich 11

      Re: Just finger trouble

      and taped a cardboard flap over the power switch

      This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs. I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee.

      1. Anonymous Coward
        Anonymous Coward

        Re: Just finger trouble

        "I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."

        There was a horizontal desktop PC case, monitor on top, that had the power switch on the front - low down and projecting. If you pushed the bulky keyboard too far back on the desk - it hit the switch.

        1. sandman

          Re: Just finger trouble

          I had one of those at work. I blame my potty-mouth on the experience.

      2. Anonymous Coward
        Anonymous Coward

        Re: Just finger trouble

        "I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."


        Much more recently that there is the Compaq 500B MT about 2010 protruding power switch on the side knee height. At least you can change the function of a soft power button to something other than shutdown.

        Anon because they are we still our main PCs

      3. Michael Wojcik Silver badge

        Re: Just finger trouble

        a cardboard flap over the power switch

        A "molly-guard". Don't they have the Jargon File where y'all work?

        This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs.

        Dell, a consistent producer of systems with crap design, had a number of these. I had a Dell tower system with a hardware reset button positioned right below the CD-ROM eject button.

    3. gypsythief

      Re: Just finger trouble

      "I managed to keep the button in, so the power stayed on"

      I did this a few times, back when computers had Turbo buttons and switches were Real Switches instead of these noncy soft-switches they are today. The trick was to very quickly let go of the button, then instantly slam it back in again. Usually, enough residual power resided in the capacitors of the PSU to carry it through, and the computer would stay on.

      1. DJV Silver badge

        Re: Just finger trouble

        My main PC is one of those with a power button on the top of the case, which is fine until the cat jumps up and lands her heel on the power switch before sitting down and turning things off with me in mid-work! A strategically placed book seems to have fixed the "problem" now.

        1. Alien8n

          Re: Just finger trouble

          @DJV and how many throws of the book at the cat before it learned the error of it's ways?

    4. Ilgaz

      IBM still use the same button

      IBM does use the similar power button on their mainframes, one really wonders when it will be used.

      1. Anonymous Coward
        Anonymous Coward

        Re: IBM still use the same button

        When it gets replaced?

    5. Anonymous Coward
      Anonymous Coward

      Re: Just finger trouble

      I've done the "inadvertently press the power button, realise I'm an idiot" thing as well - I was supposed to press the CD-ROM eject. I didn't immediately release the button, and was able to reach a piece of paper that I folded one handed then wedge into side of the power button until I could cleanly shut the machine down.

    6. Michael Thibault

      Re: Just finger trouble

      " I managed to keep the button in, so the power stayed on."

      A momentary grasp of reason saves the day!

    7. notowenwilson

      Re: Just finger trouble

      Our work PCs have one of those nice setups with a bunch of USB ports, headphone and microphone jacks on the top edge of the case. Very convenient. Except the power button is directly below the headphone jacks so when you're poking around under the desk to get the cable in there's a decent chance you're going to have windows asking if you want to start saving things before shutting down.

  2. Nick Kew

    Yep. Sounds all in order. Elementary commonsense: you don't leave an important switch where someone might accidentally hit it, regardless of their girth, clumsiness. Or indeed if they're wearing big loose clothes that swish. And in a server room, you *also* give good care to your cables. Hence some of those neat little inventions like recessed switches. Didn't you learn the principle when you were little and your parents told you not to put your glass right at the edge of the table?

    Once again, the protagonist seems to be innocent. When are we going to get someone owning up to a proper f***up?

    1. Natalie Gritpants

      Or indeed if they're wearing big loose clothes that swish

      That's why I only wear satin. It's very slippery.

    2. Flywheel

      Elementary commonsense: you don't leave an important switch where someone might accidentally hit it

      Unless you're a bean-counter trying to save the last penny by using that box of old switches you "saved money" on for a previous project...

      1. Bob Wheeler

        you don't leave an important switch where someone might accidentally hit it

        Or taking ownership of a new server room, finding that the door release button was right nest to the emergency power down button.

        Despite pointing this was a bad idea to the customer, it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons.

        1. La Barbe D'Action

          Re: you don't leave an important switch where someone might accidentally hit it

          When I worked in the server support team for my local health board about 25 years ago, an engineer was called out to fix a faulty open reel tape drive for a development server that sat on a table in the data centre. It was quite a heavy old bit of kit and the table was next to a wall, so space was fairly limited. The engineer picked up the drive and, apparently without thinking through the manoeuvre he was about to make, swivelled round and bumped the emergency power off button on the wall with the back of one of his hands. The data centre was then a hive of busy people, trying to bring up the ICL mainframe, comms kit and multitude of midrange servers that lived there. Once he realised the chaos that was unfolding, said engineer started to sweat profusely as he continued his task of repairing the rarely-used tape drive. Some time later, once services were restored and things reasonably back to normal, a wooden box was placed around the power off button, with a hole cut in it so that it could be pressed when actually required.

        2. not.known@this.address

          Re: you don't leave an important switch where someone might accidentally hit it

          Similar situation happened to me at the Home Counties-based headquarters site of a well-known British aerospace company in the late 80s - we had to have the electricians in to do some work on one of the PDUs and, as they went to leave, one of them says "Is this the exit button?" and before we could yell "NOOOOO!!!!!!!!!" he pressed the big red emergency power off button...

          We weren't sure whether we should have been relieved that we didn't need to spend the next few hours rebooting various mainframes (the IBM and VAXs wouldn't have been much of a problem, the biggest headache would have been the ICL 2966s...) or extremely p***ed-off that the Emergency Power Off - intended to save our lives if anything went wrong like someone connecting themselves to 415v AC, remember - did absolutely nothing.


          Not a gorram thing.

          We went with p***-ed off, and the Data Centre Manager made sure scalps were collected from the muppets at fault...

        3. Alan Brown Silver badge

          Re: you don't leave an important switch where someone might accidentally hit it

          "it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons."

          Vs about £6 for a flip cover?

          Lots of other examples around.

          Schnieder even make a transparent one:

    3. phuzz Silver badge

      "When are we going to get someone owning up to a proper f***up?"

      When enough time has gone by that my coworkers have forgiven me...

    4. Anonymous Coward
      Anonymous Coward

      Once again, the protagonist seems to be innocent. When are we going to get someone owning up to a proper f***up?

      When the Plastic Surgery has healed and the Panamanian Passport has been received !!!

      Any day now :)

    5. rskurat

      the designers & marketing people don't care if they work, they just care that they look good. And even if the suits who sign off were once engineers, they aren't anymore.

  3. Anonymous South African Coward

    How many times would I told an user to "keep on saving your work, F2 (Turbo Pascal's shortcut to save) will do it"... only to have said user ignore my advice blithely, carry on typing a beautiful and perfectly working program, only to have the PC crash.... (It was my experience with programs and PC's to keep on saving frequently, especially when doing Turbo Assembler stuff. Back then there was no virtual machines).

    v2 of the same program was buggy and full of errors. Shame. NOT.

    1. Anonymous Coward
      Anonymous Coward

      I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.

      For most people it takes a while before paranoia becomes second nature. My friends laugh at me for having at least an idea of the potential "Plan B" (C,D,....) when I do things. They are learning the hard way....

      1. Doctor Syntax Silver badge

        Muggins was then expected to recover their data when they had a problem....My friends laugh at me for having at least an idea of the potential "Plan B" (C,D,....) when I do things.

        They do have a plan B. You.

        1. Omgwtfbbqtime

          Plan B

          From Schlock Mercenary:

          Commander Kevyn: So we're supposed to evacuate the rest of them?

          Commodore Tagon: Sure, that would be a great solution. Let's call that "Plan A."

          Captain Murtaugh: Also known as "Ablative armor that probably won't protect Plan B."

          Captain Tagon: Plan B always takes the bullet for Plan C. That's why the alphabet has more than three letters.

        2. Anonymous Coward
          Anonymous Coward

          "They do have a plan B. You."

          That's usually their Plan A.

          It's like Cassandra and the usual Greek gods' attitudes. Gifted by the gods to warn everyone about a foretold disaster - fated to have to suffer the consequences with them. The Ancient Greeks knew human nature - or at least their story tellers and playwrights did.

      2. BebopWeBop

        I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.

        I can be smug here - I did buy the kids USB disks - and then made a royal pain in the arse of myself b buffing them to use them! Paid dividends in one (of three) who has evangelised ever since...

      3. Anonymous Coward
        Anonymous Coward

        That sounds familiar. Complete lack of gratitude for a save tells you there's something wrong with your rates; adjust them upwards to find out who your friends - and valuable customers - really are.

    2. This post has been deleted by its author

    3. Anonymous Coward
      Anonymous Coward

      "v2 of the same program was buggy and full of errors."

      VBA in Excel 2010 has an interesting quirk. If you Ctrl-Z a few times to undo an edit - it regularly corrupts whole swathes of the module. It deletes unrelated chunks of statements in a totally different part of the module - which you only discover when you try to compile.

      After any significant change I make copies of the project directory with a yyymmdd_hhmm suffix.

      Using the same technique on VB.Net hit an unpredicted problem. Norton security*** regularly quarantines one of the project files that is a .dll type - because it is "new" and has no "reputation". You then have to do the arcane ritual to reinstate it and make it "safe" for the future. According to Norton support that only protects for that specific version's directory name.

      This doesn't happen all the time. It was horrifying the first time to see it purging the e-sata back up disk of all instances of that file going back years.

      ***Norton works for me - and often cleans up people's disks who have used a different AV. YMMV

      1. detuur

        > After any significant change I make copies of the project directory with a yyymmdd_hhmm suffix.

        Sounds like you just need a decent version control system like git.

        1. Stevie

          a decent version control system like git

          Me: Can I please have git installed on the Project Millstone server?

          My Boss: What's git?

          Me: It's a version control and software repository tool. Our vendors use it. I could use it to comply with your order to "stop using .old files to save stuff that gets changed, dammit!"

          My Boss: No. Everyone must use the Microsoft version control tool we have.

          Me: Okay. What's it called?

          My Boss: Dunno. Microsoft something-or-other.

          Me: Where's the documentation for it?

          My Boss: Dunno. On a share somewhere. Maybe somewhere on sharepoint.

          Me: Who owns it? Who's the SA?

          My Boss: Dunno. One of the windows guys.

          Me: Is there training for it? Can I have some please?

          My Boss: No. The budget is all used up.

          Time Passes In Month Chunks

          Me: I've written my own version control. I run a script that just appends ".old" to everything of the same name before you edit.

          My Boss: That's the opposite of what you were told!

          Me: I know. I was faced with insurmountable difficulties in implementing your methodology.

          My Boss: How in hell do you tell one so-called version from the rest? what happens if you need to reinstate a previous version?

          Me: You figure out how many versions you need to go back and count the ".old"s.

          My Boss: Whut?

          Me: It works very well.

          My Boss: It's stupid! And you've soft-linked this "script" to all the servers and filesystems, I'll bet.

          Me: Of course not!

          My Boss: Well that's something at least ...

          Me: I simply copied the script into every directory on every unix computer we own.

          My Boss: WHAT!

          Me: Very quick and cheap, if a tad cheerful. Works though.

          My Boss: Get rid of it!

          Me: With pleasure. Can I have git please?

          My Boss: No. Everyone has decided to use Toad's built-in version control. I don't know what it's called before you ask.

          Me: I do. It's something called "git".

    4. Mark 85 Silver badge

      A sign with "Save early, save often" should have been hung on his monitor. We actually had some printed up on card stock and hung on the developers/programmers monitors.

  4. TRT Silver badge

    College students...

    stretching their legs out under the desk was a frequent cause of problems as toes pulled out power leads, keyboard and mouse cables variously.I had told them that proper cantilever desks with cabling trays were required rather than reusing the old 4-leg plain top wooden tables that had been there since 1953.

    1. Nick Kew

      Re: College students...

      Good diagnosis, bad solution. Yes, cabling should be kept clear of the meatware's legroom. Real-life computer desks get that horribly wrong by restricting the legroom: sadly I hadn't heard of "constructive dismissal" when I was forced out of office-based working by desks that forced me into postures that were incompatible with my back. The right solution is to keep the legroom but provide alternative safe spaces for cabling!

      1. The Oncoming Scorn Silver badge

        Re: College students...

        On a few days backfill work for a insurance co (When they were still called Norwich Union), a lady wrapped her leg around a power bar\cable & brought down a entire row of computers.

        Having rectified the issue, she then went & did it again.

        Thus prompting a better solution in cable management.

    2. A Non e-mouse Silver badge

      Re: College students...

      When I was at Uni, we had diskless Sun SLCs as our workstations. There were two SLCs which had massive 1GB HDDs to serve the other SLCs in the room. Unfortunately, it was very easy (by accident) to knock out the ethernet cable from the SLC with the HDD and bring the room to a standstill. (And I'm ignoring the joys of 10Base-2!)

      1. TRT Silver badge

        Re: College students...

        What about 10Base-5? The tripping snake.

  5. DainB Bronze badge


    You can't arrange server racks in L-shape and arrangement have nothing to do with hanging cables.

    Calling BS on this one.

    1. Anonymous South African Coward

      Re: L-shape

      Ponder Stibbons have noted your comment with distaste, and want to take you on a tour of Hex's facilities.

    2. Blockchain commentard

      Re: L-shape

      Down one wall, at the end turn left (or right) and continue with the racks. What's so difficult with that?

      1. MikeOxlong

        Re: L-shape

        Well if you're running your racks down a wall, then I suspect getting to the back of them may prove to be a little tricky as a starter for 10! (But in principle, I agree that L shaped configurations of racks in rooms is entirely plausible, feasible and indeed, practised in some areas.

    3. Anonymous Coward
      Anonymous Coward

      Re: L-shape

      As the rather clumsy protagonist responsible for this submission, I will call your BS claim as the story is indeed true and happened to me roughly 5 years ago. Perhaps a bit more detail would help you visualise things as there is the odd minor inaccuracy in the story as it happens:

      The server room in question was a small, square room. As you enter, the main server rack is directly facing you (with a rackmount-capable UPS sitting next to it at the left of the rack). Turn 90 degrees to the left from the door and you have two more racks. The rack in the corner of the L shape was the comms rack and was facing directly towards the UPS and the side of the main server rack. Hence when patching phones in the comms rack, the UPS and main server rack were actually directly behind me (not next to me). Crappy diagram time:

      < /\


      Hence, crouch down in-front of the comms rack, arse protrudes out and neatly catches one of the trip switches at the top-rear of the UPS behind me. The cabling issue then came when I poked my head around the back of the rack to check the power cables running to our Oracle server and the movement of the cables dislodged the poorly-seated feed into one of the distribution units in the rack.

      Not my cleverest moment, even if sensible cabling would have prevented an outage...

    4. Charles 9

      Re: L-shape

      Especially since parallel racks just seem to beg for domino-ing by a large enough butt.

    5. Stoneshop Silver badge


      That's twice in a week or so that you proclaim something that happened to be impossible.

      You have clearly never been at the coal face of hardware installation, upgrades and maintenance, so kindly keep your trap shut on those matters because it annoys those of us who do and have done so for decades.

    6. Prst. V.Jeltz Silver badge

      Re: L-shape

      You can't arrange server racks in L-shape and arrangement have nothing to do with hanging cables.

      Calling BS on this one.

      whut? to continue the Pratchett theme.

      I'm calling "Try to articulate a comprehensible sentence" on this one. I genuinely cant work out how your post.

      1. Prst. V.Jeltz Silver badge

        Re: L-shape

        "I genuinely cant work out how your post."

        Hmm, after writing several guesses why he was confused, then giving up to get some work done , I seem to have created my own unfinished sentence!

    7. anothercynic Silver badge

      Re: L-shape

      @DainB, I call BS on your BS. Anyone can visualise this scenario (I can. I've seen these kinds of designs before...)

      That's a downvote from me...

  6. Anonymous Coward
    Anonymous Coward

    I'm just glad he got to the bottom of the problem.

  7. frank ly

    "... your tail of woe..."

    I see what you did there. (I strap mine to my leg so it doesn't wag against anything important or delicate.)

    1. Nick Ryan Silver badge

      Re: "... your tail of woe..."

      I see what you did there. (I strap mine to my leg so it doesn't wag against anything important or delicate.)

      You sound like a most considerate mog/mawg, most wouldn't have thought of such a thing! At least in any of the documentaries that I've studied anyway.

  8. Sandtitz Silver badge
    Thumb Up

    #metoo with a big arse

    One client had the single server rack installed into a stupidly tight space in a corner. There was barely space to stand in front of the rack for the times when you needed console. And behind the user was a big fuse board with circuit breakers aplenty.

    Apparently I wasn't the only one who managed to turn off several of those breakers with my posterior and thus turning off several office computers, lightning, and worst of all, the coffee machine. I felt pretty stupid for doing that.

    You couldn't remove or install servers in the rack unless you carefully turned the rack at least 45 degrees or so. Didn't really help that the ethernet patch panels were built into the back of the same rack...

    1. Anonymous Coward
      Anonymous Coward

      Re: #metoo with a big arse

      A customer had offices in a row of ancient houses several storeys high - with an internal labyrinth of interconnecting doorways, corridors, and stairs. One comms fixture was in a small old toilet room. The tall rack was backed up against the wall with the cables at the front.

      To get to anything on the rear of the cabinets the lower section of the rack was empty. You sat on the floor and rested the back of your head against the wall in the small gap behind the rack. That was the day I found I would now need close-up spectacles to read some essential label just beyond my nose.

      1. John Brown (no body) Silver badge

        Re: #metoo with a big arse

        "A customer had offices in a row of ancient houses several storeys high - with an internal labyrinth of interconnecting doorways, corridors, and stairs. One comms fixture was in a small old toilet room. The tall rack was backed up against the wall with the cables at the front."

        That sounds remarkably like a certain brewery in Tadcaster.

        1. defiler

          Re: #metoo with a big arse

          That sounds remarkably like a certain brewery in Tadcaster.

          I'm thinking a solicitor in Edinburgh. That being said, I know somebody who worked in both buildings...

    2. Nick Kew

      Re: #metoo with a big arse

      Apparently I wasn't the only one

      Now we're into the territory of everyday life. Like when the meeting room is overcrowded and someone trips a switch just by squeezing in. Those light switches at around shoulder height by the door (elbow height or bum height work too), and sometimes switches that operate something more entertaining.

      1. T-Bo

        Re: #metoo with a big arse

        Indeed - we have a large, well-equipped meeting hall with powered window blinds, powered projector screens, various spot-lighting arrangements, etc ... all controlled by a cosmos of switch panels strategically placed at hip-height about the perimeter of the room ... rarely does an all-hands meeting pass without blinds, screens, and/or lights being triggered at intervals by folks leaning on the wall ... endless fun.

    3. Stoneshop Silver badge

      Re: #metoo with a big arse

      and thus turning off several office computers, lightning,

      They were building Tesla coils and Vandergraaf generators?

      1. Robert Carnegie Silver badge

        Re: #metoo with a big arse

        Should I say "bums and arses against the lightning!" now, or read down to see if anyone else has, or, not bother?

        It is an obscure reference. And bums and arses are the same thing, anyway - I claim poetic licence. And it clearly should be "Bums and arses against the lighting", anyway.

        And it may not even be funny. And yet aren't bums always funny?

  9. Alan J. Wylie

    VIP visitors

    Many years ago, when computers filled a room, we had lots of important visitors being shown around. I can't remember anything being powered down, but I do remember the "write protect" button on a washing machine sized disk drive (RP06 perhaps?) being pressed during one such visit.

    Here are a couple of photos of our visitors. One had to be very circumspect at such times - no rushing into the room shouting "which blithering idiot?".

    1. TRT Silver badge

      Re: VIP visitors

      Dennis was THERE to take the blame. That was his job.

    2. 's water music

      Re: VIP visitors

      Here are a couple of photos of our visitors.

      Bizarrely my work web filters the domain for 'hate speech'. Google's cache of the home page offers no clues for why that might have happened

    3. paulf

      Re: VIP visitors

      @Alan J. Wylie

      Suggested caption for the second photo:

      "Hahaha I've found the shutdown button!"

    4. Antron Argaiv Silver badge
      Thumb Up

      Re: VIP visitors

      Saaaay...the terminal the bearded chap is typing on in photo #2 -- that wouldn't by any chance be a Tektronix storage tube graphics terminal (4010?), would it?

      Wrote a graphics package for one of those in uni.


      Buttons right out there on the front, easy to back in to.

      // VIP visitors are always "fun"

      1. Alan J. Wylie

        Re: VIP visitors

        Tektronix storage tube graphics terminal (4010?)

        Yes - indeed it was.

        [I] wrote a graphics package

        In that case you might be interested to know that the man standing to Prince Philip's right in photo 1, in the striped suit, tie, finger pointing, is Peter Woodsford, author of GINO

        1. The Oncoming Scorn Silver badge
          Big Brother

          Re: VIP visitors

          Peter Woodsford - I thought for a second it was Dinsdale Landen as Doctor Judson (Doctor Who - The Curse of Fenric), which also featured early computer development as part of the plot to break the Enigma machine.

          However I thought first initially & going for Herr Lipp, but Dinsdale seemed more appropriate.

    5. Stoneshop Silver badge

      Re: VIP visitors

      no rushing into the room shouting "which blithering idiot?".

      Most assuredly a CLM, but correct nonetheless.

  10. Anonymous Coward
    Anonymous Coward

    Not me, but you know how old Shuttle PCs had sticky-out power and reset buttons?

    I was typing, someone came to talk to me, etcetera.

    Thankfully, Shuttle learned and later generations didn't have them.

    1. Chris King

      I remember seeing a full-size desktop case with a similarly stupid design, but with the power and reset buttons about an inch or so from the bottom of the chassis. Just the right height to be pressed by someone pushing a keyboard against them.

  11. jake Silver badge

    Was filling the truck with diesel ...

    ... and noticed a youngster at the next pump getting out to go in to pay in advance. He left the car out of gear, and no parking brake. It started rolling back, towards my truck. I yelled, and lunged for his vehicle .... and caught my toe in the fuel line. Landed face first, with only the thought to get up and stop the car crashing into my truck, which I did. But not before the nozzle popped out of the tank unnoticed by me, and proceeded to pump diesel all over the station's forecourt. Failed auto-shutoff.

    The attendant said that it was only about three gallons lost between the time he he noticed and managed to hit the emergency shut off, but have you ever spilled three gallons of diesel under pressure? What a freaking mess. That stuff gets EVERYWHERE! Fortunately, it's a small town, so we called the kids father ... who made him clean up the mess. In foggy weather the forecourt was slicker than owl shit for months afterwards.

    1. John Robson Silver badge

      Re: Was filling the truck with diesel ...

      And that's why in civilised countries we don't let the pumps operate without a person holding the valve open.

      Oh, and we dispense fuel then go and pay - rather than guessing how much fuel might or might not be needed. Multiple trips in to 'pay' for fuel is a crazy way of doing things...

      1. anothercynic Silver badge

        Re: Was filling the truck with diesel ...

        Ahhh, @John Robson, but that's why those civilised countries have a higher incidence of fuel theft and have to resort to ANPR camera usage to prevent that.

        At least in this instance, it actually makes sense to pre-pay (although I've never liked this weird way of doing things). :-)

        1. Anonymous Coward
          Anonymous Coward

          Re: Was filling the truck with diesel ...

          Prepay works fine: if you've overestimated you just have to go in once more and get change, otherwise you just drive off. Pay at the pump is even better but very often you have vouchers etc. that require you to go in the kiosk.

          1. Anonymous Coward
            Anonymous Coward

            Re: Was filling the truck with diesel ...

            Many years ago I was doing a long run from Denmark to Luxembourg via a short stop in The Netherlands. In the middle of the night I discovered that Dutch petrol stations on major routes weren't open 24 hours.

            The pre-payment machine only took a limited range of Dutch bank notes. I had only a few of them - and then discovered it only liked the couple that were in mint condition. Just enough to get home.

        2. Anonymous Coward
          Anonymous Coward

          Re: Was filling the truck with diesel ...

          "and have to resort to ANPR camera usage to prevent that."

          Many years ago I pulled into the pumps behind a car that was just finishing filling. I noticed that his rear number plate appeared to be covered with tape - and then he drove off at speed without paying. The till operator was immediately on the phone to the police. Apparently he always noted the front number plates of cars arriving late at night.

          1. anothercynic Silver badge

            Re: Was filling the truck with diesel ...

            I noticed that his rear number plate appeared to be covered with tape

            That's why ANPR in the UK uses cameras for the front plate on the forecourt and there are some that check the rear plate as well as people drive in. :-)

            1. David Nash Silver badge

              Re: Was filling the truck with diesel ...

              Doesn't even need ANPR. simple CCTV would work, I doubt there are so many fuel thieves that it needs to be automatic.

        3. John Robson Silver badge

          Re: Was filling the truck with diesel ...

          > Ahhh, @John Robson, but that's why those civilised countries have a higher incidence of fuel theft and have to resort to ANPR camera usage to prevent that.

          I prefer pay at pump myself (at which point pre-auth is sufficiently easy to make sense - particularly since the 'refund' doesn't then need the card a second time) - but fuel theft is no different to theft in other shops (except that ANPR makes it really easy to ID the culprit).

          1. anothercynic Silver badge

            Re: Was filling the truck with diesel ...

            prefer pay at pump myself (at which point pre-auth is sufficiently easy to make sense - particularly since the 'refund' doesn't then need the card a second time)

            The newer way of Pay-at-pump systems (for card payment) make a lot more sense... no need to ever go and pay someone in person.

        4. The Oncoming Scorn Silver badge

          Re: Was filling the truck with diesel ...

          Pre-pay became compulsory a few years back in Alberta, after a gas station attendent was unfortunately killed after chasing non paying clients.

          The latch to allow the nozzle to dispense gas without needing a human is bloody useful at -27C (Especially if driving a truck that takes approximately 120 litres).

          1. John Robson Silver badge

            Re: Was filling the truck with diesel ...

            > The latch to allow the nozzle to dispense gas without needing a human is bloody useful at -27C (Especially if driving a truck that takes approximately 120 litres).

            So you're not dressing appropriately for the conditions?

            Some issues which are resolved:

            - Can't drive off with the hose still in.

            - Remaining grounded so you can't ignite the vapour with a static discharge (and then pull the nozzle out in a panic and spray (now ignited) fuel over the forecourt...)

            1. Stevie

              Re: Was filling the truck with diesel ...

              John, you are being a tw*t.

              In Alberta -27C doesn't include the windchill, and no you won't be standing outside in any amount of "appropriate dress" in that because even breathing is uncomfortable if you've ever actually done it at those temps. Which you obviously never have.

              The hose-in drive-off is still depressingly possible with a manual-only pump gun as five minute browsing YouTube will show. I'll grant it is slightly more likely with a pump latch.

              And in "Civilized Countries" (you started it) the pumps are designed to recover the vapor (no "u" because we are being snotty) so there is nothing to ignite with the static electricity you feel will be coursing around the forecourt.

              Why you feel that a person holding the metal trigger with gloves graded for -27C and wearing waterproof rubber-soled boots good for -27C would provide any sort of reliable ground anyway is a bit of a puzzler. Which I suppose could be why the pumps themselves are grounded. Long shot I know, but that sorta works for me.

      2. Stevie

        Re: Multiple trips in to 'pay' for fuel is a crazy way of doing things...

        What the flock are you talking about JR?

        You either say "I want X amount of gas" or "I want X dollars worth" to the attendant (EXACTLY like it used to be in the UK back in the late 70s when "pay before you pump" was first initiated) or you use the almost ubiquitous pay-at-the-pump credit/debit card technique.

        Where do the "multiple trips to pay" come in?

  12. chivo243 Silver badge

    I can relate

    I've also brought down a full rack. A colleague and I were having trouble getting a network card on an older UPS communicating with the outside world. We used an incorrect serial cable to connect to the UPS. Now when a new UPS comes, the serial cable is tagged and paired with its owner.

    The room gets extremely quiet when one rack goes dark! We still laugh about that day.

    1. kain preacher

      Re: I can relate

      FUCKING APC. I've had that pain. Use the wrong (non standard ) serial cable and it turns it off.

  13. FlamingDeath Silver badge


    I'm a ninja, this stuff never happens to me

    1. TRT Silver badge

      Re: D'oh

      Eat a Snickers.

      1. The Oncoming Scorn Silver badge

        Re: D'oh

        Eat a lot of Snickers (Or Marathons) & donuts.

  14. AustinTX

    Whoopsy Daisy

    OK then, FINE. I *may* have brought down a big U-shaped robotic tape storage vault by feeding it a cassette which I'd just dropped on the floor. In all fairness, no-one actually told me why the vault had to be shut off the next day so someone could go inside and cut out a wad of tape which was jamming one of the readers.

  15. Chewi


    Whilst building my new desktop last year, I was hunched over it to reach the back when I discovered that I had the ability to switch it on with my balls. The power button is located on the top.

    1. Anonymous Coward

      Re: Balls

      Very trendy, taking your sexuality to new places. That's in the general. But at risk of TMI in the particular.

    2. DuchessofDukeStreet
      Paris Hilton

      Re: Balls

      Braggart :-P

    3. onefang

      Re: Balls

      "I had the ability to switch it on with my balls."

      I guess that's better than the ability to turn them off with your balls.

    4. Montreal Sean

      Re: Balls


      I know this is a British site, but that's not an appropriate tea bag use!

  16. GruntyMcPugh Silver badge

    We'll call him,...

    A former colleague of mine, we'll call him Suhail,. because that was his name, did exactly this. We worked a kind of shift, ish, some of us would come in at 08:00hrs, so we had time to rectify any minor niggles before the muggles clocked on, and then others would work until 18:00hrs, so someone was available, allowing whoever was on call to get home. So muggins is staying until 18:00hrs, when said colleague called at 17:00hrs, and fesses up to knocking the power lead out of a disk array down in the DC, which for some reason only had one feed. He said he'd love to stay, but he had a pressing engagement (not the buttock against the kettle lead this time) and would I do him a solid and make sure everything came back OK. Luckily, it did.

  17. Tiggrrr42

    Tap, tap...

    Mumble years ago, we had a small department network (Netware 2.x), thin Ethernet[1]. This had been set up with floor ports under desks. Well, I say floor ports, but I mean angled plates in an inverted V formation sticking out of the floor, with a connector on each side, to which a cable was connected.

    For reasons of not spending much money, the end users had what were known at the time as diskless workstations - PCs which booted their OS (DOS) from an image on the server. So, in the event of no network, these were slightly useless.

    We had a user who, in his non-working hours, was a drummer. And he used to tap his feet a lot while sitting at his desk.

    And one day, he managed to move his tapping a bit further under the desk and CRUNCH. One totally destroyed BNC connector and no network in his department....

    [1] Coaxial cables joining one computer to the next with a terminator plug thingy at each end. Had the slight drawback that one broken or disconnected cable would drop the whole network

  18. Anonymous Coward
    Anonymous Coward

    ITT: Fat People

    That is all

  19. AndyMulhearn

    Not a crash but

    Ten years ago I ran a small infrastructure team for a large news/data organisation. We had a bunch of Sun and HP kit in a small comms room in St. Katherine's Dock. One day we get a fire alarm, so we all troop out then back in only for another one to kick off and we all troop out again.

    Third time it happens, as we're heading back in my Unix guy calls me saying "Can you nip to the comms room, some of the boxes are reporting over temperature." You can no doubt imagine the conclusion I drew so my response was "Very funny" and I hung up.

    He's back on the phone 30s later "No, I'm not taking the piss...". Turns out the outflow from the de-humidifier had blocked so the air con had shut down, or some similar cause. 30 odd boxes had to be shutdown to cries of "I have a demo" and "How can we develop" and me responding to the effect that they should go and get a coffee. Ot something similar related to travel.

    Having said that, there was one evening I was manually patching a remote windows box and selected shutdown rather than reboot. It would have been OK but for the fact that box was in a remote data centre and we had no details of the rack. Or in fact an understanding of which of the many remote sites it happened to be in... Fortunately we narrowed it down to one, small, site and a local ops guy was kind enough to push the on button on the one HP DL 380 in the room that had no lights flashing on it.

    Happy times.

    1. Antron Argaiv Silver badge
      Thumb Up

      Re: Not a crash but

      ...shutdown rather than was in a remote data centre...

      Been there (about an hour away), done that.

      // it involved rebooting with a bad network configuration...

      1. AndyMulhearn

        Re: Not a crash but

        Been there (about an hour away), done that.

        This was late one night. Startup was the following day :-(

    2. Jay 2

      Re: Not a crash but

      Was that in a building known as Commodity Quay by any chance?

      1. AndyMulhearn

        Re: Not a crash but

        Was that in a building known as Commodity Quay by any chance?

        It may well have been but I'm saying nuffink.

    3. Charles 9

      Re: Not a crash but

      "30 odd boxes had to be shutdown to cries of "I have a demo" and "How can we develop" and me responding to the effect that they should go and get a coffee."

      How many of them countered, "I have a deadline and the board's on my neck!" AND "I hate coffee."

  20. Wanting more

    my worst mistakes

    I was supposed to be refreshing the test products database from our live database. Both databases were on the same host. I accidentally mixed up the database names and refreshed the live database with the test data. At least the backup system worked to restore the data when I realized what I'd done. Fortunately this was 15 years ago and most of our workload was batch with a few screens for data maintenance. If I did this now it would of affected online websites and whatnot. The databases were moved on to separate hosts soon after.

    Another time I was coding a database trigger so that on delete of one record it would delete some child records in another table. Unfortunately I missed the where clause from the child delete and it emptied the entire table of records. Furthermore the tester didn't catch the issue (it had deleted the record they were expecting and they didn't notice that all the rest of them had gone also). However it was quickly noticed when the code ran on the live database for the first time...

    1. Anonymous Coward
      Anonymous Coward

      Re: my worst mistakes

      DB triggers for child table cleanup? Let me guess, MySQL with the awful MyISAM table type that doesn't support foreign keys. I so hate that mock database engine.

  21. Paul Cooper

    Problems in polar regions

    Well, the worst situation I ever encountered wasn't in a purpose-built computer room. This was back in about 1983, and I had developed a system to log radar data from an airborne ice-penetrating radar for use in Svalbard. State of the art at the time - an S100 bus Z80-based single card computer! No OS, of course; everything had to be programmed in Assembler, and my development system was an Osborne 1 with Wordstar, asm and link - remember them? All was well in the office in Cambridge, but of course, it wasn't quite as we wanted it when we went off to the polar regions, so we took the development system with us. No problem - the Osborne 1 was (just about!) portable.

    Unfortunately, the room we worked in had a nylon carpet, and of course, the humidity in the polar regions is extremely low. It was so bad that every time we brushed against a metal filing cabinet next to the door, we got a serious spark - enough to sting! It took a few goes for me to remember to touch metal BEFORE touching the keyboard of the Osborne 1; if you didn't the poor little thing just died. Fortunately, the whole setup was such that I always saved stuff (to floppy discs!) at regular intervals.

  22. MJI Silver badge

    Our home PC

    Reset button at toe level

    Me once

    Wife once


  23. TRT Silver badge

    If a particularly long and sticky out mains lever switch...

    Gets caught up your butt crack, does it cause a brown out?

  24. Jon 33

    Belly's goona get ya!

    An unnamed BT/OpenReach engineer once used his belly to hit the emergency stop button in our server room.

    He couldnt be bothered to use his hands to move his security pass from his belly to the pass scanner to get out of the room so belly flopped it onto the pass reader (and hit the emergency stop button located 30 cm's underneath)

    Think of "Fat B$$stard from the Austin Power movies....

    1. defiler

      Re: Belly's goona get ya!

      Ewww. Even I've realised that it's bad to get to that stage. And I'm peculiarly lazy and snacky!

  25. Anonymous Coward
    Anonymous Coward

    Buttocks of power

    Been there. Done that.

    Some years ago I was involved in an EU funded project to put learning centres throughout SE Scotland in various small business, libraries and learning establishments. This involved installing a server, workstations and (if required) structured cabling and network kit.

    In one Uni in Edinburgh I was shown the cabinets in the server room and pointed to the space where the new server was to go. Except in this case read Dexion racking for cabinets and basement for server room.

    Now, I am not a particularly tubby guy but I struggled to manhandle an IBM Netfinity server down the gap between the wall and the back of the "cabinets" as they hadn't left a huge amount of space. Part way down my journey along the wall I felt a click on my backside. Looking round, I saw that the geniuses had put a PDU with a honking great rocker switch on the back of the racking which I had managed to switch off with my arse, turning half of their kit off.

    I pushed my backside against the switch again to turn things back on, dumped the server and left in a hurry. On my way out I heard one of the admins wondering what was going on with their DNS servers.

  26. I Am Spartacus

    On the receiving end of this

    When the Sequent engineer came in to register all of the new SCSI disks we had spent a week installing and configuring. This was in the days when 4G SCSI disks were considered the best thing ever and we had a full rack of these, populated front and back. It ran the Customer Care database, the incident logging DB, and most important, the transaction log that we were required to keep by law.

    cue the Sequent engineer kneeling down to read the lower disk IDs, over balancing, reaching out, and yes, he hit the main power contactor. "It was only off for a second", he told me, as the DBA's started wailing that their database had gone down. And then wouldn't come up because it seemed we still had the transaction log on the same disk array.

    Cue backup taper recall from the safe store, reinstall, and oh? Where are the historical table spaces? It seems that the Sequent backup program for Oracle doesn't backup readonly table spaces at all.

    Next thing happening:

    1) Me , explaining to the regulator, why we couldn't fulfil their request for information we had to have;

    2) The DBA's doing a repeat after me: "A backup you haven't tested is not a backup at all"

    3) Sequent issuing a fix: namely a page to insert in to the manual set saying the readonly tables spaces don't backup.

    Very expensive and painful lesson learnt, and we too put cardboard flaps over the very exposed main switches!

  27. Alien8n

    Server rack

    Installing a new phone system at a previous place and dislodged a wooden panel at the base of the server rack. Panel dropped into the floor space under the rack and managed to land perfectly on the single power switch hidden underneath the rack, resulting in the whole rack going down. Turns out the 2 large UPSes had been disconnected as well by a previous person so there was no UPS backup for the servers. Took 20 minutes before everything came back up again.

  28. keith_w

    Crashed Mainframe - Twice

    Many years ago when I was a system programmer we had a big red ermergency power off button on the wall by the exit. I was working on a communications box which was directly under the button and straightened up and hit the button with my shoulder shutting everything in the computer room down. The second time it happened I suggested that we cover the button with a plastic flip up cover, my boss threatened to fire me if it happened again instead. The third time it happened, I was in my cubicle and popped up and yelled "Not Me! I'm here!". The plastic cover got installed.

    1. Anonymous Coward
      Anonymous Coward

      Re: Crashed Mainframe - Twice

      To save my repeating myself again here's one I posted earlier honed to its essentials.

  29. Stuart Castle Silver badge

    A couple from me. One was my mistake, the others I just witnessed..

    The one that was my mistake.

    It was my first job after getting my degree. We had what we called a "server room" that was actually a bench occupying half my then office. It had a few assorted servers, including the company webserver and our NT domain PDC and BDCs.. All protected by a UPS (or so I was told).

    One day, the boss was off. I had a CD walkman I bought in every day, I was doing work on one of the servers,so I plugged my CD walkman in to listen to it. Bang. The circuit breaker tripped, and the UPS lasted just long enough to shut everything down safely. When I queried it with our FM department, they said that the electrical circuit in question was up to it's limit, and my boss had been told there was too much equipment plugged in to it.

    The second one was a company I worked at had a relatively new building, with a nice, riverside location (as nice as one can get in Woolwich, anyway). They spent a fortune cabling up the building for a new network and built a state of the art server room in the basement. The problem is, being on the bank of the Thames, the basement of the building was prone to flooding. They moved the server room a couple of years later.

    The final one was when I left school. I worked for Network Southeast, in a local office, just doing general admin work. I heard a loud bang come from the lift next door. Had a look, and there was a rather large computer at an odd angle on the floor. I knew they were moving some offices around, and I figured it was just a delivery man dropping a PC. It wasn't. I heard several more bangs throughout the day, I queried it with my boss, he reported it. Apparently, he reported it, and found out that it was actually a removals company who'd been hired to move the servers in one room to another room.

    1. Anonymous Coward
      Anonymous Coward

      In Africa - a long awaited mainframe memory upgrade was finally delivered from the UK. In those days that was a 6' (1.8m) high cabinet for all of 128KB.

      The engineers opened the door - to see the internal rack frame now looking like a no-longer rectangular rhombus. A tell-tale mark near the top of the case showed where it had been abruptly stopped after falling sideways at some point in its journey.

      1. Anonymous Coward
        Anonymous Coward

        "The engineers opened the door - to see the internal rack frame now looking like a no-longer rectangular rhombus."

        Don't you mean parallelogram? The only way it could be a rhombus is if it had been a square originally (as both are defined as being equilateral but the square demands right angles).

  30. Big_Boomer Silver badge


    I have previously refused to even enter a customers server room after seeing network and power cables running left, right, and centre across the space they expected me to go through to reach the server I needed to work on. Nope, yer not blaming me for downing your servers! Fix your cabling disaster first, then I'll fix your server. Came back 2 weeks later and it was tidy and accessible.

  31. DataWombat

    I was in the optician's trying to get a retina photo - nothing on the monitor. Cue much switching the monitor on & off etc. Eventually I looked where the cables were going, and found the PC mounted under the table cunningly positioned so my knee had hit the off switch. Meanwhile the wife was waiting outside thinking I'd got horrendous retina problems. Apparently it was all my fault...

  32. Syn3rg

    Late 90's Netware Gefingerpoken

    It was the late 90's, and there was a known condition that would arise on one NetWare 4 server. The workaround, from Novell, was to cold-boot the box. So I roll into that data center at 3AM, still somewhat groggy despite a 75 minute drive, walk to the server, open the door and push in the power button. It was at that exact moment that I realized that my finger was on the button of a nearly identically named server (two transposed letters in the middle of the name). Needless to say, I was now fully awake. I then proceeded to do the one-handed-shutdown boogie from the KVM in the next rack.

  33. Anonymous Coward
    Anonymous Coward

    Pull up chair and let me tell you tail of IT misery and woe.

    This the story of the time and MCI data center went down.

    During a routine inspections of their sprinkler systems they use a long pole to inspect them and see if the senor worked. Now some cheeped out and did not provide an cover the EPO button and you can guess what happened.

    No biggie just turn the power back on. Only if it was that easy.

    See kids this is were danger (side kick ) housed their servers. To make things worse this was an enhanced data center meaning the did voice and data.

    Just when you think it could not get worse. This is were ATT and MCI did physical peering and was marked off line by a DNS update.

  34. Mohawk

    That's Nothing

    I had a colleague a bunch of years back working in a large Datacentre that was pulling in a bundle of Cat5 cables for a new install through the big floor tile trench they had opened & just as he got to the end of the pull the whole place fell silent ! Turns out his derrière had bumped into an emergency power shutdown button that did NT have a protective cover, Oops ! What me, nah run away :)

  35. bristolmoose

    Timely article. This happened to me today. I have one of those stupid big Alien PCs at work with the stupidly big stupid alien head as the power button, which stupidly stands proud of the surface of the unit. My colleague once shut my PC down by kneeing the button as she rolled her chair over to my desk to ask me something. So I changed the power button to do nothing. Problem solved or so I thought. Today she managed to hard reset the PC by crossing her legs and resting her boot against the stupid button. Thank the $deity$ for autosave in Excel!

  36. Grant Fromage

    Sometimes there is literally a hitch

    Spacing between bays, hmm.

    Simple job, one company that i had a few days per month with regularly hired in stuff for sports or concerts and all the downstream odds, my job was plug it, test it and link it without taking anything running off air. ( or fail and no more work)

    The video edit/severs had arrived, there was a vtr +extras loom in the bays in the racks, nothing in the rack behind me, empty bay just a loom. Concentration on job. GigE extras from the stuff for Hd to dangly lead later.

    First server was at about knee height (26U up?), so I squatted down and felt a tug on the back of my belt and a sound of ripping, but from above, not behind.

    A 6/8 port GigE switch on air with several other servers behind me, was tagged to me after parting from velcromount. Non-latching power from a wallwart, mains via an iec10 to Bs-13A not cable tied in.either.


    I worked out the best sequence of movement to stand up with no pull on the cables and did it slowly until I could reach to put it all back. and did so. NOT funny. !!!

    Logic says, hooks on the fixed mounts, furry on the wanderer so it is easier to clean and picks up less crud in transit, commonsense. It also stops it latching onto the back of your fleece.

    There was a 3-ish metre cat 6 cable terminated in cat5 ends, hoods don`t fit connectors stressed and also this means thelatch hooks are out. and I had been caught. by one as sat down

    I do not get why the bodge of cat6 .

    10, even 20m of cat 5I work to GigE spec for all but counterfeit cables, Cat 6 3m of installation cable as regular pluggable/? as flexi as made of pasta and no cable grips possible, daft.

    I`m losing mine slowly, but there is a world shortage of commonsense IMHO.

  37. rnturn

    This reminds me of...

    ... one of the evening sessions at DECUS many years ago where one speaker lamented the fact that the controls for a certain model of disk drive (this was back when disk drives were washing machine sized) were the same height as his rear-end and, with a wallet in your back pocket, it was very easy to take disk drives offline by backing into the drive.

  38. Peter \Horne

    ...and the band played on.

    This is from a Rock and Roll Lighting perspective rather than IT, however we do use lots of IT kit these days. This happened one summer about 5 years ago, I had done a UK Stadium tour with a big Alternative Rock band, and during overnight programming sessions I had noticed all the APC UPS's dropping back to Battery power every time the Lighting Designer hit a big look - Lots of strobes and audience blinders - and I realized the sensitivity settings of the UPS's were too high for working on generators. I got the warehouse to send out a USB to serial interface and a serial cable and armed with the DOS based update program I went round all the equipment racks and changed the sensitivity to Low without having to turn anything off or disturb the Lighting Designers programming time.

    A year later I was looking after the racks understage a large UK festival (big Egyptian shaped stage). I had noticed when loading in that there was one UPS that was too sensitive and had obviously not been on the tour the year before when I had reset all the others. Again I requested the necessary kit from the warehouse but it did not arrive until the last day of the festival. Undeterred, I had asked for the kit, so I was going to use it - and in my opinion ALL the UPS's the company owned should be set to Low Sensitivity by default as we often run on generators.

    I go round the back of the rack, plug in the serial cable and fire up the software. No UPS detected, I try again, same result.

    While I'm sat there pondering why this isn't working - I had no problems the year before - another member of crew comes running into underworld shouting that "ALL THE LIGHTS ARE OUT".

    Panic Time - Slash is onstage above my head - there are 60,000 people on the other side of a black drape from me and they are looking at a band playing in a dim stage (thank god it wasn't properly dark)

    I run round the front of the wall of mains distribution racks and the whole data rack I was working on is off, no reassuring blinky lights to be seen. I turn the UPS back on and a minute later everything is working again.

    It was unfortunate that the Intercomm Masterstation which powers the headsets to talk to the LD at Front of House (and has big flashing strobes and sirens to attract attention) was in this data rack. It had taken theOperator several minutes to get hold of somebody at stage over a radio, amid all the noise, to come and find me.

    It transpires that the serial cable needed is a 'special' one - with non-standard pinout - with the APC logo embossed on the connector. This time the warehouse had sent out a generic serial cable, liuterally the moment I plugged this cable into the back of the UPS the UPS had turned off - no beeps, no warning just off and I could not tell from the back.

    I sent the serial cable back to the warehouse cut into lots of short pieces.

  39. ICPurvis47

    When I was working for a very large electrical engineering company, building motor control gear for a well known maritime organisation, I also caused mayhem with part of my anatomy. One of the units was in Test, and the testers were doing a heat run, running the equipment at full chat whilst the observers from the customer looked on. I had gone into Test to take some photographs for the Instruction Manual that I was preparing (I was in technical Manuals Department at the time) and I had to scrunch myself up into one corner of the roped-off area in order to get all of the cabinets in shot. Suddenly everything went dark, and the high pitched whine of the invertors wound down the scale to inaudibility. Cue furious shouts from the Test Engineers, I had inadvertently backed onto one of the emergency shutdown buttons that were located at various points around the department, and that had shut off all power to the Test area and surrounding parts of the building. A complete morning's heat run ruined, and the customer's observers were distinctly unimpressed. The heat run had to be rescheduled for the next morning as it had to start from cold. Needless to say, I was NOT allowed into Test whist a heat run was being performed on that or any further equipments.

  40. onefang

    It pays to plan ahead.

    A few years ago I was the sysadmin for a medium sized company. They had two racks in their server room, and had grown to the point where they needed a third. It was gonna be a bit of a tight squeeze, luckily they had the foresight to measure me up at the same time they where measuring up the room. So they could fit in their third rack, and still leave room for me to squeeze between them when I needed to. I joked at the time that if they had not done that, I would have had to swap jobs with one of the programmers, a tall skinny guy.

    Much further back in time, when I was a programmer at some other company, I had noticed the hardware guy had a habit of powering up clients computers very quickly after powering them down when he needed to power cycle them. I warned him a few times that he should wait at least five seconds between off and on, which he ignored. Hey, I'm just the software guy, what do I know? Much to my amusement I had the joy of watching him do that to his own computer one day, with a loud bang and smoke just before my " told you so".

  41. defiler

    I don't believe it.

    The clue is where the boss signed off for overtime...

  42. fobobob


    Just last week, I had reason to read a manufacturer number off of a part in a pedestal server... needed a flashlight.... oh hey, cell phone! I touched nothing in the case and got what I needed, but when I emerged from the server room/storage closet (small-ish business), I was hearing cries of despair, the main local repository of most of our company's work has suffered, um, Total Inability To Support (insert something funny, I'm not funny, and also American, so help me out here)! (a whee bit of exaggeration there)...

    Reboot, everything was working again. Didn't think too much of it at the time. Fast forward to Monday, I go investigating another unrelated issue and notice that the RAID was missing two disks... Seems "something" caused one of the disk controllers in the box to glitch out and lock up, resulting in a soft lockup of the system, and software RAID6 coming back up with two disks missing... survivable, etc. The disks appeared to be fine, and it was clear a controller crap-out was the cause. Anyhow, I re-add the disks (which are all in perfect health), and while it's merrily rebuilding, the controller flakes out again... this time very noticeably, given the wall of crap on the console. After the reboot, the RAID6 is stuck in some silly state where two of the disks seem to have become.. independent .. of the other 4, and unfortunately, those two were part of the formerly working array...

    Fast forward to today, we've cloned the disks, made backups, and forcibly reassembled the RAID, fsck'd the filesystems, etc., and the thing is back chugging along with a new disk controller, now with only 2 disks on each.

    Could've been a lot worse, as we became acutely aware that our backup target had become completely full (I had been trying to keep up on that, but non-IT workload had become a bit excessive), and the last full backup had occurred some time in mid-December. Lessons learned. Getting fresh NAS hardware, expanding the offsite backup target, and revising our policies/procedures...

    Root cause? Dunno, flaky Marvell onboard SATA controller (which had worked perfectly for several years prior) and possibly RFI from the cellphone... anyone here ever encountered that sort of thing?

    edit: Actually, my father had suggested, in passing, the possibility of some sort of photosensitivity issue, a la the Raspberry Pis that could be crashed/rebooted with a camera flash. Having operated the thing for many years, and shone many flashlights in it, it seems unlikely... still an interesting thought

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like