back to article Thanks, boss. The accidental creation of a lights-out data centre – what a fun surprise

Another tale involving buttons of the big and red variety arrives for today's deposit in the Who, Me? archives. There are some things that are best left unsaid. And unpressed. Naming no company names, our reader (regomised as "Jim") told us "I was a shift leader in a data centre that ran the European IT for a big American …

  1. Admiral Grace Hopper

    Push me

    Everyone who has been in the industry long enough has witnessed a Father Dougal moment.

    1. ShadowSystems

      Re: Push me

      I once had a poster with the Hand of God reaching down out of the clouds & poised to push a big red button. The caption read "I wonder what this button does?"

      I bought a USB-attached big red button under the clear plastic "missile launch" cover, the words "Emergency Use Only" across the top. You flipped it open, it made the classic warning klaxxon siren, and the button lit up even brighter. Press the button & the light/siren turned off to be replaced with a cheery female voice "Thank you for pressing the Big Red Button. You have aproximately 5 seconds to kiss your ass goodbye. 4... 3... 2... 1... Too late!"

      My posting of the poster above the physical button on my desk made my coworkers a bit amused/nervous. I don't know why...

      *Wanders off whistling innocently*

      1. Admiral Grace Hopper

        Re: Push me

        I'd managed to get this far into Monday without hearing of something I wanted to buy. If I ever see the inside of an office again I'm getting one of these.

        1. ShadowSystems

          Re: Push me

          I don't know if the original is still floating out there on Ebay, but the Staples (office supply store) "Big Easy" button might be easier to find. Scrub the original letters off, replace them with well-crafted Tippex lettering, & give it an internal RPi to give it the ability to play prerecorded audio clips of your choosing.

          *Hands you a pint*

          Just remember NOT to wire it up to the orbital LART controls lest you turn someone into a smoking glassy crater "on accident". =-D

        2. Velv

          Re: Push me

          I got mine from everyone's favourite online tat bazaar many years ago. Here's what it looks like :)

          1. Woza

            Re: Push me

            A friend I used to work with pushed the Big Red Button by accident twice within a couple of months. We bought him one of these as a leaving gift.

            (His leaving was unconnected to these incidents)

            1. Eclectic Man Silver badge

              Re: Push me

              I have an old style computer keyboard button in bright red plastic with the word "PANIC" on it. I never did manage to fit it to a computer :o(

              I got it from a shop a the Museum of Modern Art in San Diego (I think) (I was on a conference there at the time).

              1. W.S.Gosset Silver badge

                Re: Push me

                [Username checks out]

          2. Shalghar

            Re: Push me

            I found a battery operated big red button at technisk magasinet in stockholm (some kind of swedish tech shop ranging from audio tech, electric unicycles over soft air weapons to all kinds of gadgettery you may or might never imagine).

            This thing (after removing the telltale white writing on top) looks like your typical 3 inch/9 cm emergency stop button, only that it shouts "Bullshit" whenever pressed.

            And yes, i already used it as a decoy, two times effectively preventing the real e-stop button ( around 3 meters down the corridor) to be pressed.

            1. David 132 Silver badge

              Re: Push me

              And yes, i already used it as a decoy, two times effectively preventing the real e-stop button ( around 3 meters down the corridor) to be pressed.

              All fun and games until someone walks up to the joke button on the wall, laughingly says "listen to what happens when I press this, it's hilarious!"... and realizes they are 3 metres further up the corridor than they thought...

              Definite shades of Chief Inspector Dreyfus and his joke gun-shaped cigarette lighter... "Don't just stand there, call a doctor, and then help me find my nose..."

      2. scoldog1

        Re: Push me

        I once had a poster with the Hand of God reaching down out of the clouds & poised to push a big red button. The caption read "I wonder what this button does?"

        "Sir. There appears to be a large object in the path of our-"

  2. b0llchit Silver badge

    Access denied

    The last data-center I was working at had a strict policy of non-interference. That meant, the "Boss" did not have the key to the rooms where both active and passive equipment were located. Only certified people were allowed entrance.

    1. St Deiniol

      Re: Access denied

      An entirely sensible policy. So sensible I'm genuinely surprised!

      1. PM from Hell

        Re: Access denied

        I had the same policy, plus all guests had to be authorized by me, in advance so I could ensure we had a couple of sysprogs to act as guides / wardens.

        Imagine my surprise returning from a meeting to find a tour of school children in the data center, no I hadn't been informed, no there were not sufficient people to safeguard my DC and yes it had been organised weeks ago by a county Councillor who hadn't bothered to tell anyone.

        It was to late to stop that one but I did have an interesting 'chat' with my manager involving resignation threats, and a press release in the case of an unplanned outage. I was completely serious about both. I'd put my body and soul into taking a team who were not meeting any SLA's to a 99.999% availability across 3 different mainframe technologies and was not having a return to the bad old days.

      2. John Brown (no body) Silver badge

        Re: Access denied

        "An entirely sensible policy. So sensible I'm genuinely surprised!"

        But was the policy planned in advance or from hard learned experience?

        1. David 132 Silver badge

          Re: Access denied

          "How do you avoid making mistakes? Experience.

          How do you get experience? By making mistakes."

          1. Thomas Steven 1

            Re: Access denied

            The only thing better than learning from your mistakes is learning from somebody else's mistakes.

            1. Anonymous Custard Silver badge

              Re: Access denied

              Knowledge is knowing what to do.

              Experience is knowing what not to do ever again.

      3. vogon00

        Re: Access denied

        Thanks to the similar stories on El Reg, I hope to never suffer something like this, as management are told to keep their hands in their pockets in the lab and they do not have write access to the important production-related network shares.

        In fact, they are read-only for me too by default!

    2. ShadowSystems

      Re: Access denied

      Only certified personnel allowed? That's fine, I'm certifiable so I should get in, right? =-D

      1. Eclectic Man Silver badge

        Re: Access denied

        At one company I worked they had a most enlightened employment policy of hiring people you would not normally consider for jobs particularly in senior management positions - I guess that about half of them were patients on day release from Broadmoor, judging by their people and other skills.

        I'll get my coat, its the one with the sleeves that meet round the back.

        1. ShadowSystems

          Re: Access denied

          I used to carry a printed plastic card in my wallet about the same size as a credit card. It claimed to be an official day pass from the local mental hospital "extremely twisted ward". Whenever anyone asked me if I had escaped, I'd pull out my wallet, show them the card, & gleefully proclaim "I didn't escape, they gave me a day pass!"

          I wish I still had it, I could use it to gain free munchies from time to time when showing it to restaurant wait staff. =-J

    3. Eclectic Man Silver badge

      Re: Access denied and Big Red Buttons

      We had the same thing at BT data centres. Many's the time when I had to inform a prospective client that the requested contract clause that they could turn up and inspect the servers running their service without notice was not possible. I would carefully explain that the sites were secure, some ran Critical National Infrastructure (CNI) such as 'Blue Light' (999) services, larger supermarket chain logistics systems and suchlike, and that even the Chairman and CEO turning up would not get in without an appointment.

      If they wanted to come in and test the power fail-over in the event of a mains failure, they could arrange to conduct the once a year test. One of the data centre managers told me that only half of the customers who had arranged to literally 'press the big red button' and turn off the mains power actually had the courage to press it during their visit.

      I once, as a consultant visited Shell's North Sea modelling data centre, getting into the computer room (very secure actually). And yes there was a 'big red button' - I felt a strong urge to press it, but managed to keep my hands in my pockets.

      1. MarkET

        Re: Access denied and Big Red Buttons

        I worked for BT in the 80s at a DISU. 12 floors of IBM, not a red button in sight. Had some prefcat rooms though. Used to think it was the officers mess.

    4. Velv

      Re: Access denied

      To be fair, most places I've worked at least half the certified people shouldn't have been granted a key either.

      Nobody had access on their proximity card and everyone had to sign out a second card for access. And no, even still the Boss was not on the list.

    5. big_D Silver badge

      Re: Access denied

      But nowadays, the big red button is often outsourced through TeamViewer...

    6. Korev Silver badge

      Re: Access denied

      When I was a manager this was exactly the case. I couldn't access the machines, but I still responsible for them...

    7. Evil Auditor

      Re: Access denied

      Seems very sensible.

      Can't remember how many discussions with C-levels I've had, initiated by having a look at the data centre access list. Answers ranging from "I'm the CEO of the company and need to have permanent access everywhere" to "we're chillin' the beverages in there".

      And then there's the story for which I don't have any proof (a.k.a data centre legend) of a boss hitting the Big Red Button at the eve of a long bank holiday weekend: "now you've got all weekend to test the IT contingency plan."

      1. vogon00

        Re: Access denied

        "now you've got all weekend to test the IT contingency plan."

        I hope my response would have been just as legendary - 'No, you have. I quit, you twat. Seeya'.

        1. Eclectic Man Silver badge

          Re: Access denied

          Plus "On my way out do you want me to let the PR and the customer support teams know what you've just done or are you going to let it come as a surprise?"

    8. The Oncoming Scorn Silver badge

      Re: Access denied

      Manglement are usually certifiable.

    9. uccsoundman

      Re: Access denied

      I'm very surprised that the Boss did not self-certify, you know, because he is god.

  3. Stuart Halliday

    If you display a big red button, expect it to be pressed. Stupid IT.

    1. swm Silver badge

      At Dartmouth in the days of time sharing we had a room full of mainframe boxes. There was a big red button near one of the entrances. College students would regularly come in to the machine room (it was never locked). There was even a demonstration by students that packed the room solid with protesting students. None of them ever pressed the button.

      However, the chief operator pressed it twice: once when the room started filling up with steam after an adjustment to the A/C and another time for a perfectly good reason (I forget what). The button worked both times.

      1. Wexford

        The former time seems just as perfectly good a reason to emergency halt as the latter. If I saw my DC filling with mist or condensation from the AC I'd be pretty keen to protect whatever I could while I could!

  4. IHateWearingATie

    Obligatory quote...

    I feel a Terry Pratchett quote coming on....

    "If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH,' the paint wouldn't even have time to dry."

    1. ShadowSystems

      Re: Obligatory quote...

      *Presses the button*


    2. b0llchit Silver badge

      Re: Obligatory quote...

      So, why hasn't this 'End-of-the-World Switch' been available?

      I am certain many have the urge to press such button, even before it gets painted, with hanging wires and all, me included.

      1. Eclectic Man Silver badge

        Re: Obligatory quote...

        There is a theory that if ever anyone finds out what the universe is for it will immediately vanish and be replaced by something even more bizarrely inexplicable.

        There is another theory that this has already happened ....

        (Obligatory HHGTTG reference.)

        1. The Oncoming Scorn Silver badge
          Thumb Up

          Re: Obligatory (Better) quote...

          Arthur Dent: What happens if I press this button?

          Ford Prefect: I wouldn't-

          Arthur Dent: Oh.

          Ford Prefect: What happened?

          Arthur Dent: A sign lit up, saying 'Please do not press this button again.

        2. ShadowSystems

          Re: Obligatory quote...

          And then there's those of us that have realized that not only has it happened before, it has happened repeatedly, and Loki will continue to press the button until the results are to his liking. =-j

    3. Psmo

      Re: Obligatory quote...

      Presses the paint.

      Creates a ticket for the finger covered in paint.

      And a second for the forehead with fingerprints without apparent cause.

  5. chivo243 Silver badge

    Hands in Pockets!

    Anytime I visit a place like this as a visitor.... keep my hands in my pockets! No touchy nothing, nada!

    My coat has pockets!

    1. jake Silver badge

      Re: Hands in Pockets!

      Many moons ago I took my daughter to SLAC on take your kid to work day. At the ripe old age of 9, she had been there many times before and knew the ropes, but I figured she deserved a day out of school.

      She told me as we were walking in that it'd cost me ten bucks for her to not push any buttons. I gave her the money.

      On the way back out, I told her that it'd cost her ten bucks for me not to tell her mother she was running a protection racket. She made a face and paid up ... and promptly told her mother as soon as we got home. They both still laugh about it :-)

      1. ShadowSystems

        At Jake...

        With skills like that, your daughter will rule the world someday. Please put in a good word for me to be fed to the sharks last, ok?

        I, for one, approve of our new Evil Overlordess. =-D

      2. Eclectic Man Silver badge

        Re: Hands in Pockets!

        Jake: "Many moons ago I took my daughter to SLAC on take your kid to work day. "

        Oooh, Oooh, can I come next year, please, pretty please?

        I promise I'll be good, and not press any buttons, except in the lift or on the coffee machine.

    2. Arthur the cat Silver badge

      Re: Hands in Pockets!

      Anytime I visit a place like this as a visitor.... keep my hands in my pockets!

      You're forgetting the surprisingly strong attractive force between Big Red Switches and glutei maximi.

    3. harmjschoonhoven

      Re: Hands in Pockets!

      My father taught me to never point at a machine with moving parts. Yes, I have an aunt with nine fingers.

      1. The Oncoming Scorn Silver badge

        Re: Hands in Pockets!

        Unless your family comes from Norwich (Icon) or related to the late great Dave Allen.

        Your aunt has 7 fingers (Webbing optional) & presumably still has two thumbs.

        1. harmjschoonhoven

          Re: Hands in Pockets!

          That same aunt later saved with her 7 fingers and two thumb a man's life

          who lost an arm in an industrial accident.

        2. vogon00

          Re: Hands in Pockets!

          "Unless your family comes from Norwich"*

          I've said this before (if you can be bothered to look), but I fail to see why people pick on Norwich or Norfolk when there is a much better target just south of us...

          While we are on the subject, 'N-F-N' is usually known as 'Normal For Norfolk' and allegedly (?) stems from use in the patient notes/records of some medical doctors. I prefer the one disclosed to me by a local GP...'PRATFO', which was used for patients with hypochondriac tenancies - 'Patient Re-assured And Told to Fuck Off'.

          I bet senhor Scorn hails from Scunthorpe Suffolk :-)

          1. Eclectic Man Silver badge

            Re: Hands in Pockets! - N-F-N

            At least it wasn't T F Bundy (Totally F***ed But Unfortunately Not Dead Yet).

            After the Data Protection Act in the UK gave people the right to see their medical records, these 'idiosyncrasies' have somewhat gone out of fashion.

      2. uccsoundman

        Re: Hands in Pockets!

        I had a father-in-law with 9 fingers. It seems he was once a soldier, drunk on leave outside Ft. Knox, and somebody at a local bar pointed a gun at him. He somehow decided that sticking his finger in the barrel would keep him from harm.

  6. Giles C Silver badge

    Hmm well I remember a few years ago that as you left a comms room there were two buttons by the door.

    The top button released the lock on the door, the bottom one was a big red button.

    Yes someone (not me) pushed the wrong button to open the door.

    After that the red button got a cover that had to be hit hard (punched) before the red button would operate.

    1. Anonymous Coward
      Anonymous Coward

      A&E light switch

      Was in A&E at the weekend for a few hours. We were all being kept amused by people trying to leave. Sign on door said "press button on wall to open". Which meant the waiting room kept being thrown into darkness by the wrong button pressed...

      One button was small and unlabelled and looked like a light switch. The other was a large six inch square metal button with Door Open on it in large green letters. Easy mistake to make...

      1. My-Handle Silver badge

        Re: A&E light switch

        On the subject of poor automatic door design, I have frequented several shops / banks etc that have upgraded their doors to operate by push-button. I have no problem with this.

        The problem I have is that a lot of the doors open at the speed of an arthritic snail, they can't be opened manually, and that it isn't immediately obvious that the doors are automatic at all. So you have a consistent flow of able-bodied people ploughing into the door at pushing force. I've seen doors that either open automatically and quickly, or can be both pushed open and opened via button. This makes the above design just baffling to me.

        1. Gene Cash Silver badge

          Re: A&E light switch

          "they can't be opened manually" - how does that get past elfin sarfty? On this side of the pond, the Fire Marshal would be gleefully writing up a violation notice.

        2. adam 40 Silver badge

          Re: A&E light switch

          If you have to use a button IT IS NOT AUTOMATIC.

          So if the sign says "AUTOMATIC DOORS" I expect a PIR device or similar to open them for me.

          Too many doors have the sign, but manual button-pressing is required.

        3. Martin an gof Silver badge

          Re: A&E light switch

          Having to look after several doors with this kind of mechanism, all I'll say is that they are often mechanically dubious. Speed can lead to premature failure. However, all the ones I've met do have sensors which can tell when they are being pushed and open as if you have pushed the button, or simply disengage the motor so that it's easier to open "manually". One of the doors at work is so sensitive that if a gust of wind causes it to blow open just a little, it then proceeds to open the whole way.

          Many also have infra red sensors to tell if someone's blocking the door. and may refuse to open (or close) if they detect an obstacle.

          What they all also have is some way of programming them - so sometimes even the ones which *can* open on a push have been set not to do that by the owner, which is a bit of a pain if the 9V battery in the push button (which invariably is connected by radio, not a wire) runs out.

          Sometimes these controls are brought down next to the door in the form of a keyswitch, so you can tell what has been set. This is often the case where the operating mode is changed during the day - for example so that people cannot come *in* to the shop ten minutes before closing, but can get out.


      2. Anonymous Coward
        Anonymous Coward

        Re: A&E light switch

        Button "noun - a small device on a piece of electrical or electronic equipment which is pressed to operate it".

        Six inches square is not small, and not a button.

        "Press pad on wall to open" may have been more helpful!

        1. Anonymous Coward
          Anonymous Coward

          Re: A&E light switch

          > "Six inches square is not small, and not a button."

          As the anon who posted the A&E story, I'll also argue that a lightswitch is also not a button but a switch.

          A big metal thing with Door Open written on it is more obvious. Big metal thing presses small hidden button.

          To be fair to the hospital, I cannot remember exact wording of the message due to being distracted by my finger falling off at the time. And sign on door was not big enough to include full dictionary definition of button.

          > "Press pad on wall to open" may have been more helpful!

          Telling them to do that would have the pressing the poster thinking it was a writing pad.

          1. Andy Non Silver badge

            Re: A&E light switch

            "my finger falling off at the time"

            So the door got you anyway. ;-)

      3. Already?

        Re: A&E light switch

        Trouble is that doors with electrical locks in that situation are an invitation for that sort of mistake.

        For a start you're in a hospital which by definition is a place you wouldn't be at unless somethings bad has happened. You're either relieved to be fixed or stressed, if you have kids with you that's additional stress, and the last thing you want is a door that doesn't just open but also has an instruction just of out of eyeline and a choice of switches probably a bit further out of eyeline. And you've already made yourself look a bit of a numpty by walking into a door that didn't open so you're even more enthusiastic now about just getting out. You scan the environment for clues, see the words Switch, Exit and Press and hit the first one you see.

        The people who fit these things aren't ergonomics experts or sociology graduates, they're electrical fitters who have a practical plan and approach to doing it. As everyone knows (on here at least ) if there's more than one option and one of them has an undesirable outcome, it's better to make the desirable outcome the default that would be hard to get wrong. Don't rely on people in unfamiliar situations having the time or the wherewithal to weigh up all the options and to carefully consider the various options and outcomes before executing the default or - to their eyes - the easiest action. Sod's Law still applies.

        1. Anonymous Coward
          Anonymous Coward

          Re: A&E light switch

          > For a start you're in a hospital

          This. The only place I've been to that has more notices per square foot of wall space is my GP's surgery. :-( There are notices everywhere: when I had to go to the hospital for some tests a couple of years back I had to literally stand still at the entrance to the ante room and scan my eyes over dozens of notices before I found the one explaining how to let them know you had arrived (there was no one at the desk).

        2. A.P. Veening Silver badge

          Re: A&E light switch

          For a start you're in a hospital which by definition is a place you wouldn't be at unless somethings bad has happened.

          Some people wouldn't qualify having a job there as bad.

          1. John Brown (no body) Silver badge

            Re: A&E light switch

            Those who work there probably know how the doors work :-)

            1. irrelevant

              Re: A&E light switch

              The senior consultant I saw walk up to an external door with a prominent bright red sign "emergency exit only - door is alarmed" and push it open to the sound of a very loud buzzer.

              To be fair, I heard her on the phone to security shortly afterwards apologising; she'd just arrived back from leave and prior to her going away it had been an allowed easy way out to the car park.. I can only assume that familiarity made her not notice the new signs.

          2. Robert Carnegie Silver badge

            Re: A&E light switch

            If bad things didn't happen, there wouldn't be hospitals. Technically.

            For instance, if that guy you didn't vote for gets a life threatening disease, is that a bad thing - technically.

      4. John Brown (no body) Silver badge

        Re: A&E light switch

        "One button was small and unlabelled and looked like a light switch. The other was a large six inch square metal button with Door Open on it in large green letters. Easy mistake to make..."

        For anyone who's never worked in a place with exit buttons by the doors, yes, it can be an easy mistake to make. Sometimes, when you see something you've never seen before, and at the same time see something you recognise as familiar, both of which could fit the description of what you are looking for, the human brain can simply blank out the unfamiliar thing.

      5. Sam not the Viking

        Re: A&E light switch

        Our offices had two buildings separated by a longish yard. For obvious security reasons, each pedestrian entrance had a keypad for entry; the keypad operating an electrically-operated latch. We were not daft, each lock had a by-pass key located in the other building in case of a fault.

        A general power cut happened on the industrial estate (I was not to blame, but I was considered a major suspect), and for some linked reason the fire alarms went off at the same time so both buildings were evacuated. Obviously, the keypads no longer worked. With both access keys inaccessible, everyone was locked out and had to stay that way until the 'tamper-proof' screws were undone by a pair of pliers someone 'happened to have on their person'.

        The IT system shut down correctly on its UPS. No data was harmed in this tale.

      6. Emir Al Weeq

        Re: A&E light switch

        It wasn't me but, having worked in an office whose door release buttons looked exactly like light switches, my Pavlovian training would have me immediately pressing such a switch if I spotted it before the Big Metal Button.

        Given your reason for visiting A&E, the irony of your story is not lost on me, but I'll refrain from any wisecracks and wish you a speedy digital recovery.

    2. Korev Silver badge

      My A level college had a climbing wall with a fire exit in the middle of it. Next to it was the fire alarm button which the PhBs didn't want obstructed in case of a fire so there was no cage fitted.. This was fine right up until one of the teachers found the alarm button was in the "right" place for her foot during a tricky move...

      1. Kubla Cant Silver badge

        a climbing wall with a fire exit in the middle of it

        I know what you mean, but I can't get rid of a mental picture of a bright red fire door ten feet off the ground. In case of fire, you have 60 seconds to teach yourself rock-climbing.

        1. Gene Cash Silver badge

          In case of fire, you have 60 seconds to teach yourself rock-climbing.

          "Blimey! I've never seen more-motivated students!"

          1. Eclectic Man Silver badge

            The very old climbing wall at my not so local sports centre was basically a corridor with holds on either side and a fire exit at one end. The fire brigade closed it down as a climbing wall with the not entirely unreasonable attitude that it should not be obstructed by ropes or other climbing paraphernalia. So I'm surprised that your climbing wall was allowed to exist with a fire escape in the middle.

            1. Korev Silver badge

              > So I'm surprised that your climbing wall was allowed to exist with a fire escape in the middle.

              This was in a sports hall big enough for multiple tennis courts etc.

              1. John Brown (no body) Silver badge

                ...And they couldn't find a blank bit of wall to use in a space that size without building it over one of the fire exits?

    3. Xalran

      Strange I have almost the same story... a stack of buttons by the door: one open the door, one turns off the lights, one is the Big Red Button, one turns on/off the intrusion alarm, and one is the fire alarm (and the tied inert gas release)

      I don't rememberr the exact orderof the stack, I visited the site once during the day and you had to be extra careful pressing the correct button to get out... but my colleague that ended up exiting the datacenter at 4AM, even being extra careful ended up pressing the wrong button. Luckily just The Big Red Button was pressed...The consequences were bad enough.

      The only good thing was that datacenter was within walking distance of the office so when the inevitable FUBARED hardware happened we had all the spares needed 10 minutes ( walking ) away.

  7. Steve Kerr

    Sound of silence

    At the beginning of each shift, there was a meeting to discuss what had happened and anything special for that shift, at the end of the meeting, everyone traipsed to the bridge in the computer room.

    Access was via a card held near the wall at a certain point and an electromagnet held the door in place.

    The door magnet was a bit weak so with a bit of a shove it was possible to open the door without the door being unlocked (there were manned desks in the next compartment so anybody coming in was seen).

    Did I mention the emergency shutdown button on the wall behind covered with its nice plastic cover and the door stop that was next to it.

    Well, as "someone" (not me) shoved the door, the door release was activated at that exact point by the card in their pocket, door acclerates into the door stop which promptly gets pushed into the wall behind it as it was only a weak partition wall, shattered the plastic cover and hit the emergency button.


    Off went the IBM mainframes, the ICL's, the VAX's, the AS/400, the HP's, the stratus non-stop (they stopped!), the System/38's and whatever other collection of stuff was in the room - all in the process of their overnight batch runs.

    The only saving grace was it was going into a night shift - the evening shift were a little unhappy about having to stay for a couple of hours longer as everyone rushed to get it all up and running and restart batch runs at their checkpoints and other stuff.

    The emergency shutdown was moved off of the wall behind the door and the door magnets fixed.

    Fun times!

    1. Terry 6 Silver badge

      Re: Sound of silence

      I've heard so many of these stories, and actually seen a few stupidly placed big red buttons.

      So the thought is readily prompted..... Someone got paid real money for that decision.

      1. John Brown (no body) Silver badge

        Re: Sound of silence

        "Someone got paid real money for that decision. "

        Yeah, the sparky got told to fit the button near the door. That was probably the entirety of the plan. So the sparky, with possibly no idea what the button did or the consequences of an accidental operation of said button, was the most likely person to have made the decision. That decision in turn was probably influenced mostly what whatever building code was in force at the time that the sparky was following.

        1. The Oncoming Scorn Silver badge

          Re: Sound of silence

          I've told this story a few times here, but needs a retelling (& retyping alas):

          So it came to pass that a remote Admin (From CalTech (Calcutta Technical college)) who had taken over the on-site network & general admins role) was tasked with an operation, but still needed the on-site admin (Now moved to a new position of site security & physical network operations) to act as smart hands.

          On-site guy takes lots of photos of the server room for the remote guy who spots one of several red button's, pours over the other schematics, then demands for whatever reason that specific one be pressed.

          On-site counters that this is the plant shutdown big red button & not the server room shutdown, it will stop everything & no he's not going to press it, brings in the site manager into the conversation to boot.

          Thus it was agreed upon that they would press it if remote guy's boss would sign off on it with a statement that it was absolutely vital this button be pushed after informing them again of the buttons purpose & at a mutually agreed time of their choosing (Shift handover, damage limitation & preparing for the inevitable).

          So the time came, management sign-offs & responsibilities printed off & stored, Site manager & on-site guy stand ready along with the maintenance team......

          The exchange is paraphrased (Feel free to read it in the required accents).

          Please be pushing the button now.

          Are you sure you want me to press the button?

          Yes I am very ready please be pushing the button now!


          [CLICK] Sudden but expected silence descends in the server room & everywhere else.

          Ohhh goodness gracious me....this is very bad..why is your site no longer on the network.

          You were told repeatedly that button is the plant shutdown, not the emergency server shutdown, you insisted & persisted despite repeated being told so by myself & the site manager to you & your manager & you both signed off on it.

          Now we have to send the evening shift home fully paid & get the maintenance teams to bring the plant back to operational status through the night & I have to bring up the servers so everything is back up by 6am start for the next shift & the site managers report will include the fact you identified the wrong button & were frequently advised of your error. Excuse me but I have work to do now, whether you do is another matter....

          If I recall correctly, the plant was ready for operations at 2am or 15 minutes shy of the 6am shifts start.

          Though the latter start time may be the story of the manager who plugged in a USB stick with the ILOVEYOU virus (Helpfully provided by way of his daughter & her homework) into his computer & brought global operations to a screaming halt at 8.12 local time at the same site.

  8. PM from Hell
    Paris Hilton

    Radio Keys

    One DC I worked in had radio keys to control access to the DC. These, we about the size of a small mobile phone today and were generally kept in your trouser pocket all day if you required regular access to the DC. The Keys required charging every night so had to be locked into the charging station before you left for the day. It was a major pain if you took one home overnight and fessed up a lot of fuss was made. In reality if you didn't use it much it would just about last a second day. There was zero lag on the system so we got used to running through the door if there was an emergency (this was in the early 80's so they were horribly common) one of the ops found out that his key was dead when running into the DC. The door didn't open but he carried on, that's when he found out that the glass panel above the door wasn't shatter proof. He was lucky to get away with some deep lacerations to his chest. A post incident review lead to the glass being replaced with shatterproof wired glass and a relaxation in the rules around using a 'spare' key, whilst usage had to be logged and the users original key had top be present being charged it was no longer a disciplinary offence taking a key home by accident.

    The operator was a resilient chap and actually was pretty happy a couple of weeks later, he'd been booked to go to Spain for 2 weeks and his holiday insurance company refunded the money as he could t take part in the water-skying and diving he had planned so he got a free holiday (lounging by the pool drinking as usual). His holiday was taken as sick leave and he got bunged a few thousand pounds in compensation for his injuries.

    Paris as he apparently got a lot of attention from girls who were very sympathetic about his bandages.

  9. ColinPa

    put your hands in your pockets and leave them there

    On a trip to support a very large organisation (where they had men with machine guns on the outer perimeter, all bags were x-rayed, and we were body searched on entry), we had to go into the inner sanctum operations room. We were told by my company "put your hands in your pockets and do not touch ANYTHING".

    We could not touch a (dumb) screen we had to stay 2 inches from it. This means pointing to the error was a challenge.

    Someone had been caught on video, pressing the "next" button to display more error diagnostics. This caused a huge stink, involving senior managers having to apologise.

    As we kept our hands in our pockets we could not get out of the room unaccompanied. Even in the "unsecure" areas we were not allowed to go to the toilet unaccompanied, but at 0300 in the morning no one was too worried.

  10. Anonymous Coward
    Anonymous Coward

    The magic loo-roll and your pockets are too big.

    Two "red button" issues.

    The big red button was at the end of a row of tape racks. One balmy after while the junior was putting away the tapes, they inadvertently bumped into the unprotected BRB - silence issued. The ICL's the PICK, and other assorted equipment goes silent. Rather than the flashy plastic, up-market flip cases all the "rich" computer rooms above seemed to go for, we put a trusty loo-roll centre over the button - this worked.

    The best story, a hapless (and rather senior) tech support person was in the computer room having a chat, and walked past the ICL 2966 - WOPR (sadly not as advanced and blinkly lights as the War Games one) - just a monitor and keyboard station on a stand. And caught the reset switch with his pocket. Queue the requisite running around to get the borough back up and running so people could pay the rents and rates. The fix for this, was to remove the reset switch - that just slide out and leave it on the top so it could be popped back in when required.

  11. Anonymous Coward
    Anonymous Coward

    Not quite the "red" button

    We have a small data centre hosted about 1/2 mile away from the office, but still on site - not quite a red button setting, but the dedicated building is split into 2 parts - ~75% machine room, and 25% maintenance equipment, air con power/controls etc.

    Had some members of the estates and buildings team in doing some work who decided to save power by switching off some buttons on the way out. Including the air con. Didn't take long for the temperature alarms to trip.

    The members of the server team started grabbing fans and getting up to the room to try and keep the equipment cool until the cooling systems could be restarted and the room back to normal temperatures.

  12. Pen-y-gors Silver badge

    I may have mentioned this one before...

    Back in the days of big iron, a friend worked for one of the utility companies. They too had problems with bosses showing people around and showing off their shiny.

    Problem was that sometimes they'd turn up during a monthly billing run, when the box was just crunching its way through calculating thousands of bills and spooling the output. As a result this was a good time for the Ops to sit back, drink coffee and read the paper, because no intervention was needed for hours. Basically the blinken lights blinked, a tape drive or two inched their way round, and that was it.

    Bosses not impressed with this apparent lack of activity when showing off.

    So the Ops bods had a chat with some developers and between them put together a program which, when loaded, suspended the current (useful) operations and made all the machinery work at once. The blinken lights went berserk, tapes rewound, things ejected, printer covers went up and down, things that go beep all beeped, and ops ran around looking busy. Very impressive. Once the grockles went they just shut it down, and got back to real work (and the crossword)

    They were the same team who worked out that printing certain combinations of characters on the line printer made the chain vibrate at a certain note. After a bit of experimentation they put together a card deck that would get the printer to play God Save the Queen.

    1. Roger Kynaston Silver badge

      Re: I may have mentioned this one before...

      That last paragraph deserves five upvotes on its own.

    2. Admiral Grace Hopper

      Re: I may have mentioned this one before...

      Management sprang a lot of cash on a huge flat panel display to show the systems dashboard to all sundry. This was very useful, as a flashing red graphic does catch the attention. Management then realised that they also caught the attention of visiting bigwigs, so someone *coff* wrote a script called green.bat that turned the dashboard to a more senior management friendly colour. The reset script was truth.bat

    3. Morrie Wyatt

      Re: I may have mentioned this one before...

      As I recall, Beneton did much the same with an F1 Renault engine.

    4. The commentard formerly known as Mister_C Silver badge

      Re: I may have mentioned this one before...

      -> Imperial March from Star Wars. On a 3D printer:

      ->Pirates of the Caribbean:

    5. jake Silver badge

      Re: I may have mentioned this one before...

      I was at a meeting of the Homebrew Computer Club in 1977ish when Steve Dompier demonstrated similar music making skills with an Altair 8800. It took him about 30 minutes of toggling switches to get it to play "Fool on the Hill" or "Bicycle Built for Two" in RF picked up on an AM radio. Someone watching (Roger Melen? There were several CROMEMCO folks there that day, if I remember correctly ...) was overheard to say that it was the most useful thing he'd ever seen a personal computer do. At that stage of the game he may have been right!

      I'm fairly certain that everyone witnessing this thought it was a computing first ... I found out later that the Ferranti Mark 1 had a function that would allow variable pitch operator feedback, and someone had programmed it to generate music in the very early 1950s[0]. A couple decades later I found out that the Australians had beaten us all to the punch, having programmed their CSIRAC to make music in 1949 or 1950[1].

      [0] "God Save The King", of course (among others).

      [1] "Colonel Bogey".

      1. Precordial thump

        Re: I may have mentioned this one before...

        Take the Williamstown line train to Spotswood, pleasant stroll down to Scienceworks if you want to see it in (simulated) action.

        1. jake Silver badge

          Re: I may have mentioned this one before...

          "Take the Williamstown line train to Spotswood"

          You can't get there from here.

          "stroll down to Scienceworks if you want to see it in (simulated) action."

          I could toggle them into my period Altair 8800 again, but somehow I think once is enough. I should have taken a video.

          If you have access to an actual Altair 8800 and want to try this for yourself, here is a PDF of Dompier's article in the May 1975 People's Computer Company newsletter, as reprinted in the February 1976 issue of Dr. Dobb's Journal of Computer Calisthenics & Orthodontia, including code for "Daisy, Daisy" and "Fool On The Hill".

          It would seem my memory of watching the demo in 1977 was a tad off. Mea culpa.

          1. Anonymous Custard Silver badge

            Re: I may have mentioned this one before...

            Not music, but reminds me of an undergrad lab I did student demonstrator work on when I was doing my masters. Undergrads were supposed to build a circuit with two waveform outputs, wired to a scope to display both simultaneously.

            Particular student managed to screw up both the circuit and the settings on the scope (had the two inputs driving X-Y mode rather than both on X-T mode), and rather than generating some rather boring sine-type waves, managed to generate an almost perfect silhouette of a penguin.

            Of course he had no idea of how he managed it, but it earned him is one and only A mark he ever got in that lab, and it was still talked about occasionally years later when the same lab lesson rolled around to the next batch of victims^H^H^H^H^H^H^H undergrads.

  13. a_yank_lurker Silver badge

    From the dark ages

    With your best German accent, from the dark ages:


    Das computermachine ist nicht fuer gefingerpoken und mitten grabben.

    Ist easy schnappen der springen werk, blowenfusen und poppencorken mit spitzensparken.

    Ist nicht fuer gewerken bei das dumpkopfen.

    Das rubbernecken sichtseeren keepen das cotten-pickenen hans in das pockets muss; relaxen und watchen das blinkenlichten.


    1. batfink Silver badge

      Re: From the dark ages

      Ah yes - I remember this being printed out lo-res on good old-fashioned stripey fanfold paper so it could be stuck on the ops room wall.

      The Good Old Days.

      Clearly both you and I are getting too old for this kind of thing...

    2. jake Silver badge

      Re: From the dark ages

      Pretty Much the definitive history of blinkinlights here.

      Here's a plaintext addrerss for those of you who quite sensibly refuse to simply click on links without having at least an inkling of where they might take you:

  14. Anonymous Coward
    Anonymous Coward

    Security shortcut

    I worked for a well known UK university for a long time. The buildings had RFID based access card control on external entrances/exits and also on internal doors for different areas. This of course meant you needed to press a big green button to release the magnetic lock to exit.

    One of the computer rooms on the site had internal access doors and access doors at the rear which led to a small loading dock. The rear exit was via two sets of doors linked via a short corridor.

    For whatever reason, a member of security had apparently gone through the computer room one night and having made it through the first set of doors at the rear, were now in the dark corridor feeling around for the door release button. It wasn't the door release button...

    1. Eclectic Man Silver badge

      Re: Security shortcut

      AC: "It wasn't the door release button..."

      Guessing it wasn't the light switch either?

      1. Anonymous Coward
        Anonymous Coward

        Re: Security shortcut

        You're right. It also wasn't the concierge button.

        Presumably, if one of those had existed on campus, it would have been located in the VCs office!

  15. StephenTompsett

    Door Bell?

    A major utility company used to have an 'Emergency Power Off' button outside each door exiting from the main machine room.

    The person delivering a pallet of equipment who had been directed to a door opening onto a corridor at the back of the room said: "I thought it was the door bell"...

  16. DS999 Silver badge

    The Big Red Button

    Should work like fire alarms in US schools and squirt you with a purple ink if pressed. While the perpetrator of a big red button press is almost always known, that would be an extra penalty on any high level management type who was dumb enough to do it as it would ruin his suit and tie.

    Bonus points if it stains your skin for weeks so he'd have to walk around with that badge of shame everywhere he went at work and everywhere else!

    1. jake Silver badge

      Re: The Big Red Button

      "fire alarms in US schools and squirt you with a purple ink"

      They do? I've never seen this. Cite?

      1. DS999 Silver badge

        Re: The Big Red Button

        When I was in elementary school a kid pulled the fire alarm. He had a purple stain on his hand for about a week.

        I'm sure not all schools had this, and maybe it is less necessary now with cameras everywhere, but I think it was pretty common as I know people who went to school in other states had the same setup for their fire alarms.

        1. jake Silver badge

          Re: The Big Red Button

          Some establishments put a gel (called a tamper dye) on the top of the fire pulls. Activating the pull gets the dye on the finger tips. Attempting to wash it off spreads the dye and turns it a rather fetching shade of blue, not purple ... the myth always says "purple spray", but I've only seen topical blue tamper dye. There is no squirter on any fire pull that I am aware of, and I've spent a good deal of my life mucking about in the wiring of schools, new and old. I've been asked to apply the dye to new kit on several occasions, but I've always declined, citing allergies[0].

          [0] 'tis true, kinda ... I'm allergic to setting booby traps. Totally uncivilized, IMO.

    2. David Hicklin

      Re: The Big Red Button

      UK College in the 1980's, we were all stood around the corridor bored waiting for the lecturer , friend leaning on wall next to fire alarm...

      no need to guess what happened.........

    3. Robert Carnegie Silver badge

      Re: The Big Red Button

      I think I've worked out that "The Stopper", used in the U.S., isn't an unbreakable shield fitted around a fire alarm box to make it impossible to use it, although it sort of is that, but actually it's a cover which you can open easily... but a loud alarm sounds straight away, so people know you've done that. It's important to understand that that is not the fire alarm, and you still need to do the fire alarm, if you were planning to.

      There also are pictures of an alleged patented fire alarm box from about a hundred years ago that when you sound it, it clamps down on your hand, so that you will be found trapped there later. After the fire for instance. That's not a good idea, is it!

  17. Brian 3

    I saw someone do this with a fire suppression system in a restraunt kitchen once - right after he finished cleaning the whole place. He pulls the knob off - nothing - then proudly goes to show how the chain it's connected to needs a bit more pulling - DING DING DING - cue the glycol nozzles!!

    1. The Oncoming Scorn Silver badge
      Paris Hilton

      Hell Man

      He pulls the knob off.

      I'd be wary of trying anything with mayonnaise from his kitchen.

      Icon - Baby Batter.

  18. Sparkus Bronze badge

    Same exact story

    bigwig, showing off, etc, etc.......

    Put a Full Stop to the tours though.

  19. Sam Therapy

    I think someone's found the big red button at the Guardian's server centre. They just flatlined a few minutes ago.

  20. Muscleguy Silver badge

    I have a real one

    At work I have a big red button I can push. I’m a school science tech and to get gas to the taps and use a bunsen I insert a key into a box of electronics by the door and turn it. A couple of minutes later gas begins to flow (after pressure testing the system).

    It also has a big red button which instantly cuts off gas supply.

    I’m new to the job, the gas pipe in the fume hood was missing. After looking at one in a hood in a classroom I spotted it doing service as a spout on a water tap. I juggled things around and had it back where it belonged with a different proper water spout on that tap.

    The tap needs a spout on it so I can put the inlet hose of the water still on it. Fiendishly clever design these water stills, there is no way to distill anything other than water in them. Same source does cooling water first which then tops up the boil but right by the outlet pipe . . .

    1. jake Silver badge

      Re: I have a real one

      "there is no way to distill anything other than water"

      Fortunately glass tubing is easy to work with a bunsen burner ... how else does one make proper glass cleaner?

  21. Robert Carnegie Silver badge

    In this story the manager looks the IT crew in the eye and presses the "shut down the company" button.

    What you should do is, without speaking, or looking at each other, all just walk unhurriedly out of the room, the building, across the car park, keep going...

    Aside from the correct effect on your stupid management, when you get far enough away, you can scream loud and long, and THEN go back in and fix it.

  22. LordHighFixer

    Molly guard


    molly-guard (plural molly-guards)

    A physical barrier to protect something from unwanted contact, especially a shield to prevent accidental tripping of an emergency shutdown or power switch.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021