back to article Wham, bam, thank you scram button: Now we have to go all MacGyver on the server room

Monday has arrived, and with it another tale to send administrators scurrying for their event logs, and engineers reaching for the coat hanger. Yes, it's Who, Me? "Jon", for that is not his name, was working for a now long-defunct IT giant back at the end of the 1990s. Fortunately for those depending on the big iron for …

  1. Chris Miller

    Many computer rooms suffered similar accidents. Before the days of online systems, often the ops were the only ones who were aware of them. The simple solution was a cover for the big red power button. But the bigger danger was the fire alarm that dumped halon into the room - which took more time to recover from.

    1. BebopWeBop
      Holmes

      I am given to understand that some of the BOFH's victims (sorry righful targets) might never have done so.

    2. Venerable and Fragrant Wind of Change

      Isn't molly-guard the usual term for such protection, when this story appears in this column?

      The trains have had "in emergency break glass" hammers for as long as I can remember, but they're well-protected against accidental access.

      I think my personal nearest to this anecdote is leaning on the wall in an overcrowded conference room.

  2. Fabrizio

    world-wide DC access

    When I left a certain company, back in the beginning of the new millennium, HR had a list of company property they wanted me to return and when they were done with their list I asked them to add a single item to the list:

    "Data centre access badges and keys". Just 3 access badges and 7 (or 8) keys opened all global data centres 24/7 and about 95% of all racks.

    I got a phone call 2 weeks later by the VP of IT how the hell I amassed so many... Told him: solve lots of critical issues for more than 10 years and people will just tell you: "Keep them for next time you have to perform magic"

    1. qwertyuiop
      WTF?

      Re: world-wide DC access

      In one of my jobs I was home-based, although I spent most of each week travelling to offices around the UK. The company did me proud, setting me up with a decent laptop, docking station, two monitors and a decent printer for my office at home.

      On the day I left the organisation I duly returned all of my kit to the desktop support team. Their manager asked why I'd bothered to lug two monitors, a docking station and a printer 100 miles up the motorway to return them. I reminded him that they were company property. He told me that, with the exception of my laptop and mobile phone, he had no record of what kit I'd been issued so I could just have kept it!

      When I pointed out that I was the company's InfoSec manager....

      1. BebopWeBop

        Re: world-wide DC access

        I see a missed opportunity.....

        1. qwertyuiop

          Re: world-wide DC access

          Yeah, so can I. But then I weighed the advantages - £500 worth of kit (secondhand value) if nobody realised - against the disadvantages that my prospects of continued work as an InfoSec manager would suddenly become vanishingly small.

  3. This post has been deleted by its author

  4. ColinPa

    Dont have your machine room at the top of a building

    I was at a IT conference, and we were discussing horror stories. As it was in the bar, some of the details are hazy. One guy said that someone spotted smoke coming from a display, pressed the scram button, there was power down, and the halogen system was activated.

    To reactivate the halogen system, they could not get the kit up in a lift, so they had to take out an exterior wall, and have a huge crane lift in some kit (Halogen tank?) to the 13 floor, rebuild the wall, and "clean" the machine room.

    There was a 4 month outage all because some insulation on a screen was starting to smoke.

    Not all details are accurate, but I hope you get the idea.

    1. Chloe Cresswell Silver badge

      Re: Dont have your machine room at the top of a building

      Makes me think when I was told about cam.ac.uk having a halon dump on a false alarm.

      That's when they found there was plumbers grease in the halon pipework. Well, there was. It was atomised and deposited all over and in their new mainframe.

      1. Vulch

        Re: Dont have your machine room at the top of a building

        In the early 80s the computer centre of what's now University of Bangor was moving from its home in the old MANWEB maintenance base to a shiny new purpose based building with lots of equally shiny new kit. Commissioning started with the aircon being fired up for the first time which proved its effectiveness by coating everything with a fine layer of cement dust from the open bag that had been left in a duct by the builders. I'd graduated by then, but several friends landed summer holiday jobs cleaning it out again.

        1. Sequin

          Re: Dont have your machine room at the top of a building

          I worked in a government department in Bootle, just north of Liverpool. When the building was being put up (you can see it coming down here https://www.youtube.com/watch?v=z3i2baydpCM) it was involved in the longest strike in British history, and the shell was open to the elements for about 5 years.

          After the strike ended, they started fitting out the building, putting in windows etc, but found that hundreds of local feral cats had moved in in the interim. Somebody decided it would be impossible to catch all of them, so they decided to seal up the building and pump it full of poison gas.

          It proved impossible to find all of the corpses , so they just left them to decompose. We used to come in every day to find our desks and computers covered in dust and gritty stuff - the management tried to insist that it was just dust, but we all knew it was mummified cats!

          1. TheFurryCircle

            Re: Dont have your machine room at the top of a building

            What actually happened is the cats all survived, and by their very nature as cats had to win against mere humans, and proceeded through several generations to dump asbestos through the vents as revenge. Should have just let them colonise the building, it's safer in the long run.

          2. Andrew Norton

            Re: Dont have your machine room at the top of a building

            That wasn't the HSE building was it?

            Used to do a bit of work there, and had some friends that worked there 20 years ago, and remember something about 'dusty desks'

        2. vir

          Re: Dont have your machine room at the top of a building

          We had central air in one of the houses I lived in, with a duct right over my bed. One night in the early summer, when we first turned on the A/C, I was reading a book in bed and noticed a tiny spider on the page. I looked over and the bed was coated with them; there was a nest in the duct that got blown out into my room. I'll take cement dust any day.

      2. Stevie

        Re: Dont have your machine room at the top of a building

        "Plumbers grease?"

        When I go a-plumbing I carry a tin box full of stuff including a blowtorch, squeezy flint lighter thingy, solder, emery cloth, wire brushes, a bunch of copper pip and fittings, a tin of flux and a brush for same in a little jar.

        There is no grease. I'm not sure in what capacity it would ever be called for. You certainly wouldn't want it contaminating any lines you had just carefully soldered together.

        1. DubyaG
          FAIL

          Re: Dont have your machine room at the top of a building

          Plumber's grease is used to lubricate various rubber parts like O-rings and seals in things like water filters and valves. There must have been some valves in the system that probably had tags on them that said, "Do not lubricate" and were duly ignored.

    2. Adrian Harvey
      Flame

      Re: Dont have your machine room at the top of a building

      I seriously hope it was Halon, not Halogen. Halogens (Flourine, Chlorine, Iodene, etc) are highly reactive and would be an “interesting” choice for a fire control system. Good for the BOFH though, very effective for wrapping up stories.

      Icon may well appear on Halogen cylinders.

      1. Anonymous Coward
        Anonymous Coward

        Re: Dont have your machine room at the top of a building

        You're not kidding. Iodine is solid, Bromine is liquid, so the only way you're going to have a cylinder of halogen is if you're dealing with Chlorine or Fluorine. Chlorine has been used as a war gas - it's nasty stuff. Fluorine isn't used as a war gas as it's far too dangerous to be chucked around like that...

        1. DJO Silver badge

          Re: Dont have your machine room at the top of a building

          Bromine is fun being one of the few elements that is liquid at room temperature.

          When I was a nipper I knocked over a bottle of it but luckily it was in a fume cupboard so no problem, the fact that the fume cupboard discharged over the lower school playground and bromine vapour is a lot heavier than air really didn't concern us in the slightest.

        2. Stoneshop
          Mushroom

          Re: Dont have your machine room at the top of a building

          Fluorine isn't used as a war gas as it's far too dangerous to be chucked around like that...

          That didn't stop the Germans trying out ClF3 in flamethrowers as it's hypergolic (also with asbestos, sand and test engineers) so you don't have to have an ignition source, but they actually considered it too nasty to work with. Which must have been quite a high bar to clear, considering what they used for powering the Me163.

          1. Terje

            Re: Dont have your machine room at the top of a building

            80% hydrogen peroxide (the oxidizer used in the Me163) is a LOT safer and easier to handle compared to ClF3 and there is no way to extinguish it if it starts to burn, and then we have not even started on the lovely combustion products you are likely to get.

            1. Alan Brown Silver badge

              Re: Dont have your machine room at the top of a building

              80% peroxide..... that reminds me.

              There was an incident on the M25 some years back involving a shipment of 85-90% peroxide on a lorry full of other stuff. Some might remember it for the reports of blue buckets strewn across the motorway.

              http://news.bbc.co.uk/1/hi/england/london/4197500.stm

              The peroxide in question was jetpack fuel...

            2. Stoneshop
              Mushroom

              Re: Dont have your machine room at the top of a building

              Still, both components are quite toxic and prone to cause explosions which is something you'd like to avoid if there's a test pilot with their bum more or less on top of 1500 liters of T-Stoff (the 80% peroxide), with 500 liters of C-Stoff (hydrazine hydrate plus methanol) uncomfortably close.

          2. Agincourt and Crecy!
            Mushroom

            Re: Dont have your machine room at the top of a building

            ClF3 is very nasty stuff but it does have a much nastier relative, dioxygen difluoride O2F2, also known as FOOF, that stuff will even cause ice to ignite.

            1. Anonymous Coward
              Anonymous Coward

              Re: Dont have your machine room at the top of a building

              Derek Lowe's article on FOOF is quite amusing, as usual. Quote from the paper he was quoting:

              "When a drop of liquid 02F2 was added to liquid methane, cooled at 90°K., a white flame was produced instantaneously, which turned green upon further burning. When 0.2 (mL) of liquid 02F2 was added to 0.5 (mL) of liquid CH4 at 90°K., a violent explosion occurred.”

              Note: 90 K is -183 °C / -298 °F.

              https://blogs.sciencemag.org/pipeline/archives/2010/02/23/things_i_wont_work_with_dioxygen_difluoride

              Of course, the aforementioned ClF3 is ugly, too: (hypergolic = autoigniting)

              "It is also hypergolic with such things as cloth, wood, and test engineers, not to mention asbestos, sand, and water-with which it reacts explosively."

              https://blogs.sciencemag.org/pipeline/archives/2008/02/26/sand_wont_save_you_this_time

    3. Anonymous Coward
      Anonymous Coward

      Re: Dont have your machine room at the top of a building

      Also don't have it under toilets or sewage pipes.

      It's not great when a pipe gets blocked on christmas eve and literally **** rains into your server room for 4 days whilst they try to fix it.

      1. Anonymous Coward
        Anonymous Coward

        Re: Dont have your machine room at the top of a building

        Or a building with a flat roof prone to leaking.

        Queue bins with plastic bags sat on top of equipment to keep the drips off!

        1. J.G.Harston Silver badge

          Re: Dont have your machine room at the top of a building

          A queue of bins? Where were they going?

          1. Hero Protagonist

            Re: Dont have your machine room at the top of a building

            “A queue of bins? Where were they going?”

            Waiting to be emptied, Shirley

        2. stiine Silver badge
          Unhappy

          Re: Dont have your machine room at the top of a building

          Or you can be enterprising and use the plastic clamshell shipping containers to create a roof over the switch chassis that you located...under the trap in the hvac system above...

          FYI, just because the ductwork is wrapped in insulation doesn't mean there isn't a hidden drain plug just above the optimal location for your expensive hardware.

      2. TeeCee Gold badge

        Re: Dont have your machine room at the top of a building

        The water tank for the building's fire suppression system is bad enough. When a seam splits on that you really know about it.

        Also there's nothing like claiming for flood damage on the 1st floor to make an insurance company start asking difficult questions.

        1. The Oncoming Scorn Silver badge
          Alert

          Re: Dont have your machine room at the top of a building

          Especially when it happens at a branch of Norwich Union......

          One of my earliest contracting jobs & the managers moved printers around to try & get their documents out. (Didn't work, different subnets & IP's). Patching cables & inputting new IP addresses wasn't what I was there for, but by golly its what I did (for the most part) for 2 days.

          I think they wanted me back two weeks later to do what I had been booked to do\take over the role on a more regular basis, but by then I had a longer contract elsewhere.

      3. Nunyabiznes

        Re: Dont have your machine room at the top of a building

        Or the basement of a building in a flood plain.

        Cue IT staff (myself included) lugging sandbags and bilge pumps.

        1. Jakester

          Re: Dont have your machine room at the top of a building

          I worked at a bank where the server room was on the bottom floor and the top floor (single-floor building with a flat roof). Unfortunately, roofers were not able to fix leaks in the roof for more than 2 years (only rains a few times a year). We had to have plastic sheeting over all equipment, supply shelves and desks when it rained, which made working and equipment ventilation difficult during periods of rain or melting snow.

      4. Alan Brown Silver badge

        Re: Dont have your machine room at the top of a building

        Not being under the main water tank for the building is also good advice.

        And CHECK the build if you can, because even if the original spec is NO WATER PIPES IN THIS AREA WHATSOEVER, you may find the architect has ignored this requirement and every single piece of plumbing passes through the ceiling space of your data store (This happened. The people concerned "no longer work for the company", but that didn't get rid of the unwanted plumbing)

        1. jake Silver badge

          Re: Dont have your machine room at the top of a building

          "even if the original spec is NO WATER PIPES IN THIS AREA WHATSOEVER"

          ... That's when The Boss decides she needs an Executive Powder Room, and the only place the lowest bidder can figure out to run the fresh, grey and black water pipes is through the ceiling of the data center. Naturally, they all leaked at various times in the ensuing decade. Fortunately, my warranty was made null and void by this bone-headed mistake.

    4. Alan Brown Silver badge

      Re: Dont have your machine room at the top of a building

      "There was a 4 month outage all because some insulation on a screen was starting to smoke."

      Yup. there's got to be some level of "sense" about when you want the gas system to fire.

      Inergen systems are reputed to be about $5k a bottle, and that's apart from the other costs(*) - I tend to get quite paranoid about loose crap under the floor - especially drilling debris and insist that all areas worked on are thoroughly vacuumed afterwards. I've had contractors regularly gripe about me being anally retentive and how noone will ever see it, why does it matter, etc

      (*) I think we've all seen the "don't shout at your drives" video

  5. Korev Silver badge
    Windows

    Some former colleagues had the opposite problem; they were setting up the PKI infrastructure in a "secure area" of the datacentre (effectively a large cage); they managed to get locked in there and had to wait for people to unlock the door to escape...

    Probably what they looked like once they'd escaped -->

    1. Nunyabiznes

      I hope they had at least trash bins for relief.

  6. EVP

    Not his fault really

    You are supposed look around and be careful with your protruding body parts while wheeling a trolley or doing whatever you do in a production environment. Still, I would not blame “Jon” and his unnumbered peers for sudden and uncalled-for electricity losses. Blockhead who insisted placing the button right there where everbody and his butt passes by is to blame. That person, if anyone, should receive all the bollocking, not anyone else >:/

    What comes to his actions to hide his deed and not to tell anyone... well, next time he knows not to scratch any buttons.

    1. Captain Scarlet Silver badge

      Re: Not his fault really

      Its more annoying when such buttons don't just have a cover you lift up and then press the button.

      Thankfully the only such button we have is just for show.

    2. The First Dave

      Re: Not his fault really

      All well and good so long as the "Do not power-on this server any more" sign is in place where necessary...

    3. Anonymous Coward
      Anonymous Coward

      Re: Not his fault really

      I am amazed when I hear all these stories about unprotected master shut down buttons. I think I've only ever seen one such button without a cover, in dozen of datacenters I've been in.

      1. Anonymous Coward
        Anonymous Coward

        Re: Not his fault really

        "I think I've only ever seen one such button without a cover, in dozen of datacenters I've been in"

        Each one of those datacenters was the source of one of those stories (back before it was decided to cover the BRB).

        1. Alan Brown Silver badge

          Re: Not his fault really

          "Each one of those datacenters was the source of one of those stories"

          EVERY SINGLE DATACENTRE I have had to deal with has been installed with unprotected BRBs

          and it usually takes stand up screaming arguments with the beancounters to get that sorted.

          (locking them in the server room and pushing the gas discharge button is a last resort but oh so tempting)

      2. dave 76

        Re: Not his fault really

        "I am amazed when I hear all these stories about unprotected master shut down buttons. I think I've only ever seen one such button without a cover, in dozen of datacenters I've been in."

        In one building where I worked (back in the 80's), the computer room isolate was just *outside* the Ops room and was triggered at least twice by people thinking it was the button to get our attention. It certainly did that!

      3. DanceMan

        Re: unprotected master shut down buttons

        A small arena used for rock shows I work in had a new led house light system installed. As a backup for the mini-controller hidden in a room under the seating there were two ordinary wall switches located in a vomitory* just above head height. They could instantly kill or bring up the house lights. Said vom also led to the only men's washroom in the building, They were enclosed in a box maybe a year later, thankfully without incident.

        *Definition of vomitory

        : an entrance piercing the banks of seats of a theater, amphitheater, or stadium

    4. The Oncoming Scorn Silver badge
      Paris Hilton

      Re: Not his fault really

      "You are supposed look around and be careful with your protruding body parts while wheeling a trolley or doing whatever you do in a production environment. "

      A friend of mine while transporting a butchers device at a trade show, managed to stick it in the wrong place during the move & amputated his finger.

      Icon - Obvious.

    5. Alan Brown Silver badge

      Re: Not his fault really

      "You are supposed look around and be careful with your protruding body parts while wheeling a trolley"

      Lesson one: If it can be whacked by a passing body part or trolley part, it WILL be whacked by a passing body part or trolley part. Plan accordingly.

      Lesson two: If whacking said item will cause a claimable injury, then you will get a claim and your insurers will gripe at you. Again - plan accordingly.

      Lesson three: Someone will ALWAYS go "what does this button do?" whilst pressing it, unless it is under a protective cover of some sort that can't be broken by simply walking into it (see #1) very clearly labelled what it does, how much pressing it will cost and that there's a camera monitoring it.

      1. jake Silver badge

        Re: Not his fault really

        "Lesson three: Someone will ALWAYS go "what does this button do?" whilst pressing it, especially if it is under a protective cover of some sort that can't be broken by simply walking into it (see #1) very clearly labelled what it does, how much pressing it will cost and that there's a camera monitoring it."

        FTFY

  7. Anonymous Coward Silver badge
    Pirate

    And this is why...

    I keep a set of lock picks in my toolkit. Those rack locks are trivial to pick; would probably have them all open before he managed to source that coat-hanger.

    1. KittenHuffer Silver badge

      Re: And this is why...

      I have a titanium toothpick and a pliers multitool on my keyring as part of my EDC kit.

      Recently a cow-orker was trying to open their desk drawers first thing in the morning because they'd forgotten their key. Depress the tumblers using the toothpick, apply torque using the screwdriver blade from the multitool, and jiggle! To their amazement I opened their desk on the first try, in less than 5 seconds!

      Says a lot about the quality of the locks that the average desk is fitted with, and also a lot about how effective your average 'clean desk' policy is in a secure environment!

      1. Pascal Monett Silver badge

        Quality ?

        Their only quality is that they semi-prevent the drawer from being opened when it is "locked".

        A while ago I was consulting in a company that worked with vast open space offices. We're talking a hundred people in the same room, working on a hundred desks. I was obviously told, on my first day, that the desk had to be clean every evening - papers (and valuables) were to be put in the drawers.

        I settled in and my neighbor wasted no time in getting me up to speed on the clean desk policy. It would seem that it was common knowledge that there were only three different lock types, meaning the key on my drawer could open at least a third of all desks present. On top of that, he showed me a trick to get the top drawer open : some desks didn't have the desktop strongly fitted to the drawer section, and if you could lift it just a few millimeters, you could just slide the top drawer open like that.

        I dutifully put my papers in the desk, but my personal belongings stayed with me for the duration of that contract.

      2. Anonymous Coward
        Anonymous Coward

        Re: And this is why...

        Ha! They couldn't be bothered to find me a proper key for the desk I was assigned when I hired in. Instead, I found a key which was "almost" right, and would use it to "pick" the lock on my desk every morning and afternoon. The selected key had all of the tumbler pins low, except for the last one, which was high. So, it was just a matter of pushing it into, and pulling it out of, the lock, while holding a rotating pressure on it. When it cleared each tumbler, the rotating pressure would cause the tumbler to stick in the open position. Once the last tumbler had been strobed, the cylinder rotated, and the desk opened/locked.

      3. Chris G

        Re: And this is why...

        For the average Eurolock, just push any key that will slide in to a point about two thirds in, apply torque to the key and give it a tap in with a hammery type object. The tumblers will bounce to neutral and the lock will open, takes a little practice but once you have the knack.....

        Courtesy of Da Toobs!

        That's why I spend a little more on my home security!

        1. Alan Brown Silver badge

          Re: And this is why...

          "For the average Eurolock"

          Except the ones which have specfic protection against this - and cost a whole pound extra.

        2. Kiwi
          Pint

          Re: And this is why...

          That's why I spend a little more on my home security!

          If you're spending money on "home security" you've already been robbed.

          Most burglars would just smash a window or door, don't give a shit about alarms, and don't worry about cameras or cops. Your lock is only as strong as the weakest part of your door or your windows. The alarms will annoy the neighbours who will sit inside griping about your alarm going off, few will ever look. The 'monitored alarm system' guards will phone and politely wait for the phone to be answered before taking action, and if someone does answer the phone and sounds like a drunken homeowner struggling to remember passwords then they'll give the burglar's mates plenty of time to act. That's if they bother responding in the first place, and then their nearest guard could be more than 30 minutes away. Cops won't be called till the guard says they're needed, and won't come unless there's some compelling reason to (a guy holding a gun to your kid's head probably isn't compelling enough!)

          A good dog may be the better option. 3 good dogs will keep most pairs of burglars at bay. Really good dogs will have the perps ringing the cops themselves and begging for rescue (exceptionally good dogs will leave the perps completely unchewed yet psychologically messed up for life).

          Oh.. And cameras.. They can be ripped out and re-sold to people who're foolish/inexperienced enough to think they make a difference.

          (How I know - much comes from living in an area where a lot of places got hit, and watching defences not so much crumble as sublimate. Best thing we did was start our own neighbourhood walking tours at night, and a couple of planted press stories about how some of us were facing charges for being a little too rough with someone caught trying to break into a house (and a big reason why I don't trust the press, I know how easily stories can be planted ;) )

          Best thing - watch the 'Adam ruins security' and use his main take-away - "don't worry".

    2. MJB7

      Re: And this is why...

      I used to work for a manufacturer of Hardware Security Modules. The procedure for signing firmware was elaborate and involved hardware that was kept locked in a cage next to the meeting room with a hard drive kept in the safe in the accountants office. Some of the other signing procedures were less elaborate, and the associated hardware was less securely protected. The hard drives were kept in a locked filing cabinet in the main office (in open sight). Unfortunately there were seven people who needed access and only six keys.

      Not a problem: the chief cryptographer also had an interest in physical security and would unlock the cabinet with a paper clip (about as quickly as those using a key). This had two major advantages: a) it saved having to get another key cut; b) it prevented anyone having any illusions about the security properties of the filing cabinet.

    3. jake Silver badge

      Re: And this is why...

      Came here to say much the same thing.

      Sadly, it would seem that in the countries many of ElReg's commentards live in, the simple act of carrying a set of picks is considered a crime, even if you have no intent of doing anything criminal with them.

      1. Anonymous Coward
        Anonymous Coward

        Re: And this is why...

        Indeed, and this is why the lock-picking community, where skills and techniques are shared, is somewhat secretive.

        There's a fairly hefty intersection with the techy end of the InfoSec community, esp pentesters, as you can well imagine.

        1. Dabooka

          Re: And this is why...

          Isn't there an old adage that simply states 'locks keep honest people honest'.

          That is to say in most cases anyone with even a passing interest in circumventing the lock can do so with ease (as we've heard).

          1. phuzz Silver badge

            Re: And this is why...

            Locks slow down anyone trying to bypass them so that they can be detected by your other security.

            Unless you don't have other security, in which case you've just got more clean-up to do.

            1. Baldrickk

              Re: And this is why...

              Have you watched any Lock Picking Lawyer videos on youtube? Most are under 3 minutes long if they don't incude a tear-down. the majority of that time is talking about the lock beforehand, and how it's easy to pick afterward...

          2. Anonymous Coward
            Anonymous Coward

            Re: Isn't there an old adage that simply states 'locks keep honest people honest'.

            That, and it means that if someone did get in, past the lock, (and were caught), that we can assume that it was very unlikely they are an honest person who opened the drawer/cabinet "by mistake" or "looking for a pen", or whatever other excuse might be proffered.

            1. Herby
              Joke

              Re: Isn't there an old adage that simply states 'locks keep honest people honest'.

              "Looking for a pen"??

              So that's why they call them "Pen Testers"

        2. DJV Silver badge

          Re: And this is why...

          "...the lock-picking community, where skills and techniques are shared, is somewhat secretive"

          Or not: https://www.youtube.com/channel/UCm9K6rby98W8JigLoZOh6FQ

          1. Graham Dawson Silver badge

            Re: And this is why...

            I opened that link with the tool Bosnian bill and lpl made.

    4. Prst. V.Jeltz Silver badge

      Re: And this is why...

      "I keep a set of lock picks in my toolkit."

      I used to find you could put anything in the lock and turn it and the entire lock would rotate in the circular hole in the glass.

      i guess they dont make em like that anymore !

    5. J. Cook Silver badge

      Re: And this is why...

      ... and if they are from a certain manufacture, they are all keyed the same. Every last one of them, unless you've replaced the cylinder, which is semi-nontrivial. (i.e., our in-house access control people managed to do it, although the replacement cylinder they fitted isn't *quite* the same as the stock one, and the stock one can't be re-keyed.)

      There's a DEFCON presentation on the internet from a penetration testing company that lists the keys that he has that will get him into ~80-90% of buildings, enclosures, and at least one type of car.

      1. Nunyabiznes

        Re: And this is why...

        Late 70s Ford pickups had about 7 key profiles. Chances are if you owned enough of them (I had many as parts rigs, fix and flips, etc) you had a key for any of them. Came in handy a time or two helping out people who had lost their keys or locked them in the truck.

        1. Sgt_Oddball
          Pint

          Re: And this is why...

          Modern Dacias are the same.

          Got told by an auto lock Smith that in his home county they only had 7 keys and 3 immobiliser codes... he found this out after going back to his car drunk to sleep out off and drive in the morning, waking up to find the wrong CD in the stero before realising his car was 3 cars further down the street.

          Got him a career out it in the end though.

        2. Anne Hunny Mouse

          Re: And this is why...

          Wasn't just pickups, Ford UK used about 6 keys for the Cortina.

        3. Tim99 Silver badge
          Facepalm

          Re: And this is why...

          I had a new light blue Volvo in the 80s (Yes, I know - It was an overreaction from the previous car, a somewhat unreliable FIAT). Mrs Tim99 and I stopped for a lavatory break, and an evening snack at a Little Chef (That dates it) in the depths of rural Lincolnshire. When we returned to the car, unlocked it, and I tried to start it, I realised that this car had a manual gearbox - Ours was an automatic. Our car was parked on the other side of a van parked next to the "wrong" Volvo.

    6. JohnFen

      Re: And this is why...

      "Those rack locks are trivial to pick"

      This is true. Even better is that somewhere north of 80% of all of those racks take one of four different standard keys (all of which are readily available from the rack manufacturers and/or eBay). The expectation is that the customer will replace the locks with something else if they care, but most don't care.

    7. Alan Brown Silver badge

      Re: And this is why...

      "Those rack locks are trivial to pick"

      Almost all of them only use 3 key patterns. Why use a pick?

      1. jake Silver badge

        Re: And this is why...

        "Why use a pick?"

        Because it's faster. Your way, you might have to try three keys. My way, I only need to use one pick. And that one pick works for most other makes of office lock, too.

    8. spold Silver badge

      Re: And this is why...

      I travel a lot, so I also carry a lock-pick set, but also a Leatherman Wingman multi-tool, not only deals with all sorts of work crap but it fixes a myriad of things in hotel rooms that would take the hotel an age to get around to fixing, it also has a nice blade I've sharpened that opens equipment boxes in a jiffy.... However I'm on number 6 now as I've way to frequently got to the airport security line and gone to turf out the pockets... oh fuck forgot to put this is back in the suitcase...

  8. OzBob

    Yep, had a very dodgy HP tape array in the noughties

    that was only about 8 u high (and was double-stacked). The door release mech never worked properly when pressing the LCD display, so we just used to stick a half-unfolded paper clip into the appropriate hole and release the damn thing manually. Actually taped it to the front on a piece of string so we did not need to recreate it if we lost the old one.

    1. GlenP Silver badge

      Re: Yep, had a very dodgy HP tape array in the noughties

      a half-unfolded paper clip

      Isn't that standard equipment for all IT departments? I've certainly got one in my desk drawer!

      Main use has been for retrieving CDs/DVDs that have been left in drives and the system powered down of course.

      1. macjules

        Re: Yep, had a very dodgy HP tape array in the noughties

        Bent paperclip uses:

        #1 opening iPhone sim slots

        #2 Router reset operations (why on earth do you not include a paperclip Netgear?)

        #3 Cleaning buildups of ear wax from Apple in-ear phones.

        1. Nick

          Re: Yep, had a very dodgy HP tape array in the noughties

          and, with the addition of a bead of blue-tack, for retrieving small screws from below that utterly pointless raised base-plate that covers only 90% of the bottom of any given enclosure

        2. Robert Carnegie Silver badge

          Pardon?

          I started mentally composing a strong warning against inserting paperclip wire in your ear... Oh! Ear-phones! All right, then. Actually, still be careful. But for insertion in ear, "be careful" means "don't do it".

          1. Alan Brown Silver badge

            Re: Pardon?

            "I started mentally composing a strong warning against inserting paperclip wire in your ear..."

            If you insert it too far, you WILL reset youself to factory presets

        3. swm

          Re: Yep, had a very dodgy HP tape array in the noughties

          In 1965 we were working on a DN-30 with bit buffers that connected to teletypes. The field engineer installed everything but didn't put jumpers in a block to connect the bit buffer to the dataset. It was late at night so I got a bunch of paper clips and started bridging various connections (using a volt meter to find some useful voltage sources). The whole mess was sort of fragile as none of the paper clips could touch. The field engineer came in the next morning and took one look and pulled out all of the paper clips and used proper jumpers etc. It didn't work so we had a discussion to get it right.

      2. dfsmith

        Re: Yep, had a very dodgy HP tape array in the noughties

        I keep several of them.

        1) Look for bent paper clip.

        2) Cannot find one, so bend a new paper clip.

        3) Decide to store the bent paper clip in a special place.

        4) Discover all the other bent paper clips.

        1. Robert Carnegie Silver badge

          Re: Yep, had a very dodgy HP tape array in the noughties

          You could keep your bent paperclip with your... more bent paperclips. I recognise you probably have a reason not to.

        2. Olivier2553

          Re: Yep, had a very dodgy HP tape array in the noughties

          That is why I have one on the same key-ring where I keep the bunch of all 2 different keys that open all the racks in our institution. With a big red cable tie to make the thing stick out in the middle of a mess.

      3. Flightmode

        Re: Yep, had a very dodgy HP tape array in the noughties

        At a previous job, my colleagues and I always kept a metal teaspoon in a rack at a remote site. It fit perfectly for the screw heads for Cisco GSR line cards.

    2. Andytug

      Re: Yep, had a very dodgy HP tape array in the noughties

      Someone should do an appendix to the BOFH manual, on the documented 101 uses for a bent paper clip.

      Mine: to remove 2FA smart cards inserted into the gap between motherboard and CD drive, conveniently placed 3mm below the actual smart card slot (Thank you HP for that inspired piece of design!).

    3. Doctor Syntax Silver badge

      HP Tape & bent paperclips

      We had a 1/2" reel HP drive. Periodically its loading mechanism would get into a state where it failed to work - much blowing of tape into the slot and re-winding. The solution was to reset it by shorting two pads on a circuit board. The spacing was just right for a bent paper-clip.

      1. The Oncoming Scorn Silver badge
        Pirate

        Re: HP Tape & bent paperclips

        I have used a paperclip in the reassembly of the paperfeed\seperation* assembly when reassembling HP LJ4000 - 4100 series printers.

        *I don't recall exactly what the assembly was after all these years, but there were two nasty springs under tension that needed to be in place during reassembly.

        1. John Brown (no body) Silver badge

          Re: HP Tape & bent paperclips

          Likewise, I shaped a spring out of a straightened paper-clip which was then wrapped around a pencil to make a running repair on a dot matrix printer many years ago. AFAIK it lasted at least a couple of years when the printer was finally replaced.

  9. chivo243 Silver badge
    Thumb Up

    Lucky Man!

    A lot of our cabinets have Glassdoors. Some have the mesh doors, will have to remember this if I ever get locked out of a cabinet and need to restart.

    1. Prst. V.Jeltz Silver badge

      Re: Lucky Man!

      in a glass door you can usually just rotate the whole lock in the hole in the glass , as it has nothing to hang on to exepct a little spring clip

  10. jake Silver badge

    Coathangers are handy.

    Here's an old post of mine from 2009, originally titled "That's just the switch, they do that!" ...

    The scene: Old house in Mountain View, converted to Vet clinic.

    The time: Late 1999.

    The job: Convert the Vet's database from non-Y2K compliant PSI/Idexx on HP/UX to Cornerstone on Win98 (I know, I know, but that was what the Vet wanted).

    Considering that Cornerstone included the database conversion in the cost of the 8 hour staff training session, my job was basically setting up the Windows boxes, installing software, and pulling wire. Easy. In fact, I did the complete network setup back in my lab, so all I had to do was make space for the computers, printers, label makers etc., plug it all in, and turn it on.

    Unfortunately, the house was built in the post-war building boom, and originally had 2 prong plugs for mains power. Some had been converted to three prong, but not all. I had to convert the six locations where the Vet wanted the computers installed. Three new breakers, pull some wire ... Again, pretty easy.

    Day of install, the Saturday afternoon before the Sunday the Cornerstone rep was to hold the staff training session. The Vet wanted me to plug a scanner into her personal machine. The floppies that came with the scanner didn't come with Win98 compatible drivers, so I used the new-fangled V.92 modem to dial into my ISP ... But the connection speed reported as 2400 baud instead of the expected 56K (or so). I broke the connection, redialed, same result. And again. Now, I had already used the modem from my lab, just to verify it worked, so I knew it wasn't my hardware ... I picked up the phone. Line noise. 60hz line noise, to be precise.

    I turned to the Vet and asked how long the phones had been buzzing ... she brightly answered that "That's just the switch, they do that!" ... WTF? After further questioning, it turned out that instead of having six phone lines coming in, she had a small switch with six extensions. It was installed when she took over the practice about a decade earlier. (You can probably see where this is going ...).

    She lead me to a storage and washing machine room in the back, and pointed at the wall. Hanging there was a dusty, cobweb covered, slightly sad looking $TELCO supplied switch. It had a three-prong cord ... plugged into a two prong socket. The $TELCO tech who did the install had cut the ground pin off the power cord to fit the availabe mains power. I asked her to go get me a cordless 'phone and grabbed a wire coathanger. With phone to ear, I grounded the case of the switch to the cold water pipe with the coathanger ... No more AC hum.

    I explained the situation to her, and she went ballistic. After she calmed down a little, I asked if she wanted me to call $TELCO for her. (I could have put a new cord on it myself, and put in another 3-prong socket, but if you touch $TELCO gear, they will refuse to fix it if anything goes wrong in the future.) On further questioning, she allowed as to how in the last decade she had had techs out half a dozen times or so to service her telephones, including what sounded like a firmware upgrade to the switch itself.

    Long and short of it, we had a $TELCO tech out 45 minutes after I hung up the phone.

    1. Venerable and Fragrant Wind of Change
      Thumb Up

      Re: Coathangers are handy.

      Long and short of it, we had a $TELCO tech out 45 minutes after I hung up the phone.

      Good for you.

      Brits can only dream of that kind of response from Openreach. Let alone from Virgin's Kafka-inspired castle.

      1. rhydian

        Re: Coathangers are handy.

        "Brits can only dream of that kind of response from Openreach"

        To be fair, Openreach doesn't supply PABX switches, that's a BT/BT Business/$Whoever-else service

      2. This post has been deleted by its author

      3. Anonymous Coward
        Anonymous Coward

        Re: Coathangers are handy.

        Brits can only dream of that kind of response from Openreach.

        They do it all the time. However, if you're worth £3 a month to them, safety is not an issue and there is no SLA in place you're not exactly front of the queue. Among the grown ups give your service provider a call and they're generally contractually required to be on site within two hours of the referral. The service is there for whom it matters.

    2. Manolo
      Happy

      Re: Coathangers are handy.

      You also reposted this last year :-)

      I remembered the story, but could not imagine that I remembered it from ten years ago!

      Still, worth retelling.

  11. Franco

    This is why for years I've been collecting rack and server keys. New racks invariably arrive with lots of keys, which also invariably are kept in a locked cabinet that no one has access to. More than once I've driven an hour to a customer's site to find that the server bezel or cabinet is locked and the only person with the key is the MD who's gone home because the server has crashed and needs a manual reboot (I.e. a hold of the power button or a yank of the cable)

    Thankfully much less common now with DRACs, iLOs etc but the keys are still in my bag.

    1. A.P. Veening Silver badge

      the only person with the key is the MD who's gone home because the server has crashed and needs a manual reboot

      In that case the MD can return and the waiting time is billed for 100%. After a couple of times the MD will learn better than to disappear in situations where he should remain.

      1. Franco

        That wasn't usually a conversation I had to have being a lowly engineer, pretty sure that's what my MD at the time would have told them though.

        1. A.P. Veening Silver badge

          Of course you wouldn't have to have that conversation, you should just have written the hours and made sure you got paid. After that you just delegate it to your "superior".

      2. Nunyabiznes

        At A.P.

        You would think. I've spent a lot of time on the clock cooling my heels waiting for a key person to come back to the mess they made and then left without a care. Multiple people at my current org have done this. They are all still employed at considerably more base rate than I. Sometimes I have a problem with speaking truth when silence would have been a better choice - so naturally I have a much bigger HR file than any of said people.

        1. Kernel

          "I've spent a lot of time on the clock cooling my heels waiting for a key person to come back to the mess they made and then left without a care."

          Yep - I once spent about 4 hours wandering around a beach at the customer's expense, while waiting for a fibre jointer to turn up to pull in and splice another pigtail on the line fibres of a DWDM system. Due to the analogue nature of the line optics of DWDM systems commissioning has to proceed in sequence from A to Z and then Z to A, it's not possible to miss a site and go back to it later.

          The one they'd put in originally was brand new, but had a massive scratch across the core face at the connector which caused it to be rejected by the test software on my inspection scope. Supposedly the original pigtail had been inspected (as per documented work procedures) before being installed.

    2. Unicornpiss
      Meh

      Keys

      "This is why for years I've been collecting rack and server keys"

      I do the same. I had accumulated a whole box of them at one point, which pretty much covered every situation.. Until.. someone "helped" us clean our messy IT room one weekend and threw away the whole box of them. Which had me angrier I think than all the other little atrocities committed when someone with no clue about your day-to-day work "helps out" This same moron was throwing away brand new CAT-6 cables still in the bags because "too much clutter".

      It brightened my day when I heard he had been let go some months later--he apparently couldn't do his job either, much less ours.

      1. Nunyabiznes

        Re: Keys

        Let me guess: Facilities guy? Ours insist on having unrestricted access to our data closets and centers.

        One day I came in to find a contractor, not escorted, busily installing a (home) ac unit in one of our centers to alleviate the heating issues we were having. The facilities boss insisted we didn't know what we were talking about when he spec'd the original ac unit, and now that was a failure he raided our budget to install a home unit with an el cheapo contractor. Lovely seeing sawdust floating towards the intakes of your equipment. Even better when you realize he has his equipment plugged into your rack UPS.

  12. Prst. V.Jeltz Silver badge
    Paris Hilton

    scram button!

    Do you really need that button?

    In what situation , in a server room, would the "instantly cut power to everything" button be used?

    fire? earthquake? solar storm? DDOS?

    1. defiler

      Re: scram button!

      Basically for the things you can't think of ahead of time. But for the love of the drive heads, put a cover on it, or make it a twist-switch or something!!

    2. macjules

      Re: scram button!

      There has already been a very good example of this: NotPetya. Read about Maersk's need to instantly cut power here

    3. Korev Silver badge

      Re: scram button!

      >In what situation , in a server room, would the "instantly cut power to everything" button be used?

      If there was an electrician with a higher current flowing through them than is normally advised?

    4. Stevie

      Re: scram button!

      In what situation , in a server room, would the "instantly cut power to everything" button be used?

      Electrocution in progress?

  13. Prst. V.Jeltz Silver badge
    Alien

    SCRAM!

    "I reversed into the scram button."

    The origin of the name of the big bad button is tricky to pin down.

    I think its something to do with "Charlie and the Great Glass Elevator"

    1. J. Cook Silver badge

      Re: SCRAM!

      That button was labeled "Up and Out", IIRC.

      1. Anonymous Coward
        Anonymous Coward

        A warning from the Vermicious Knids was Re: SCRAM!

        "That button was labeled "Up and Out", IIRC."

        Yes, but also:

        <https://investorshub.advfn.com/boards/read_msg.aspx?message_id=133722360>

  14. Marty McFly Silver badge
    Facepalm

    Helping out...

    Same solution, different story....

    I came across a hapless soul who had lost her car keys while swimming in the lake. After spending two hours wandering the shallows looking for keys, I managed to unlock her car with a couple sticks and duct tape. She had left a window down a bit (out swimming, so it was a hot day). Taped the sticks together, slid them through the window, and bumped the unlock button on the other side. She could now get her cell phone and phone a friend to bring the spare key.

    Granted had she known their phone number, instead of relying on the iThingy to remember it, she could have simply called them in the first place.

    1. John Brown (no body) Silver badge

      Re: Helping out...

      "Granted had she known their phone number, instead of relying on the iThingy to remember it, she could have simply called them in the first place."

      To be fair, that is so ubiquitous nowadays that I don't blame here for that. I think the only two numbers I know from memory are my own landline and mobile numbers these days. I used to have a veritable phone book stored in my brain, but there's almost zero need for that these days.

      1. jake Silver badge

        Re: Helping out...

        My own land-line is the one number that I can never remember ... probably because I almost never dial myself.

        1. doublelayer Silver badge

          Re: Helping out...

          But you probably give that number to people or enter it on forms from time to time. If not, you might not need that number. My mental phone book is sadly pathetic. I can give you the phone number of friends I had in 2007 when I placed calls by manually dialing on a land line, but people I've met since then, including those I call or send messages to frequently, are only known by my contacts database. I have that backed up, but maybe I should try memorizing the numbers.

          1. entfe001
            Facepalm

            Re: Helping out...

            Same for me. This includes people that have changed their phone number.

            The old numbers are so engraved in my memory that more than once I end up calling the old number.

            1. Kubla Cant

              Re: Helping out...

              Same here.

              I use phone numbers from fifty years ago when I need a secret numeric passcode. But I can't remember the mobile number of somebody I call every day.

  15. Dave 32
    Pint

    Big Red Button

    In our case, it was the contractor in charge of changing the burned out light bulbs who caused the first incident, when he sat his ladder next to the door on the way out. One of the rungs hit the Big Red Button perfectly. Ker-CHUNK! Whoopsie.

    So, to prevent that from ever happening again, another contractor was hired to put a shield around the Big Red Button. The first thing this contractor did was to bring his ladder into the room and lean it up against the wall by the door. Ker-CHUNK! Whoopsie!

    1. Nunyabiznes

      Re: Big Red Button

      Another case of million to 1 odds happening 9 out of 10 times.

  16. PassingStrange

    SCRAM

    Back in the early 70s the "perceived wisdom" was that SCRAM stood for "Start Cutting Right Away Man" (vide the instructions for the Atari game of the same name). In reality, I suspect that there's a 99.9% chance that the name came first, and every "explanation" is, basically, a backronym..

    1. PassingStrange

      Re: SCRAM

      (Or, possibly, "Start Chopping..." - memory is fallible, and it's been nearly half a century. But I agree with the the article linked to below by @Jake - the slang term "scram" seems by far the most likely origin. It was certainly sufficiently in use around the time to have even reached the UK; my father - UK career soldier - used it regularly, when I was under his feet.)

  17. jake Silver badge

    Etymology of "the scram button", according to the U.S. NRC.

    My daughter just sent me this link:

    https://public-blog.nrc-gateway.gov/2011/05/17/putting-the-axe-to-the-scram-myth/

    And now you know.

  18. jake Silver badge

    Scram button, revisited.

    As a side note to my fellow IT techs ... If you ever come across a drag racing advocate in your travels, don't assume that the "scram" button, often on the steering wheel, means "power down all systems". In fact, it means quite the opposite, and usually adds between 10 and 15% more power to the motor for a last-second improvement in ET and MPH. It's basically a press to pass button, and can activate one or more of several different methods of pushing a motor past it's normal limits for a second or two.

    It's short for "scramble", which some racers call it.

    The "All systems off" switch is located at the rear of the car where track personnel can access it in an emergency, and is usually well marked.

    1. Nunyabiznes

      Re: Scram button, revisited.

      I put another master switch within reach of me when I'm belted in. One switch cuts ground, one switch cuts positive. I race off road and sometimes you are your own safety crew. Being able to cut power and release your own fire extinguisher (same for co-driver) may be all that lies between you and a bad day.

      You have to make sure the alternator charge line is included in the positive master switch. Luckily, I found out my very basic mistake (late night wiring before the race) during testing and not when it counted.

      I've never had a "scamble" button. Would that be like the WW2 fighter planes that had "emergency" power levels available for short periods? I've used nitrous in drag cars, but that's a portion of total power output and is in use every run, subject to tuning.

      1. jake Silver badge

        Re: Scram button, revisited.

        I usually have a kill switch and fire suppression within arms reach, too. Prudent. The switch itself isn't protected by a molly-guard, but the arming switch for the fire suppression is.

        Yes, just like the aircraft emergency power button, but often to the point of destroying the motor just as you cross the finish line. It can activate another level of nitrous (dry or wet), increase the redline (if you normally run a rev limiter), tell the wastegate to increase the boost, adjust the timing, or more usually some combination. It is normally only used in the semi finals or the finals because of the chances of doing internal damage to the engine.

  19. Anne Hunny Mouse

    Knew of a UK Gov related data centre which regularly had accidental shutdowns as the shutdown button was where you would expect door release to be.... In addition it was the data centre without resilience as they were too tight to pay for it.

    On another note, recently picked the locks on some Telecoms NTE cabinets that nobody had keys to.

    One was in the process of being removed, the other needed the coaxes switching and hadn't seen an engineer for years

  20. Unicornpiss
    Pint

    Similar, but different story..

    I didn't take down a server room, but I did find myself in a large, noisy dyno room, standing on top of a control room that I had to reach by climbing a 15-foot ladder. I had inadvertently taken down a locked rack of switch gear and one fairly critical server that was located up there. The equipment was not accessible and I had no idea who had the key, but I suspected it was in the other building down the road, held by a different team. It was too noisy up there to even make some phone calls to try and find out. At the time I was a contractor with less than a year in with the company and the lowest on the totem pole. I didn't think I would get fired, or even yelled at too badly, but the event wasn't going to do much to improve the general opinion of me either.

    I stood up there for a while, thoroughly discouraged, listening to the almost painful roar of the screaming dynamometer, feeling my head pounding, and having had a long, fairly rotten day already, didn't have the energy to climb back up there again, much less go on a snipe hunt for the key. Fortunately locks on equipment cabinets are not all they could be in many cases, especially with cheap cabinets. I spotted a snipped off bit of zip tie in the grit where I was standing, and with a little wiggling was able to open the simple wafer lock, start everything back up, and relock the cabinet just like it was.

    A half hour later our Helpdesk, consistently a day late and a dollar short, called to inform me of an outage with the equipment, as I was driving home. I told them it had already been taken care of. Beer icon for obvious reasons after a day like that.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like