back to article Keep your cables tidy. You never know when someone might need some wine

We've all heard the one about the cleaner in the hospital pulling the wrong plug. But managing to push back the schedule by months through sheer brute force? Welcome to Who, Me? Our story comes from a reader Regomized as "Tom" and concerns the time his employer, a major parcel delivery company, was planning to open a new …

  1. Pete 2 Silver badge

    fail - over?

    > more than 3 months before we could finally get the local hardware support company to re-install the parts

    So an annualised uptime of less than 75%

    1. Howard Sway Silver badge

      Re: So an annualised uptime of less than 75%

      Annual Company Report : IT Service availability exceeded "One Seven" levels for the fifth year in a row.

      1. RichardBarrell

        Re: So an annualised uptime of less than 75%

        This is one of the reasons why I advocate that most services should be targeting a nine-fives SLO rather than five-nines.

      2. NoneSuch Silver badge
        Coffee/keyboard

        Re: So an annualised uptime of less than 75%

        Secure.

        Your.

        Server.

        Room.

        No yobbo's or lookie-lou's allowed.

        1. Evil Auditor Silver badge
          Facepalm

          Re: So an annualised uptime of less than 75%

          No yobbo's or lookie-lou's allowed.

          Erm, yes. That is, until I came across that client: the complete staff was allowed to enter its server room. I did not like it and asked them to restrict access to those who actually need to access it. And their response: "We did." - ??? - "It's the coolest room and we store the beverages in there so everyone needs access."

          1. gandalfcn Silver badge

            Re: So an annualised uptime of less than 75%

            DHL? Hong Kong?

    2. Dave314159ggggdffsdds Silver badge

      Re: fail - over?

      24/7 is 3.42857142857...

      1. Paul Herber Silver badge

        Re: fail - over?

        American Pi in some states.

        1. fidodogbreath Silver badge

          Re: fail - over?

          Pi is just a conspiracy by the global elites to dupe the sheeple into believing that circles exist.

          1. John Brown (no body) Silver badge

            Re: fail - over?

            "Pi is just a conspiracy by the global elites to dupe the sheeple into believing that circles"

            I think your post got cropped.

            1. mistersaxon

              Re: fail - over?

              Rounded down

          2. low_resolution_foxxes Silver badge

            Re: fail - over?

            All those spherical-earth globalists need to learn to stay away from our parallelograms.

            There are over 9000 reasons to keep them away from our edges.

  2. Red Sceptic
    Joke

    Leg pull, surely?

    I’ll see myself out …

    1. Anonymous IV
      FAIL

      Re: Leg pull, surely?

      Just someone putting the boot in...

      1. Sgt_Oddball Silver badge

        Re: Leg pull, surely?

        I'd rather say it was boot error leading to a localised network transport issue. The liquid cooling problem however appears to have been a root cause.

        1. logicalextreme Silver badge
          Pint

          Re: Leg pull, surely?

          fantastique

    2. aerogems Silver badge

      Re: Leg pull, surely?

      You'd better leg it before the mob shows up with pitchforks and torches.

    3. bpfh Silver badge
      Joke

      Re: Leg pull, surely?

      But leave the back plane at the front desk with the petit fours please.

  3. lglethal Silver badge
    WTF?

    I hope whoever agreed to allowing catering to use the server room for cooling received their pink slip before the day was over. Dear lord that's a horrendours decision.

    Who could ever think letting a load of unknown, low paid, catering staff to run loose in your brand new server room was ever a good idea. They maybe saved $100 by not having to hire a refrigerated trailer to store there stuff, and look what it cost them!!!

    My Flabber is well and truly gasted...

    1. b0llchit Silver badge
      Alert

      And please, management should be publicly flogged and fired on the spot because they allowed critical infrastructure accessible without proper locks and other restrictions.

      1. Giles C Silver badge

        At a previous company I remember walking into one of the comms rooms to find I couldn’t get to the network switch to trace a fault because facilities had filled the room up with pallets (yes plural) of bottled water.

        The water was for a promotional thing to the staff they got to told to remove it and the question came back why was it a problem ?

        I left that for management to deal with

        1. Captain Scarlet Silver badge
          Trollface

          Ah yes I would climb over phone site manager and then go, I can't get to a fire exit.

          No BOFH should let anyone store shit in their server rooms, as soon as management see there is free space it ends up as a meeting room in my experience.

          1. aerogems Silver badge

            To be fair, a good BOFH would make sure the raised floor was like something out of an Indiana Jones movie and if you stepped on the wrong tile you'd be facing a long drop into the inky blackness.

            1. Paul Herber Silver badge

              a long drop into the inky blackness

              '... a long drop into the inky blackness.'

              Otherwise known as the Marketing Dept.

              1. aerogems Silver badge

                Re: a long drop into the inky blackness

                That's just regular blackness. INKY blackness is beancounter central.

                1. Boris the Cockroach Silver badge

                  Re: a long drop into the inky blackness

                  Still wrong

                  Proper inky blackness is the chute that ends in the lift shaft... where you've also been dumping that left over toner and inkjet cartridges

                  Can we have a BoFH icon please?

                  1. bpfh Silver badge

                    Re: a long drop into the inky blackness

                    I thought there was one a long long time ago…?

                  2. aerogems Silver badge

                    Re: a long drop into the inky blackness

                    But if you fill that space up with random trespassers to the server room, how will you run your "recycling" program for ink and toner carts?

                2. Anonymous South African Coward Silver badge

                  Re: a long drop into the inky blackness

                  The real inky blackness is where grues lurk.

            2. NXM

              Most inky blackness is just very very very very very very VERY very dark blue

              1. Astarte1

                That would be an ecumenical matter.

    2. UCAP Silver badge

      Who says anyone "allowed" the caterers to use the server room as a fridge? It is entirely likely that they, or someone low-life they talked to, made the decision themselves without bothering to consult Those Who Would Actually Know Better (tm). The real question in my mind is - why wasn't the server room properly locked down with some serious access control?

      1. PM from Hell

        Facilities Access

        I have come across so many issues over the years because the facilities team have access to every room and just see the DC as an equipment store.

        Now so many rooms are dark I dread to think how much crap is scattered through them.

        1. JimC

          Re: Facilities Access

          Or the building designers put an aircon component that requires maintenance in a roof space only accessible from the server room...

          And the server room rarely requires access by IT staff it may be some considerable time before you discover that the Facilities staff have decided this is a good place to put Christmas decorations with disintegrating metallic tinsel...

          And if you suggest that the Facilities staff need to relocate the decorations in the manner of the good fairy and the Christmas tree then you may find yourself in trouble with HR

          Don't ask me how I know...

    3. chivo243 Silver badge
      Windows

      Brand new server room and cables hanging willy-nilly? I think lots of pink slips should have been handed out that day. Starting at the top...

    4. Anonymous Coward
      Anonymous Coward

      I would say that it happened in earlier times, where things where things were more relaxed

    5. John Brown (no body) Silver badge

      "I hope whoever agreed to allowing catering to use the server room for cooling received their pink slip before the day was over."

      It was described as a "server closet" and "at the turn of the century", so I'm guessing we aren't talking about "server central" and BOFH + PFY roaming around. More likely a rack or two and barely room to swing a cat.

      "Dear lord that's a horrendours[sic] decision."

      Despite the above, I still agree with this bit :-)

    6. phuzz Silver badge
      Facepalm

      It all depends who came up with the idea. I've worked plenty of places where an otherwise sensible IT decision (eg, 'don't use the server room as a fridge') would be overruled by someone higher up. After all, if they get paid more money that me, then they must know better right?

      Of course, when said idiotic decision turned out to be a bad idea, it wasn't the execs that took the blame....

      ("why didn't you tell us it was a bad idea", 'I did, here's the email', "I didn't read that, you should have told me in person", 'I did!', "well, I don't remember that, so it's all your fault").

    7. atheist

      Of many root cause analysis tickets undertaken, from a customer's clustered SAN and server logs was able to time/journal the cable disconnects, fibre, NIC and power, to show that it happened from top to bottom, starting half way down, the inference being someone/thing fell against it.

  4. Anonymous Coward
    Anonymous Coward

    Russia

    "Who could ever think letting a load of unknown, low paid, catering staff to run loose in your brand new server room was ever a good idea."

    Do you really think the oligarch was fired?

    1. Strahd Ivarius Silver badge
      Trollface

      Re: Russia

      Russia?

      It was in Florida...

      1. Doctor Syntax Silver badge

        Re: Russia

        Different oligarch.

        1. UCAP Silver badge

          Re: Russia

          They call them "ex-Presidents" in that part of the world.

          1. Korev Silver badge
            Joke

            Re: Russia

            Isn't that Point Break?

  5. elsergiovolador Silver badge

    Blame shifting

    A multiple-month delay thanks to a bit of careless footwork on the part of a caterer

    Not a fault of caterer for being a clumsy caterer. The blame is on anyone responsible for the server room security.

    Also when you run a server room yourself, you need to have some redundancy. I know servers are expensive, but that's just how it is. Even one spare server would get them out of trouble.

    1. Jou (Mxyzptlk) Silver badge

      Re: Blame shifting

      If you let people with no computer knowledge and beverage loose in a server room you'd need a two-digit-million setup to get that redundant.

      For the rest: I agree.

  6. Anonymous Coward
    Anonymous Coward

    far flung kit issues...

    VOIP Telco in early 2000's (startup so many ethical corners cut - oh the memories...)

    wanted a local gateway in a particularly high volume destination of our phone cards (remember them?)

    anyway... official telco licenses were impossible, we had support of a local general, who was willing to house equipment at his house and arrange the local E1 drops - for a rental fee and revenue share deal.

    so we shipped out half a dozen preconfigured AS5300's they got escorted through customs to the generals house, and appeared on-net rapidly. Traffic flowed and the backhanders seemed to have paid off.

    until the rainy season.

    boxes went down constantly. came back up, but were particularly flaky.

    we sent out an engineer to "get it back to the reliability we had at the start" on landing at this tropical hellhole and getting transported to the generals pad was pointed to the stack of boxes in a corner of the veranda. he took a picture (with a real camera) of the cables coming down the building acting as a perfect rain conduit directly to the boxes.

    asked if they could get moved in to the house until the rain stopped was met with grumbling... made comments about revenue share needed the boxes up to work...and it happened. they called in a local labourer to literally knock a hole in the wall so the cables could get fed through (hole stuffed with newspaper afterwards to "seal") . once in the lounge the kit worked without issue

    eye opening all round.

  7. Doctor Syntax Silver badge

    "snarled up in local customs for more than 3 months"

    Local customs!

    Maybe the caterer should have been told off to pay all the local expedited clearance fees.

    1. Anonymous Coward
      Anonymous Coward

      Additional Customs clearance chargers

      Sometimes you have to be pragmatic either fly the parts out with an engineer with a completed low value customs clearance form and a pocket full of cash, or have a conversation with the customs agent locally about the additional inspection charges required to release the kit.

      Its wrong, irritating and probably illegal locally and if you can plan for delays without interrupting progress then do so, but the cash amounts are normally so low they don't even appear as a blip on the project budget.

      1. Dave@Home

        Re: Additional Customs clearance chargers

        It can also be illegal back in your home country to offer to pay a bribe overseas

      2. Anonymous Coward
        Anonymous Coward

        Re: Additional Customs clearance chargers

        Also illegal in this country. Don’t say ‘pragmatic’ when you mean ‘illegal’, it’s a very slippery slope.

      3. xeroks

        Re: Additional Customs clearance chargers

        the whole import & export of hardware can be fraught with entirely legal difficulties, never mind illegal ones. I've heard of servers & laptops being confiscated -legally - by governments. And we're not talking third world governments here: I'm sure the US was one of the worst culprits.

  8. ColinPa Silver badge

    two tales

    I remember being told about the connection between 2 mainframes - the cable was a maximum of 5 metres long. The usual configuration was ... 1 Metre from socket, down to the floor ... 3 m under the floor and 1 m up to the other machine. The customer's problem was the machines were 4 m apart, so it went across the room at waist height, over other kit and a chairback etc.

    This worked fine till someone tripped and backed into the cable and pulled it out of the machines.

    There was a bank of DASD - 7ft high . 3 ft square in slightly the wrong place (2 inches). They put their backs into it - and pushed with the legs... and moved the disks. As they stood up - someones's belt got caught on the Emergency Power Off button - which was pulled - and dropped the power. It took a day or so to recover!

    1. Anonymous Coward
      Anonymous Coward

      Re: two tales

      Was at a client site in NW London in the 90s and had to go behind their server racks (reason lost in mists of time) into a total rat's nest of wiring. Needless to say, I tripped and yanked the power cable out of a UPS outlet that was supplying 'our' server. Sheepishly reinserted, but as I knew the server in question had been specified with dual-redundant PSUs I was quite confident of having got away with it. Until, upon emerging, the office is all a-fluster. They had decided to save electricity and unplugged the UPS for the second PSU as it was apparently 'redundant'.

  9. Barry Rueger

    My girlfriend did it

    Managing a community radio station in Ottawa, Canada. Needless to say by 1999 the radio control room was a no-liquids zone. Late one night I get a frantic phone call telling me that a snottty kid calling himself DJ T*****B*** had dumped a can of Coca-Cola into the mixing board.

    I phone the twerp the next day to chew him out, and his very first words to me weren't "Gosh, I'm sorry." They were "It wasn't me, it was my girlfriend, here's her phone number. "

    Followed quickly, when I explained that he was now an ex-volunteer radio host, by "You can't do that! I'm incredibly popular in Europe, and your name will me mud!"

    * It seems that "DJ T****B***" is still floating around these days these days, so I'll anonymize him.

    1. johnfbw

      Re: My girlfriend did it

      Theres no DJ TB in this list.

      https://en.wikipedia.org/wiki/List_of_club_DJs#T

      Feel free to name and shame (besides - he isn't tech literate enough to read The Register!)

      1. Doctor Syntax Silver badge

        Re: My girlfriend did it

        He might periodically google his name.

        1. JimC

          Re: He might periodically google his name.

          And being a DJ almost certainly does. Frequently.

      2. Robert Carnegie Silver badge

        Re: My girlfriend did it

        It's never Tony Blackburn? :-)

    2. Martin an gof Silver badge
      FAIL

      Re: My girlfriend did it

      As I think I've recounted here before, I had exactly that happen at an ILR station in Cardiff back in the late 1990s. DJ in question was doing his last live show before moving on to pastures new and a well-wisher had sent him a bottle of cider.

      Of course, it wasn't he who called it in at any point in his mid-evening (7 - 10pm?) show, nope, it was the poor sap who followed him who had to page me and listen to me moaning.

      One thing which really annoyed me was that both jocks flat out refused to move to the spare studio. Granted this was in the days when they had boxes of CDs to shift, but the spare was literally five steps away and ok, yes, the "offer, accept" handover system was a bit awkward to do on your own, but not impossible.

      Instead, both carried on with half a working desk, making do with the guest microphone, one CD player and the playout system if I remember correctly. Maybe the phone TBU too.

      On the positive side, the desk surface was just some sealed switches, sealed pots and Penny & Giles conductive plastic faders; all the electronics were in a pod in a rack. The desk carried on without a glitch (apart from the faders covered in sticky cider), the drain holes in the bottom ensured that jock went away looking as if he'd had "an accident" and I was able to revive the faders by running them under a tap and replacing a couple of wipers that may have been close to needing replacement anyway.

      But I could have done it at 8pm when the accident happened, rather than at 11pm.

      M.

      1. low_resolution_foxxes Silver badge

        Re: My girlfriend did it

        Wasn't there a 'martin the goth' who played on Xpress FM /Metros for a while?

        1. Martin an gof Silver badge

          Re: My girlfriend did it

          May have been, but it were nothing to do with me. My name refers to Myghal Josep an Gof / the Smith, who was one of the leaders of the Cornish rebellion in 1497... and we all know how well that went. The DJ in question's name escapes me right now (Chris something?), he was only with us for a couple of years and headed off to Leeds (Aire) I think, or maybe not. Our greatest claim to fame was probably Bobby McVay, briefly famous for the Eurovision song "I'm never giving up" and latterly as a "spare" fourth member of Bucks Fizz. In those days we had a thriving newsroom with close ties to the Cardiff School of Journalism, and several alumni are still active in the field.

          M.

      2. cosmodrome

        Re: My girlfriend did it

        ...and this is the reason why "Broadcast Class" equipment costs an arm and a leg (of your first born).

    3. Martin
      Happy

      Re: My girlfriend did it

      Back in the very early eighties, a new multiprocessor system was being demonstrated at a trade fair in Germany. It was literally the prototype - the only one that was working in the world.

      The boss managed to spill his beer into it.

      Every card was removed from the rack, and carefully rinsed with water and dried. The backplane was also carefully washed and dried. It was all put back together, with fingers crossed and a prayer or two offered up to the gods of technology. Power was applied....

      It sprang into life, like it had never been away.

      1. Phil O'Sophical Silver badge
        Pint

        Re: My girlfriend did it

        A clear example of why it's much better for you to drink beer than Coke.

        1. yetanotheraoc Silver badge
          Pint

          Re: My girlfriend did it

          Also, beer > hot chocolate. Don't ask how I know.

      2. Anonymous Coward
        Anonymous Coward

        Re: My girlfriend did it

        The old custom IBM Model-M (and even older Westinghouse) keyboards at BA LHR checkin had God-knows what poured into them (various bodily fluids included I sustpect..). Standard recovery was to put them, intact, through the high-temp industrial dishwashers from which they would emerge spotless and working perfectly. The later rubber-membrane stuff sadly didn't respond so well...

    4. C R Mudgeon
      FAIL

      Re: My girlfriend did it

      The time an unopened can of Coke spontaneously exploded, and the resulting splatter took out our only copy of the installation floppies for some mission-critical software.

      Several fails, obviously: only copy; food on the same shelf; all of it by a window (don't know, but I'm guessing that the can was in the sun).

    5. herman Silver badge

      Re: My girlfriend did it

      Wow, you got her name and number and then you passed the opportunity for some payment exercise?

  10. Anonymous Coward
    Anonymous Coward

    This is why

    There are only 3 keyholders for my server room. And the MD isn't one ...

    1. Lis

      Re: This is why

      I run my own business so I suppose I am sort of an MD.

      As for keys to the server room, (actually a cellar) why would I want to have a key?

      I pay someone else to look after that sort of thing. It's not like I haven't got many other things to do.

      Love the way you refer to the company that hires your services server room, as MY server room.

      1. nintendoeats Silver badge

        Re: This is why

        I'm quite sure the employer is happy for their employees to (incorrectly) "feel ownership" over their work. You are much more likely to stay late you fix "your server room" than "Jeff's server room".

        1. General Purpose Silver badge

          Re: This is why

          Quite. And in many companies (probably most large ones and of course all charities) the MD is another employee, not the owner.

        2. cosmodrome

          Re: This is why

          Don't tell them or HQ is going to fire everyone for embezzlement.

      2. Doctor Syntax Silver badge

        Re: This is why

        "Love the way you refer to the company that hires your services server room, as MY server room."

        You appear to be unfamiliar with BOFHs. You are advised not to enter any stariwells or lift shafts in the company of any of your IT staff and to keep clear of any windows when they're about. Parking your car away from anywhere where a heavy object may fall fromm a building is also a good idea.

  11. reeferman

    Advanced Server Room Dynamics

    Poor planning surely? In times (and jobs) gone by it was common to lift the raised floor tiles to stash beer in the CRAC-cooled void space. A mini-Threshers accessible only to authorized server room entrants.

    1. Doctor Syntax Silver badge

      Re: Advanced Server Room Dynamics

      Caterers might not know that and in this case the evidence points to there being no such facility.

  12. Hero Protagonist
    WTF?

    So many questions

    My question is, how is it that the backplane was so easily pulled out of the server instead of the the cable connector breaking?

    1. Loyal Commenter Silver badge

      Re: So many questions

      On cable = cable connector breaks.

      One bundle of cables = whatever they are plugged into breaks.

    2. ColinPa Silver badge

      Re: So many questions

      If you are often taking the cover./backplate off, people don't screw it on - knowing they'll be in it next week. Touch it, and it comes off (whether you wanted it off or not)

    3. doublelayer Silver badge

      Re: So many questions

      The tensile strength of a lot of cables at once can end up failing to pull out or break just the cable. Even one cable with a secure enough mechanism (E.G. the ones with extra pins just to keep the plug connected) can transfer a lot of the stress to the thing it's connected to. In addition, the strength of someone pulling with their leg is a lot higher than with their arm, meaning the stress on the backplane would have been one quick pull, whereas pulling the cables with your arm would have taken longer and given some of the cables a chance to break off and transfer the strength to others.

    4. Anonymous Coward
      Anonymous Coward

      Re: So many questions

      Stumped me too. Lots of WTF-HTF around the tech team that day!

  13. anothercynic Silver badge

    Familiar...

    I believe a certain software vendor in the home counties had an impressive wine collection gathered by one of the two co-owners in one of the server/patch rooms for exactly the same reason. It was cool, it usually was dark, not many people had access (and those admins who *did* have access knew not to touch said alcoholic goods).

    And apparently that arrangement persisted way past the day the owners signed their baby over to some investment bankers for a *lot* of wonga... until someone pointed out that since the baby wasn't theirs anymore, it wasn't appropriate for them to use said room in the building to store personal property. I don't know what happened next, but I suspect said personal property must've been relocated to a more suitable venue...

  14. fidodogbreath Silver badge

    Frame of reference?

    To be fair, not a lot in IT goes particularly well when alcohol is stirred into the mix

    Has anyone ever tried to determine whether IT goes well when alcohol is not stirred into the mix?

    1. Jou (Mxyzptlk) Silver badge

      Re: Frame of reference?

      Yes and in detail.

  15. Jenny with the Axe

    Yeah, that happened

    Back in late 90's at a national ISP, one of the sysadmins didn't see any reason to avoid alcohol just because he was on call. His favourite drinking place was just around the corner from the office anyway, so it's not as if he'd need to be able to drive there in case something happened!

    Something did happen. He got back to the office and up to the server room... and stumbled, bringing an entire rack down.

    I think he sobered up fairly quickly thanks to the adrenaline. I also don't think the customers ever knew the cause of that outage.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022