back to article Beware the big bang in the network room

Cables can be unkind, especially when one has the confidence of youth but not the dark cynicism of experience. Welcome to an edition of Who, Me? to tug at, if not the heart strings, then certainly the RJ45s. Today's hero is a Register reader Regomised as "Colin." Colin had accepted his first proper role as a network engineer …

  1. Pascal Monett Silver badge

    "Lessons learned after event tend to be the toughest ones"

    Indeed. You can't go back and sort things out anymore so, more often than not, you're looking at one of your failures. Or, if you're really, really lucky, one of your closest brushes with disaster.

    1. Anonymous Coward
      Anonymous Coward

      Re: "Lessons learned after event tend to be the toughest ones"

      Traditionally expressed as:-

      "Experience is a hard teacher because she gives the test first, the lesson afterwards."

      When you do have an expensive mistake like this, you ensure that the maximum amount of learning is done from it by other people, as learning from somebody else's mistakes is a lot cheaper than letting you commit them yourself. An entire new generation of people may be helped to avoid certain mistakes via this article.

      Alas, the young engineer here didn't hear "Always multiply your repair estimates by a factor of four." else he'd have had a maintenance window plenty large enough. ;)

      1. Anonymous South African Coward Silver badge

        Re: "Lessons learned after event tend to be the toughest ones"

        I follow the teachings of one Montgomery Scott.

        And it's always a wise move to plan for a longer window than required.

        1. derrr

          Re: "Lessons learned after event tend to be the toughest ones"

          Aye.

    2. ShadowSystems Silver badge
      Joke

      Re: "Lessons learned after event tend to be the toughest ones"

      Brushes with disaster? I'm more of a scrubby loofa sponge on a long stick sort of person myself, especially with disaster -- the longer the stick, the further away I can poke it. =-)p

  2. Dan 55 Silver badge
    Unhappy

    Maintenence window, gosh how quaint

    Everything's got to be 24/7 now. Even on ancient systems which were designed with a maintenance window in mind and to be offline while overnight batch processes are run, you find sales have sold it as some shiny new always-available thing, so now you've got to persuade it do something to do it was never designed to do.

    1. Fred Daggy Bronze badge

      Re: Maintenence window, gosh how quaint

      Hmmmm. "Sales sold it as ... ", and I think that at certain times, the server should mysteriously go offline.

      Not to punish the customer, but to alert management to the frailty of the setup. Network errors are normally the least detectable, ahem, traceable to an individual, as they can be done with a single physical cable. YMMV, some setups are more secure than others. but we all know the weak spots.

      You might find then that cash will flow for proper redundancy. If not of hardware, then at least people providing TLC.

      1. Korev Silver badge
        Joke

        Re: Maintenence window, gosh how quaint

        By proper redundancy do you mean the systems or the staff?

        1. Anonymous Coward
          Anonymous Coward

          Re: Maintenence window, gosh how quaint

          With any luck, the sales staff who sold the client something as something it is not in the first place.

          I can think of at least one such coke-hoover* where I work who has cost the company several weeks unpaid development time in the last year, due to not-entirely-truthful statements made to clients. this has included me having to learn, and make changes to, a product, which in any reasonable universe would have been end-of-lifed at least a decade ago.

          *metaphorically speaking, I'm sure none of my colleagues would ever touch the Bolivian Marching Powder, and any behavioural traits that might be associated with such are purely coincidental.

          1. Doctor Syntax Silver badge

            Re: Maintenence window, gosh how quaint

            Seeing the prospect of being caught between management and customer in that situation was part of the reason for leaving one job. It had been sold as a replacement for the existing customer's system. Having seen that and knowing the database design of our I could see it wouldn't be the drop-in replacement they seemed to be expecting.

        2. Dvon of Edzore

          Re: Maintenence window, gosh how quaint

          Ambassador Kosh says "Yes."

      2. GreyWolf

        Re: Maintenence window, gosh how quaint

        Public executions of the staff who sold it with lies also work quite well.

        1. Anonymous South African Coward Silver badge

          Re: Maintenence window, gosh how quaint

          Or a good old-fashioned tarring and feathering...

        2. A.P. Veening Silver badge

          Re: Maintenence window, gosh how quaint

          Public executions of the management who bought it with lies also works quite well and is a lot cheaper in the end.

        3. Anonymous Coward
          Anonymous Coward

          Re: Maintenence window, gosh how quaint

          A bit difficult when the person lying is the managing partner, and he's ridiculously oversold one's skills & experience

    2. dl1jph

      Re: Maintenence window, gosh how quaint

      To be fair, the problem in many cases isn't that it can't be done, but that it's far more expensive than the people making all the promises realized... Usually, the math on that only gets done after the promises are made.

  3. .stu

    I was expecting more sparks from the title :(

    1. El blissett

      I thought the loosened power lead was going to light up the networking room for sure, looks like Colin got let off easy.

      Honestly for getting as much done on his own without proper equipment as he managed, surely conflagrations are in order.

    2. jake Silver badge

      I was expecting our hero running across The Boss in flagrante delicto in the dark corner behind the servers ... It happens. It's soul scarring. Mind bleach doesn't exist. Whisky helps.

      1. Doctor Syntax Silver badge

        "Whisky helps."

        It usually does. Can you remember what the problem was?

        1. jake Silver badge
        2. short a sandwich

          Problem?

          In BOFH parlance this is an opportunity, especially if there is security video footage.

      2. ColinPa Silver badge

        Horizontal appraisal

        I heard from a customer who went into the machine room to find the (female) manager giving a "horizontal appraisal" to a member of staff as part of the year end assessment.

        BTW The engineers went in to the machine room in pairs.... for safety reasons.

      3. Anonymous Coward
        Anonymous Coward

        @jake

        This has nothing to do with the article, but your comment reminded me of an incident a few decades ago.

        I was a dietician/diet cook and had started work in a hospital. I can't remember what I was cooking but I needed something from the 'fridge. Said 'fridge was a big "walk in" ' fridge. On my "journey" to the 'fridge I ignored the couple of people who told me it was not a good time to go to the fridge. I assumed that as I was the new guy they were taking the piss.

        So I opened the door and was confronted with a bang. Actually, a more accurate description would be "banging". The company bike was being banged to fuck by someone (I never knew who, all I saw was his back and his arse doing about 3000 thrusts per second).

        I mean, what the fuck do you say when confronted with such a situation?

        I did the only possible thing. I forgot that I needed whatever and said, "sorry" (yes really) and turned around and shut the door. And then put up with weeks of people piss taking me.

        And Whisky is not good enough. I find Vodka way better. At least, that is my excuse...

        1. Anonymous Coward
          Anonymous Coward

          "I mean, what the fuck do you say when confronted with such a situation?"

          Next.

          1. Anonymous Custard Silver badge
            Trollface

            "I mean, what the fuck do you say when confronted with such a situation?"

            "Could you pass me the salt please?"

            Or perhaps "Ah yes, sausage would work with the dish" (or chipolata if you're feeling mean).

      4. Anonymous Coward
        Anonymous Coward

        In our old office, there was a tiny server room in the building's roof, which was more like a large cupboard. For some reason there was a mattress in there. I'm sure it was purely concidental that the network manager and one of his staff, both of whom were married to other people, were "at it"; it was a pretty well-known open secret.

    3. Anonymous South African Coward Silver badge
      Unhappy

      I thought so too, and expected a silent and dark server room, but was sorely disappointed.

      Is this a new type of clickbait?

    4. KarMann Silver badge
      Mushroom

      I was expecting much the same at first, but then, after a bit of reading, I thought it sounded more like there would be a server or switch falling off the rack, or the rack tipping over, resulting in the titular 'big bang', especially with that talk of 'tug[ging] at, if not the heart strings, then certainly the RJ45s.' And even that still didn't happen.

      Where is the Earth-shattering ka-boom? -->

      1. Anonymous Coward
        Anonymous Coward

        Did you know a wall of cables from a stack of 5 48-port switches will prevent ALL airflow through a 19' comms rack? Also, if you use separate colours for each type of server (esxi, pdu, hsm, etc), that the waterfall of cables looks like a 1960's bead curtain.

        anonymouse because while its been 9 years since this was done, its still in production. and while I don't go all zip-ties and velcro bands, neatness DOES count. for 6-8 port devices proper cable management means if you disconnect all of the ports to replace the hardware, they'll only reconnect, without straining, to the correct ports.

        1. DougMac

          Plus proper cable management.

          Always plan out your racks with enough.

          I can't imagine having to work through a waterfall of cables.

        2. Alan Brown Silver badge

          My instant reaction to _ANYONE_ waterfalling cabling in my racks is to remove it.

          I simply won't tolerate it and on a new site it's the first cleanup job on the list

          Cable management is there to ensure things can be swapped out quickly in an emergency and the horizontal row of rings below/above the switch isn't there for decorative purposes (nor are patch runs so taut that you can't trace a cable by tugging it and feeling the other end of the bundle - something picked up in telco days))

          It also means that for switch swaps as described, the plugs aren't going anywhere once removed and you have an opportunity to check the rear of the cabinet

          What do you mean you don't have a couple of trouble lamps?

          What really irks me about most large switches is the way they tend to require 2 people to install - one at the back to hold the rear up whilst guiding the unit into place. My solution to that is to use ventilated 1U "L" rails(*) (beware the solid ones. Most switches use sideways airflow, not front to back)

          * eg: https://www.network-cabs.co.uk/acatalog/1U-19-inch-Universal-Server-Rack-Rails---Adjustable-Depth----450mm-to-600mm-Fitting-AD1UADJSR_300.html

        3. Anonymous Coward
          Anonymous Coward

          "for 6-8 port devices proper cable management means if you disconnect all of the ports to replace the hardware, they'll only reconnect, without straining, to the correct ports."

          I haven't met one that might trip an issue at the 8-port size yet, but on 10, 12-port, or larger devices, that only works if said replacement device has its ports in the same physical configuration as the old, ie, only one row, or in two rows.

          Or if the idiot designers haven't shifted the bank of ports from one side to the other.

  4. Giles C Silver badge

    There are a lot of people who should never be let near a RJ45 patch panel.

    I have seen some bad ones the worse was a cabinet at a company my employee at the time had bought. The cabinet sat in a corridor and the patch leads extended in a pile of spaghetti which extended almost a metre in front of the cabinet. I upset the network person on site with my reaction.

    I said get a maintenance window and unpick each cable label them and it back beater.

    Mind you I have done the same pulling a cable and finding someone has wound something else round it or even worse used spiral wrap on a few cables.

    1. Arthur the cat Silver badge

      All networking aspires to the condition of macramé.

    2. John Brown (no body) Silver badge

      "The cabinet sat in a corridor and the patch leads extended in a pile of spaghetti which extended almost a metre in front of the cabinet."

      I can't beat that on amount of spillage, but maybe on time reach spillage. Network cab in a school. Not a big one, maybe 6-8U, about 8 feet up on the wall. The door will no longer close due to the mess of cables. This was only a few week after I'd installed it as part of an entire wiring up of the school. All wall points went back to the patch panel. All wall points were live as I'd carefully and neatly patched every patch panel port into the switches, everything labelled and colour coded. There was absolutely no reason I could fathom as to why someone had gone in and started re-routing cables. After all, we'd designed it all for them, got the sign-offs, set up all the proper VLANS and wired the correct ports to the correct VLANS etc. I can only imagine that the school technician had forgotten all that and possibly after some classroom re-arrangements had re-patched the cables rather then edited which ports were in which VLAN.

      1. Notas Badoff

        It *was* beautiful

        So part of finishing a good job of properly installing a network cabinet is to get pictures afterwards? And then before repairing after the passage of time, get pictures?

      2. Jellied Eel Silver badge

        My favorite place for spillage was the original Telehouse. Floor voids full, ceiling voids full, so trays laid inside the walls. The shrinking corridors were quite disconcerting.

      3. Alan Brown Silver badge

        " I can only imagine that the school technician had forgotten all that "

        You imagine he retained it in the first place, or that whoever was assigned the job even knew about VLANs

        The solution to that is to produce photos of the "before" and say "that's not how we left it and this isn't supportable, it'll cost to put right"

  5. jake Silver badge

    But did he learn the biggest lesson of all?

    In my opinion, learning to pick locks it's a "must learn" skill for any well-rounded sysadmin, network engineer, consultant, or whathaveyou.

    1. Korev Silver badge
      Coat

      Re: But did he learn the biggest lesson of all?

      You mean it's a key skill?

      1. John Brown (no body) Silver badge

        Re: But did he learn the biggest lesson of all?

        Or even a KPI for the next performance review :-)

    2. Anonymous Coward
      Anonymous Coward

      Re: But did he learn the biggest lesson of all?

      Or a big bunch of 'spare' keys in the bottom of the toolbox

      A few months back had an idiot manage to lock the keys to his desk inside the desk(*). Couldn't find the correct key on the bunch but found one that worked if you only partly inserted it then 'j-j-j-jjiggled it, Granville'

      (* he has a 'cash box' of vital USB sticks inside with the key on the same bunch. Instead of unlocking the drawer and pulling the key straight out, he unlocked the drawer, turned the key back to 'locked' before withdrawing it... and the drawer went 'click' as he nudged it closed)

    3. Prst. V.Jeltz Silver badge

      Re: But did he learn the biggest lesson of all?

      It was nice in the old days when these cabs had glass in the front. Presumably cooling wasnt a priority.

      the lock would generally just spin in the round hole in the glass door if required.

      1. Anonymous Coward
        Anonymous Coward

        Re: But did he learn the biggest lesson of all?

        Sod's Law says you get the only cabinet where the lock fits a notch that prevents you spinning the lock

        1. imanidiot Silver badge

          Re: But did he learn the biggest lesson of all?

          No, no, Sods law gets you a glass door with a notch which doesn't appear to prevent you from spinning it but when you do it puts pressure on just the wrong spot and the whole hardened glass door drops to the floor in tiny little bits

          1. A K Stiles

            Re: But did he learn the biggest lesson of all?

            Well, you then still have the access you need...

            1. imanidiot Silver badge

              Re: But did he learn the biggest lesson of all?

              True, but you'd have to either work standing in glass (which contrary to popular belief about safety glass is still feckin' sharp) or you have to spend a lot of time cleaning the debris up before you can get to work proper. And in either case you have to spend either time, money or both satisfying the boss/customer/client/landlord who's cabinet door you broke.

      2. Jellied Eel Silver badge

        Re: But did he learn the biggest lesson of all?

        Glass doors are so manglement can look at the leds. Important if you've modified your optics to reprogram anyone who gazes too long or too deeply.

        As for doors, sometimes ignoring the lock is the best approach. Doors often have spring loaded hinge pins top and bottom that can easily be persuaded to open.

        Ventilation gets more fun. So if AC's forced air underfloor, then you probably want fan trays in the top of the rack, with panels all round to direct airflow up & out. If it's a hot/cold aisle setup, racks should be set up for that. Becomes FUN! as racks fill up, holes in floor get blocked and doors get left off cos there's to much cabling to close them.

        But look on the bright side, most kit these days lets you monitor it's temperature.

        1. Dante Alighieri Bronze badge
          Coat

          *cough*

          blinkenlights

          in this vicinity please

        2. imanidiot Silver badge
          Windows

          Re: But did he learn the biggest lesson of all?

          Doors on cabinets are usually more to keep the rif-raf uneducated "the server is down so I must powercycle the server" yobbos out than to provide actual security.

          --> Those sorts of people. We all know them -->

          1. jake Silver badge

            Re: But did he learn the biggest lesson of all?

            Actually, those particular locked doors are to keep Bosses from flipping switches ... To keep the rest of the riff-raff out, we put locks on the doors of the server room (data center, whatever). After many decades, here's my list of folks who don't belong in Corporate Data Centers:

            Middle management.and higher (applies to management track folks only).

            Any corporate bod with a non IT related Doctorate.

            Most children of employees (there is nearly always an exception or two).

            Tour groups.

            Vendors.

            Janitorial staff/cleaners.

    4. vogon00

      Re: But did he learn the biggest lesson of all?

      Either pick the lock, or use the patented 'yale-latch-defeating-disposable-loyalty-card-round-the-frame' trick to get at the single cable in someone else's comms room that you desperately need to move on a Friday afternoon when they (Corporate IT) do not have anyone available to open the door *.

      *Except when the silent alarm goes off and two of the feckers turn up with 'security' in tow 15 mins later. I got away with this as my boss significantly trumped theirs, this ended a previously un-diagnosed and mysterious 6-week outage, and it was Corporate's fault anyway - the 'feckers' did some cable & equipment moves 6 weeks ago (Without telling anyone like wot they were supposed to!) that took a rather important Ethernet segment out to ~130M.. No wonder all them collisions were 'late' :-)

    5. Alan Brown Silver badge

      Re: But did he learn the biggest lesson of all?

      There are only 6-8 universal keys for most cabinets

      1. jake Silver badge

        Re: But did he learn the biggest lesson of all?

        I have one lockpick that'll open virtually all of them.

  6. Anonymous Custard Silver badge
    Alien

    Had he never seen Star Trek?

    "...and to always declare a much longer window than you think you need."

    A factor of four, according to the Scotty Principle

    1. Doctor Syntax Silver badge

      Re: Had he never seen Star Trek?

      Hofstadter's law. It always takes longer than you think even when you allow for Hofstadter's law.

    2. Fonant

      Re: Had he never seen Star Trek?

      I often use "multiply by two and increase the units by one":

      5 minutes -> 10 hours

      1 hour -> 2 days

      1 day -> 2 weeks

      1 week -> 2 months

      etc.

      1. heyrick Silver badge
        Unhappy

        Re: Had he never seen Star Trek?

        Sounds like the logic used by the maintenance crew around here. Changing a bolt that has worked loose? Five minutes work. Factor in all the farting around, risk analysis, toilet breaks, having a coffee or two, you're looking at hours to change a bolt. Ten hours? I don't think it's been stretched quite that far, but not for lack of trying.

        1. TomPhan

          Re: Had he never seen Star Trek?

          "just for my records, can you tell me which moon of Neptune you're based on?"

      2. TRT Silver badge

        Re: Had he never seen Star Trek?

        If we did it by the book like Lt Savik, hours could seem like days.

    3. chivo243 Silver badge
      Go

      Re: Had he never seen Star Trek?

      Aaah, Laddie you didn't say how long it would really take?!!

    4. cookieMonster

      Re: Had he never seen Star Trek?

      I used to roll two dice (or is it die?), anyway, and depending on what the task was , the numbers represented hours or days, worked like a charm for a number of years.

      1. TRT Silver badge

        Re: Had he never seen Star Trek?

        How many years???!!!

        It's dice. Die is singular.

        But I always believed the railway running each day was determined by a modified version of Owzat! With minutes delay on one bar and if you scored an Owzat! Roll the other bar marked with Bridge Strike, Persons on the Line, Derailment, Signalling Problem, Trackside Fire and "Leave 'em Guessing"

  7. MiguelC Silver badge

    Re: "as a prime example of why you should never go 'big bang' with a swap-out like this, not to work onsite alone, and to always declare a much longer window than you think you need."

    Whenever you're calculating a time estimate you always - ALWAYS - put a healthy extra for unforeseen events. We call it the fear factor (that you use to multiply your real estimate by)

    1. jake Silver badge

      It's not fear. It's pragmatic.

    2. Anonymous Coward Silver badge
      Thumb Up

      I'm actually quite open about this these days. I go with "it might be done in x, but allowing for unforeseen issues, we should allow y"

      It leaves the customer with some hope that things will be back up somewhere between the two but leaves plenty of wiggle room when things go wrong.

    3. Jellied Eel Silver badge

      It's one of those things you learn by experience.

      So switches have lots of ports, which means lots of cables. Which then usually means a rats nest of unlabled cables. And may also include some 'spares', or faulty cables that have been left dangling. There'll probably also be at least one with a dodgy crimp, and some devices will keep their Ethernet ports shut down after LoS.

      You need access front and rear. Trying to thread a switch through a nest of cables is never fun, and increases the risk of dislodging or damaging cables. Having some form of console or syslog access can help spot devices going down when they shouldn't.

      Then for time allowance, always include enough time to roll back the change, and when to abort.

      And ideally, always use patch panels, never cable directly to a switch.

      1. Loyal Commenter Silver badge

        And ideally, always use patch panels, never cable directly to a switch.

        If I could upvote you more than once for this alone, I would.

        Decent software engineering abstracts things into layers, which can be treated as modular and swapped in-and-out. This is the hardware equivalent, and should be standard practice. It's the difference between having your car's wheels bolted to the axle, and welded on.

        1. TRT Silver badge

          Or use one of those spring loaded patch bays.

    4. Snake Silver badge

      "Whenever you're calculating a time estimate you always - ALWAYS - make sure you have full access to all devices by having keys in hand for all locked cabinets"

      FIFY

      It 100% was not Colin's fault. You assigned him a project yet did not grant him access to all tasks the project could entail - you failed to give him the keys to the rear of a server panel.

      Then tried to blame him when he needed said access but could not acquire it.

      Blame the management, not the worker. Forcing a mechanic to work on your car but then saying "You can't use the 10mm socket" will NOT get you guaranteed results.

  8. Anonymous Coward
    Anonymous Coward

    red alerts

    Good preparation, but a number of red alerts were ignored, here.

    "I felt some resistance and pulled gently since the RJ45 jacks on Cat 5 cables can take some strain without damage, and all the cables were out anyway..."

    Red alert one: if there is resistance and you can't explain it right now, stop, and only restart until you're confident it has been dealt with.

    "The Data Centre helpdesk team had not shared their step ladder due to "health and safety""

    Red alert 2: if you don't have every tool you'd potentially need, refuse the operation.

    "Colin hadn't been given the key to get into the rear of the rack."

    Red alert 3: like #2

    Transformation is about sorting out years of mess, so you'll always step onto some crazy temporary "fix" from a long time gone permie or contractor.

    It's really like walking on a mine field. Every bizarre thing counts.

    1. Martin Silver badge
      Happy

      Re: red alerts

      Come on, be fair. Colin was young. When you're young, you think red flags are for decoration.

    2. Gene Cash Silver badge

      Re: red alerts

      Also, red alert #4, it sounds like the first time he looked at the setup was when the maintenance window started.

      So apparently he agreed to do something and set a maintenance window time without even eyeballing the situation.

      There's lots of times when "it's a simple printer switchout" has turned into something like "the printer has been permanently installed in a custom built alcove" or the "printer is something from the '70s that uses neither USB, parallel, or other bus known to modern man, and has a custom interface board."

      So there's been times I've have to NOPE the hell out, and agree to fix it on a "best effort" basis.

    3. Prst. V.Jeltz Silver badge

      Re: red alerts

      Red alert one: if there is resistance and you can't explain it right now, stop, and only restart until you're confident it has been dealt with.

      I'd say thats more of a Bronze alert. Or mauve.

      Even if it does mean changing the bulb.

      A subtle jiggling of the unit at "below plug releasing levels" will give you geographic indication of the issue.

      1. phuzz Silver badge
        Unhappy

        Re: red alerts

        The trouble with IEC13 plugs (ie kettle leads) is that they sometimes manage to work their way out of the socket, to the point where something as gentle as another cable brushing past it can break the connection.

        I managed it once while just swapping some network cables into a rack, one knocked a power lead in a blade enclosure.

        Now, this enclosure did have multiple redundant power supplies, but the one that ended up drawing all the load was bad, because it immediately popped, and the remaining two couldn't handle the load and shut down, taking all twelve blades with them.

        All of this was silent, so the first I knew about it was my boss bursting into the room, frantically asking what I'd touched.

        1. Alan Brown Silver badge

          Re: red alerts

          "they sometimes manage to work their way out of the socket"

          More often than not it's dozy pricks not pushing them fully home in the first place.

          I've never seen one work its way out unless snagged

      2. Alan Brown Silver badge

        Re: red alerts

        I've seen it happen often enough that I insist on using self-anchoring cord sockets. These solve a LOT of aggravation later on (think: someone fumbling in badly lit back of rack knocks out black power lead from black socket in a dark environment. Do you think they'll notice it?)

    4. Alan Brown Silver badge

      Re: red alerts

      Agreed.

      The moment the stepladder was denied I'd cite my own Helath and safety plus "local obstructions" as a reason for cancelling the job (and charging for the hours spent onsite or XYZ minimum charge)

      When manglement get a few bills like that, they tend to start asking pointed questions of underlings

  9. ColinPa Silver badge

    Talking about cables

    Off topic...ish

    A shipyard in Scotland is trying to build two ferries. They are over budget and late. The latest problem is the data cables are not long enough.

    The cables were long enough when they were ordered a couple of years ago, but someone decided to move the displays and instruments, so the cables are no longer long enough. Swapping the cables around is not an option, so even more will have to be ordered.

    The obvious answer would be to move the displays back again - but that's another can of worms.

    1. Anonymous Coward
      Anonymous Coward

      Re: Talking about cables

      I know someone who carefully specified 'line termination should be within 2m of the router' because the Cisco routers came with a 6ft(1.8m?) cable for free.

      He learned his lesson (order 5m cables, just in case!) after discovering that at more than one site the line termination had been mounted half way up the wall opposite the comms cabinet because the line guy had thought "I'm about 6ft fingertip-to-fingertip, and it can touch both the line termination and the router!"

      It was obviously quicker to pick up a few 5m cables from PC World than argue with BT/Virgin/whoever(*) to get them to re-site the termination somewhere sensible

      (* Virgin were taken off the list of 'approved suppliers' because... well, life's too short!)

      1. Alan Brown Silver badge

        Re: Talking about cables

        > (* Virgin were taken off the list of 'approved suppliers' because... well, life's too short!)

        If you think that's bad, try getting an outage fixed when Virgin and BT are pointing fingers at each other, so won't go out in less than 24 hours (on a 24*7 4 hour onsite contract)

    2. H in The Hague Silver badge

      Re: Talking about cables

      "but someone decided to move the displays and instruments, so the cables are no longer long enough. "

      Thanks for the info. I'd read about the issue and wondered what caused it. Mind you, that whole project seems to be jinxed.

      1. Ian Johnston Silver badge

        Re: Talking about cables

        Mind you, that whole project seems to be jinxed.

        True dat. The sunk costs fallacy kicked in years ago: it would now be far, far better and cheaper to sell both hulls for scrap and get someone else (the Poles, probably) to build a couple of new ferries, but far too much political capital has been invested.

    3. Ian Johnston Silver badge

      Re: Talking about cables

      Are you sure about the display movements? I haven't seen that reported anywhere. As I recall, they have already had to replace all the power cables in the Glen Sannox once because Ferguson's put them in out-of-order when they were waiting for other problems to go away.

      The only good things about the ferries clusterfuck are (a) they give us all a good, if expensive, laugh and (b) they make the Edinburgh trams project look pretty good.

  10. Grundleberry

    Learning Excercise

    Look at it this way, the company has now just realised that their biggest client has no redundancy. They should be thankful it was only a cable that was unplugged and not the switch being busted with some of the lead times coming from people like Cisco now, we're getting quotes of delivery in a year now.

    1. Anonymous Coward
      Anonymous Coward

      Re: Learning Excercise

      I suspect there's plenty of Huawei kit around now :).

  11. Anonymous Coward
    Anonymous Coward

    You just know what's coming...

    I read as far as "...And ever so gently, he began sliding the old switch out of the rack..." and immediately knew what was coming next. How did I know? Well, um, *cough*, tra-la.... oh look, a butterfly!

    A/C b/c Obviously

  12. Paul Hovnanian Silver badge

    Nip it in the bud

    I wasn't directly involved with this. But a new electric substation was built at a utility I used to work for. Racks were installed in the control house and the electricians dutifully installed the protection relays and wired them per the provided diagrams. But they failed to leave enough slack in the cabling or utilize the (provided) hinged cable brackets which would allow the equipment to be rolled out for maintenance on its slides.

    The relay technicians came to the job site next to calibrate and commission the equipment. Upon finding that the cabling had not been installed properly to facilitate future maintenance, they proceeded to take side cutters and cut every cable in the racks. And then call the electrician crew back to do it right this time.

  13. Boris the Cockroach Silver badge
    Pint

    Ahh the forest of cables

    and labels.

    Every cable tagged with a code number , C93 for the arm overtravel sensor for example.

    So when the arm overtravel alarm goes off, it must be a doddle to swap the sensor out.... except the sensor is fine and cable is blown somewhere.

    Open the control box...... C93........... C93 .............. C93............ ahhh there it is.... coupled to the 24 volt DC supply ......... eh?

    OK resort to the wiring diagram......... C93 .......... C93 ............. C93............. theres a C92.. but that says its on the HP sensor..... but its wired to Wrist motor encoder line on the arm.....<looks on the arm> that says D12 and the manglement are waiting .. oh so waiting..... with the looks.... the glances..... the wrist tappings.....

    resorted to tying a string to the cable , pulling it out, and seeing where it was realy plugged in... A47....

    Beer because I needed one

    1. TomPhan

      Re: Ahh the forest of cables

      All of our PCs have a non-removable sticker with the host name in big friendly letters, so if there's an issue the user knows what machine they're on. And if the PC isn't accessible, for example inside a kiosk or behind a wall mounted TV, then the sticker is put on those.

      Except the workflow changed, and now all the stickers are put onto the PCs when they're received, so we now have to keep look-up tables saying what they used to be going back three generations.

      1. Loyal Commenter Silver badge

        Re: Ahh the forest of cables

        My work laptop didn't have a label on, so I stuck one on there. I was forever having to go and look up the host name (seemingly random alphanumeric string) whenever I needed to put the local SQL Server instance name into a connection string to test something (and no, "localhost" doesn't work in all situations). I got fed up with it enough to get the label printer out from the back of a drawer.

    2. Anonymous Coward
      Anonymous Coward

      Re: Ahh the forest of cables

      I remember a guy getting some Saturday overtime to hardwire some RS232 connections (~10 wires per connection) to a remote test desk

      Come Monday we ran a few checks but just got weird results. Looked at the tag block in the back of the test desk, expecting to see a neat pattern just saw a jumble of colours... same at the other end of the cable

      NOW he tells us he's colour-blind!

  14. jpennycook
    Pint

    This reminds me of the time when I was working late when my colleagues and boss had gone to the pub. I was just about to leave to join them, and when I walked past the server room I could hear lots of UPSs complaining. I went in the server room, traced the cables back and found far too many devices were powered from a single 13A mains plug, which had partially melted. After reorganising the mains cables so they used more than one socket, I was then very late for the pub, and no-one seemed bothered that there could have been a fire or something. If I didn't have such sensitive hearing and wasn't so dedicated, I could have had more beer (and probably had to perform the BCDR process the next day).

  15. TomPhan

    Service Windows

    Ours can only be between 9PM to 5AM, and you have to state in advance what the start and end times will be. Fine, except recent reorgs meant many people now only work 8AM to 5PM, and there's some systems where nobody can make any changes because of this.

    1. uccsoundman

      Re: Service Windows

      Or better yet, the only person who either had permissions and/or knew how to do the maintenance quit long ago because the company across the street was offering 2X the salary and was offering benefits like not being on-site 24/7/365 because you were the only person on staff. And BTW, he was the only one that knew the key-codes to get in the server room door and the only one who knew how to change them. So, as time goes on, the company has less and less access to its computers.

      Or, how about maintenance is required but nobody knows where the machine is? Indeed in one case they didn't even know what CITY the machine was in.

    2. NITS

      Re: Service Windows

      I get to the customer's site to troubleshoot something in their control room, a.k.a. IT closet. Hard start, 0600 hours (Eastern, AKA GMT-0500). Not allowed to be in the control room after the store opens at 0800. I call the number I was given for their help desk, and get a recording along the lines of "Your call is very important to us. Please leave a message, and we'll get back to you as soon as we get into the office, at 0900 Pacific". Which would be noon, Eastern time.

      Aarrggh!

  16. Anonymous Coward
    Anonymous Coward

    Beware moving cables

    I remember doing a cabinet tidy up which involved upgrading some servers. All went well and as we were also rationalising cabling, we decided to move one server to a different network port. This was not part of the plan, but all was working just fine when we packed up for the evening.

    Fast forward to the next day - some, but not all client machines on part of the network were falling offline.

    It took some time to realise that the reason was the port we'd moved the server to was untagged on the correct VLAN but also tagged on several others.

    The DHCP server on this machine didn't understand VLANs and was responding to every DHCP request both on its own VLAN and all the others too.

    As there was a DHCP server on the other VLAN this explained why only some clients were failing. Some swift switch reconfiguration fixed the issue and that server's DHCP was fixed some time later.

    1. Muppet Boss

      Re: Beware moving cables

      Long time ago, during one migration the main DHCP server was switched off and on again. Turned out to be running the DHCP client itself, hence not getting the IP address and of course no one knew the local password. Like someday it got the IP address via DHCP, kept it after the old DHCP server was gone, eventually got promoted to the DHCP server role and no one spotted anything until the reboot who knows how much later. A temp DHCP server quickly got it back online but the whole situation was quite hilarious. It was probably seen too important to be rebooted at all.

  17. Conrad Longmore
    Alert

    Elfin Safety

    Years ago when I was working (mostly) desktop support, facilities wouldn't allow IT to have a trolley because we might break something by bumping into it. So we had to carry everything ourselves, back in the day when desktops, printers and of course servers could be massive. LaserJet IIISi printers were a favourite at over 40kg each. Apparently IT staff were more expendable that the doors and furniture.

    1. Alan Brown Silver badge

      Re: Elfin Safety

      " facilities wouldn't allow IT to have a trolley because we might break something by bumping into it"

      I've solved that one in the past by invoking elfin-safety and insisting that facilities staff carry the heavy stuff instead of risking expensive technician injuries

      The moment they started using trollies it got photographed. Of course it was all about "demarcation" and some petty manager pissing over everything to maintain his empire. That's fine but if it now takes 3 weeks to move heavy items, that manager will be the one who frustrated end users (and higher manglement) get directed to.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022