back to article If we plug this in without telling anyone, nobody will know we caused the outage

What? Monday? Again? Didn't we do Monday last week? OK, fine. Welcome once again to your soft landing pad into the working week – the oasis we call Who, Me? in which Reg readers unburden themselves with tales of tech tribulations. This week, meet a reader we'll Regomize as "Padraic" who spent some time in the 1990s working in …

  1. AustinTX
    Flame

    Let's Check the Server Room Access Log

    Someone is going to figure out that the problem started while these guys were in the server room and then later stopped when they went back!

    1. biddibiddibiddibiddi Silver badge

      Re: Let's Check the Server Room Access Log

      They didn't bother to clip the SCSI connector in, and couldn't detect that a drive/device was disconnected during who knows how long a period of frantic troubleshooting, so I'd only give it 50/50 odds at best that there was a server room access log.

      1. Aleph0
        Joke

        Re: Let's Check the Server Room Access Log

        Or the access logs were themselves saved on the downed storage device ;)

        This story reminds me of the old saying "SCSI isn't magic. There are fundamental technical reasons why it's necessary to sacrifice a young goat to your array every new moon"

        1. Sam not the Viking Silver badge
          Pint

          Re: Let's Check the Server Room Access Log

          My recommendations of a ritual sacrifice have never been approved.

          On projects which become 'awkward' I undertake a procedure where mantras are invoked to aid the excision of demons. Those nearby think I am just muttering..... I leave a very small text file to maintain protection and also make an appropriate libation ---->

          1. Doctor Syntax Silver badge
            Devil

            Re: Let's Check the Server Room Access Log

            Don't confuse demons with the daemons that make it all work.

            Icon - closet available to BSD logo.

            1. Zippy´s Sausage Factory
              Joke

              Re: Let's Check the Server Room Access Log

              Never confused daemons with demons. I did that once, and my hard disk was haunted for years. That said, the house did ring with demonic laughter at precisely 7.33pm every day for years.

            2. Benegesserict Cumbersomberbatch Silver badge

              Re: Let's Check the Server Room Access Log

              Nor let us confusion excision with exorcism - we can cut that out right now!

              1. Zippy´s Sausage Factory

                Re: Let's Check the Server Room Access Log

                Confusing exorcise with exercise is also not a good idea. And that's why I'm banned from the local gym.

          2. Eclectic Man Silver badge
            Happy

            Re: Let's Check the Server Room Access Log

            Sam not the Viking Those nearby think I am just muttering

            Well, I guess that kept them from bothering you while you solved the problem. Des it get you an unoccupied seat next to you when travelling by bus / rail /air?

        2. swm

          Re: Let's Check the Server Room Access Log

          In the 1960's at Dartmouth we were running time-sharing on some GE computers. Every so often something would fail and the technician would run diagnostics and find nothing wrong. Bringing up the system we couldn't find anything wrong either. The technician caught on and ran diagnostics every morning. When asked, he said he was warding off evil spirits. Reliability was much improved.

          One day his diagnostics failed but time-sharing ran perfectly. Further checking revealed the failure was in the "bit change zero" instruction that converted EBCDIC to friden flexowriter code or something like that. We couldn't figure out a use for this instruction so we didn't use it.

      2. aerogems Silver badge
        Coat

        Re: Let's Check the Server Room Access Log

        The odds are exactly 50/50. Either they had access logging or they didn't.

    2. IGotOut Silver badge

      Re: Let's Check the Server Room Access Log

      Access logs in the 90s?

      You mean the mechanical code lock on the door....if it was even that secure?

      1. HandleBaz

        Re: Let's Check the Server Room Access Log

        They probably had a corporate open door policy.

      2. disgruntled yank Silver badge

        Re: Let's Check the Server Room Access Log

        Back in the early 1990s, I worked on a US government contract. The server room access log was ring binder with forms in which you might write your time of arrival, your task, and your time of exit--office buildings commonly have such at the front desk. The boss contractor mentioned as an additional reason for their use that they could get one out of trouble: somebody accused of smoking pot in a stairwell was shown as have been in a server room at the time of the alleged offense. I don't think that I was particularly conscientious about filling them out.

        1. Rich 11

          Re: Let's Check the Server Room Access Log

          I don't think that I was particularly conscientious about filling them out.

          Too stoned to remember...

        2. Shalghar Bronze badge

          Re: Let's Check the Server Room Access Log

          The non electric multimode analogue data system (pen/cil and paper) is still used.

          Last time i experienced the pure paper variant was in a railway maintenance facility in Solna (near Malmö/sweden) in 2019.

          The main reason for the use of paper was that if there was a fire or power outage, any kind of emergency where you need to make sure everyone including contractors is out of the buildings, a computer system that might be incapacitated by said emergency wont help much.

          Airbus, Mercedes and Volkswagen are using electronic access logs, the only pen and paper play is the signing of that ridiculous GDPR paper where Airbus is giving out pens with digitizer cameras to capture your signature.

          Metal working facilities and chemical plants also tend to use paper alongside electronic access logging, so it seems that any plant where something really big and nasty might happen does not like to rely purely on electronic victim records.

      3. Why Not?

        Re: Let's Check the Server Room Access Log

        server room?

        My first one in the late 90s was at the back of the stationary cupboard, key at reception. It was an improvement because when I arrived we had an AS400 dumped on the service dept floor, imagine how many toolboxes etc were rested on it.

        In the early 90s Very few of the server rooms I visited in London had logged key card access, only the big market makers, traders and technology companies. Enron oddly enough had one!

        1. biddibiddibiddibiddi Silver badge

          Re: Let's Check the Server Room Access Log

          The managed service provider (third-party IT for SMBs) I started with in 2012 had their one small server hanging from industrial hooks in the bathroom. It did get moved to a shared rack in a closet with a co-tenant in another building when we moved at the end of that year.

        2. Excused Boots Silver badge

          Re: Let's Check the Server Room Access Log

          Even now, you’ll be surprised of how many companies simply have no concept of the importance of their servers and other network infrastructure and simply stick them wherever. Even when a purpose designed ‘room’ is set aside for servers, they have barely settled in before said room becomes a general storage facility for stationery, bicycles (don’t ask), etc.

          There was one client of mine some many years ago who had their servers and network kit in a rack in a nice air conditioned room, but they were a magazine publisher and as such often got sent ‘samples’ of various products for review. One day we found the server rack entirely surrounded with boxes and boxes of perfumes and deodorant sprays. I’ll always remember my colleague at the time telling the office manager that ‘if they ever have a fire, they’ll find bits of your fucking servers on the moon!’

          Once, once, I managed to get it done properly. An advertising/marketing company were moving into a complete new build facility and I managed to get on the design team. I wasn’t interested in the general office payout, but they had reserved a space for ‘the server room’. I managed to get this space divided into two, one part was just big enough to get the four 42U racks in, with sufficient space front and back to install equipment. The other part was much larger and was intended as a ‘build area’ / storage area for IT. Guess what happened to the outer room?

          The inner one, being far too small and narrow to put much else in, got left alone. Plus I kept changing the combination on the door lock, so it didn’t take too long for everyone to get the message.

    3. Doctor Syntax Silver badge

      Re: Let's Check the Server Room Access Log

      System stopped while they were there but continued when they weren't & started when they were there. Problem doesn't correlate well with their presence. Couldn't be them.

    4. Dabooka
      Joke

      Re: Let's Check the Server Room Access Log

      Lol, check the 'access log'

      Good one that :D

    5. Just Enough

      Re: Let's Check the Server Room Access Log

      This is why some of these Who Me? stories, while being very amusing, I take with a pinch of salt. Any sysop faced with this situation is immediately going to identify the guys working in the server room as suspects number one, and go check the hardware.

      1. NorthIowan

        Re: identify the guys working in the server room as suspects

        Yes, I remember he said they were drilling holes. I was expecting a destroyed cable in the wall.

      2. gnasher729 Silver badge

        Re: Let's Check the Server Room Access Log

        Only if you suspect the hardware.

        I worked at a place where one day every wax in panic because continuous builds didn’t build. It took two hours trying to get it working until someone decided to go to the server room.

        They found a window broken open, and no server.

        1. Killfalcon

          Re: Let's Check the Server Room Access Log

          That's the problem with remote servers, you stop thinking of them as hardware in the first place. Even worse is "cloud" stuff where everyone remembers the decade old sales pitch of virtual machines being seamlessly passed between servers without ever impacting client experience, yet, inevitably, your "cloud application" has been running in a single unchanging rack in a single warehouse somewhere in Dublin for the last six months, and won't move without an outage.

          1. Jou (Mxyzptlk) Silver badge

            Re: Let's Check the Server Room Access Log

            I don't get your complaint. That is actually good! Why move a VM which is running fine to another host for no good reason?

            1. Zippy´s Sausage Factory

              Re: Let's Check the Server Room Access Log

              Because cloud! It saves money because we don't have to pay techies! And we don't know where our data's physically hosted so we can use that as an excuse on data protection enquiries*! Everybody wins!

              * didn't Facebook actually do this once, I seem to remember?

      3. Excused Boots Silver badge

        Re: Let's Check the Server Room Access Log

        “ This is why some of these Who Me? stories, while being very amusing, I take with a pinch of salt. Any sysop faced with this situation is immediately going to identify the guys working in the server room as suspects number one, and go check the hardware.”

        True, except that assumes that a) the Sysop is competent, and b) is actually aware that someone has gone into the server room.

        Although, in theory the sysop, or whoever has that title, ‘should’ be aware of what is going on, in reality, often not so much. The server room / general stationary store / coat drying room / bicycle shed, might have more traffic going in and out than a major railway station!

  2. Anonymous Coward
    Anonymous Coward

    The sure fire 10p investment...

    Ah the 10p saving that ends up costing thousands – classic. If you've ever found yourself in a mess that could have been prevented by some spare change, let us know about it

    Buts lets exclude 10p invested in a firearm round. We all have stories that could have been best avoided by that simple investment.

    1. diguz

      Re: The sure fire 10p investment...

      let me guess... you're from the western side of the Atlantic Pond.....

      1. Anonymous Coward
        Anonymous Coward

        Re: The sure fire 10p investment...

        TBF there are a few lads on the Emerald Isle who favour a calibre based coercion approach

      2. Ghostman

        Re: The sure fire 10p investment...

        let me guess... you're from the western side of the Atlantic Pond.....

        I don't know of anyone on the correct side of the pond to use the phrase "10p". Purely a Brit saying since here we say "10 cents". We don't have pennies, we have cents.

        1. Killfalcon

          Re: The sure fire 10p investment...

          Canadians have pennies!

          1. Sherrie Ludwig
            Headmaster

            Re: The sure fire 10p investment...

            Canadians have pennies!

            The Royal Canadian Mint stopped producing and distributing pennies in Canada as of February 4, 2013 due to rising costs relative to face value and the significant handling costs of the penny for retailers, financial institutions and the economy in general. Reference Canadian Bankers Association.

            1. David Hicklin Silver badge

              Re: The sure fire 10p investment...

              > Royal Canadian Mint stopped producing and distributing pennies in Canada as of February 4, 2013 due to rising costs relative to face value

              Same happened to the UK half penny (decimal), amazed our 1p has not gone yet

    2. heyrick Silver badge

      Re: The sure fire 10p investment...

      "We all have stories that could have been best avoided by that simple investment."

      WTF?

      I don't. I'd bet that there are many people living in civilised countries who haven't had any need in their lives to escalate any problem to one involving lethal force.

    3. Sorry that handle is already taken. Silver badge
      WTF?

      Re: The sure fire 10p investment...

      What in hades possessed you to write and then post that comment?

      1. Anonymous Coward
        Anonymous Coward

        Re: The sure fire 10p investment...

        Yes, that comment would have been avoided by a simple investment in a USD 0.10 rubber some years ago...

  3. DS999 Silver badge

    Ugh I hated SCSI cables

    The OG SCSI-1 Centronics style connectors with the clips were fairly secure (if they were properly clipped) but they went away when SCSI went wide. So by the 90s those had fallen by the wayside for wide/ultrawide SCSI using the high density connector which was a nightmare!

    I can't count the number of times that intermittent problems could be traced to a dodgy 68 pin UWSCSI cable that had one pin that was slightly bent or had been pushed in a bit, so that that it didn't make solid contact. It could be properly connected and have the screw things on the sides actually screwed in (which were another problem altogether...) but still be in a situation where wiggling the connector or pushing it hard up or down or in could make an intermittent problem go away. But heat cycles or the slightest bump of the cable would undo that magic and it would again become intermittent. The only real fix was to trash the cable in favor of a new one, and hope that once it is plugged in it is never moved - because repeated plug/remove cycles were what created the bent/pushed in pins in the first place!!

    I'm glad I got into storage consulting when fibre channel was going mainstream, so I only had one contract where I ever had to deal with servers connected to a Symmetrix with SCSI! Unfortunately JBODSs were still attached with SCSI so you still had to deal with those terribly designed 68 pin HD connectors for server boot drives/scratch storage.

    1. IGotOut Silver badge

      Re: Ugh I hated SCSI cables

      Ahhh the bent pin nightmare. I had a custom long nose plier that I'd ground the end into a flat nose, specifically so that I could get in and straighten them back up.

      1. Kevin Johnston

        Re: Ugh I hated SCSI cables

        I kept a wirewrap tool for exactly that purpose. It was unusual to find a plug where I could not gently slide the tool between the surrounding pins and straighten any bent ones.

      2. J.G.Harston Silver badge

        Re: Ugh I hated SCSI cables

        Yeah, I had one, until somebody "borrowed" it to tighten some bolts and destroyed it.

        1. The Oncoming Scorn Silver badge
          Windows

          Re: Ugh I hated SCSI cables

          I recently had to straighten a bent pin(s) on a VGA cable for the first time in years.

    2. Will Godfrey Silver badge
      Coat

      Re: Ugh I hated SCSI cables

      I'll see your 68 pin SCSI and raise you a fully populated 120 pin EDAC - complete with a lovely tangle of A/V cabling.

      1. I Am Spartacus
        Coat

        Re: Ugh I hated SCSI cables

        I'll see your 120 pin EDAC and raise you the complete back plane of a VAX 11/750, when the massbus started playing up. Me (sysop for the VAX) and a Cray engineer (very bright guy) spent a happy early morning with a pulse mode oscilloscope trying to figure out why the VAX occasionally crashed the Cray it was connected to. Yup, two bent pins on the backplane. Feeling of achievement when we tweaked a pin by less than a mm and everything was rosy,

        Mines the one with the HP Oscilloscope manual in the pocket.

    3. Anonymous Coward
      Anonymous Coward

      Re: Ugh I hated SCSI cables

      > trash the cable

      Cut in it half first! So someone doesn't fish a "perfectly good" cable back out of the trash and "recycle" it.

      Not that I've ever recognized a bad cable I trashed just yesterday and am troubleshooting again...

      1. Eclectic Man Silver badge

        Re: Ugh I hated SCSI cables

        You don't need to cut the cable in half - just hit the pins on each end with a 10oz hammer.

        You're welcome!

    4. Jou (Mxyzptlk) Silver badge

      Re: Ugh I hated SCSI cables

      This is what I love SCA connectors for. One plug, mechanically much better, done.

    5. ShortLegs

      Re: Ugh I hated SCSI cables

      Never ever ever had a problem with SCSI. Ever. Possibly because we insisted on using high quality full wired and shielded cables.

      And I wonder if the fundamental reason was really "missing a connector" as cables came with connectors as standard

      - 50pin SCS-2 centroncs interface on the equiomennt had two clips to lock onto the cable.

      The female version had the same built onto the cable

      - 68pin SCSI-3 *aka Wide-SCSI" had a spring latch built into the cable connector

      -DB25 abortion used a standard screw connector

      So... no clips, 10p or otherwise. So either

      - someone hadnt fitted correctly

      - something had been broken and nor replaced

      - someone disconnected a cable..............

      1. DS999 Silver badge

        Re: Ugh I hated SCSI cables

        I assumed the article was talking about the Centronics style with the clips and someone had just neglected to engage the clips. That was often ignored, just like the "screws" on the side of the DB25 style or 68 pin UWSCSI cables were ignored (the ones with spring latches were worse than the ones with the screws assuming the screws were properly engaged which was its own problem given that depending on location you might not have room for a very long screwdriver or any vision of what you're trying to screw)

  4. Lazlo Woodbine Silver badge

    At my last job, we had to be very, very careful when working on one of our racks, because one of the fibres didn't quite fit soundly in the back of one of the servers.

    We all knew this, so we never had a problem.

    Well, we didn't until the aircon engineer visited.

    The aircon engineer who dropped his screwdriver.

    The screwdriver that just happened to graze that exact fibre as it fell to earth.

    Cue lots of frantic teachers calling because they'd lost access to all the vital powerpoints held on that server...

  5. ComputerSays_noAbsolutelyNo Silver badge
    Holmes

    Philosophical question:

    If the fasteners would have avoided the outage, is it really Padraic's fault?

    ... or is it the fault of the people who failed to install the fasteners, albeit with a really long lead time?

    1. Fred Daggy Silver badge
      Windows

      Re: Philosophical question:

      THAT! Would be an ecumenical matter.

      1. Gene Cash Silver badge

        Re: Philosophical question:

        There's never enough holy water to drown them all...

      2. Anonymous Coward
        Anonymous Coward

        Re: Philosophical question:

        Drink! Girls! Feck! Arse!

        1. Bebu
          Pint

          Re: Philosophical question:

          《Drink! Girls! Feck! Arse!》

          You stikl with us Father Jack?

          1. Uncle Slacky Silver badge
            Windows

            Re: Philosophical question:

            I'm not still on that feckin' island????

  6. Bebu
    Windows

    Ethernet AUI were a pita too.

    I think I read that one of ethernet's inventors (Bob Metcalfe?) greatest regret was the aui connector - the retaining sliders I think.

    The fairly heavy media adaptors were always falling of the back of machines.

    Fortunately once twisted pair became de rigeur the e-waste box of old cards and adaptors filled quickly. :)

    1. Adrian 4

      Re: Ethernet AUI were a pita too.

      Twisted pair followed in the tradition of AUI by having an easily broken retaining latch.

      1. Peter Gathercole Silver badge

        Re: Ethernet AUI were a pita too.

        Fixed with so called "snagless" hoods that prevented the tab catching on other cables as you tried to untangle the knitting at the back of switches and servers.

        Of course, sometimes these prevented the connectors fitting into recessed ports on the back of some servers.

        I hated the AUI slide on older systems. Where ports were close together, you needed a screwdriver or some other stiff, thin object to get to the slide to undo it. I'm sure that some of them 'fell' off because someone couldn't get their fingers in, and just pulled.

        In case you had forgotten, the cables that came out of the AUI were very heavy. They needed a quite substantial slide clip to prevent the weight of the cable pulling them out of the back of the system!

        1. I could be a dog really Silver badge

          Re: Ethernet AUI were a pita too.

          Ah, but the snagless ethernet cables came with their own hazards.

    2. Antron Argaiv Silver badge
      FAIL

      Re: Ethernet AUI were a pita too.

      I was at one of the IEEE meetings where AMP made a pitch for the slide latch.

      What a mistake. I hated those damn things. I have ONE memento of that era...an old SGI Indigo with an AUI connector on the back and an AUI to 10BASE-T adapter so I can hook it to my wired network.

  7. Anonymous Coward
    Anonymous Coward

    Reckon there's an AT&T Wireless engineer with a story like this from recent times?

  8. ocelot

    I kicked out the SCSI cable on a file server....

    Once upon a time I worked at a place where there was an Apollo Domain ring of workstations.

    For some reason the powers be that had fitted some external SCSI discs storing general user files to a slightly higher end tower form factor machine.

    It was sitting under a table in a general office space, and I stretched my legs out and kicked the cable..

    I was sitting at the machine because my simulations ran faster on that machine.

    Gradually some peoples files stopped saving .. as their machines network caches filled up as files ceased to be written any more..

  9. ocelot

    Oh and the smoke

    I once managed to push the 50 way SCSI IDC connector in upside down on my own machine despite the locating notch . The 5 volt termination power shorted to ground and the middle two cores of the cable from the computer to the disc drive went up in smoke.

    1. phuzz Silver badge

      Re: Oh and the smoke

      I've seen someone get a DIMM in the wrong way around, but not deterred by the notch being in the wrong place, they managed to jam it in hard enough to engage the latch on both ends.

      Kind of impressive in it's own way. IIRC it worked fine once the DIMM had been re-inserted correctly.

  10. aerogems Silver badge
    Facepalm

    Booooo!

    I was hoping the story would go in a somewhat different direction, like accidentally putting a mounting bolt through a cable or something.

  11. NITS

    BNC connectors

    Not inherently bad. Just certain kinds. The 75 ohm ones used in CCTV, mostly legacy now, I think. The modern compression type are pretty good. But I run into the older hex-crimp ones that were applied with the wrong tooling. They go bad (or were bad, just not detected). And there's a special corner in a very hot place reserved for anyone who thought that making, or using, the ones that don't crimp at all, they just have a female screw thread to capture the shield. I find those causing problems a lot. I can imagine the marketing speil: Easy to apply! Saves time! No tooling! No crimping! Just twist on and go! I think I remember seeing them in Radio Shack blister packs.

    Honorable mention: the DVR makers who jam the BNCs so close together that you (or at least I) can't get fingers between them to do up, or undo, the retention rings. I find them unlatched, and the plug having worked its way out, sometimes.

  12. CorwinX Bronze badge

    Aah scuzzy clips...

    ... memories.

    The barsteward things were so irritating because they were so essential.

    Every other connector on the planet didn't need no steeking clips.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like