back to article No, you cannot safely run a network operations center from a corridor

With a whole month of 2023 already consigned to history, The Register brings you another instalment of On-Call, our weekly column in which readers share their stories of past deeds performed in the service of keeping computers copacetic. This week, meet "Nick" who once ran the Japanese branch of a storage-as-a-service provider …

  1. UCAP Silver badge

    I remember once (about 25 years ago or so) having to go over to Japan to witness some factory acceptance tests (ultimately successful, but not without a major screw-up on the way), before ending up in Tokyo to witness a site acceptance test for a new satellite NOC. These latter tests where not successful in any respect; in fact I think that I failed them on just about every item we looked at. Basically they where woefully unprepared, and I suspect where expecting me to just sign on the dotted line rather than actually check things.

    One of things I realised from this trip, however, is to be careful not to make the situation worse when dealing with the Japanese. The loss of face was already considerable, there is no need to push them deeper into the quagmire of shame. Because of the way I handled things, they specifically asked me to come out a couple of months later to rerun the tests, with results that meant that not signing on the dotted line was out of the question.

    1. ColinPa

      Losing face

      I had to go to an Asian country to help diagnose a performance problem - they wanted to go live the next week and it was not performing. The first thing I spotted was the applications still had debug code in them - they were writing out about 1000 printf's a second. Rather than tell them the problem, I showed the local team how to use the tool - the problem was obvious. The local team spotted the problem and went away and got it fixed. I then spotted they had some bad SQL statement's which could not be cached. Overnight they applied fixes, and by my second day the throughput had doubled!

      I had to go to an all hands meeting on my second day, for a project status update. There was the bank staff, the staff from the company providing the systems, me, and a local person from my company. The senior project manager stood up to speak. I could not understand what they said - but he was clearly very upset and every one else dropped their heads in shame as he berated them. I was told later the manager said things along the line of "You have been working on this performance problem for three months, and made no progress. This person comes from the UK - and in the first day finds problems and has doubled the throughput - what have you been doing for three months!!!!!! You told me it could not be fixed...." My local contact was embarrassed having to explain the rant from the senior project manager.

      1. Anonymous Custard
        Headmaster

        Re: Losing face

        Been there, done that. The joy of working for the European arm of a Japanese company.

        The worst one is asking something from them, and then getting the silence back. This actually means that they cannot do what you're asking, but they are not allowed to lose face by saying so. So you just end up swinging in the wind with silence coming one way and overly loud shouting from the other from your actual local customer who is waiting for the fix that you were asking Japan about.

        Generally speaking though they are a wonderful culture to work with (the Japanese, not the local customers), but there are certainly skills and tricks that need to be learned along the way to optimise and smooth communications.

        Similarly just knowing to accept that no matter your knowledge, experience or standing, in any discussion you will always be a gaijin and so never fully believed or trusted (and not to take it personally, as it's rarely meant that way or even something that they are conscious of a lot of the time).

        1. Fred Daggy Silver badge
          Holmes

          Re: Losing face

          "If I was to ask you to do X, who would I need to speak to first".

        2. F. Frederick Skitty Silver badge

          Re: Losing face

          Having worked as at a Japanese owned company, I have to take you to task for accepting their attitude towards non-Japanese staff. It's racism, simple as that, and saying it's just part of the culture does not make it OK.

          The whole face saving thing was a nightmare as well. Rather than acknowledge and fix problems the Japanese management would just ignore it. The result was an appalling set of software systems where any attempt to clean them up would result in the development branches being ignored in favour of a quick bodge to the main branch.

          It's also the only job I've had where I've seen a programmer sacked for not wearing a smart enough suit. The slightly rumpled appearance of the person in question upset a visiting Japanese director, who insisted the competent contractor was terminated the next day.

      2. Martin Gregorie

        Re: Losing face

        I had almost the opposite experience back in the '80s.

        We'd just completed development of a financial switch and shipped the hardware and software out to Austria and a pair of us from the development team were on site while the client's acceptance tests were run. This was about a month behind schedule because they'd shipped the, rare at the time, fault tolerant hardware to us to use for development and there had been problems arranging for it to be shipped back to Austria. Since we couldn't disband the development team until acceptance was complete, we'd used the extra time for setting up and running additional system tests while the shippers got their act together.

        Anyway, for the first few days everything went perfectly - all the acceptance tests, which had been written by the clients team, passed with flying colors. However, the acceptance team's faces got longer and longer, which we couldn't understand.

        Then, on about day 4 they found a minor defect and instantly it was smiles all round: it turned out that the deep gloom was just them thinking that, because they had not found any problems so far, their acceptance tests must be crap.

      3. Killfalcon Silver badge

        Re: Losing face

        Good on you forshowing them where the problem was, and letting them fix it. It's always tempting to just fix and be done, but you never are 'done' if you don't let people learn how to deal with issues.

        1. ITMA Silver badge

          Re: Losing face

          I agree with you 1,000,000% there.

          Mistakes from which one doesn't learn anything are bound to be repeated.

      4. Outski
        Pint

        Re: Losing face

        I was engaged to run an audit of Notes/Domino for a Chinese company. On the admin side they were good, but their appdev side had jerry-built a system to update names in the directory and all names fields in applications whenever a change was detected in AD, despite the fact that ND has a wonderful built in process to do this (Pascal Monett will back me here). When I pointed this out to the head of appdev, he didn't take it well, and I was gently steered by our contact (head of systems admin) to say that we could set up a test system to prove that a ten year old process actually worked, and that this would free up appdev time without losing face.

        We secured that year long maintenance contract (until our guy, a native Mandarin speaker chose to come home to Malaysia).

        Beer for our contact, who sent me back to KL with a lovely couple of gifts for the Outskits.

        1. mikecoppicegreen

          Re: Losing face

          I'e had similar within Europe. I was tech support for a range of digital panel indicators, and these were supplied from my English employer to our french importer to a local distributor to a panel builder for a new bakery conveyor oven for a large french bakery. The RS485 comms on these panel indicators was simply not talking to their computer.

          With a bag full of laptop and testgear, I arrived on site with the importer to find the bakery company, the oven manufacturer who was supplying the computer and RS485 controller, the panel builder and the local distributor all on site ready to give me a hard time.

          I requested 30 minutes to set up my testgear, and the crowd went off for a coffee, and I tested the indicators, and all looked good, they all worked correctly.

          When they came back from their coffee, I showed them that the indicators were all good - which was followed by a collective shrug, and the inevitable "why are they not working with our RS485 converter" I requested the manual for the converter, and it showed that they had connected everything correctly - except for two optional wires.

          I said - why not connect those?

          They said - we never need to.

          I said let's try - and 30 seconds later all was working perfectly.

          The bakery owner was furious with the oven supplier - who lost huge amounts of face by having an english guy read a french manual and add two wires that had delayed the project for several weeks. I thought our french importer was going to explode - he was trying so hard not to laugh!

    2. Arbuthnot the Magnificent

      Fresh out of school I did an industry placement at Sanyo in the 90s, and we were issued a handbook of cultural differences and do's and don'ts for interacting with the Japanese management. It was quite a shock for a kid from a backwards northern village.

  2. chivo243 Silver badge
    Holmes

    Huh?

    Where's the rest of the story?

    Did you forget the part about the cleaners??

    1. LogicGate Silver badge

      Re: Huh?

      Cleaners unplug the cable. A passing cart full of heavy stuff will grab the cable and rip it out of the junction box. Regrettably, any believable argument for this having gone unnoticed is made null and void through the action of stuffing the cable back in again.

      Small loss of face avoided through the application of big loss of face.

      1. chivo243 Silver badge
        Devil

        Re: Huh?

        At a place I worked at, the cleaners were charged with waxing the floors, all of the floor... Copiers, printers and vending machines(you read that right!) were moved with out unplugging the network cable... I got to repair the RJ45s in the wall and replace the network cables. I was also tasked with finding physical solution for preventing these unfortunate 'accidents'. My first thought came right out of Simon's rantings...

        1. H in The Hague
          Pint

          Re: Huh?

          "I was also tasked with finding physical solution for preventing these unfortunate 'accidents'."

          In the UK, gas cookers need to be fitted with a safety chain so they cannot be accidentally moved, ripping the gas hose off.

          https://www.screwfix.com/p/cooker-stability-chain-hook/30143

          Years ago I worked on a piece of scenery which was occasionally rotated by hand on a slewing ring (well, a VW Golf hub from the local breakers' yard). That had a power cable feeding lights in the unit (no battery powered lighting and wireless DMX in those days). Although the operator knew when to stop the rotation we were worried that others might play with the unit and damage the cables, so I also fitted a wire rope, slightly shorter than the cables, to limit rotation. And the slewing brake was built using the jack of my first car, an Austin Maxi.

          Here's one for the weekend -->

          1. Julian Bradfield

            Re: Huh?

            It's usually said to be to prevent the cooker tipping forward.

            Have these bureaucrats ever tried to tip a 100kg range cooker? (Or indeed accidentally move it.)

            1. Jou (Mxyzptlk) Silver badge

              Re: Huh?

              Open the front door. Sit on it. Depending on your stature it will tip and bury you, or not.

            2. Killfalcon Silver badge

              Re: Huh?

              It's at least as useful for the folk who are *intentionally* moving it that it'll stop moving before the pipe comes out.

              It's usually kinda hard to reach the gas fittings, so you need to pull the cooker out a bit to reach it, and that's when you're most likely to put in a bit too much oomph...

        2. C R Mudgeon Bronze badge

          Unwelcome movement

          Speaking of unwelcome movement, I once had to deal with a quite different sort.

          I had a contract (thankfully short, for other reasons) at a startup whose office was in a converted warehouse, or perhaps factory. You know the kind of space: exposed stone walls, massive distressed wooden beams holding up the floor above, chic lighting -- very fashionable among smallish tech companies (at least for a time; tastes might have moved on, but if so, I wouldn't know).

          The problem was, the bare wooden floor wasn't quite level -- or at least, the bit under my desk wasn't. It wasn't off by much -- walking or standing there, you'd never notice -- but it was enough of a slope that my chair insisted on rolling away from my desk. It would stay where it was put as long as it was unoccupied, but as soon as I sat down...

          It was an all-day struggle to remain within reach of my keyboard. Not an insurmountable one, but a constant annoyance, distracting me from concentrating on the task at hand.

          I finally had to "hobble" my chair. I brought in a roll of duct tape, and taped up one (just one) of the wheels so it couldn't turn. That produced the opposite annoyance, that the chair wouldn't move easily when I did want it to -- but at least that was intermittent, and mostly occurred when I wasn't deep in concentration, but was about to take a break anyway.

          I wonder what the chair's next user thought of my kludge. I imagine s/he went "WTF?", removed the tape ... and then replaced it a couple of days later once the light had dawned.

      2. Flightmode
        Megaphone

        Re: Huh?

        Tangent, but on the topic of believable arguments, I present you with an unbelievable one from real life. I swear, these guys must have had Creative Consultants on staff.

        In the early 2000s, I worked for an ISP in mainland Europe. All our international circuits (at the time an impressive mix of STM-1 and E3!) were leased from one provider that had serious quality issues. I'm not sure if it was poor documentation or crappy hardware, but we had outages in various parts of the network pretty much every day. There were several months where our SLAs kicked in and we didn't have to pay them anything at all for their services, that's how bad it was.

        Anyway, after a longer-than-usual outage between Amsterdam and Paris we got a root cause analysis back. It basically explained that over the weekend before, there had been a riotous demonstration at a train station somewhere in Belgium. A large group of people had stormed the tracks and vandalized the buildings[0]. This wasn't in and of itself the cause of the outage, oh no; one of the rioters had as part of his protest jammed a crowbar into the train tracks. A couple of days later, a commuter train[1] entered the station on that track, stopping right over the crowbar which buckled and bent under the weight of the train. When the train rolled out, the force on the crowbar was released, and as it snapped back into its regular shape it launched in a high arch - skipping over two other platforms - and hit the provider's junction box at pretty much the other end of the trainyard. It struck the door at exactly the right point for it to buckle inwards and break only our patch connection. All other cables in the box were undamaged, only our circuit had an outage.

        Anyway, the provider is no longer around. A couple of bankruptcies[2] and buy-outs later, they were absorbed by a national carrier's ISP arm and probably dissolved there. Good riddance.

        Icon for rallying crowds.

        [0] Not sure what about, we could never find anything in the papers on this.

        [1] They were very insistent on this being specifically a commuter train, not just any train, also in follow-up discussions. Still not sure why this was important.

        [2] Their head-office was in Brussels. One of their bankruptcy filings came when I was in town for a two week-training, so I was basically asked by my mangement to stay in Brussels and more or less force my way into their NOC if their staff decided to walk out to monitor our circuits in their absence. As luck would have it, I didn't have to.

        1. Killfalcon Silver badge

          Re: Huh?

          Unspecified riot at a train station?

          Probably football. Common enough to go uncommented in the news, and footie fans often take trains so no-one has to be designated driver.

          1. UCAP Silver badge
            Joke

            Re: Huh?

            ... often take trains so no-one has to be designated driver

            Someone was to be the designated driver of the train.

          2. I could be a dog really Bronze badge
            Coat

            Re: Huh?

            Some time ago I followed advice to leave the car at home and take a bus. Good advice, I found it remarkably easy to drive.

            OK, icon says it all.

        2. Bebu Silver badge

          Re: Huh?

          "...struck the door at exactly the right point for it to buckle inwards and break only our patch connection."

          Talk about a shaggy dog story. Sounds like the "dog ate my homework" all grown up.

          Perhaps at that time Mr Magoo was visiting Belgium.

          1. Charlie van Becelaere

            Re: Huh?

            Or possibly it was M. Hulot?

          2. nintendoeats

            Re: Huh?

            As said above, I would believe that it was a football related thing. The only thing that bugs me is that they would let trains through without inspecting the tracks.

            1. A.P. Veening Silver badge

              Re: Huh?

              The only thing that bugs me is that they would let trains through without inspecting the tracks.

              There is a reason both the French and the Dutch tell jokes about the Belgians.

              1. Anonymous Coward
                Anonymous Coward

                Re: Huh?

                Belgians (Flemish) tell jokes about Belgians (Walloon), and vice versa.

          3. yetanotheraoc Silver badge

            shaggy dog story (was -- Re: Huh?)

            "...struck the door at exactly the right point for it to buckle inwards and break only our patch connection."

            This kind of precision is indistinguishable from that of a crowbar wielded in two hands. I believe this story was told, but I have serious doubts this story describes the actual events on the day. Although, stranger things have happened, so what do I know?

            1. John Brown (no body) Silver badge
              Thumb Up

              Re: shaggy dog story (was -- Huh?)

              "Although, stranger things have happened, so what do I know?"

              It has just the right level of convolution and detail to be true 'cos no one could make that up :-)

    2. LateAgain

      Re: Huh?

      Or the IT manager who used the hoover with the big red "PAT test failed" tag and blew the IT office power.

  3. wolfetone Silver badge
    Coat

    "He was assured that the Japanese partner was confident that if they told their reliable and honest staff to leave the rack alone, no harm would befall it."

    Reliable and honest staff: Hold my sake.

    1. chivo243 Silver badge
      Coffee/keyboard

      Hold my sake. There are redneck Japanese?? Who knew!

      Time to make another cuppa... and grab a towel or two

      1. david 12 Silver badge

        One of the nice things about leaving Japan for Malaysia was the relief of being no longer the least polite person in any given situation.

        But one of the interesting things about Japan was the realization that in spite of the politeness and cultural differences, underneath the Japanese are human after all -- and just as likely to smoke, spit, and break things when nobody is looking.

  4. elsergiovolador Silver badge

    One time

    That one time, the owner of the company got really upset that once again someone has restarted the server, so he yanked it out of the server room and put it on a desk next to him.

    Mind you this was a small open plan office and many know the noise a server makes when it boots up and often afterwards.

    Turns out the fan noise that one was giving out was unbearable and everyone complained.

    Blasting the latest glam rock tunes to cover it up didn't help.

    After two or three hours, the owner took the server back to the server room and was working on the floor by the door keeping an eye on it.

    Then he realised nobody restarts it. It had a fault with cooling and overheating was what was causing the reboot.

    1. KittenHuffer Silver badge
      Flame

      Re: One time

      Did the fan screaming in agony not give it away?

      ------------> Where processors go when the fan dies!

    2. Jou (Mxyzptlk) Silver badge

      Re: One time

      Was it a server without the usual "ORANGE-RED GENERAL ERROR PLEASE CHECK ME" light? The number of times I've seen such a light when walking by and innocently asking that meant... Though, where I work now, monitoring is taken more serious. So I have to use the past tense here.

      1. the spectacularly refined chap

        Re: One time

        It can mean anything though. I'm most familiar with the alarm/indicator light on rack mount Sun kit. It could be turned on and off under admin control, if for example you wanted to indicate "this is the machine to put that memory upgrade in" or indeed "this is the machine we've decommissioned". Works well enough provided there are not too many machines with the light on at once.

        1. Jou (Mxyzptlk) Silver badge

          Re: One time

          The manual toggle of all warning or indicator lights has always been possible for better servers - I can certify that for Siemens servers from the Pentium 1 90 MHz class and later, since they were the first actual servers I had my hands on. For other manufacturers I had to deal with later: They too.

        2. phuzz Silver badge
          Pint

          Re: One time

          Many of the servers I've worked with have an ID button on the front and back, which you can just press, and it will light up on both ends (you can also usually control it via IPMI etc.).

          I don't know who first came up with that idea, but it's saved me a lot of time (and possibly unplugging the wrong server) over the years, so thank you very much :)

          For the inventor >>>>>

      2. John Brown (no body) Silver badge

        Re: One time

        "The number of times I've seen such a light when walking by"

        Yeah, I once got called out to a Compaq server with a 3 disk RAID array that had failed. I got there and there were two red lights showing two failed drives. I asked the on-site person (non-techy type), in conversation, when the two lights had come on. "Oh the second one came on yesterday, the other one has been on since I started here". Oops! Clearly their head office were not monitoring or checking to make sure someone on site was aware of how to at least keep a basic eye on things. No blame for the on-site non-techy person though. She had little knowledge about MS-DOS and none at all about Novel servers, that wasn't her job. She knew how to drive the PC to do her job.

        On a similar note, an "emergency" HDD replacement on a similar vintage server, out of hours, so no one on site to query other than security, Only one failed drive in this case so I go to pull the hot-plug drive and note that there is a lit "failed" LED and the "faulty" drive has been removed already. Called over the security guy to confirm what I was looking at (Always have a witness, just in case). Phoned their IT guy and asked about the state of the drives when he left for the day. He says he pulled the faulty drive ready for me arriving. I could almost hear the "gulp" when I told him he'd pulled one of the two remaining good drives and the array was now toast.

        1. I could be a dog really Bronze badge
          Facepalm

          Re: One time

          Ooh, many years ago the business I worked for had sold a an Apple Xserve to a creative business. For those that haven't seen them, they took the "minimalist clean design" way past what was good.

          https://discussions.apple.com/thread/251281172

          The disks hid behind an almost blank panel - just a couple of "pin prick" sized indicators. Each panel was a press to close, press to pop open type - and on opening it disconnected the SATA drive behind it ready to pull the drive out by the handle that the now popped out panel had turned into. One day they phoned to say the server had died and could I come round ... at which point you are probably ahead of me.

          I found the 3 disk array dead - with messages on the screen to that effect. On enquiry, I was told that someone wanted to plug a USB drive in, so had opened each "access panel" in turn looking for a front panel USB socket - obviously the first one just took a drive out leaving the array working but degraded, the second took the array out completely and froze the disk system from under the server's feet. Luckily he could remember which order he did them in, so I forced the most recently removed drive back into the array, which would have been as close as possible to get to the same as having just pulled the power cord, and then set the other to rebuild for consistency.

          To be fair, if you'd never seen one, you might well think that those small panels are just there to hide the ugly USB connectors ... which they didn't have. Needless to say, it didn't take long for everyone in the small business to know what had happened - and more importantly, not to press those panels that might be hiding a non-existant USB port.

        2. phuzz Silver badge
          Facepalm

          Re: One time

          Recent (Gen 9 I think) HP servers have disk caddies with a big red indicator lamp on them. I had to replace a failed drive in a RAID 1 array, so I naturally assumed that the big red light meant "Failed Drive".

          Nope, the light indicates "Do Not Remove". I pulled the wrong disk out, the server froze, and shouting ensued.

          (The array was fine once I replaced the good disk and the new one and it had a bit of a fsck, phew).

    3. yetanotheraoc Silver badge

      Re: One time

      "latest glam rock tunes"

      Try death metal.

    4. C R Mudgeon Bronze badge

      Re: One time

      "The owner ... was working on the floor by the door keeping an eye on it"

      Well, somebody was going to have to. Might as well be him...

  5. GruntyMcPugh Silver badge

    Several years ago I did a stint in School IT. I was diagnosing a network connectivity issue to one of their external pre-fab classrooms, so I went to the comms cabinet at the far end of the school, opened the door, and bunch of basket balls and hula hoops fell out. What we tecchies would consider to be a coms cabinet, the games teacher thought of as a handy cupboard.

    1. Snapper

      Had a client not install adequate ventilation in the small server room, so the only solution they would go for (money-wise) was to take the sides and front off the cabinet and direct a large floor-mounted fan at it. They also insisted that the 'server' room could be used for storage of anything they desired, so one of the team leaders bought his bike in.

      There was a server outage of course. The silly billy came in soaking wet one day and hung his dripping lycra up to dry on the convenient cabinet.

      Some people should not be allowed computers etc.

      1. Norman Nescio Silver badge

        The (washing) machine room.

        There was a server outage of course. The silly billy came in soaking wet one day and hung his dripping lycra up to dry on the convenient cabinet.

        Some people should not be allowed computers etc.

        The (contract) system manager for our VAXes, when he had been out on the bevvies, would sleep overnight on the couch in reception. He would wash his shirt in the sinks in the toilets, and hang it to dry in the exhaust of the VAX cooling fans in the machine room. AFAIK he never caused downtime for that reason.

      2. I could be a dog really Bronze badge

        Had a client not install adequate ventilation in the small server room

        They never do.

        My favourite was clients that would direct your attention to some random cupboard and suggest we move the "too noisy out in the office" server into it. It was really hard getting through to many of them that the reason that cupboard under the stairs was nice and cool was that it didn't have a fan heater in it - and once there was a fan heater (a.k.a. a server) in there, it would quickly get "quite toasty". In hindsight, I should have just taken a small fan heater with me and demonstrated ...

      3. Hazmoid

        optional kit drying rack

        I used to ride daily to my work (started at 5am during ESDT ) and when it rained, my kit was hung on a coat hanger at the rear of the rack. Always made sure that it was not dripping first. Those Lenovo servers certainly generated enough heat to keep the room cosy even with an aircon set at 20 degrees (Celsius)

  6. Jou (Mxyzptlk) Silver badge

    Classical "Told you so"

    Warnings and suggestions are all ignored until it happens. Not matter how obvious, nothing changes until it happens.

    Some people or companies are like children: Won't learn and budge until it really hurts.

    Seen often enough. With companies, people and children. Including myself of course.

    1. Eclectic Man Silver badge

      Re: Classical "Told you so"

      Well, to be fair, when the warnings are listened to and acted upon we don't get the tales of woe, so no 'On Call' or 'Who Me?' articles in the Register. Sadly success is not newsworthy, only spectacular failure gets publicised. It would be nice if once in a while there was an article about a successful major IT procurement, especially a government one, I mean, can they all be crap?

      1. chivo243 Silver badge
        Go

        Re: Classical "Told you so"

        I've seen some cable porn that will make most techies shed a tear! And not porn on cable...

        1. waldo kitty
          Coffee/keyboard

          Re: Classical "Told you so"

          I've seen some cable porn that will make most techies shed a tear!

          yup! every time i see such i have two immediate thoughts fighting to see which gets to the forebrain first...

          1. that's an extremely pretty flow of cabling.

          2. i sure don't want to be the one to have to cut that apart to replace/trace a cable.

      2. yetanotheraoc Silver badge

        Re: Classical "Told you so"

        "... major IT procurement, especially a government one, I mean, can they all be crap?"

        That's what they are trying to find out.

  7. Marty McFly Silver badge
    Pint

    Racks in hallways

    A decade or so ago I helped staff a tech conference with hands-on training labs. Classrooms of computers, all attaching to VM images running on rented ESX servers. Only problem was the ESX servers were overheating the 'tech office' we were assigned to work out of.

    Ended up rolling the racks out in to the large atrium / hallway that is common to convention centers. Roped off the racks, gaffer tape held the fiber cable to the carpet, and perched a warm body on a chair to keep an eye on things. Worked great! Only issue was repeatedly answering the same questions from all the techies curious about high-end kit.

    1. John Brown (no body) Silver badge

      Re: Racks in hallways

      "Only issue was repeatedly answering the same questions from all the techies curious about high-end kit."

      And the techies were probably quite impressed at the forethought of the organisers to put all the blinkenlights out on show instead of hiding them away. It was all part of the learning experience :-)

  8. Luiz Abdala
    Alert

    I'm gonna borrow a friend's story here. I wish this wasn't true.

    So, back in the 80's. Company that integrated PCs. Expanding.

    He was an Eletrical Engineer, helping with the expansion. Raised floors. Assembly lines for 50+ people. Wrist-straps all around. Everything being installed. He notices something very, very very crucial: the ground lines for the wrist straps were the same for the LIGHTNING RODS.

    Take a moment. Read that again. Wipe your desk, you just spilled on.

    He asks the responsible for that part of the project, which promptly dismiss it (!?!?). "What? Ground is ground!". He takes it to the higher-up in the Sparky Department. Cue the largest chewing he's ever heard in his life, AND him being put in charge of every. single. object. that. man. ever. authorized. The fact that man and everybody that worked in previous projects with him were still breathing was a major cause of rejoice and soiled pants.

    He had to go through everything in order to avoid the recreation of Benjamin Franklin's experiment in mass scale, plus every other building that man ever worked on. The unnamed walking manslaughter sparky was promptly fired and was never heard of again.

    1. Benegesserict Cumbersomberbatch Silver badge

      Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

      Upvote for the story, downvote for "major cause of rejoice and soiled pants" in the same sentence.

      Can't do both, settled on the former!

      1. Graham Dawson Silver badge

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        More comfortable than settling on the latter.

    2. Eclectic Man Silver badge

      Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

      Government procurement. Big IT boxes in one building, copper ethernet via ducting in trench to adjacent building where the flight simulators were located. Metal roofed buildings. Lightning conductor down side of building into ground 3 feet from ducting.

      I did mention this as a possible issue, but, you know, re-routing a properly installed lightning conductor is actually quite difficult, or so I was told. On the other hand moving the trench and re-wiring also rather difficult.

      I was once at school (yes I did go to school, inside too, so there!) when one of the buildings was struck by lightning. As it was a steel framed building no real damage, but it did short across to an oil can, which melted. It was put on display in the metalwork block.

      You certainly want to avoid being struck, however romantic it may seem:

      "Lightning injuries include cardiac arrest, loss of consciousness, and temporary or permanent neurologic deficits; serious burns and internal tissue injury are rare. Diagnosis is clinical; evaluation requires ECG and cardiac monitoring. Treatment is supportive."

      https://www.msdmanuals.com/en-gb/professional/injuries-poisoning/electrical-and-lightning-injuries/lightning-injuries

      I recall there is a rock climb where the lads were struck by lightning on the descent. Abseiling off charred ropes was a bit stressful; They called the route "Young Men on Fire".

      1. Eclectic Man Silver badge
        Alert

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        Young Men on Fire:

        https://www.lifesystems.co.uk/news/big-walling-in-the-bugaboos-jerry-gore-adventure-series

        "The wind roars all around us and I feel sick inside. We scramble back towards the bivi ledge, crampons skittering on dry rock, 1,000m of exposure dragging at us from both sides of the knife-edge ridge. The storm is going crazy all around us and there are splashes of lightning flying in all directions. Some hit the rock right in front of us and explode in a fountain of sparks, the sharp smell of burning filling our heads with fear. I feel like I'm running across a battle field with an unseen enemy firing directly at me.

        The adrenaline pulsates through our bodies as we collapse onto the snow ledge, terrified of the unknown and sensing danger in the darkness that surrounds us. We know we have to get under some sort of shelter fast and tear at the portaledge flysheet to gain protection.

        BANGGGGGG! There is a loud ringing in my ears and I feel my whole body lifted upwards and then slammed back down onto the deck. I look back at myself and feel totally numb and outside my body. I feel my heart stop and my body shudder to a halt and then re-start, like I have just been sledge hammered. I fall backwards onto the snow, winded, paralyzed and in pain. The flysheet is torn away by the wind, revealing a large smoke cloud from our burnt bodies and clothing. The first thing that I am really aware of is Warren screaming – he is yelling at me to see if I am still alive. We have just experienced a direct hit by lightning. We lie paralyzed and in pain from third degree burns across our bodies. We cannot move, at the mercy of the storm that thrashes around us."

        - Jerry Gore

      2. PRR Bronze badge
        Devil

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        > Lightning conductor down side of building into ground ...

        Much lesser event but related: We got satellite TV. Dish on the side of the house. It is supposed to be grounded, to give static and lightning a short-path around the box, TV, and any humans near them.

        Water spigot right under the dish. Done!

        They did not look in basement to see plastic interior plumbing. And I still had not twigged that there was NO good ground on that house because 1832--- all later frills were minimal add-ons.

        I got a copper rod in the patio to ground the dish, and over the years bonded-around meters and pumps and plastic utility piping with good #6 wire.

      3. Caver_Dave Silver badge
        Flame

        Re: Lightening

        I got hit in the entrance to Tathams Wife Hole in Yorkshire, England. It turns out that the cool cave entrances emit a column of negatively charged ions which attracts the lightening.

        It hit the coiled metal ladder attached to my belt on the way to ground. I was wearing a wetsuit and Wellington Boots which were all soaked and kept the worst of it with away from my body. I received a burn down my leg along the vertical seam in the wetsuit and was deaf in the closest ear for a few months.

        Icon, because that's what I could have been!

        1. Eclectic Man Silver badge

          Re: Lightening

          Until I met some cave divers (The group 'Dive Right' who know what they are doing), I was unaware that being struck by lightning whilst underground is a genuine hazard. Its seems that the 'spark gap' between the caver / cave diver and the roof and floor of the cave can be more conductive than the walls. Plus, of course, when underground you probably don't know there is a lightning storm going on above ground.

          I hope you are fully recovered now, Mr Caver_Dave.

          1. Nick Ryan Silver badge

            Re: Lightening

            Similarly people react in surprise to how often underground cables get struck by lightning. Then you explain the scenario: Lightning has just travelled 1000m of air, which is a good insulator, it's obvious that 2m of ground, which is not a good insulator, is not going to stop it.

    3. C R Mudgeon Bronze badge

      Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

      Question from someone who knows just enough about such things to be dangerous (but unlike the "manslaughter sparky", I'll never hang up my shingle as an "expert" on the subject):

      Seeing as a ground is basically a big spike hammered into the (actual, geological) ground, how far away would the two spikes need to be to make the situation safe?

      (No, I do not plan to put the answer to practical use -- see above. Just intellectual curiosity...)

      1. PRR Bronze badge

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        > how far away would the two spikes

        I Am Not A Licensed Electrician In Your Town!

        The general thing here is: roof-spikes come down to their own dirt-rods. Fusebox goes to its own dirt-rod. These are normally interconnected (at discretion of the lightning engineer and the Authority Having Jurisdiction), but are not "the same". Each fault-type has a short-path to its own dirt-rod and a long-path to the other dirt-rod.

        This is North America (land of the lightning bolts). The premise owner('s electrician) is assumed to install and maintain local dirt-rod(s), beyond what the electric company installs along the street for its own self-protection. I hear that in much of the UK the "fusebox ground" is supplied by the electric company, I assume because denser housing and better organized electrification.

        And yes it has not been "fusebox" since before my beard turned grey. (Though I do have a literal fusebox in service.)

        1. I could be a dog really Bronze badge

          Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

          Yes, in the UK, PME (Protective Multiple Earthing, TN-S or TN-C-S - see e.g. https://www.lsp-international.com/power-supply-system/ for an explanation) is the norm - though there are still places that use TT and their own earthing rod. When PME was being rolled out, typically the user's own earth rod would be disconnected - but with the latest revisions to the wiring regs, we were expecting specific advice (requirement) to install a local earth rod as well as the DNO's (Distribution Network Operator) earth.

          The benefit of PME is that the DNO provides multiple distributed earth points, resulting in (typically) a much better earth than any single earth rod (spike) would do. This means a low Ze (fault impedance of the supply to earth) and reliable tripping of over-current protection in the event of a fault. For TT supplies, the earth rod is typically not enough for this and some form of earth fault detection is required - historically this used to be a VOELCB (voltage operated earth leakage circuit breaker) which actually tripped on current in the wire to the earth rod, but since they would not detect a fault to a different earth (e.g. via a water pipe) these days it will be an RCD (residual current device, detects a difference in current between the live and neutral conductors).

          Now, to the question of what happens when there's a lightning strike. The important thing to remember is that these are very high currents, mind bogglingly high, and with incredibly high rise times (i.e. the current increases very very rapidly, before decaying almost as rapidly) - thousands of thousands of amps (i.e. mega amps) for milliseconds is roughly the order of magnitude. The second thing to remember is that "ground" isn't some mystical material with infinite conductivity - in fact, it typically had a fairly high resistance. So those that know even the most basic facts about electricity will realise that a massively high current passing through a significant resistance will create a significant voltage drop. So when a bolt hits the ground, immediately around it the "ground" may take up a voltage gradient measured in hundred of volt per meter distance close to the strike, and rapidly reducing with distance as the current is distributed though a greater mass of earth.

          I've dealt with the aftermath of this. Years ago (when serial lines to terminals was the norm), we had a client in a rural location. From the description, I think they had a lightning strike to ground a field or two away - I'd guess perhaps 200m or so. This was quickly followed by a burning smell from the room that housed their server.

          While everything shared the same electrical earth from the DNO, the combination of potential gradient in the ground, multiple connections between electrical supply earth and the building frame, and so on meant that there was significant voltage gradient between different pieces of equipment. Luckily, most of the old Wyse 60 terminals just lost their RS232 converter chips which failed (fuse like) and protected the rest of the circuits. But those in the second building, with separate electrical earth, and further from the strike, had much more damage. But the poor old server took surges from multiple connected devices which fried it's serial cards (and I think, the motherboard).

          And that is why, it is good practice to use fibre for any longer runs (even within the same building), and especially between different buildings.

          As to connecting the LPS (lightning protection system) earth to supply earth, that's not as clear cut as it might seem. Guidance in the wiring regs (BS 7671 in the UK) is that they should be connected - and you should have surge protection in the power supply system. The idea there is that should there be a strike, the LPS will conduct it to ground and there may be a difference between that ground and the supply ground - meaning that there's a risk of flashover to other services. If you connect them, then the whole system changes voltage at the same time, with the surge protection keeping live, neutral, and earth at reasonably similar potentials, and this avoid the risk of flashover. That does require a reasonable earth electrode for the LPS, which reminds me that it's something I still need to address at our church ...

          1. Luiz Abdala
            Joke

            Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

            Saying "May the wrath of God strike me down if I am lying" is a bad idea at your Church at this moment, right?

            1. I could be a dog really Bronze badge
              Stop

              Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

              Well actually not - to save money on oil for the heating, we're meeting in the church hall until the weather warms up (considerably).

              But otherwise, perhaps. The LPS earth electrode is a single rod which I strongly suspect is not very long (there's a lot of rock not far below the surface) and was driven as far as someone could do to put a token rod in. When we last had an inspection, it's resistance was ... "considerably more" than is considered acceptable - better than no LPS, but not good. Fixing it is a different matter. Apart from the cost of materials, and me finding time, there's the matter of digging a trench to put a couple of tapes in ... in ground with an unknown number of graves at an unknown (possibly not very deep due to said rock) depth. Once you want to start digging, the paperwork starts to get equally interesting (and expensive).

      2. Eclectic Man Silver badge

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        "Seeing as a ground is basically a big spike hammered into the (actual, geological) ground"

        I was talking with my dad's old college room-mate (then a senior lecturer at Wolfson College, Oxford) and he explained that a spike of metal in the ground is not an effective earthing method as a lightning strike would release so much energy that it would shoot the rod 'sky high'. You need something to dissipate the charge over a large area in the ground, like a buried metal trellis, sort of like a large chain-link fence, all connected up.

        1. Luiz Abdala
          Go

          Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

          My old man referred to these as "chicken feet".

          When one rod may not work, you add another three. And those were sufficient for a ground home installation if large enough..

      3. Jou (Mxyzptlk) Silver badge

        Re: I'm gonna borrow a friend's story here. I wish this wasn't true.

        The ground, at least in my country, is more than a spike. It has to be a thick metal plate (forgot how think but thick) the size of a few m² buried more than one meter deep. It depends on how dry the area is how deep it has to be. From that slab of metal a rod comes up, and from there on it is like the spike you mention. This is how to make sure ground is actually ground: Give it a huge contact surface.

  9. Anonymous Coward
    Anonymous Coward

    Joi Ito

    This reminds me of disgraced MIT Media Lab Director Joi Ito's claim of the first internet node in Japan. He set up the router in his tiny bathroom. He was so proud of it, he posted pics, it was a big ugly stack of rack mount boxes, minus the rack, tilting like it could topple over at any moment. He did not specify where he went for regular bathroom functions.

    1. Yes Me Silver badge
      Headmaster

      Re: Joi Ito

      Where is that claim described? Google couldn't seem to find it. He would have been 18 when Internet connectivity started in Japan, in 1984, thanks to Jun Murai and his team. That was initially UUCP, not TCP/IP. I can't find the exact date for the first TCP/IP connection in Japan but it was definitely in 1986 and almost certainly at Keio University in a computer room, not a bathroom.

      History at http://hiroshi1.hongo.wide.ad.jp/hiroshi/wide-wp/10th-Anniv/history/index_e.htm

      1. Anonymous Coward
        Anonymous Coward

        Re: Joi Ito

        First privately operated node, probably. I remember he had an old Sun workstation with a webcam you could interactively aim around his messy room.

  10. aerogems Silver badge
    Flame

    Server Closet

    I could write a novella length post on all the stupid things I witnessed in my roughly 1-week tenure at a company some years ago, but one thing that really stood out to me was how they wanted me to help set up some space they had recently leased on another floor of the same building. Naturally they waited until like a week before they wanted to start moving people in there to do silly things like establish Internet service, order any of the networking equipment that'd be needed, or even start assembling desks for people to sit at. The only UPS they knew was the package delivery company and they were literally pirating software from some of their clients. Of particular facepalm worthiness, however, was when they were showing me the space and I asked them where they planned to put the networking equipment; they pointed to a small closet. The average person would probably have to turn slightly sideways to avoid bumping their shoulders on the door frame, and then it went back maybe 2-3ft and was maybe 7ft high. It was just a coat closet, complete with a door. No ventilation, no power outlets, no network hookups, just your bog standard small closet for people to put their coats in.

    Suffice to say we had a difference of opinion and went our separate ways very shortly after. They were looking for someone to fail in place of their manager, and I wasn't interested in that position.

  11. Qwerty44

    "Because of course."

    LOL

  12. ecofeco Silver badge

    Has your advice been ignored, leading to utterly predictable problems?

    All the time.

    I can't wait to retire.

    1. logicalextreme

      Re: Has your advice been ignored, leading to utterly predictable problems?

      I was gonna say, this question is basically asking "do you work in IT?", or one of several other disciplines.

  13. Anonymous Coward
    Anonymous Coward

    Corridor story

    The mention of the word "corridor" made me remember of one story.

    A decade ago, we were to install a full rack of hyper-converge kit, from, if I remember, the only consortium providing such a stack, complete with virtualization and unified support.

    This was for a top aerospace company, and the kit would run critical workloads.

    One of our local PM (not the brightest), in charge of planning the DC setup for this kit, requested the environmental metrics (power consumption and cooling requirements) which was normal. But he also requested the noise level in Db.

    WTF ??? A DC is a noisy and generally hostile environment to human life, who cares about noise levels ?

    After the red alert was on, one senior PM, very up to date with DC stuff, was dispatched to the customer site. He reported: "well, the place is not so much a DC as a corridor really, it's tight, public and there are offices around, hence the noise issue".

    Lesson learnt: "never assume anything when you discuss a "DC" until you've been there".

    End of the story: the kit was installed and the project was a success, I think they had to close walls around the rack to make it a bit more DC-ish.

    1. Anonymous Custard
      Trollface

      Re: Corridor story

      OK, so maybe he thought DC stood for designated corridor?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like