back to article Nobody does DR tests to survive lightning striking twice

Lightning may never strike twice, but each and every Friday The Register runs another edition of On Call, our reader-contributed tales of tech support gigs that did not spark joy. This week, meet a reader we'll Regomize as "Karl" who once worked in the services limb of an enormous and venerable three-letter corporation's …

  1. simonlb Silver badge
    Meh

    No Good Deed Goes Unpunished

    Not paying for the overtime? Wow, that is really shitty.

    In a previous role I provided out-of-hours and on-call support and the official company policy was to book extra work as overtime in the first instance, or take time off in lieu (TOIL). After booking my first batch of out-of-hours overtime my manager said there wasn't a budget for this so I'd have to take it as TOIL. I checked for the company's policy document on this to see how much TOIL to take - the equivalent of the hours worked as overtime, or the equivalent of the hours worked at the overtime rate, which was at time and a half. There was nothing in the document to specify what TOIL to take, so I booked it as equivalent to the overtime rate.

    Naturally my manager queried this, so I explained that if the company were unwilling to pay me, say, four and a half hours pay for three hours overtime at time and a half, then they would need to give me four and a half hours of time off in lieu. I referred him to the policy document and said it wasn't detailed in there so I'd interpreted it at face value, but that I was more than happy to go and have a chat with my union representative to get their take on it. My manager said it was fine and that's how I booked my TOIL for over five years.

    1. Anonymous Coward
      Anonymous Coward

      Re: No Good Deed Goes Unpunished

      I did the same thing and after a while I pointed out that because of the amount of TOIL I was getting (same as you, time and a half, double at weekends), it'd be worth my while doing some small bits and pieces on the side for the clients. My boss pointed out that I can't do that because there is a clause in my contract prohibiting me from performing any work for clients on the side...so I read the clause and the clause in the contract read "...within the scope of works as set out in the client contract...". Since the sales team and legal team were very keen to stay ahead of any edge case complaints, they made the scope of work for each client extremely specific and classified anything outside of the extremely specific scope as "best endeavors", as in, they will assist with it, but it isn't deemed as "in scope" therefore if anything goes wrong, no claim can be made against the company regarding poor service, negligence etc...but here is the nice bit, all the really valuable work (server side admin, hardware upgrades, Linux stuff etc etc...basically anything that wasn't desktop support or really easy), was outside of the scope...I pointed this out to my boss..."yeah, but the scope of work in the customer contracts is really specific, there's a ton of work I can take on during TOIL and I can make a killing without having to take money out of the business"...well he went white as a fucking sheet, the lawyers stepped in and insisted on giving me a 30% payrise and they cancelled my acces to TOIL in favour of paying me 2.5x for each overtime hour I did...and yeah, you bet your ass I took the piss...we had a three shift pattern, 7am til 2pm, 9am to 5pm, and 11am to 7pm...you can bet your ass I never took the 9 to 5 slot...I took the 7 til 2 slot and booked 3 hours of overtime a day, 5 days a week for years.

      1. A.P. Veening Silver badge

        Re: No Good Deed Goes Unpunished

        I took the 7 til 2 slot and booked 3 hours of overtime a day, 5 days a week for years.

        That is taking malicious compliance to the next level, well played.

        1. Anonymous Coward
          Anonymous Coward

          Re: No Good Deed Goes Unpunished

          Yes, well I ultimately ended up in a senior position at one of their biggest clients which I was able to take based on other oddly specific clauses in my contract (and the client contract) and the wiggle room afforded in my change of status when In handed my notice in.

          Essentially, because of bean counting and insurance bullshit, they wanted to scrimp on liability while someone was on notice. Essentially when on notice you were no longer deemed to be an employee. Which had the effect of technically cancelling out a load of clauses...including non-competes, working for clients etc etc.

          To make it iron clad, I was never hired by the UK arm of this client. They hired me through the Swiss arm, which never had a contract with these clowns and is where all the employees of the client were placed.

          The icing on the cake was I got to become the European Head of IT....so it was my job to cancel the contract with my former employer.

          When they showed up for a meeting and realised I was the new head of IT. They hit the fucking roof. I think the penny finally dropped at that point because they didn't lawyer up and walked out looking absolutely defeated. One of the directors found it funny though and had the balls to shake my hand and say "well done, you're someone else's ballache now, don't fuck it up". They went bust a year later and the "hot headed twat" director ended up working at Rackspace as a third line tech. The other two (less unreasonable ones) started a new firm and they are still going I think.

          These days I am freelance, yoinking contracts on a regular basis, fighting contractual injustice...like Batman, only nerdier.

          You'd be amazed at how shitty the average MSP contract is...remember kids, if you spend too long trying to cover your ass you might forget to put a helmet on!

    2. Anonymous Coward
      Anonymous Coward

      Re: No Good Deed Goes Unpunished

      We had a weird problem with some kit on another continent and we had to send a development engineer out to fix it instead of one of the normal support team. We estimated a 3 day trip and he was gone over 2 weeks and left a very happy customer. When he put in his expenses his engineering manager wouldn't authorise a lot of the expenses cos they were outside policy. E.g. he'd only packed enough clothes for a couple of days so he'd had his clothes laundered a few time by the hotel and bought some other odds and sods he hadn't packed. The customer PM went ballistic and read the riot act to the engineering manager, who signed off the exes. The engineering boss, in a fit of contrition, sent some flowers to the engineer's wife to apologise for him being away so long. She sent them back to him with something pretty unpleasant written across the card.

      1. Doctor Syntax Silver badge

        Re: No Good Deed Goes Unpunished

        "She sent them back to him with something pretty unpleasant written across the card."

        I hope the flowers were roses with plenty of thorns on the stems.

    3. Marty McFly Silver badge

      Re: No Good Deed Goes Unpunished

      Still grates on me... Y2K and HR says "All hands on deck January 1st, just in case. We will comp you an extra vacation day."

      Nothing happened on Y2K, except we sat around the office thinking about missing the New Year's Day holiday.

      Three months later the company downsized and none of the promised extra vacation days were paid out. That was early in my career, and lessons were learned.

      1. An_Old_Dog Silver badge

        Re: No Good Deed Goes Unpunished

        I was lucky to have rather better management. Because of all the pre-Y2K prep work we had done -- about a year and a quarter of it -- we only had to have 6 people on deck for the Dec 31st 10PM - Jan 1st 6AM shift, and we had no significant problems, and we were properly paid.

        If my management had been better than they were, we'd have started in 1998, but as it was, I made more money on Y2K-related overtime in 1999 than I made on my regular salary!

      2. Anonymous Coward
        Anonymous Coward

        Re: No Good Deed Goes Unpunished

        All hands to the deck

        Bet HR were at home that day

      3. Anonymous Coward
        Anonymous Coward

        Re: No Good Deed Goes Unpunished

        At least you didn't have to go around every client checking the analog clocks because your boss is a fuckwit. I know several people that were made to do that.

    4. Scott 26

      Re: No Good Deed Goes Unpunished

      I F'ing hate TOIL booked at 1:1 to hours worked....

      You had me working out of hours and you see fit to renumerate me as if it were normal hours? Fuck that.

      Hence why I always ask "Do your timesheet for Project X have an OT component?"

  2. Korev Silver badge
    Coat

    I guess computers have moved on and many now have a dedicated lightning port...

    1. Anonymous Coward
      Anonymous Coward

      Not for much longer, now the EU has poked its fingers in…

      1. Elongated Muskrat Silver badge

        It's all Thunderbolt™ now...

        1. LogicGate Silver badge

          Naah.. Universal Surge Bus 3

        2. Anonymous Coward
          Anonymous Coward

          This is getting a bit too Bohemian Rhapsody for me.

          Cue Galileo and fandangos...

          1. Anonymous Coward
            Anonymous Coward

            and a devil sideboard....

          2. breakfast Silver badge

            Motherboard, I just killed the mains,

            When some lightning hit a tree

            Took out the circuits and the leads...

            1. Anonymous Coward
              Anonymous Coward

              Overtime, Overtime...as if money doesn't matter!

      2. Ordinary Donkey

        In the future all lightning must be cloud based.

        1. b0llchit Silver badge

          Ask SUN?

        2. Doctor Syntax Silver badge

          And nothing too precipitate

  3. Anonymous Coward
    Anonymous Coward

    At least you fixed the problem.

    I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there) during a downpour, decommissioning a load of customer kit to relocate it to another DC...the roof caved in and there was water gushing into the datacentre everywhere...thankfully though, not on my racks, which were almost dead centre...I was there all day working on my own kit and I observed precisely zero people coming in to sort out the racks that got drenched...I had a wander over to some of the, by this time, dead racks...and I noted at least a few million quids worth of kit that was now soaked and ruined...most of which were storage arrays...so quite a few people, somewhere were having a really shitty day.

    I really hated that datacentre...pretty much any datacentre in MK to be honest...it's far enough away from London to be relatively cheap, but also just far enough away for the trip to have a long boring journey...I've also never been in a nice datacentre in Milton Keynes...they're all miserable...several of them have pool tables but they all have at least 1 ball missing, cues with no tips...broken coffee machines...backed up bogs...a miserable twat on the front desk...walking around them is creepy, like walking around a derelict wreck from the 90s.

    1. Dabooka

      Re: At least you fixed the problem.

      To be honest that description of the DC extends to most of Milton Keynes in general

      1. H in The Hague
        Pint

        Re: At least you fixed the problem.

        "To be honest that description of the DC extends to most of Milton Keynes in general"

        Thanks, just what I needed to be reminded of while packing my bags to go and visit the family in MK!

        Though Gallery MK hosts some good exhibitions and the National Museum of Computing offers many moments of nostalgia to older Commentards.

        And this looks potentially interesting: https://miltonkeynesmuseum.org.uk/collections/view-our-collections/#Telephone

        A good weekend to all Commentards -->

        1. GlenP Silver badge

          Re: At least you fixed the problem.

          I'd concur on both TNMOC and MK Museum. Last time I visited the latter it was with someone who'd actually operated the manual exchanges displayed during/shortly after WW2.

        2. Simon Robinson

          Re: At least you fixed the problem.

          There's always Bletchley Park too...

      2. Wanting more

        Re: At least you fixed the problem.

        Hey, I live there! It's actually a good place to live on the whole. Lots of greenery and parks. Nice village pubs. As like anywhere there are some estates to avoid, but anything based around the old villages is usually pretty decent.

        1. Andy Taylor

          Re: At least you fixed the problem.

          MK is much maligned but I can think of worse places to be (e.g. Reading).

          1. F. Frederick Skitty Silver badge

            Re: At least you fixed the problem.

            Reading you say? I'll raise the stakes by suggesting (actually, insisting) that Bracknell is worse.

            1. Snapper

              Re: At least you fixed the problem.

              I see you and raise you Croydon!

              1. Anonymous Coward
                Anonymous Coward

                Re: At least you fixed the problem.

                Oh fuck Croydon man...in fact that general area around Croydon as well. Sutton, Mitcham, Colliers Wood and Mordor (Morden).

                Mate of mine got stabbed in the neck in Sutton for nothing (he's fine and still very much alive) and I've had the misfortune of a couple of nights out in Croydon...what a shit hole.

          2. Anonymous Coward
            Anonymous Coward

            Re: At least you fixed the problem.

            I can think of worse places to be (e.g. Reading)

            What a lot of southern centred commentards we have here.

            I see your Reading, and raise you ... Barrow in Furness. Turn off the M6 at J36 and Barrow is the end of a 40 mile cul de sac. Anon because, well there might be some Barrovians here and there's "a certain amount of risk" in criticising the town.

          3. Anonymous Coward
            Anonymous Coward

            Re: At least you fixed the problem.

            Reading is a fucking dump. It's a town full of commuting jobsworths...pretty much every major town along the M4 is shit. It's like Slough is ground zero and the general grubby wankiness spread up the M4...the further you get from Slough the less (bit still) shit it gets.

      3. ShortLegs

        Re: At least you fixed the problem.

        Now I really do not mind MKS. We have a DC there, staffed by a handful of great people.

        The Stadium Hotel is great to stay at, there is an absolutely superb Turkish restaurant - Enfes - opposite (I really cannot stress how great this place is)

        And lots of green cycling and running routes.

        Always look forwards to a trip down south. But...

        ... we are moving to Slough.

        Slough! Concrete boredom, the only place worse than Basingrad, as I used to refer to Basingstoke when with Global Crossing. For some reason people were not too happy with that label. No idea why :)

        1. spold Silver badge

          Re: At least you fixed the problem.

          John Betjeman.... Come friendly bombs and fall on Slough...

        2. drand

          Re: At least you fixed the problem.

          Amazingstoke we called it, with a skip full of sarcasm. Eurgh.

          1. Anonymous Coward
            Anonymous Coward

            Re: At least you fixed the problem.

            People outside there call it Basingrad.

        3. Anonymous Coward
          Anonymous Coward

          Re: At least you fixed the problem.

          Slough, ugh

          Always knew when you are getting close to the motorway turnoff for Slough because of the stench of the water treatment plant.

          Also the word slough could mean a small stream or depths of despair.

          I never saw a stream in Slough....

        4. FatGerman

          Re: At least you fixed the problem.

          I used to work nor far from Slough. When the wind was in the right direction the smell from the Mars factory coupled with the stench of sewage would waft over our office like a well-brewed fart. To this day even the thought of a Mars bar makes me wretch.

      4. Already?

        Re: At least you fixed the problem.

        Not the cliché MK is dull response surely?

        I moved there for a couple of years 30 years ago, am still here. It’s great: drive at NSL speeds and get across town on the grid roads in < 15 minutes unlike the peak time crawl whenever I go back to what I still call home. As mentioned in other replies it’s got Bletchley Park, MK museum (fab), the theatre, The Gallery with its well worth a visit cafe too. Everywhere is within walking distance of huge green open spaces and linear parks, the same time in any direction is a great bike ride in open countryside. Pubs, nightlife, cafes, decent shopping centre, kart track, ye olde villages within and close by outside, as well as the old towns, and London 40 minutes away. Etc etc etc. Quality of life here is great.

        But yeah - roundabouts and concrete cows. Yawn.

        1. anothercynic Silver badge

          Re: At least you fixed the problem.

          NSL and roundabouts are the best thing about MK's grid. Although getting out of MK onto the M1 in the afternoons (during rush hour) used to be the absolute pits.

          At some point I gave up with the horridness that was the last 400 yards or so from the H5/H6 roundabout onto the M1 junction and then the crawling-slow merge onto the actual motorway, would zip past the John Lewis warehouse to the road towards Bedford, and then either merge onto the M1 there (if you were lucky with light timing that was an absolute breeze) or chance the construction work (t'was the time when the A421 was being dualled between the M1 and Bedford) to skip over to the A1(M).

          These days to get to Cambs, it literally is just a case of going straight ahead through MK onto the A421, then over to the A1(M). It avoids the mess of Northampton/Wellingborough.

        2. H in The Hague

          Re: At least you fixed the problem.

          "But yeah - roundabouts and concrete cows."

          As only an occasional visitor to MK I find the roundabouts a bit difficult to remember. Here in NL they tend to put pieces of public art on them. So you recognise them by the archeologically inspired sculpture, the giant trowel, mega-tulips, etc. Makes the drive easier and more fun. But admittedly in MK you would need a v substantial arts budget to plant something on every roundabout! Just as well we've got satnav now.

          1. John Brown (no body) Silver badge

            Re: At least you fixed the problem.

            In Scotland, they seem to give them names, at least in some areas :-)

        3. Anonymous Coward
          Anonymous Coward

          Re: At least you fixed the problem.

          What are the beaches like?

      5. Anonymous Coward
        Anonymous Coward

        Re: At least you fixed the problem.

        Beat me to it, I used to work at a company where I had to frequent a certain fruity store a lot. Too many phones in an environment not suited to them.

    2. Anonymous Coward
      Anonymous Coward

      Re: At least you fixed the problem.

      I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there)

      TalkTalk and Milton Keynes are still with us, sad to say...

      II've also never been in a nice datacentre in Milton Keynes.

      Datacentre and Milton Keynes should set off very loud alarms warning you that there isn't going to be any niceness.

  4. Inventor of the Marmite Laser Silver badge

    Donkeys years sinceupon, when 8 inch disks were still a thing, a work chum was despatched from our Luton office to take a raft of bits to a water treatment works in Diss, Norfolk, which had been on the fizzy end of a lightning strike.

    I got involved when he rang in to say that he'd found the problem - an 8 inch floppy unit had been fried and he'd replaced it. Everything was then fine he said, until the second lightning strike fritzed the replacement.

    Guess who got stuck with a panic run to Diss and back.

    I think we did something about power arrangements but I've been asleep a few times since then and I'm buggered if I can remember what.

    Sadly my old chum is no longer with us but I do remember him in the occasional lightning storms we get here.

    1. Donk
      Trollface

      Surely the lightning strike did both of you a favour getting you out of the hell that is Luton, if only for a little while.

      From someone born in Luton who got out while they could.

      1. UCAP Silver badge

        Lived there for 8 years. Left when I got married. Best decision I ever made [1].

        [1] Leaving [2].

        [2] As well as getting married!

        1. Anonymous Coward
          Anonymous Coward

          You have to get married to leave Luton? That explains a lot about the place.

          1. Chris 15
            Joke

            As I understand it only counts if you marry a cousin though...

          2. Anonymous Coward
            Anonymous Coward

            That's what my dad did!

            Come to think of it, so did his two brothers. One made it all the way to NZ

          3. Anonymous Coward
            Anonymous Coward

            You have to get a visa somehow.

    2. Noram

      My memory may be playing up, but from what is surfacing I think the old adage about lightning never striking the same place twice is quite wrong, when lightning strikes it ionises the air and actually makes it easier (thus more likely) that you'll get another hit in roughly the same area.

      I'm fairly sure there is/has been research into how to utilise that to direct more lighting to where you want it hitting thus building a better lightning rod by effectively creating that ionised pathway through the air before the first natural strike.

      1. Anonymous Coward
        Anonymous Coward

        So *that's* why we send someone from marketing or HR into the office car park waving a big metal pole whenever there's a thunderstorm.

        1. Doctor Syntax Silver badge

          You think a strike might animate them?

          1. Richard 12 Silver badge
            Mushroom

            Worth a go

      2. Fonant

        Yes, I watch a video where they used a pulse of extremely high energy laser to pre-ionise the air to "steer" a lightning strike. Exciting research, if in very short bursts of activity!

        1. Prst. V.Jeltz Silver badge

          wow coolest job ever!

          #thor

      3. JamesMcP

        Correct, but not the general reason why lightning hits the same place twice, since few strikes hit twice in the second or so that the ion channel exists. (Or more accurately, most lightning strikes actually consist of more than one discharge that happens in rapid succession while that channel exists)

        The real reason lightning tends to hit the same place more than once is the same reason your finger and the doorknob get more static discharges than anywhere else in your house. Weather patterns exist, just like you shuffling from your bed to the kitchen, which tend to accumulate similar charges, and there are only so many physical locations that fall under that pattern. Some of them are simply closer and of more appropriate materials/shapes. Ceramic coffee cup? Insulators by definition aren't good electrical conductors. Brass doorknob? Excellent.

        Similarly a flat grassy field has a much lower ability to concentrate a charge than a house with a chimney covered in sharp edge. (Sharp edges and points are better charge concentrators)

        So, until there is some kind of new construction that changes becomes a more attractive lightning rod or the old lightning rod finally gives way to a couple of gigawatts of power or climate change alters the weather patterns, lightning will keep striking the same place.

        Hence that one house in every town covered in lightning rods.

        1. KittenHuffer Silver badge
          Boffin

          ---------> You forgot your icon!

        2. Doctor Syntax Silver badge
          Flame

          You've just reminded me there's a weather vane on the house.

        3. A.P. Veening Silver badge

          Hence that one house in every town covered in lightning rods.

          In my experience usually a church if there aren't more modern (and higher) high-rises around.

        4. I could be a dog really Silver badge
          Mushroom

          And the little detail that lightning rods tend to have a sharp point. The idea is that they'll form a locally very strong field at the point, and thus draw charge into the rod - so in the vicinity of the rod, the air is less charged, and so the chance - both of a lightning strike, and of it hitting your building if it does happen - is reduced.

          What I can tell you from hard experience, is that serial (all RS232 back then) links across large buildings, or even worse between buildings, do not mix happily with the very high voltage gradients in the local "earth" if there's a nearby lightning strike. Icon approximates what it did to our equipment :-(

      4. PRR Silver badge
        Devil

        > the old adage about lightning never striking the same place twice is quite wrong

        And not just for recent ionization. As others suggest, the reasons lightning struck once are often still there after the strike, unless the tree/rod/manager vaporized or fell.

        Much the same reason you go to Pat's Pub every week. It is nearby, it is easy.

        In my youth my house was in sight of a 1,000+ foot (many meters) broadcast tower. We could see it get hit multiple times per storm.

        So no large shock that we have two tales of double-hitters. All you need is a storm-line long enough to give time for a first repair before a second strike. (If 2 strikes happen "at the same time", bang--bang, for billing purposes that's just one hit; though often the potential is dropped by the first hit so will be some time to recharge.)

        Icon represents Zeus, Lightning-guy.

      5. Benegesserict Cumbersomberbatch Silver badge

        Apollo 12 cut out the middle-man and ionised the air itself. It got struck twice.

        1. druck Silver badge

          Try SCE to Aux

      6. Ignazio

        Aside from ionisation immediately after a strike, features of the area make lightning more likely to strike, so yes, lightning does strike more than once in the same place. Rarely the same living creatures or fuses, though.

        1. MrBanana Silver badge

          Park ranger Joy Sullivan holds the record for surviving 7 lightning strikes, so he says.

      7. ricardian

        That put me in mind of a quote from Pratchett's "Colour of Magic":

        “If complete and utter chaos was lightning, then he'd be the sort to stand on a hilltop in a thunderstorm wearing wet copper armour and shouting 'All gods are bastards!”

    3. Andy Taylor

      My father worked for Thames Water in the 80s and early 90s and lightning strikes were responsible for a good proportion of the repair work required on the telemetry systems he managed.

      Pumping stations water towers tend to be situated at the top of hills for some reason.

    4. breakfast Silver badge
      Pint

      I hope your extra efforts earned some Diss respect.

      1. Doctor Syntax Silver badge

        Whe he turned up on site he Diss appeared.

  5. big_D Silver badge

    Lightning strikes

    We had something similar happen to a facility on the South Coast of the uK.

    When I went over for an unrelated issue, I walked past a couple of hundred DEC VT100 terminals stacked 5 high in the corridor! A couple of days earlier, a lightning strike hit the building and shorted every terminal in the place!

    DEC VT100 being "advanced" technology that the US didn't want falling into "enemy" hands, they had to be officially disposed of and witnessed and certified by a UK government representative.

    Luckily, they was an scrapyard next door and the terminals were just pushed on trollies to the compactor of the neighbour and stuck in there, then, with the witness there, the terminals were counted and the compactor turned on. At the end, we got a signed certificate that the terminals had been properly destroyed...

    Another time, working at a Naval Dockyard, one of the suppliers had tried to take shortcuts and replaced the heavily shielded DEC VT220 terminals on the purchase order with standard terminals... A few weeks later, a US destroyer tested its radar, whilst still in port. Queue a couple of hundred dead terminals and a very red faced, and out of pocket, supplier.

    1. GlenP Silver badge

      Re: Lightning strikes

      DEC VTs were usually fairly robust.

      I'd left a VT220 in the computer room when I went on holiday as I'd been doing some tasks that were better on a VDU than the console. Unfortunately the computer room burnt down while I was away (no connection of course - electrical fault in the workshop next door).

      The two VAXen, an 11/750 and 11/780, were a total loss of course.

      The terminal was pretty much melted but for a laugh we tried plugging it in from a safe distance - yes, it came up VT220 OK, albeit somewhat faintly.

    2. J.G.Harston Silver badge

      Re: Lightning strikes

      A very long queue.....

  6. Luiz Abdala Silver badge
    Alert

    Lightning always finds a way... of least resistance.

    At least the guy that made the server rooms didn't weld the wrist straps ground to the lightning rod's ground... like someone I know...

    The friend of mine that noticed this averted a Benjamin Franklin experience en masse, and got in charge of the grounding of the building for his troubles.

    1. Doctor Syntax Silver badge

      Re: Lightning always finds a way... of least resistance.

      "for his troubles"

      And his long term survival.

    2. Anonymous Coward
      Anonymous Coward

      Re: Lightning always finds a way... of least resistance.

      "At least the guy that made the server rooms didn't weld the wrist straps ground to the lightning rod's ground..."

      You actually want to be at the same potential as the lightning rod's ground. If you're not, bad things can happen. That's why communication wiring has an intersystem bonding termination that ties all the grounds to a common point. Granted, running a dedicated wrist strap ground wire back to the ground rod can create some other issues, so typically you grab the closest grounding conductor.

      If you're grounded to a different "ground" and lightning hits the building, now your wrist strap could be at a potential that's hundreds or thousands of volts different than the ground on your equipment. That would lead to a very bad day. The problem is that the earth is a relatively poor conductor.

      If you're in the US and want to dive down a rabbit hole, read up on NEC chapter 250 on grounding and bonding. That's one of the key areas that low voltage technicians need to know for their license.

    3. Inventor of the Marmite Laser Silver badge

      Re: Lightning always finds a way... of least resistance.

      Should've gone to earth somewhere till it had all gone away

    4. swm

      Re: Lightning always finds a way... of least resistance.

      Lightning is a strange beast. A lot of it is RF. If you ground a lightning rod with a wire that makes a sharp turn, the lightning will go straight. The wire used to ground a lightning rod is a coarsely braided conductor more than an inch in diameter. All of the building gutters should be connected to the ground wires with a 1 inch wide copper strip. The actual ground to earth should be a 12 foot copper rod driven into the ground, preferably at three places around the house. The ground system should be wired to the electrical ground (that, by code, is a 6 foot galvanized pipe).

      Remember, lightning rods attract lightning so you'd better be ready when lightning strikes.

  7. Boris the Cockroach Silver badge
    Angel

    FLASH!!!!!!!!!

    Ahhahhhhhhhhhhhhhhhhhhhhhh

    there goes your data center

    (welll someone had to )

    1. Anonymous Coward
      Anonymous Coward

      Re: FLASH!!!!!!!!!

      no they didn't!

    2. Richard 12 Silver badge

      Re: FLASH!!!!!!!!!

      <brianblessed>MAINFRAME'S ALIVE!!</brianblessed>

    3. The Oncoming Scorn Silver badge
      Joke

      Re: FLASH!!!!!!!!!

      Flash, Flash, I love you, but we only have fourteen hours to run the restore!”

  8. Sequin

    I worked for the Home Office in the late 80s and was sent of to Fire Service College in Moreton-in-March in the midlands for several months - I got to go home at weekends, but it was a twohour drive each way, so I got home late on Friday night and had to travel back down on Sunday afternoon. I could claim for the travel time, so no complaints there.

    I was staying in a lovely little pub in the village - horse brasses, wood fires, good beer etc. Civil Service rules said that if you were working with somebody of a higher grade and having work discussions etc you could claim the expenses rate of the higher grade. This applied to me, as I was working with somebody two grades higher than me, so the difference was significant.

    Several months later I put in my expenses claim and it was the highest one they'd ever seen from somebody on my grade and they baulked at paying it. They tried to say that I should have stayed in dormitory accommodation at the college, but I said that if I stayed on site I wouldeffectively be on duty 24 hours a day, andthere was no rule that stated that I had to stay on site, so they had to pay out.

    A couple of months later they rolled out a new rule that if there was accommodation on site, you could either use that, or only be able to claim 50% of the usual rate if staying off site.

    1. anothercynic Silver badge
      Angel

      The burghers of Moreton-in-Marsh will have you know that they are in the COTSWOLDS, thank you very much, NOT the Midlands. The venerated Clarkson would be inclined to agree with them given he lives just a few miles down the road ;-)

  9. C R Mudgeon

    "we literally just sighed"

    My response would have been quite a bit louder and more staccato. Rather like a close lightning strike with an F in it...

  10. Gene Cash Silver badge
    Holmes

    You had replacement parts for TWO lightning strikes?

    That's what I find most implausible about the whole story.

    Most places I've worked wouldn't have enough to cover one such disaster. If we had two dozen of an item in service, we'd be lucky to have a single spare.

    1. An_Old_Dog Silver badge

      Spare Parts Availability

      It can vary radically from organization to organization. In one organization I worked at, there were plenty of spares for most things (no spare mainframe, but we had a hot-site agreement with another org, and no spare SAN, but spare discs, spare channel controllers, terminals, network cards, PCs, monitors, non-mainframe-printers and such). In a Government-owned organization which we were physically- and networkly-linked to, I was troubleshooting a user's PC and was told by their Help Desk that they'd have to get management approval for a replacement CAT-5 patch cable, a thing which by all rights should be considered "popcorn".

    2. DS999 Silver badge

      Re: You had replacement parts for TWO lightning strikes?

      If the datacenter is mostly mainframes "frying every ethernet port in the building" might not amount to more than a few blades worth on the switches. Customers with mainframes have a lot of money to spend, keeping a fully configured core switch plus spare blades doesn't seem too crazy to me.

    3. FatGerman

      Re: You had replacement parts for TWO lightning strikes?

      There's an old adage that "Nobody ever got fired for buying IBM", and a lot of that was because when things did go wrong there was a seemingly bottomless pit of spare parts they could magic up at short notice. Don't know if that's true any more though.

      1. Diogenes

        Re: You had replacement parts for TWO lightning strikes?

        Yes. Mid 1980s. I was working on contract at place and we had the production AS400 that was fried by lightening. It was replaced by IBM well within the Service Delivery contract time (the warehouse was literally at the other end of the street).

        IBM loaded the OS and we were right to go with the restore, blank tape, last weeks backup, blank tape etc etc

        Luckily I had been in on teh weekend and taken a full copy of the data as we needed to some performance testing on the software for the next stage we were releasing.

        Come in around lunchtime & people are running around like headless chickens - I reach into my bottom drawer and grab my backup which is then loaded (I knew it worked as I had installed the data on the performance testing machine). I was taken out for a very long, free lunch.

        It was a totally unimportant application - it held enrolments and marks for the year 12 exit exams (UK = A levels, USA think SAT)

      2. Andy A

        Re: You had replacement parts for TWO lightning strikes?

        IBM don't always have the parts to throw at a problem.

        Early in my programming career I was involved in a project using an IBM 360/40. Our company made money by running the cheapest kit possible. At the insurance company across the road, the programmers had full use of a 370/158.

        Our compile jobs were always pushed to the back of the queue. One week I submitted 30 compiles and only two got processed.

        Usual operating shifts finished late on Friday evenings. So we decided that we programmers would come in on the Saturday to clear the backlog. Some had been operators earlier in their careers, so we all knew the drill.

        We powered up the system (who remembers it being called IPL?) and started jobs.

        20 minutes later the system stopped with a Parity error. Restart then.

        10 minutes later the system stopped with a Parity error.

        We reached the point where the system would not even IPL. Obviously something was warming up and then failing.

        No engineering cover for weekends, so we rang the higher-ups, who could then get things moving for Monday.

        IBM found a failed power supply. One was rushed in, and the mainframe was running by mid afternoon.

        Of course we got NO jobs through that week, so we trailed in on Saturday again.

        IPL - 20 minutes - Parity Check.

        Ring higher-ups, head home.

        This time we were informed that the replacement PSU for the ancient beast was the last one in the whole of Europe. System back up by Tuesday.

        The operators were informed that under NO CIRCUMSTANCES were they to power down the system that Friday evening.

        We cleared our backlog in time for a couple of pints. The mainframe was left idling.

        About a month later the 360 model 40 was replaced - with a 360 model 50 !

  11. Anonymous Coward
    Anonymous Coward

    If this is the same Mediterranean hotel chain I'm thinking of...

    ... we worked for the same three-letter company, and I have a story which I can't decide if should be featured under "On Call" or "Who, me?".

    The annual DR test was supposed to be held at the end of a long, hot Spanish summer. We were supposed to simulate a power loss at the main site, wait until the generators kicked in, and then test the swing to the secondary site. Preparations for te big date took the two previous weeks, every runbook/manual/contact was double checked, and on D(R) day, the mains switch was flipped, and power was cut to the Production data center.

    We remained connected for about 10 minutes, under UPS power, more than enough time for the generators to kick in and take the load. Except it didn't happen. We lost connectivity from outside the data center, and decided to call the hands & eyes team. They picked up the mobile phone (the landline was as dead as the network links) and we asked them to go to the back of the building and check the generators. 5 minutes after, they called back, stating the generators were dead and closing with an urging "ahora vuelvo, voy a por gasóleo a la estación de servicio!" ("be right back, I'll go get some diesel at the gas station!").

    We had our phone on speaker, and the moment the H&E guy hung the phone, we busted out laughing until tears came out. We couldn't believe it: the only thing missing from the check list was to verify the amount of diesel in the generators' tanks. A quick note was taken on the DR runbook, we asked another member of the H&E team to reconnect the external power, and called the DR exercise a failure, to be rescheduled a month later.

    1. Doctor Syntax Silver badge

      Re: If this is the same Mediterranean hotel chain I'm thinking of...

      So the disaster can be recovered but only if two weeks have been spent preparing for it.

      Seriously - you could only call the exercise a failure if you didn't learn to have regular checks made on the generator fuel. DR exercises are to be learned so that when the D happens for real so does the R.

    2. Anonymous Coward
      Anonymous Coward

      Re: If this is the same Mediterranean hotel chain I'm thinking of...

      > .. the only thing missing from the check list was to verify the amount of diesel in the generators' tanks ..

      And check for water contamination collecting at the bottom of the tank. And do your DR test more than once a year.

    3. An_Old_Dog Silver badge
      Happy

      The Devil is in the Details

      I'd call this one a "learning success."

    4. DS999 Silver badge

      I've heard of more than one case

      Where sites did monthly generator tests, but neglected to check the fuel level. With the result that when a real disaster happened and the generator had to power everything it didn't last nearly as long as had been expected due to the monthly tests running down the fuel and no one checking/refilling it.

      1. David Hicklin Silver badge

        Re: I've heard of more than one case

        Or the other classic of having a big fuel storage on site that relied on mains electric to run the refill pumps....

        1. Doctor Syntax Silver badge

          Re: I've heard of more than one case

          Summer grade diesel still in the tanks in the winter. It contains wax that solidifies in the cold.

          I suffered that one, not in a data centre but a signal failure on a miserable wintery evening's commute.

          1. DS999 Silver badge

            Re: I've heard of more than one case

            Can you leave winter diesel in the tank year round? If not, how are you supposed to have a generator always at the ready if you have to replace the diesel twice a year? Two tanks??

      2. Ken G Silver badge

        Re: I've heard of more than one case

        Yes, it was so common it was a cautionary tale in DR preparation training.

        Like the cleaner unplugging the UPS to hoover, funny because it was true.

      3. PRR Silver badge

        Re: I've heard of more than one case

        > ...when a real disaster happened and the generator had to power everything it didn't last nearly as long as had been expected due to the monthly tests running down the fuel...

        My school did not make that mistake.

        Instead they assumed the fuel trucks would be running. This was not a deep snow area--- at worst they might wait a day.

        A once-a-century blizzard (the second one that decade) came through and overwhelmed the snowplows. Few even tried to defy Governor's Orders to stay home. The fuel trucks actually did run the best they could, following the few plows, because many homes were on fuel oil and you don't want old ladies found frozen in their homes.

        The bulk of the school was also crying for heating oil before water-pipes froze.

        The IT staffer snowed-in at the datacenter saw the generator start and run fine, then the oil tank dropped and dropped for 3 days before it quit.

        The next year there was a MAJOR rebuilding. Bigger tank, a new pipeline and street-gas generator on the side, and do-over of the legacy power distribution. Cut-over to the new breakers and BLAMMM! Some forgotten low-bid contractor had done something dumb not noted in the new job's bidding. I think Chuck had to fly halfway across the country for more of the special breakers to be made.

  12. Anonymous Coward
    Mushroom

    Lightning strike borks power conditioner

    I'm not an expert on hardware or power supplier or anything but: I would have thought that if you are tasked with designing a power conditioner, you would designe it to survive a near-by lighting strike or at minimum, self-destruct to prevent damage to the ethernet switches and sundry other hardware.

    ps: Never ever do unpaid overtime.

    1. Doctor Syntax Silver badge

      Re: Lightning strike borks power conditioner

      No plan survives contact with reality.

      Moved into new (to us) building in the autumn. Returned from Christmas hols to find a lightening strike had taken out the thyristors in the UPS. The main building power was OK and so was all the equipment. We ran without the benefit of the interrupted UPS for many months. It's not always DNS; sometimes it's the UPS.

      1. Anonymous Coward
        Anonymous Coward

        Re: Lightning strike borks power conditioner

        Well, an UPS saved our bacon once. Mountaintop, rack with some very specialized and expensive hardware, with a small 2k VA UPS protecting it. And that UPS had a feature to run the network cable through it too, to protect it against surges, so we did ("why not?").

        Long story short, some time later lightning struck the building, a surge apparently traveled along the Ethernet cable(s) doing quite some damage, but our rack was saved (at the expense of the UPS, which was fried). Very fond of this UPS, it saved us a prohibitive (for us) amount of money. RIP, little black box.

    2. Excused Boots Silver badge

      Re: Lightning strike borks power conditioner

      Fair enough, but the sheer amount of power dumped into a system by a lightening strike is enormous. Attempting to have a solution that can cope with that would be stupidly expensive and whatever you did, nature would probably at some point conspire to ‘think you have engineered against any contingency, well try this…..’!

      And having sacrificial devices in the chain which ‘should’ lay down their lives to protect….., is fairly normal, and probably mostly works. Well until nature decides to throw a curve ball and…..!

      It is absolutely impossible to 100% protect against any and all issues to do with lightning, lightning really doesn’t play fair! At best you try to mitigate against most issues, and the more you are prepared to spend (or waste, as the bean-counters see it as), the closer you get to 100% reliability on an asymptotic scale, so you can never actually get there.

      On the other hand, your point about unpaid overtime, is really spot on.

    3. DS999 Silver badge

      Re: Lightning strike borks power conditioner

      Lightning is one of those things that never acts according to plan, so no matter how well you think you are prepared for a strike it can always do something you haven't seen before.

    4. Anonymous Coward
      Anonymous Coward

      Re: Lightning strike borks power conditioner

      I'm not an expert either, but having to deal with manufactor's kit in the "real world" is very different to what the designers think happens.

    5. Robert Carnegie Silver badge

      Re: Lightning strike borks power conditioner

      As explained earlier in the story, lightning is the stuff that hits a tree and the tree explodes. In a thunderstorm do NOT shelter under a tree.

  13. cmdrklarg

    SHAZAM!!!

    n/t

  14. that one in the corner Silver badge

    Under emergency lighting

    > Nothing quite enhances the senses like a dark, quiet and smoky datacenter.

    As you stalk you way through the servers, looking for zombie processes to kill. There's one now, stuck in a pipe.

    1. Anonymous Coward
      Anonymous Coward

      Re: Under emergency lighting

      And Gordon lifts his crowbar...

  15. an.other_tech
    Mushroom

    Primary, secondary, tertiary, quantilary ?

    When your backup needs another backup but your primary got fried in the first strike, you naturally move to the second and so on.

    However, when the owner doesn't understand I.T and the need for any backups because he owns the building, it's a really tough uphill struggle to get anything done, let alone a backup schedule and regime.

    So, the workaround?

    An on site tape backup, a daily removable database backup (thats what made the money), an off site hdd that only came in on Thursdays, a random backup that was done twice monthya d went elsewhere. Plus I shoved a separate server elsewhere in the building, think it was in a loft space, inside a modified waterproof cabinet. That had a dedicated ups, various surge protectors, plus I could remote access over, a, wait for it, 56.6k modem.

    Oh the heady days.....

  16. Inventor of the Marmite Laser Silver badge
    Pint

    This all reminds me.....

    The seller of lightning rods arrived just ahead of the storm. He came along the street of Green Town, Illinois, in the late cloudy October day, sneaking glances over his shoulder. Somewhere not so far back, vast lightnings stomped the earth. Somewhere, a storm like a great beast with terrible teeth could not be denied.

    Thanks Ray Bradbury. Here's a pint for you, wherever you are

  17. LateAgain

    Lightning always strikes twice !

  18. Anonymous Coward
    Anonymous Coward

    Ha!.....The Twentieth Century....I Can Barely Remember.....

    -----backups, off-site copies, Iron Mountain.....DR weekends.......

    So, so......old fashioned!!!

  19. TooOldForThisSh*t

    ZAPP

    Had a customer long ago back in the NetWare days who got hit by lightning. Phone system was destroyed along with melted thin-net ethernet cables. Compaq server and desktops all survived but the cheap off-brand clones they bought elsewhere all had fried motherboards. Needless to say they bought replacement Compaq's from us :)

    1. Anonymous Coward
      Anonymous Coward

      Re: ZAPP

      ... but did the customer survive?!

  20. Hazmoid

    speaking of lightning strikes

    I recently had to replace 2 network switches and use the backup link for a site that experienced a near miss. It appears that the strike caused a surge in the existing link (ethernet cable) that blew up the network switches at each end. Fortunately, when we ran the link cable, we ran 2 in case there were issues with one.

  21. Baucent
    Meh

    Nothing stops lightning...

    'Nothing stops lightning, and no one wins but the lawyers'

    The trick is to get the two to meet.

    Venerable Clarkson? Venerated by whom? I concede he has a peculiar imagination when writing about your Royal Family which apparently hasn't been entirely to their liking.

    In any case I will try to avoid driving in the Cotswolds as I don't imagine he has many careful driving awards.

  22. Alan Brown Silver badge

    This is why...

    in a BIG 24*7 operation (like say, a global hotel chain), your DR includes an alternate site (and switching between them is TESTED)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like