back to article Do you come from a land Down Under? Where diesel's low and techies blunder

We go Down Under for today's Who, Me? with a slightly NSFW tale of an incomplete checklist, a surprise outage, and an even more surprised gerbil. "Bruce", as the somewhat unimaginative Regomiser has dubbed our reader, is today's contributor and tells us of an event that occurred in the 1990s. He was working for a telecoms …

  1. GrumpenKraut Silver badge
    Windows

    looking at the DNS log

    back when dial up was a thing. Some URLs where telling, others not so much. Content was invariably "OMFG!". Never did it again.

    1. Anonymous South African Coward Silver badge

      Re: looking at the DNS log

      Checked a pr0n filter log on a company firewall.

      Only once. Never again.

      1. Anonymous Coward
        Anonymous Coward

        Re: looking at the DNS log

        From a former company, overheard conversation between CEO and IT Manager:

        CEO: You mean it's possible to check exactly what sites our employees are visiting from the firewall logs? Why don't you do this then?

        ITMan: Because if I did, I'd have to tell you what I found. Do you really want me to? Really?

        CEO: <thinks carefully for a bit...> Err, no. Best not.

        1. This post has been deleted by its author

        2. Potemkine! Silver badge

          Re: looking at the DNS log

          Alternative answer:

          ITMan: Because if I did, I would also see what sites you visited. Do you really want me to? Really?

          Expect the same answer from CEO.

          1. jmch Silver badge
            Happy

            Re: looking at the DNS log

            "Alternative answer"

            The actual answer was the same as teh alternative answer. Just not so blunt!

        3. keith_w Bronze badge

          Re: looking at the DNS log

          That's exactly what happened at my work when we first set up internet access.

          1. Anonymous Coward
            Anonymous Coward

            Re: looking at the DNS log

            >> exactly what happened at my work when we first set up internet access

            Yes.... the most depressing period in all my history with IT.

            Once the rumors of internet abuse reached upper management, setting up the monitoring of internet activity was assigned to me. The results led directly to employee dismissals and demotions. Additional monitoring of phone logs led to one more dismissal. Either due to type of content or amount of time being wasted.

            Employees learned quickly to be more careful with company resources.

            1. Anonymous Coward
              Anonymous Coward

              Re: looking at the DNS log

              Unfortunately I was given the task and nothing was ever done with the info. I had several users who should have been shown the door for doing f-all during work hours.

              Instead the word leaked out I was the internet police and suddenly became quite a bit less popular.

              And I'm not claiming Prom King popularity to start with.

        4. JimboSmith Silver badge

          Re: looking at the DNS log

          Worked.for a company where at the dawn of the internet PR0N there were no written rules about net surfing. However the company had signs up saying that it was being logged but didn't mention it being checked at all. HR did do some checking and saw some naughty sites being visited by a fair few people.

          HR decided to have a quiet word with everybody concerned. One staff member was looking at a BDSM site quite a lot after hours. She was invited to a pesonal meeting where this was brought up. She said the PR0N was lifestyle related as she was in a committed BDSM relationship with her Master. She was looking for inspiration about painful things he could do to her. HR lady said in the meeting that she didn't have to suffer like that, it wasn't normal and the company could get her help.

          She explained she didn't need help they'd been together for 15yrs and she was perfectly happy. So happy she'd married him a few years previous. If she'd been watching lesbian PR0N and announced she was gay in the meeting would their reaction have still been it wasn't normal and they'd get her help? "Nope thought not' was what she told me she'd hissed back at them.

        5. cdegroot

          Re: looking at the DNS log

          I had something similar, late '90s. We installed some internet-facing stuff at a customer site and a couple of weeks later, during a follow-up call, complaints about performance. I got sent in to troubleshoot and of course started with a traffic log analysis, conveniently available from their outgoing gateway (an AS/400, but that's a different story). Within ten minutes, it was clear what caused the performance issues, and the logs also gave with the IPs of the culprits' PCs where company time and resources were spent, well, "watching gerbils".

          Wasn't the most fun chat with management to tell them what I found. And, indeed, I said that while I could give them the IPs of the offenders (I didn't tell them I already had the data), I strongly suggested that they would operate under the assumption that that info did not exist and just send a stern warning email around.

          Which, luckily, they did.

      2. TeeCee Gold badge
        Alert

        Re: looking at the DNS log

        At one point in a place I worked, one proxy served teh hole wurld and I could see the logs.

        I still have no idea why sex with donkeys is so popular in Egypt and I'm not sure that I want to find out.

        1. Manolo
          Joke

          Re: looking at the DNS log

          Not just Egypt, seems to be a thing in North Africa:

          https://nypost.com/2017/08/16/teens-treated-for-rabies-after-alleged-sexual-contact-with-donkey/

          1. sebacoustic

            Re: looking at the DNS log

            It's the NY Post so consume with 10^100 grains of salt

            1. Manolo

              Re: looking at the DNS log

              Sigh....

              Alright then, straight from the horse's errr donkey's mouth:

              https://www.moroccoworldnews.com/2017/08/225797/15-teenagers-treated-for-rabies-after-engaging-in-bestiality-with-donkey

              But indeed, apparently they did not contract rabies:

              https://mobile.ledesk.ma/desintox/zoophilie-sur-une-anesse-ni-rage-ni-bestialite-mais-un-travestissement-de-putes-clics

      3. ShadowSystems

        Re: looking at the DNS log

        I was a brand new PFY at a small ISP when I got assigned to the security side of the place. My boss wanted to know who was looking at what & when, so had me checking the aforementioned logs. My boss, a rather nice lady that inspired my hormonal young self to go the extra mile so as to hopefully cause her to smile, was definitely non-plussed when the person I found visiting the most disturbing porn sites & the most often turned out to be the CEO... Her husband.

        I'm glad I was safely at home when she exploded on her hubby, but the aftermath was obvious when I returned to work on Monday. Her hubby no longer worked there, his office was empty, and I was taken aside by her to ask me most politely to not discuss the real reason for his departure with the rest of the staff. I agreed, she smiled, I floated on cloud 4.5 for a while (it's halfway to Cloud 9 because it's tinged with that icky feeling in the pit of your tummy that YOU may have done something that'll get you fired) and I tread *very* lightly from then on.

        Nothing like the knowledge that your every site visit is recorded for eternity in the bowels of the corporate archives & that info can be dredged up to not just "bite you on the arse" but bite your head off, shite down the hole that used to be your throat, & get you fired so hard that even large calibre cannons get jealous.

        *Nervous squeak*

        1. JimboSmith Silver badge

          Re: looking at the DNS log

          Back in the dialup days I was asked help a friend of a member of my family. She had switched on the monitor and on one of the IE windows was a smutty site. Convinced this had to be her son she wanted proof of all the Desmondesque sites he'd visited. After a virus scan proved negative I looked at the logs rather than the totally deleted history. There were quite a few of them and she was hopping mad. I asked when her son went to bed and it was early say 8pm, the daughter half an hour later. All but one of the sites were accessed after 10pm so that ruled the kids out. The daughter confessed she'd gone to a previously normal domain (that had lapsed and been bought by a smut peddler) and saw some smut.

          That accounted for the earlier one and I got the feeling her husband would be getting a right telling off that night. The son was obviously playing Wolfenstein obsessively given the game was installed and the maps stored under the keyboard etc.

      4. Anonymous Coward
        Anonymous Coward

        Re: looking at the DNS log

        Anonymous since I and the others still work at the same place...

        Many years ago I was asked to watch the web logs for a specific PC as there had been complaints that one of the kitchen staff was browsing porn in the chef's office. Over several days I'd randomly check the logs and if there was activity going on, let HR know who would rush over and nonchalantly enter the chef's office on some pretext, and find it empty and PC off. After about a week of this I found that I was accidentally watching the wrong PC, and was actually watching the traffic for a different staff member with the same first name who, it turned out, was also browsing porn in an open plan office shared with 3 women without being spotted.

        Became a bit of a dilemma for HR as to whether to take action or whether the monitoring was invalid but wasn't asked to do the same again...

    2. Gene Cash Silver badge

      Re: looking at the DNS log

      I remember one time getting referred by a link to a most interesting article on the manufacture of the Commodore 64... on playboy.com

      This was back when IBM ATs roamed the earth and C64s were a current topic.

    3. Wexford

      Re: looking at the DNS log

      Takes me back to when I'd set up a basic keyword log filter at work, and was suprised at the domain names used by Pen Island (calligraphy consumables) and Experts Exchange (tech forum) at the time.

  2. Chris G Silver badge

    No pr0n without diesel

    Give 'er a good filling and crank 'er 'ard!

    1. Nodrog

      Re: No pr0n without diesel

      Easy if she's self-priming, less so if you have to prime 'er by hand first...

      1. Sgt_Oddball Silver badge

        Re: No pr0n without diesel

        I always thought you started with the mechanical aid first then finished it by hand?

        1. John Brown (no body) Silver badge

          Re: No pr0n without diesel

          No, the mechanical aid is for when you get tired of the manual method and/or to prevent the forming of blisters and friction burns. Those hand cranks are exhausting!

    2. Gene Cash Silver badge

      Re: No pr0n without diesel

      I remember a situation where the generator ran fine for 20 minutes (past the usual 15-min test) then failed because the tank vent was blocked, and the vacuum stopped fuel from flowing.

      1. Anonymous Coward
        Anonymous Coward

        Re: No pr0n without diesel

        That sucks.;-)

        1. A.P. Veening Silver badge

          Re: No pr0n without diesel

          Like a Microsoft vacuum cleaner ;)

  3. Anonymous Coward
    Anonymous Coward

    Alternative steps

    1) On the quiet, install camera to find out who's siphoning diesel out of the storage tank.

    2) Review footage.

    3) Present footage to CEO, and wait for a sudden resignation. In our case, of the building services manager, who was earning a bloody sight more than I was, but still felt the need to steal a few quid's worth of diesel once a week.

    Optional step 4: While officially declining to give a reason for manager's sudden departure, make sure that unofficially the entire company knows exactly why they went, and exactly how they were caught. This last step can dramatically reduce all sorts of thefts, not just of diesel.

    1. Electronics'R'Us Silver badge
      Thumb Up

      Re: Alternative steps

      Some years ago (about 2005 or so) I was working in the automotive third party business and we had a contract with one of the large waste disposal companies in the UK.

      They were using a lot more diesel than they should be (big trucks but even so...) so a bright individual came up with an idea to fit a RFID tag at the fuel cap of each of their vehicles and put a reader on the fuel nozzle.

      The reasoning was that the pump would only operate if it read a valid tag so we implemented the system for them such that a hash of the tag was available from what was basically a password hash file.

      Diesel consumption went down by about 25% almost immediately.

      No points for guessing who were most put out by the new improved fuel delivery system.

    2. Manolo
      Facepalm

      Re: Alternative steps

      In this legislation, applying cameras to cath theft is only legal if everyone is informed first.

      It hasn't occured to the legislator that this only helps preventing further thefts without being able to punish the pepetrator.

      1. lglethal Silver badge
        Angel

        Re: Alternative steps

        But that's brilliant. You see you dont actually need to set up the cameras! Just tell everyone that due to a series of thefts, you're having hidden cameras set up in "certain" areas, no need to mention specifically where, and boom thefts will stop overnight.

        As all the guilty parties (even the ones you werent aware of) will stop assuming it was them you were after...

      2. Cynic_999 Silver badge

        Re: Alternative steps

        It most certainly has occured to the legislator - in fact it is the main reason for the legislation. Same reason that speed enforcement cameras in the UK must be conspicuous and have advance warning signs.

        1. Manolo

          Re: Alternative steps

          Whereas in this legislation, they are unannounced and hidden, because they don't serve the purpose of traffic safety, but filling government coffers.

          Populistic drivel?

          Well, let me give some additional evidence:

          - When people started behaving better, i.e. generating less speeding tickets, the fines went up instead of down. Cashflow had to be maintained. In a normal country, fines go up when they DON'T work.

          - No speed controls near schools.

          - Average speed trap set up on a stretch of motorway that had only seen one lethal accident in the past five years, and that had happened on the parking lot of a gas station. But is was a very busy stretch of motorway, so lots of revenue.

          When I was in Canada, I saw signs at road works that read "Fines doubled when workers present". Very sensible. Here they are doubled whether all workers are safely home in bed.

    3. Doctor Syntax Silver badge

      Re: Alternative steps

      "find out who's siphoning diesel out of the storage tank."

      I'd never thought of this aspect before but is generator diesel marked in the same way as agricultural "red diesel"? If so a quiet call to HMRC and a spot check on the offender's tank's content might save the need for involving the CEO.

      1. DarkwavePunk

        Re: Alternative steps

        Back in the day when I lived on a canal boat we had red for agricultural, blue for leisure vehicles (which included house boats for some reason). and standard. There was talk about removing the blue designation. Don't know. Boat sank for other reasons. There's no difference except the price as far as functionality is concerned.

        1. Anonymous Coward
          Anonymous Coward

          Re: no difference except the price

          And in the US, that price difference is probably all tax. Like don't charge the agricultural the road use tax and so on.

          Probably not a big house boat lobby so you were most likely paying road use taxes.

          1. W.S.Gosset Silver badge

            Re: Tax

            Can confirm: the only difference is tax. (And colouring additive)

            Hence the harsh penalties if caught with unauthorised red diesel: you are literally committing tax fraud.

            Agricultural: purpose is: subsidy for Food. Ie, fruit and veggies (primarily) are cheaper. Target beneficiaries: low income earners.

      2. Cynic_999 Silver badge

        Re: Alternative steps

        Yes, you are permitted to use red diesel for generators. Whether a company goes to the trouble of doing so in order to save a few £ on something that will only be used occasionally is a different matter.

        Personally, I filled our genny tank from diesel ferried in cans ferried from the nearest petrol garage, though I could have instead bought it from a more rural garage 20 miles away that sells red diesel - but that would have taken all day.

      3. Aussie Doc Bronze badge
        Black Helicopters

        Re: Alternative steps

        Here in Oz, as far as I'm aware, diesel is diesel.

        Major users such as farmers and truckies can claim significant tax relief (or at least you used to be able to) but there was no difference between what went into the tractor or into my Merc Sprinter in either colour or performance.

        Chopper doesn't use diesel, though ----------------------------------------------------^^^

    4. Eclectic Man Silver badge

      Re: Alternative steps

      Umm, these days you need a legal opinion before you can install cameras 'on the quiet' to 'spy' on staff.

      In my previous employment, we had a presentation where a senior manager presented a story of someone who had been stealing laptops. Basically there was a laptop set up team who received literally hundreds of new shiny Lenovo laptops a week, built them to the company standard and shipped them out to new joiners, people who needed an upgrade, managers who had left theirs in the car overnight and discovered it 'missing' and a broken window in the morning* and others who just wanted a new toy to impress the clients.

      Well, it seems that the company was buying a few more laptops than made it out of the build team to staff, and 'no one' knew what was happening to the others. So eventually, and with police authorisation, covert cameras were installed one weekend. The miscreant was caught on video, bold as brass, taking four laptops in boxes and one out of its box (his company car wouldn't hold five in boxes in the boot, poor chap). When his home was searched there was £60,000 (sixty thousand pounds) in banknotes wrapped up in clingfilm, some stolen laptops and various other dubious items.

      He was adamant that the company expected a certain amount of loss of laptops and that he was entitled as part of his job to take them. I believe that he went to prison for several years.

      The important thing about the police involvement and approval for the cameras is that you are also spying on potentially completely innocent employees too, which is currently against the law (at least for companies to do on their own in the UK).

      *That was an interesting de-brief session. I even got to help the PR team to prepare a range of press releases depending on what level of breach might be revealed if things went seriously bad. But that is another story ...

      1. DS999 Silver badge
        Thumb Up

        Re: Alternative steps

        He was adamant that the company expected a certain amount of loss of laptops and that he was entitled as part of his job to take them

        Well that's certainly bold as criminal defense strategies go!

        "Well your honor, hospitals expect a certain number of patients in the ICU will die, so I'm entitled as part of my job as an ICU nurse to kill my ex husband's new wife by injecting air into her"

        1. Eclectic Man Silver badge

          Re: Alternative steps

          DS999: "Well that's certainly bold as criminal defense strategies go!"

          Well, it does remind me rather of a short scene in an episode of the northern detective series 'Vera'

          Vera Stanhope was telling a teenage girl that there was once a little girl who tried to find and save a man from anti-social activity (euphemistically 'philandering'). The teenage girl asked; "That girl was you, wasn't it?"

          Stanhope; pause "yes"

          Girl: "And did you? Save him that is?"

          Stanhope: pause "No, love, he didn't want saving. They never do."

          Criminals generally believe that what they are doing is 'right' in some sense, until they actually meet the victims of their actions. Then they either have a sort or revelatory moment, or genuinely don't care. Stealing 'a few laptops' from a large multinational company just did not seem to be so bad to this guy.

  4. I Am Spartacus
    FAIL

    Happened to me as well!

    At what was once described as Europe's Biggest Data centre . have no idea if this was true, but as a 23 year old, it was pretty damn big. Think a large aircraft hanger filled with mainframes, tapes, printers and drum storage. The local power builders managed to put a JCB through the heavy duty three phase power supply, which mist have brought tears to the operator.

    We ran a motor-generator set, with a large fly wheel to keep the generator going until the diesel kicked in. Which it duly did, coughed spluttered and died. We had forgotten that diesel can wax, which ours, unused for years, duly had. So the injectors clogged and the thing refused to work. The embarrasing thing is that we were an oil company, and we were supposed to know about such things!

    1. Roger Greenwood

      Re: Happened to me as well!

      Generators that run all the time are easy, standby ones need care. I love how it is called "polishing" the diesel.

    2. mmonroe

      Me too!

      When I worked for a large facilities management company, they decided to do a disaster recover test at our site. The power company shut down the electricity supply, the UPS's told the genny to kick in, which it did, and management were all beaming smiles, showing the assembled clients how safe our site was. Beaming smiles right up until the genny ran out of diesel after about 10 minutes. Seems the blokes who drove the nightly backup tapes to the off site storage had been filling up the van from the genny tank.

      1. Doctor Syntax Silver badge

        Re: Me too!

        Moral of the story - if you're going to do a customer demonstration make sure you rehearse it first.

        1. Dwarf Silver badge

          Re: Me too!

          As they say, never start a demonstration with something so prone to failure as "Watch this !"

          1. Nunyabiznes

            Re: Me too!

            Always say "Hold my beer and" first.

    3. John Robson Silver badge

      Re: Happened to me as well!

      "The embarrasing thing is that we were an oil company, and we were supposed to know about such things!"

      Brilliant - I've only been at one company that did DR processes well, and it doesn't come cheap (but it does come with great reliability, which is correctly more important in certain sectors).

      The closest I've come to the failed diesel generator is working next door to a company that, supposedly, had it happen.

      Given the number of people here who have claimed to have had it actually happen to them... I am more inclined to believe the reports.

      1. Antron Argaiv Silver badge
        FAIL

        Re: Happened to me as well!

        Have seen the effects and got the explanation from someone I knew (broadcast engineering stories are great!)

        The location shall remain nameless, but a major market broadcast station suddenly went off the air. UPS kicked in, generator started but did not power the facility because the wiring was incorrect. 'Twas a brand new facility and the electricians had screwed up.

        1. Will Godfrey Silver badge
          Facepalm

          Re: Happened to me as well!

          Electricians? Over the years I've come across an alarming number of them that have no idea what they are doing, both in industry and domestic environments. It seems they memorise just enough of the current regs. to get though their exams then forget everything.

          Oh, including one who was muttering about the three phase motor he had to fit, as it was going in the wrong direction. When I said offhand "You've probably swapped two of the phases", his dismissive reply was "What difference does that make?"

          Yet I'm not considered capable of adding a spur to a domestic ring main.

          1. Eclectic Man Silver badge
            Joke

            Re: Happened to me as well!

            Mr Godfrey: "Electricians? Over the years I've come across an alarming number of them that have no idea what they are doing, both in industry and domestic environments. "

            You seem surprised by this. They are obviously management trainees on their ways to higher and greater things.

            1. A.P. Veening Silver badge

              Re: Happened to me as well!

              You seem surprised by this. They are obviously management trainees on their ways to higher and greater things.

              I'll have to report you for incorrect usage of the Joke Alert! as this is the sad truth.

          2. Rob Daglish

            Re: Happened to me as well!

            Yes, I know what you mean. Some particularly resourceful chaps round my way managed to cause chaos with their wiring of a backup generator: utility power failed, UPS kicked in fine. Generator started, system attempted to move to generator, building went completely dead. Somehow, the cut out tried to send the generator power back into the grid, frying a decent sized area, which took most of the weekend to sort out. I think there were probably three or four follow up visits to the site once it had been fixed as it was tested, very thoroughly, afterwards. The owner of this particular building? Well, that would be the local electric supplier.

          3. W.S.Gosset Silver badge

            Re: Happened to me as well!

            One of the largest Aussie bushfires last season was started by a major regional transformer blowing.

            The utility's engineers had got the cables round the wrong way.

    4. Anonymous Coward
      Anonymous Coward

      Re: Happened to me as well!

      Happened at an Andover MA fab facility in the early 2000's. They'd run the generator regularly but forgot to keep the tank filled. As usual, the union shielded the idiots who didn't do their jobs.

      1. A.P. Veening Silver badge

        Re: Happened to me as well!

        As usual, the union shielded the idiots who didn't do their jobs.

        No problem, as long as the union also pays the bill (at which point the idiots need a shield from the union).

    5. Anonymous Coward
      Anonymous Coward

      Re: Happened to me as well!

      Had to evacuate a building due to a gas leak.

      IIRC, they made gas boilers....

  5. chivo243 Silver badge

    My first gig

    20+ years ago, I was managing a computer lab for high school students... this was before we had a content filter! Not enough mind bleach... not enough!

    1. DailyLlama
      Alert

      Re: My first gig

      This kind of thing is why I no longer fix people's personal computers on the side... *shudder*

      1. CountCadaver

        Re: My first gig

        Ditto...male AND female (and the computers of the latter are often worse for having oddball smut and sexual harassment no matter what the meeja say.....)

      2. heyrick Silver badge

        Re: My first gig

        Helps to be able to turn a blind eye. It's a lesson I taught myself after helping to fix the computer of a sixty-something year old woman who lived by herself and had multiple things on her backdrop and bookmarks that would suggest that she really liked gay men dressed as what's-her-name from the Wizard of Oz.

        From then on (and until a different expat made me refuse to ever do that sort of thing again) I just paid no attention whatever to anything not directly related to the job in hand.

      3. Zarno Silver badge
        Coat

        Re: My first gig

        I raise you the "back end computer" from a pizza shop.

        Stopped going to that one because there was no way that I was eating pizza stretched on hands that created that browser history and folder structure.

        Professionally repaired it, made no mention of the history/favorites, left the cache of pics, and never returned.

        Shame, because it was pretty good pie for a reasonable price.

        Mine is the one with Purell in the pocket back before everyone had a bottle.

        1. yetanotheraoc

          Re: My first gig

          "pretty good pie"

          Maybe I need to reconsider my pie standards.

      4. CuChulainn

        Re: My first gig

        I used to maintain the computers and network for a friend who had a chain of builders merchants locally.

        The kind of people he had to employ (knew the business and the parts) had the filthiest minds imaginable. But one time he was having a particular recurring problem with a computer at one branch and he asked me to look into it for him. The computer was brand new, but it kept getting various malware on it that stopped it working for the sales and inventory purposes it was primarily intended for.

        The Event Log was revealing enough (packed with pr0n), but it didn't show who was doing it. He wanted to know, so I installed a keylogger (he specifically requested it when I told him it was one possible way).

        It identified the one person doing it after only a few days - log-in details, and search terms. This was over 20 years ago, but what he was looking for was stomach-churning,

  6. KarMann Silver badge
    Joke

    Gee…

    [P]articularly randy wombats wanting a root?
    I didn't even know wombats ate roots!

    (They actually do, even if that's not quite what Bruce—no, not that one, the other Bruce—meant by this.)

    1. mmonroe

      Re: Gee…

      Generally they eat, root, shoot and leave. Ooooo Matron!

      1. W.S.Gosset Silver badge

        Re: Gee…

        "Wombat: eats roots and leaves"

        Text from an old Australian children's encyclopaedia, which immediately entered Aussie legend.

        (For the uninitiated, "root" is Aussie slang for --how should I put this? Faux-formal, methinks-- sexual intercourse, typically of a casual nature or lacking emotional involvement.)

        Unknown author Liz Truss much later nicked the joke and mangled it for non-Aussie consumption to entitle her Punctuation book, "Eats shoots and leaves".

        1. Cederic Silver badge

          Re: Gee…

          Lynne Truss. Liz sells cheese and may yet become Prime Minister.

          1. W.S.Gosset Silver badge

            Re: Gee…

            "I can't tell these support undergarments apart."

  7. SimonF

    Checklists

    The Air Force have a saying that "Checklists are written in blood"

    Someone else has invested heavily in adding each item.

    1. Anonymous Coward
      Anonymous Coward

      Re: Checklists

      The number of diesel generators for true high availability (i.e. military) is four. Two load balancing, one fully off to survive any sudden environmental change and one to do rolling maintenance

      1. WonkoTheSane
        Mushroom

        Re: Checklists

        "sudden environmental change"

        Such as an EMP (Icon chosen for obvious reasons)

        1. Dwarf Silver badge

          Re: Checklists

          If there was an EMP, then what would there to be power once the new generator came up ?

          .. and secondly ... I though it made no difference if things were on or off in regard to an EMP.. being that the pulse is induced into the wiring of everything, so this will be both sides of the magic power switch.

          1. Eclectic Man Silver badge

            Re: Checklists

            In the event of a major solar flare causing a global EMP it is, it seems, best to leave everything connected up so the pulse can be distributed across as much infrastructure as possible. If something is in the 'off' position, there is a spark gap for the induced current to leap across. Curiously one of the dangers of cave diving is lightning storms. Being the bearer of a metallic cylinder in a water-filled cave is apparently not a good idea if the ground above you is struck by lightning.

            Oh, and try to live on sedimentary rather than igneous rock.

            https://www.livescience.com/38848-emp-solar-storm-danger.html

            https://curious-droid.com/1349/how-prepared-are-we-for-a-carrington-level-solar-storm/

  8. Jay 2

    At my last company we moved into a new building which had a huge flywheel generator thingy on the roof. It was frequently tested, so everyone was under the impression it would do the job if it came to it.

    Obviously one day a JCB (or similar) went through some nearby power lines and it was time for emergency power... or it would have been had someone remembered to flip the switch from "testing mode" to "do something when the power goes out mode". Oops.

    This didn't effect me as I wasn't in the nice shiny building, but the old building a lot of the company didn't know existed which housed the computers/phone switches etc where we toiled in the basement. Ironically it couldn't be made a machine room down there as "it might flood".

  9. John Sturdy
    FAIL

    Make sure you connect the sense wire to the right side

    One (large) place where I worked had an automatic diesel generator set (ISO-container sized IIRC) set up to come on automatically on detecting loss of incoming mains, and off again once mains was restored.

    I can't remember whether this was on a deliberate test, or a real mains failure, but the first time it detected loss of power and switched itself on... it immediately detected that the power was back, and switched itself off again.

    1. My-Handle Silver badge

      Re: Make sure you connect the sense wire to the right side

      I thought you were going to tell me that it was wired up to the UPS output, so it only kicked in when the UPS ran out. But your story's better :D

  10. UCAP Silver badge

    An old story ...

    ... but a good one. Learnt this from a colleague who was present at the time.

    A certain US hospital way back in the early 90's had installed a highly resilient power system, mostly because of certain issues when they had previously lost power. Basically they had a substantial UPS to handle critical power circuits (e.g. the ICU) for up to 30 minutes, plus a backup diesel generator system that could handle the entire hospital for as long as they kept the fuel tanks topped up. The fuel tanks where regularly checked and the diesel cleaned (no waxing here!); also the whole system was checked out every month by starting up the generators and making sure that they produced the correct power. The UPS only had to handle the load for about the 5 minutes it took the generators to come online (all automatic of course) and spool up to full capacity.

    One thing about this setup, the fuel tanks where about 100 yards away from the generator for safety reasons (you do not want large fuel tanks catching fire right next to a hospital) so they hand a pretty impressive fuel pump to get the diesel from point A to point B. Again, this was checked every month.

    One day, the inevitable happened - the mains power went off. UPS took over the load fine. The diesel generators powered on and started to handle the load as designed - for about 30 seconds, then they died and could not be restarted. Major panic started; fortunately power was restored before the UPS expired, but it was a close run thing after which many trousers need to be changed.

    On investigation it was discovered that the fuel pumps (remember those) had been wired into the mains power feed only, and could not pick up power from the UPS or generators.

    1. Manolo

      Re: An old story ...

      Heard the same story about a Dutch bank.

    2. sebacoustic

      Re: An old story ...

      ...generators checked out every month by starting up the generators

      That kinda stores up problems for the service life of the diesels - they're built for running rather than being started from cold a lot, this will wear them out long before their time is due.

    3. swm Silver badge

      Re: An old story ...

      When I was in college during the 1965 blackout the associated hospital had emergency power (of course). The New Hampshire power was so unreliable that the system was used regularly. During the blackout the hospital was a blaze of lights surrounded by darkness. Sometimes these things work.

  11. trevorde Silver badge

    Collect your printouts!

    Worked at a company many years ago where the sysadmin had a *very* liberal attitude to content filtering ie there was none. He believed in treating people like adults and letting them have 'personal responsibility' for their web surfing. This worked very well until he had to have a talk to one of the guys:

    [name redacted], you know I don't look at the firewall logs for your web browsing and, frankly, I don't care. But if you do print things out, for Christ's sake please collect them from the printer!

  12. Lazlo Woodbine

    One place I worked we were given advanced notice by the electric board that they were cutting our power for a whole weekend.

    This meant the regularly tested 1 hour UPS wasn't going to cut it, so we went up to the roof to check on the big Cummings diesel genny, which after 5 years of standing idle surprisingly kicked into life almost immediately, and then it died. We dipped the tank, empty.

    So we ordered a barrel of diesel which was delivered that afternoon.

    In the loading bay we had a pump to squirt the fuel up to the roof, but it wouldn't work, it had seized up because it was about 15 years since it had last been used.

    So we rolled the barrel over to the freight elevator, which we then realised stopped one floor short of the roof, so 4 of us laboriously stepped the 55 gallon barrel up the emergency steps and onto the roof, where we siphoned the dirty diesel into thirsty generator's fuel tank, we fired it up and it worked just fine, delivering a reasonably steady 230v, which the online UPS would clean-up before it hit the servers.

    Then back in the office we got a call from Electricity North West telling us they didn't need to cut our power anymore as they'd found a different way to fix the grid without cutting us off...

    1. KBeee Silver badge

      I was amazed once when visiting a Castrol oil processing place to see a fairly small guy spinning two 45 gal. imp/55 gal. US drums on their edge, each controlled by one hand as he walked along. A final flick and spin, and the drums spun off to stand upright in their proper positions. I felt like clapping.

      1. Potty Professor
        Boffin

        Expert rtyre handling

        I worked for a tyre fitting company for a short while, just about when the "Super Single" tyres were first being fitted to semi- trailer tankers. I was only quite small then, about 8 stone, but I was sent on a course on how to fit and demount them. My first callout was to a tanker in the Services at Watford Gap on the M1. A whole gang of truckers were sitting in the transport caff, watching me and expecting me to struggle with this tyre/wheel assembly that weighed almost as much as I did. Remove wheel from axle, keeping it upright, insert specially shaped tyre levers at 11 o'clock and two o'clock, then let the wheel fall towards your legs, snatching the ends of the levers upwards just as the tyre is about to hit you in the stomach. Tyre stands up again, and wheel shoots out between your legs. Nonchalantly roll dead tyre away to sound of astonished applause from truckers in transport caff, before dropping replacement tyre onto now horizontal wheel. Only stayed with that company for three months, until I landed a draughtsman's job at a local engineering works.

    2. John Brown (no body) Silver badge

      "Then back in the office we got a call from Electricity North West telling us they didn't need to cut our power anymore as they'd found a different way to fix the grid without cutting us off..."

      Did anyone send the power company a thankyou note for teaching you the lesson that you need to check your fuel tank and fuel pump more often?

      1. Lazlo Woodbine

        Most of us didn't even know we had a generator on the roof.

  13. Bogbody

    Check you're phases....

    Part one..... when fitting a genny make sure that phase and neutral aren't swapped over behind the wiring panel ......

    Part two ..... big 3-phase UPS ..... make sure that the incoming power sense leads are actually conected to all 3 phases....

    Part 3 .... dont let a <redatcted> Manager demolish a wall in the warehouse using a JCB.....

    Part 1 -- bang, bang ... bang went the PC power supplies

    Part 2 -- the lights are on but why is the main frame not .... oh ....

    Part 3 -- big blue flash followed by much darkness and a bit of silence ......

    1. My other car WAS an IAV Stryker Silver badge

      Re: Check [your] phases....

      Our church is still "small" but a large enough building to require a 3-phase feed from the lines at the main road, fed to different parts of the building.

      Last Easter (yes, 2020, during lockdowns) when the church was only doing Facebook Live (and the worship team still came in person to perform), one phase was dropping intermittently, and we had no way to cross-connect, probably to avoid overloading the others. Half the sanctuary lights were out as were the outlets on one side, which fed:

      • Audio "digital snake" (Allen & Heath) for instrument microphones/inputs and monitor outputs
      • Keyboard, digital drums (also the sizeable Hammond organ that wasn't being used that day)
      • Powered monitor speakers
      • PoE switch feeding two "smart" headphone monitor mixers (Allen & Heath ME500)

      The main AV system, including main mixer, main speakers and front monitors, screens, HDMI camera, iMac, etc. were fine, as well as the Wi-Fi and cable modem elsewhere in the building.

      The sounds and visuals as the phase went in and out were scary: lights flickering, HVAC ramping up then going quiet again. Finally a member who was an electrician -- who had brought his daughters that day to sing -- figured it out and shut the affected main panels down completely.

      On a "normal" Sunday, we could just use the piano (baby grand) and acoustic guitar and carry on. But no, this year we HAD to stream the service, so we HAD to have working audio. Thankfully, saving our Easter merely required some extension cords across the sanctuary stage from still-working outlets. Those outlets were never used, so we had capacity to spare.

      (The local utility had things back up and running the next day, and by His grace we didn't toast any equipment in the process.)

    2. TimMaher Silver badge
      Joke

      JCB

      Wasn’t Boris was it?

  14. John Sturdy
    Mushroom

    UPS risks

    Years (well, decades) ago I went on a tour of the Mullard Radio Astronomy Observatory, and was told that the UPS for the 5km telescope was a motor-flywheel-generator set which stored enough energy to shut down the computer centre and park 8 90-ton dishes in a high wind. They reckoned that if the flywheel came off its bearings, it would roll about 25 miles, so it was aimed to have as clear a path between villages as they could arrange.

    1. Eclectic Man Silver badge
      Happy

      Re: UPS risks

      I hope the DR plan included a telephone number for emergency services in case the flywheel went walkies. It would make chasing a Double Gloucester down a hill look positively tedious.

      https://theculturetrip.com/europe/united-kingdom/england/articles/a-quick-guide-to-cheese-rolling-englands-strangest-sport/

      https://www.youtube.com/watch?v=cvuktushEhY

    2. Potty Professor
      Mushroom

      Flywheels, was UPS risks

      My father was an "His Majesty's Inspector of Armaments" at a large factory during WW2, where they produced camera lenses, gunsights, and bombing predicters. The factory was on the north side of Clapham Common, and there was a huge steam engine in the basement driving a generator to power the machines in the factory. One night in 1942, Jerry came over and bombed the shit out of the factory and surrounding area. The seismic shock snapped the 12" diameter crankshaft and the 10 foot diameter by 2 foot face width flywheel climbed out of its pit, demolished what was left of the basement wall, and trundled off across the common, eventually ending up spinning like a giant coin and settling down in a pond. They never rebuilt the factory, it was relocated in Barnet, on the northern outskirts of London, so Dad had to relocate as well, so I was born in Barnet General Hospital some five years later.

  15. An0n C0w4rd

    Check the gnerator exhaust

    One building I heard about never tested their backup generator. Cue power failure. Generator kicked in. Yay!

    No-one thought to check where the fresh air inlet for the HVAC system was. The tech support team (in the same building as the data centre) got gassed with the exhaust

  16. keith_w Bronze badge

    keeping the generator going.

    During the 2004 eastern North American power failure, I was working at a generic drug manufacturer who kept a large amount of their output in very large refrigerators. The diesel generators kicked in but they only had 4 hours of fuel. The warehouse manager tried to order more, but all the diesel deliveries were now unavailable. Some one noticed that the gas station at the end of the road still had, for some unknown reason, power so they started running 55 gallon drug transfer barrels to the gas station on handcarts and bypassing the cars in the fuel queue to keep the generators running.

  17. ColinPa

    Just do it!

    A major bank had a power outage. The operators opened the procedures book and started the fail over to the backup site - a well practised procedure.

    A was senior executive was passing and came to assist, saying "We have these big generators in the car park for times like this. Restart the systems in place"

    The operators said "That's not in the book - it has not been tested". Senior Executive "If you can restart in place it will be back in no time! - read my job title"

    They did what they were told - only to find the generators did not provide enough power for the machine room, and they could only start half the kit.

    The operators made the bold decision to wait for the power to come back. They said that they thought they could not successfully fail-over to the backup site once they had started the restart. Not the sort of thing you want to first try in production.

  18. mickaroo

    Need a bigger UPS

    Back in the day, I was leading a team qualifying the server room for a government service responsible for medical supplies across the entire country. The project scope included building security, server room air handling, server room fire suppression, and emergency power.

    The power test was to take down the main power to test the generator auto-start, then take down the generator to test the switch over to UPS, then bring the generator back on line, then switch back to main power. I had been told by the IT manager that the UPS was good for forty, maybe forty five minutes.

    Take down main power feed: Reassuring rumble from the roof as the generator kicked in.

    Take down generator: Everything still running as the UPS kicked in.

    IT Support Guy: "You better get the generator back up. The UPS is overloaded; it's only good for about ten minutes" WHAT!!! I was told forty! IT Support Guy: "Who told you that?"

    Frantic walkie-talkie call to guys on roof to fire up generator... NOW! Reply from guys on roof, "It needs a five minute cool down before we can restart".

    The generator came back online with about two minutes to spare.

    1. UCAP Silver badge

      Re: Need a bigger UPS

      And the moral of the story - talk directly to the technical guys, they almost always have a better idea of things than the PHBs.

  19. Sam not the Viking
    Mushroom

    Diesel Generators

    As a post-grad student at my first real employment, it is one my deepest embedded memories of a 1MW diesel-generator starting to fail after operating at 120% load for a few hours; it was OK, we knew what we were doing.... The alternator caught fire, evidenced by streams of smoke from the engine cell. The reaction was exactly like an episode of the Keystone Cops: guys in overalls running in every direction, across, into and out of the smoke, I thought for our entertainment.

    The Chief turned the load off. Island application, so limited fall out.

    It was running again a few hours late at a lighter load..... The 'Cops' were a bit shamefaced for evermore.

    It was the first time I had experienced that 'electrical overload' odour that has proved useful more often than I care to admit.

  20. Robert Sneddon

    Didn't happen to me but...

    I was told this tale by an IT support guy I worked with back in the day. He did a lot of shop and small office work and one of his customers was a large store which had a couple of back-office servers plus PCs for stock control etc. which were supported by a smallish UPS. If there was a power cut the store would be closed but the UPS would allow the office staff to shut down everything tidily before the UPS power ran out. One of his regular duties any time he visited the site was to check out the UPS under load, make sure it worked and that the batteries were swapped out every couple of years to keep them functional.

    Day came, there was a power cut, the UPS ran for less time than expected and the servers went bonk! before they could be soft-landed. Oops. Investigation discovered someone had taken the microwave out of the staff break room and plugged it in to a UPS-fed socket over by a stock control PC to heat up their lunch while they waited for the mains power to come back on. Ding!

  21. Hazmoid

    Worked for a Stockbroker in Perth on the Esplanade back in the 90s and the building we were in had a large genny set in the basement to provide essential power. During a major winter storm, the power supply to Perth was taken out for about a week. During that time, the building management had to bring in tankers of diesel to keep their generators running. I remember having to remind staff that the red outlets were for essential computers only and no you could not plug your mobile in to get it recharged.

  22. Grey_Kiwi

    It happens Down Under too

    When Auckland City had a major, weeks-long power outage [20 Jan – 27 Mar 1998 according to Uncle Google], lots of businesses found out two unexpected things about their backup generators.

    The first was that, it being summer time, the generators tended to get all hot & bothered if they were required to run for hours & hours at a time, rather than the one hour every other Tuesday lunchtime that the testing regime required. A lot of sheet metal was bashed into airflow ducting and had fans installed to keep fresh coolish air blowing onto the generator.

    The other was that the diesel tank went from full to empty in between sixty and seventy hours, so needed a refill every other day. Since the tank was down a twisty narrow central city service lane, a "mini tanker" was always used. The fuel companies ended up getting every mini-tanker in the country to Auckland to feed the generators - they even drove one from Invercargill to Auckland, including ferry across Cook Strait, using a relay of drivers.

    Fun times ... not

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021