back to article Making the problem go away is not the same thing as fixing it

Welcome once more to On Call, The Register's weekly column featuring readers' turbulent tales of their tech support troubles. This week, meet a reader we'll Regomize as "Mal" whose first job in tech was programming a mainframe at a small mutual financial services outfit. "I worked on a financial package written in COBOL called …

  1. SonofRojBlake

    Not an IT bod - chemical engineer here. But you might be surprised to learn how often large chemical factories - some of them handling things that are frighteningly expensive, toxic, explosive or all three - "solve" nuisance alarms by looping out the instrument that's causing the alarm to go off. It happens less and less nowadays, in fairness... but it hasn't stopped.

    1. Jou (Mxyzptlk) Silver badge

      Watch the follow-up of large accidents on the U.S. Chemical Safety and Hazard Investigation Board Youtube channel. I DO believe you. And Germany, especially the former DDR, had its fair share of chemical accidents due to negligence.

      1. SonofRojBlake

        I have had to watch a lot of CSIB videos in the course of my work, and the most terrifying thing about them is the coda where they describe how, having investigated the incident described, they identified the cause and disseminated the information to other plants operating the same process.... who ignored the advice because it would cost them money to implement, and the basically identical incident was repeated x days/weeks/months later, with y injuries and z fatalities.

        1. Joe W Silver badge

          Also interesting: "Well, There's Your Problem"...

    2. Headley_Grange Silver badge

      I had to put a guy on a final warning because his desk was right below the fire alarm bell and he got fed up the weekly tests so one day he took the bell off on the morning of the test. He came within a gnat's chuff of being sacked.

      1. SonofRojBlake

        That sounds like an open and shut case of deliberate defeating of a safety system without a work permit, gross negligence instant dismissal marched off the site with your personal effects in a box. I've seen it done.

        1. Prst. V.Jeltz Silver badge

          jeez chill its one bell during a drill ,

          nothings that B&W

          its not like disabling the temp gauge in a nuclear power plant to avoid getting a call out

          Wrong ? Yes ! all I'm saying is "instant dismissal marched off the site" is not a tactic to deploy every chance you get , some consideration sometimes required.

          I used to put some ear defenders on during the weekly bell test, I hope that wasnt "deliberate defeating of a safety system"

          The offence described in the story by the predecessor on call seems much more negligent (and should be sued imho) , albeit not safety related.

          1. Phil O'Sophical Silver badge

            It may be just one bell, but if he forgets to repair it, or is taken ill and leaves before he can do so, then what? I doubt if the people injured in a real fire later because there was no bell to alert them would see it as a matter to just "chill".

            Never mind just being sacked, he was close to criminal endangerment charges.

          2. JimboSmith Silver badge

            Lived in a flat years ago and one night a car alarm sounded outside at around 2am. It was a posh car and even after the alarm had stopped it bleeped loudly every 30seconds to let you know the alarm had been triggered. I had ear defenders and put those on but someone who didn't left a note on the offending car stating that they hadn't slept last night. They continued about how it was insane to have that fecking annoying bleep goiing off every 30 seconds especially as it was a false alarm. something else about how the next message would be left on the windscreen with a brick attached.

            1. Alan Brown Silver badge

              In the UK an alarm like that is a statuory noise nuisance under the Environmental Protection Act 1990 and a council or police CAN order the owner/operator to disable it immediately if it's deemed to be a serious nuisance when they check it

              https://www.westminster.gov.uk/planning-building-control-and-environmental-regulations/environmental-policies-regulations-and-guidance/noise-pollution/keep-your-noise-down/car-alarms

              Westminster Council explicitly say that if the owner can't be located they may disable the alarm themselves after an hour (breaking a window if necessary) and may impound the car

              Many other councils have similar policies but on the flipside they don't have out of hours complaint numbers or staff who can attend

          3. probgoblin
            Flame

            "It's just one bell during a drill" will probably not hold up in court should he forget (or otherwise fail) to reenable it and it's need for something that isn't a drill. When you create a potential liability for your employer, you should not be surprised should they decide they no longer want to be your employer.

            And that's before we get into the whole "morality of risking your co-workers lives" thing.

          4. SonofRojBlake

            "nothings that B&W"

            Au contraire Blackadder. On a upper tier COMAH site, you'll find that a lot of things very much are that black and white. It's not about how critical THIS safety system you deliberately defeated was THIS time that you defeated it. It's the fact that you're the kind of person who will defeat ANY safety system EVER without orders signed in triplicate - the kind of person who needs to be removed with extreme prejudice from any facility featuring a major accident hazard, lest your judgement fail and fill the housing estate/hospital/school next door with chlorine or something.

            And to state the bloody obvious, putting ear defenders on during an alarm test is not "defeating a safety system" since (a) it didn't stop anyone else hearing the alarm and (b) it didn't actually stop YOU from hearing the alarm, just dulled it a bit and stopped it causing actual damage to your hearing.

      2. Caver_Dave Silver badge
        Thumb Down

        When fingers in the ears is a problem

        One place I worked there was one of those very piercing electronic alarm sounders right next to the urinals, so it was difficult to put your fingers in your ears when it went off.

        Annoyingly the weekly test was any time within a 2 hour period, so that had 2 problems:

        * Everyone waited for at least 30 seconds to see if it turned off i.e. was a test

        * You dared not go to the toilet during those 2 hours until it had gone off - and then there was a rush

        1. DS999 Silver badge
          Devil

          Re: When fingers in the ears is a problem

          You dared not go to the toilet during those 2 hours until it had gone off - and then there was a rush

          If it was enough of a shock when it went off then you wouldn't need to join the rush to the toilet!

      3. nijam Silver badge

        > ...desk was right below the fire alarm bell ...

        Well, that's a H&S issue straight away, unless he's been issued with earplugs.

        Reminds me that I once work at an organisation that tested the fire alarms weekly by slackening off the screws on a cover of the " in case of fire, break glass" units. Yes, you guessed... the threads eventually wore away. Tested to destruction, I believe the phrase is.

        1. Anonymous Coward
          Anonymous Coward

          One employers had a two stage alarm which was largely ignored when it went off. First stage was an alarm (smoke/break glass) has been triggered we're looking into it stay put. Second stage was evacuation and could be triggered at the alarm control panel or if the first stage wasn't cancelled within five minutes. During a weekly test the bloke from facilities raced round the building checking you could hear the alarm from all the alarm speakers. People would place weekly bets on whether he'd make it to all floors and back to the panel before the second stage was triggered. He could only just make it so wasn't cut and dried either way. Only had a real activation of the second stage once whilst I was there, everyone was very shocked and evacuated but relieved that it turned out to be a false alarm, the culprit was a guest smoking.

      4. Terry 6 Silver badge

        In the Uk, if I remember my training right, that would be a criminal offence.

      5. Anonymous Coward
        Anonymous Coward

        Why would there need to be weekly tests of a fire bell? That starts sounding more like management annoying employees than an actually useful system test.

        1. 080

          High Risk Sites

          On high risk sites not only is a weekly fire alarm test mandatory but each alarm on the site has to be witnessed as working on a monthly routine. Our test was at exactly 10.30 on Thursday with staff positioned to witness alarms on a schedule this was followed by a test of the evacuation alarm. Long due to the staff having to walk around several alarms and very loud with sirens.Any interference with fire alarm systems was a disciplinary offence.

        2. Rob Daglish

          It’s box ticking. When the building burns to the ground and seven people are turned to charcoal, how else do you prove to their grieving relatives it wasn’t your fault because the fire alarm worked, and it was nothing to do with all the crap you had strewn across the exit routes?

          I also think it’s silly to do it at the same time every week, because people become habituated. One office I work in does it half ten every Wednesday, so if there’s ever a fire then, nobody will take any notice. Much better to move the time around but publish when you’re going to test it I think, but hey ho.

          1. Robert Carnegie Silver badge

            Testing the alarm at a set time does mean that you know it's probably a test when that happens. You make a good point that it might not be a test.

            When our alarm goes, first of all I check whether it's time for the test. If it isn't that time, then I leave the building. Otherwise, long story short, if it doesn't stop straight away then it still isn't a test. But it may be an evacuation drill.

            If test time comes and the alarm doesn't ring, then that's obviously a problem.

            I would like us to be e-mailed about 2 minutes before the test, so that we can be prepared for it. Or I suppose I could set a reminder alarm for myself, but it is only once a week.

        3. Mike007

          We have a weekly test of what I believe are called the "oops" sirens. The local naval yard is certified to do engineering work on nuclear reactors, therefore they are required to have (and regularly test) a siren that the entire city can hear...

      6. FeepingCreature Bronze badge

        Is it just me, or is it stupid, counterproductive and borderline health hazard to do weekly tests of an alarm system while people are working? Sure, if you're testing evacuation procedures that makes sense (though I'd never test those *weekly*, that just trains people to treat every alarm as a test), but if you're just testing the alarm itself, surely you'd do it outside of working hours?

        Forget "final notice", if it was me I'd assume I was being hazed and start looking for another job.

      7. SMDTS

        I would bet that if you were to measure the spl of the alarm above his desk it would exceed the legal threshold and put him at risk of permanent hearing damage if it happens every week. Really should of been him taking the employer to a tribunal for breaching h&s when it could have been solved by moving the sounder, or attenuating it and fitting an additional sounder elsewhere to compensate but that would cost money.

    3. Anonymous Coward
      Anonymous Coward

      Another technique

      Another chemical engineer here. Many years ago while on shift experience at a refinery I was shown an alternative technique of using a matchstick to hold down the alarm acknowledge button to "solve" a nuisance alarm. I suppose at least the sight of the matchstick sticking out was a reminder that something was amiss.

      1. Anonymous Coward
        Anonymous Coward

        Re: Another technique

        One control room I worked near used a pen cap to hold down the shift key. But it was only to prevent auto-logout, and didn't conceal/autoacknowledge any alarms.

    4. Dr Dan Holdsworth
      Black Helicopters

      The ultimate cause of the Chernobyl accident was similar to this. The reactor design was one which, if you put in fuel rods containing depleted uranium, could be used to make plutonium. Plutonium is military stuff that goes bang, therefore methods of making it are secret and everything around secret stuff has to be secret, or so goes military doctrine.

      One of the things that was made secret was an annoying little misfeature of the reactor design. The core was graphite and graphite expands when hot. The control rods were engineering fit rods, and as such tended to stick a little when the reactor core was hot and had expanded a little, thus making the fit a little bit tighter. To solve this, the first eight inches or so of each control rod was graphite, which as well as a nuclear moderator is also a good lubricant.

      The problem with all of this was simple: graphite is a moderator; stick graphite into what was an empty hole and the reactor initially starts to run even better than before, until the rest of the rod descends into place and the cadmium starts absorbing neutrons and shutting down the nuclear reaction.

      Standard "Oh golly, shutdown" procedure with a nuclear reactor is to drop control rods into it as fast as possible, even all at once to shut down the chain reaction FAST. Do that with this sort of reactor and you suddenly make it run a lot lot better which makes the core hotter which jams the control rods at that point, and shortly afterwards the whole thing goes seriously bang.

      Correct shutdown procedure here is firstly keep all coolant systems running full belt, and secondly insert rods quickly but sequentially. You don't shutdown the reaction quite as fast, but you do live to tell the tale.

      All because that little foible was a military secret that the civilian plant operators weren't cleared to know.

      1. Vometia has insomnia. Again. Silver badge

        My understanding is that RBMKs weren't used by the military in spite of their on-load refuelling, a similar feature to Magnox (which were used for that) and AGR (which aren't). The military had their own dedicated reactors for that.

        The design flaw was that the control rods had graphite "tips" which is a misnomer as they were several metres long; but crucially not the full height of the reactor, so when the rods were lowered, the graphite part entered the bottom zone of the reactor and instead of slowing everything down, they made it accelerate. Sometimes a lot. Excursions weren't unknown, at least to the design team, and there'd been a previous albeit much smaller one in Chernobyl Reactor 1 (or 2) and elsewhere. Ironically one of the things the specialist engineers (i.e. the ones who knew what was otherwise kept secret, even from the heads of ops) were going to do that evening after the reactor had been shut down following the power-failure contingency test was to install a fix to prevent the problem occurring. Sadly, because of an extremely unfortunate series of events whose consequences were unforeseen... well, the rest is history.

        I'm not sure what the fix was, possibly to winch up the short control rods that lived under the reactor first and only then drop the others.

        There's a lot of misinformation that the AZ5 switch (SCRAM) was used after things went wrong but this wasn't the case, it was just the standard means of shutting down the reactor. There's a video of its neighbour, Reactor 3, being shut down for the last time 14 or 15 years later where you can see the operator doing the same thing.

        I'm not sure where the theory about the rods sticking comes from; possibly because they descended at an absolutely glacial rate, just under the force of gravity but crucially against the flow of cooling water. IIRC they reversed the direction of water flow in the technical channels (i.e. control rods, sensors etc) to solve that problem.

        1. Alan Brown Silver badge

          RBMKs might not have been used for that, but they were direct descendants of the reactors which WERE used to produce plutonium and when they were announced as a power design in the 1960s the Soviets were implored not to proceed with them them precisely _because_ of the risk presented which caused Chernobyl to burp

          (Ie: the problem and risk was known about even before the RBMK designs had their foundations marked out)

      2. Terje

        Unless my memory fails me (which it very well may, it's Friday after all) One of the main issues was that the documented operational procedures for the reactor were ignored during the tests, when the energy output got to low you had to shut the reactor down and restart from scratch, the director didn't want that and when trying to bring power up it caused a hotspot and follow on bad things.

      3. Catkin Silver badge

        The graphite tips were there because RBMKs are built to be as economical as possible and their absence would have led to insufficient fuel burn up in the top portion of the reactor. Simply moving the rods further away wouldn't have been sufficient because that would have left a void in the fixed moderating graphite. The swelling is a separate issue with graphite in low temperature reactors.

        On the night of the disaster, the rods sticking was almost immaterial; the reactor was in such an unstable state from xenon poisoning that the lower portion of the core went prompt critical. What should have happened is that as soon as power was unable to be raised further, a full shutdown should have been initiated, rather than trying to coax a meagre power level through withdrawing almost all the rods.

    5. An_Old_Dog Silver badge

      La-La-La-La I-Can't-Hear-You / Humperdinck! Humperdinck Humperdinck!

      Walking in an urban industrial area early this morning, I heard wailing sirens, and saw a flashing green light (yes, it was green) from inside a fenced-off petrochemical tank farm. No personnel could be seen outside, though I'd expect to see some briskly-moving people somewhere inside there. "It's my coffee break, man. Don't be a killjoy."

    6. anothercynic Silver badge

      One word for that: BHOPAL.

    7. Alan Brown Silver badge

      I saw a multi-year, multihundred million dollar space project go awry because staff did something like this when actual results didn't match the predictions by kludging the output tpo make it fit

      Thankfully the project leader eventually spotted it (several years in) and fired the people concerned. The result was some new discoveries but he was spitting feathers as the entire thing nearly got published with bad data embedded in it

  2. Pascal Monett Silver badge
    Mushroom

    So, shoot the messenger is still well and alive

    When a so-called professional "solves" a problem by removing the alert, said individual should be reassigned to the deep-fry section of a burger joint and has nothing to do in a technical environment whatsoever.

    1. UCAP Silver badge

      Re: So, shoot the messenger is still well and alive

      I'd argue that frying burgers is probably well above their level of technical competence!

      1. Handel was a crank

        Re: So, shoot the messenger is still well and alive

        To be fair, he didn't say that said person would be operating] the deep fryer...

        1. RichardBarrell

          Re: So, shoot the messenger is still well and alive

          Don't serve long pork to customers!

          1. Strahd Ivarius Silver badge
            Devil

            Re: So, shoot the messenger is still well and alive

            You want to change the recipe used in all fast-food joints in Louisiana?

      2. SVD_NL Bronze badge
        Flame

        Re: So, shoot the messenger is still well and alive

        Please no! They would disable all fire alarms because they keep going off when his lack of object permanence makes him forget about the french fries!

    2. Filippo Silver badge

      Re: So, shoot the messenger is still well and alive

      Depends. There may be incompetent design behind the alert, too. I've seen many cases where the existance of a warning was actually a mistake in the control software. Also, sometimes things just change and a warning can become obsolete.

      1. Eclectic Man Silver badge
        Facepalm

        Re: So, shoot the messenger is still well and alive

        I was given a Toyota Yaris (1.4l, 4 speed automatic since you ask) by my father when he decided a year ago that he probably should not drive at the age of 90-something. Anyway, driving it on the M42 back from visiting him the engine warning light came on. I drove the next 20 or so miles to the next service station in some trepidation and looked it up in the manual. It said - take the car to the local service centre. Just that, nothing else. It could be a fault with the engine, transmission, anything. Well, I drove home rather tentatively noting that the car handled normally.

        Took it to a service centre a few days alter, and they charged me a mere £30 to tell me it was a transient fault on the automatic transmission, and they had turned it off.

        The next time I visited my father exactly the same thing happened, except I just left the light on until it went out by itself. I have since visited him again, and yup, the engine warning light is 'on'. It takes a few days to go out, so should be ok by Sunday, I hope.

        1. dak

          Re: So, shoot the messenger is still well and alive

          A £12 OBD reader off eBay will bring peace of mind.

          1. Doctor Syntax Silver badge

            Re: So, shoot the messenger is still well and alive

            "OBD Reader"

            That and a bit of Googling revealed that the engine fault light on a Subaru was due to the fact that going downhill a switch was wrongly indicating the gear box was in neutral and the ECU was complaining it couldn't control engine speed.

          2. Dave314159ggggdffsdds Silver badge

            Re: So, shoot the messenger is still well and alive

            £12? Oddly specific number. You can get one for a quid or two that will do the basic job, or pay more for a decent one that can read the model-specific codes. Depending on age you can also connect a couple of pins in the OBD port and the car will flash the codes using the dashboard lights.

        2. cookieMonster Silver badge

          Re: So, shoot the messenger is still well and alive

          Reminds me of that scene in Big Bang Theory

          1. cookieMonster Silver badge
            Facepalm

            Re: So, shoot the messenger is still well and alive

            Seriously, a thumbs down, for a reference to a comedy where there’s actually an engine warning light scene. WTF???

            There are some seriously fucked up humans on this planet, jeeeeezz.

            Asshole.

            1. KarMann Silver badge
              Holmes

              Re: So, shoot the messenger is still well and alive

              FWIW, I was borderline on thumbs-downing it myself (but didn't), but if I had, it would have been for just referencing 'that scene' as though we're all supposed to know just what that scene is. Not for referencing that comedy, but for referencing it too vaguely.

        3. Kevin Johnston

          Re: So, shoot the messenger is still well and alive

          Had a similar issue with a Ford Fusion where the light came on and after carefully getting it to a mechanic it turned out to be the super-secret Ad-Blue tank behind the read bumper had reached the low limit (was then another £200 for Ford to top it up and clear the flag since only they have the required tools) - shysters

          1. Mint Sauce
            Boffin

            Re: So, shoot the messenger is still well and alive

            Hmm, AdBlue needs to be topped up (fairly) regularly so is unlikely to be hidden. So I wonder if Ford were also using a similar system to Stellantis - i.e. a hidden fuel additive tank or pouch (Eolys, pat fluid, other names) that the car decides is empty after certain parameters are met (e.g. 100,000 miles travelled / x fuel ups, or somesuch). THat sounds more likely including the price to refill.

            I had the emissions light come on on a Subaru once. Of course dealers want to replace lots of expensive parts. Some internetting later and I discovered the issue was that I hadn't closed the fuel cap enough 'clicks' after the last fuel up. This lead to a sensor somewhere getting upset and throwing the code...

            Sadly main dealers no longer employ mechanics with diagnostics skills. They seem to be just parts-fitters who are presumably on comission for each unnecessary part they replace!

            1. Martin an gof Silver badge

              Re: So, shoot the messenger is still well and alive

              As the owner of a Berlingo with AdBlue I'll have to have a good scout around for a secret tank of special sauce.

              Had the emissions light come on in the Berlingo a year or so ago - but not until about 20 miles after I'd noticed a huge plume of vapour from the rear end and hastily pulled over, expecting it to be a blown gasket or something. Stepping out of the car and finding my eyes watering with the smell of urea I reasoned the thing was safe to drive after all. Then the light came on along with the little warning "take the car to the garage now, or it will refuse to start in x hundred miles". Took it to the bloke who usually does our cars who said "known fault - had one in last week - it's the AdBlue injector. It falls apart and dumps the entire tank into the exhaust in one go". Bosch (IIRC) marked "Citroen" was quite literally twice the price of the same Bosch part without the marking.

              And then, a few thousand miles later, the timing chain snapped. 50-ish thousand miles and the timing chain (not a belt, mark you) just disintegrates. Again a "known fault" but this time Citroen have admitted it's a manufacturing fault and repaired the car - though the story behind getting to that stage is long and convoluted. Frankly I think it should be the subject of a recall, so if anyone here owns a Stellantis vehicle with 1.5dCi AdBlue engine, post about 2018 I think, it might be worth asking if it's one of the affected models before you end up with a bag of scrap rattling around in your sump only to find that they've withdrawn the free repair service.

              M.

            2. rhydian

              Re: So, shoot the messenger is still well and alive

              "Hmm, AdBlue needs to be topped up (fairly) regularly so is unlikely to be hidden. So I wonder if Ford were also using a similar system to Stellantis - i.e. a hidden fuel additive tank or pouch (Eolys, pat fluid, other names) that the car decides is empty after certain parameters are met (e.g. 100,000 miles travelled / x fuel ups, or somesuch). THat sounds more likely including the price to refill."

              This is the answer. Ford used DV4 and DV6 Peugeot/Citroen diesels in the Fiesta, Fusion and Mondeo range in 1.4 and 1.6 capacities, and IIRC they were fitted with an Eolys aftertreement system which as you say needed refilling and recoding with the correct tools.

              1. Anonymous Custard
                Trollface

                Re: So, shoot the messenger is still well and alive

                Yup my old Peugeot sprang the Adblue one on me too.

                Generally I'd had it services at the dealers (laziness as they're across the road from work), but that time I'd had it done at Kwikfit.

                Cost about half as much, but presumably the dealers had topped up the Adblue as part of their service which Kwikfit hadn't.

                Anyway up pops the warning about Adblue needing topping up within 1000 miles (or somesuch) or else the car would refuse to start.

                Quick scan of the service manual and a hunt turned up the filler cap under the floor in the boot, and £10 at Halfords got me a 5l bag of Adblue to fill it with.

                Only real issue is the car design meaning that even with a funnel it was a pain to get the liquid actually into the tank without some ending up in the boot (knowing when to stop filling so you didn't end up with excess in the funnel and no way to drain it) as the cap was basically horizontal and level with the boot floor, but that's yet another of the design niggles that Pugs are famous for I guess...

                1. adam 40 Silver badge

                  AdBlue - literally taking the piss

                  Well, that's another new car feature that I must avoid - thanks for the heads up!

                  As it's urea and water, well you get that in piss, so that's what you should top up with.

                2. Rob Daglish

                  Re: So, shoot the messenger is still well and alive

                  We had a Citroen C4 with the 1.6BlueHDi diesel in, and it was fitted with the bloody awful adblue system.

                  I eventually got rid of the car as Citroen had made the AdBlue tank a sealed unit, so no air could get in to replace the liquid being used, which necessitated one set of recalls that missed us out. Then there was the fact they used a plastic sensor which was eventually dissolved by floating in acid for 5 years or so, and then the fact that at around the same point, everyone who had one of those rotten vehicles found that the AdBlue pump, which was fitted to the tank in such a way that it was a whole tank replacement, failed pretty much simultaneously, causing months of delay as they didn’t have stock of tanks to fit them. Oh, and you had to go to a Citroen dealer, as nobody else could code the Body Control Unit that ran the damned thing. Don’t ask why it wasn’t on the everyone programmable ECU instead…ours died and wouldn’t run before the tank arrived, so it was effectively written off at that point.

                  Incidentally, the first time our local (usually knowledgeable) garage looked at it, they wanted around 400quid to do the whole Eolys replacement as the computer was complaining the tank was empty, cue a confused conversation with them about why they wanted 400 quid for topping up the tank in the boot with 20 quids worth of AdBlue, and it turned out that the ECU was miscondigured to start with.

            3. Dave314159ggggdffsdds Silver badge

              Re: So, shoot the messenger is still well and alive

              "Sadly main dealers no longer employ mechanics with diagnostics skills."

              They never did.

              "They seem to be just parts-fitters who are presumably on comission for each unnecessary part they replace!"

              And yet more proof that there is literally nothing the commentariat here can't turn into a weird conspiracy theory.

              If you let people play parts-roulette instead of diagnosing the fault, that's your choice.

            4. M.V. Lipvig Silver badge

              Re: So, shoot the messenger is still well and alive

              "...and I discovered the issue was that I hadn't closed the fuel cap enough 'clicks' after the last fuel up. "

              That is a Small EVAP Leak code, and is one I turn off on any vehicle I tune. It's a nuisance code, mandated by the same sort of people who mandated those stupid valves on gallon gas cans that keep you from having a seeping vapor leak by spilling half a gallon of fuel everywhere when you try using it.. There's two levels for it, and only the Large EVAP Leak means there's an actionable problem with the system. When that one comes on it usually means the EVAP cannister has a dry rotted hose or road debris has bounced up and split the housing.

              1. jake Silver badge

                Re: So, shoot the messenger is still well and alive

                "those stupid valves on gallon gas cans that keep you from having a seeping vapor leak by spilling half a gallon of fuel everywhere when you try using it"

                Every single plastic "safe and non-polluting" gas can that I've ever used has spilled more gas than the "unsafe, polluting" steel jerrycans of my yoot, Until a friend turned me on to Rotopax gas cans (rotopax.com), retro-fitted with aftermarket jerrycan-style spouts. These spouts are plastic, but because of their sealing gasket/o-ring/gromet they don't leak. (Try amazon or ebay, search on "rotopax aftermarket spouts").

                Rotopax gas cans come in all shapes and sizes to fit most needs, and have a very useful and secure mounting system. They are spendy, but they work, don't spill or leak. and they last. I have a few knocking around here that are over six years old, used near daily, and show no signs of quitting on me. Beware of chineseum imitations, get the real thing. Highly recommended.

        4. Strahd Ivarius Silver badge

          Re: So, shoot the messenger is still well and alive

          This does look like a warning by the car that your are approaching the location of its previous owner, doesn't it?

        5. Anonymous Coward
          Anonymous Coward

          Re: So, shoot the messenger is still well and alive

          Folks had a diesel smart. Killed exhaust sensors and went into limp mode. "Fixed" three times under warranty. Same one every time according to obd reader. Not sure if they replaced a faulty sensor when that was never the problem or just killed the code. Great shame, it was like driving a very frugal roller-skate!

        6. Eclectic Man Silver badge
          Happy

          Re: So, shoot the messenger is still well and alive

          Today is Sunday. I drove the car. Engine warning light no longer illuminated.

        7. Cliffwilliams44 Bronze badge

          Re: So, shoot the messenger is still well and alive

          My wife's car has a similar issue. Every so often the traction control light will come on, indicating that Traction control is off, when it is obviously on. I've looked this up extensively and everyone who owns this model car says the same thing. It is a fault in the electronics and to fix it, disconnect the battery, wait 10 seconds and reconnect it, this reboots whatever electronic device controls the Traction control.

          Ironically, my wife's car is a newer model of her previous car, a Scion XB (Yup, Toyota). Her previous car had the same issue. When we took that car to the dealer that r3eplaced the O2 sensor, which did absolutely NOTHING to fix the issue!

      2. Will Godfrey Silver badge
        Facepalm

        Re: So, shoot the messenger is still well and alive

        Indeed, but when that occurs it helps to remove both the test and action, not just the test, thereby forcing the action - not that I know anyone who ever did that.

    3. DS999 Silver badge

      Removing the alert isn't a bad option

      So long as you let everyone know that's what you did and that it was only to avoid the problems that would otherwise result, while you looked into the problem at greater depth to properly solve it.

      But yeah just removing the alert so you never get paged again, and decide to assume it was a useless warning, should be a firing offense unless you told your supervisor that's what you did and he was OK with it. Not that you can fire someone if they are already gone, but sometimes a guy is hired to replace someone who failed up and is now his manager.

  3. Admiral Grace Hopper

    "oh yeah ... that"

    Back in the times of mainframes and driving to the Ops room when called out, I came in one morning to find that the team that sat next to us were all wearing frowny faces, bar the one angry and tired face and the guy with a face full of shame. Tired And Angry had been called out the previous night to discover that Shamefaced has been called out for the underlying problem the week before, fixed the problem by commenting out a loop with the intention of fixing it properly before the next weekly run and then forgetting all about it because beer. Without the loop the next run fell over in a bigger and better mess at the next run. Never underestimate the power of taking notes and holding proper handover meetings/stand ups.

    1. Alan Brown Silver badge

      Re: "oh yeah ... that"

      "Never underestimate the power of taking notes and holding proper handover meetings/stand ups."

      That's fine if manglers and other staff will actually let those happen

  4. J.G.Harston Silver badge

    It's the old burning toast + smoke detector problem. Stop burning toast? Nah! Disable the smoke detector.

    1. Jou (Mxyzptlk) Silver badge

      Hence why in overregulated-Germany smoke detectors are NOT mandatory for the kitchen.

      1. Headley_Grange Silver badge

        Scotland's recent legislation on domestic detectors requires a heat detector in kitchens.

        I used to live in a house where the detector in the hall was too near the kitchen so it went off without much provocation. It was great for grilling meat. Put the bacon/chicken/chops under the grill, leave the door open and retire to the living room to watch telly. When the smoke alarm went off it was time to turn the meat over. The meat wasn't burnt - it was just right. I think it was fumes from the fat that set the detector off because it was always a bit late if the grill pan was clean.

      2. Michael H.F. Wilkinson

        I have found many smoke detectors happily go off when enough steam develops in the kitchen, or just when you are frying or grilling something, which is why I put a CO detector in ours (and another one near the central heating system). It doesn't respond to usual cooking activity, but should respond to an actual fire.

        1. Dr Dan Holdsworth
          FAIL

          Steam sets off smoke detectors just fine. How do I know this?

          Many years ago I was living in what amounted to a hall of residence for Rothamsted Research Station; the old Rothamsted Manor house. This was an old and rambling house that was mainly wooden and was essentially one great big fire trap; if the alarm went off you got out fast.

          Anyway, some muppet had decided that the best place for a smoke detector was next to a bathroom with two showers in it. Not just any shower design, but thermostatic showers that were always that little bit too hot, so you always ran them full on to try overload the hot water input and cool the shower a bit. Given how hard the water in that area is, pumped straight from chalk aquifers, thermostatic valves vary between being cleaned of limescale and being in need of cleaning; these were a silly idea.

          Anyway, one day I was in these showers, bathroom window closed because it was brass monkeys outside, and the inevitable happened: steam set off fire alarm, alarm went off, I got out, dryish and dressed in record time and thereafter was straight into "Baldrick, deny everything!" mode, convincing nobody at all but proof was not really available.

          The powers that be installed an extractor fan on a timed PIR thereafter.

          1. Peter Galbavy

            Oh, same same. My home alarm-tied smoke detector is in the upstairs hallway outside the bathroom. At the time it was also linked to a monitoring service.

            Cue lodger having a nice hot shower, opens door, alarm goes off. Not fun. She turns it off on the alarm panel and thinks nothing more for a few minutes until the fire brigade turns up with sirens on and all - she opens the front door in her towel and much red face.

            The alarm company (an independent and NOT one of the big boys) replaced the detector with one that is not set off by steam or hot air and no issues since.

            1. Anonymous Custard
              Headmaster

              You'll often see notices in hotel bathrooms about this very issue, as presumably with the number of rooms/showers involved it would be happening with annoying regularity...

      3. Martin an gof Silver badge

        Hence why in overregulated-Germany smoke detectors are NOT mandatory for the kitchen

        Smoke detectors are not mandatory in a UK (domestic) kitchen either (I suspect also commercial, but I don't know), and advice is to fit a Rate of Heat Rise detector instead. There are also types of smoke detector which are less susceptible to cooking fumes or bathroom steam (I think the optical type is better than the ionisation type). Then again, apart from the RHR in the kitchen, we have optical detectors throughout and - I dunno - one of them must have been faulty or something because we had regular false alarms, often in the middle of the night. I took to disconnecting them one by one. All I have to do now is buy a compatible replacement (they are interlinked). The little switch in the utility room with the button that silences all except the alarm which triggered was very handy, except that in most cases the triggering was obviously transient because pushing the button caused all the alarms to silence :-/

        In Wales, of course, new builds now have mandatory sprinkler systems too. It's amazing how many people believe that it only takes a match to set one off, and that once one goes, they all go.

        M.

        1. Phil O'Sophical Silver badge

          Mandatory sprinklers, 20MPH limits, I'm surprised they let you leave the house without hi-vis vests and a hard hat.

          1. adam 40 Silver badge

            Maggie

            No, they sacked all the miners.

        2. keith_w

          Too many movies where the hero/ine holds a lighter up to 1 sprinkler and they all go off. I don't think I have ever seen a bad person do this.

          1. Alan Brown Silver badge

            I saw someone do it in a student union lounge - but that was to a heat detector which uses the simple method of having a spring held in tension by being soldered (low temp solder) to the sensor plate

            5 seconds with a lighter, alarm, evacuation and excuses to not deal with exams

            Oh and a $500 bill from the brigade for false alarms

      4. Caver_Dave Silver badge

        Making the problem go away is not enough

        While my wife's elderly parents were alive, she would fetch them on Christmas Day morning and I would cook the dinner.

        When they eventually passed, my wife said that she would happily cook the Christmas Dinner if she could be left in piece.

        She did not see the funny side when I put her shower cap over the smoke alarm in the kitchen/dinner.

        1. Strahd Ivarius Silver badge

          Re: Making the problem go away is not enough

          "left in piece"?

          were you in the habit of disassembling her?

        2. Anonymous Coward
          Anonymous Coward

          Re: Making the problem go away is not enough

          What did they pass, a cooking course?

  5. muddysteve

    Are you sure you want to go from amber to red alert? It would mean changing the bulb.

    1. This post has been deleted by its author

  6. El blissett

    Pretty sure COBOL programmers can now charge far more than overtime and a pizza for any work, as if you need them you're totally fucked if you can't get them.

  7. Jou (Mxyzptlk) Silver badge

    I feel reminded on QUITE some house wifes (well, and men)...

    ... who turned the CO alarm off 'cause "that noise causes headache".

  8. Michael H.F. Wilkinson
    Megaphone

    A former colleague of mine disabled the loudspeaker in his office due to the deafening sound of the test alarms set off each first Monday of the month, by opening the box up and snipping through the wires. This wasn't as daft as it sounds, because there was another speaker in the corridor outside his office, which produced such an ear-splitting volume that if you couldn't hear that with the office door closed, you were probably beyond help already.

    1. Anonymous Coward
      Anonymous Coward

      Snipping them is too obviously deliberate. Better just to have one "come loose".

    2. swm

      At Xerox they piped in musak through the same speakers used for alerts. I watched a scientist snip wires until the musak was gone. Xerox finally wised up and put a switch to turn off the musak.

      1. An_Old_Dog Silver badge

        Semi-wise

        That Xerox didn't completely disable the Muzak shows management there believed someone enjoyed it.

        1. Martin an gof Silver badge

          Re: Semi-wise

          Worked in a radio station where the loudspeakers in the office had a switch... to swap from the FM service to the AM service. It was sort of mandatory for staff to be aware of what we were broadcasting.

          Us "engineers" (both of us) down in the basement had our own HiFi in the transmission racks because my boss wasn't a fan of the types of music we shovelled out and would rather bring his Prog Rock CDs in. Or occasionally Michael Nyman.

          M.

      2. damienblackburn

        I used to work as a contractor at an office that had TVs around them and of course displayed the usual news and propaganda messages. This company had the bright idea that one of the videos that played would do a little jingle. So every 45 seconds or so there'd be a jingle. And this was right outside my cube.

        This was during the time of IR blasters being on a lot of phones, and I had at the time IIRC a Galaxy S5 which had said IR blaster. Quickly put that bastard on mute and enjoyed my quiet work.

  9. SVD_NL Bronze badge
    Boffin

    Sensible to a point

    To be fair, if the issue is difficult, doesn't lead to a critical failure (tricky one to assert), and it's the middle of the night, just disabling the logs temporarily to get the production up and running is a somewhat sensible move. However, calling it a day and cashing in your paycheck is negligent if not malicious.

    I preface every ticket comment with TEMPORARY FIX - NEEDS FOLLOW-UP when i end up making a temporary solution, don't want too many skeletons in my cupboard...

    1. MOH

      Re: Sensible to a point

      Yep. In fairness, batch support on those sort of jobs what always about getting the jobs finished.

      Need a SQL to exclude a bunch of policies from the batch, or suppress a runaway warning message, or whatever - fine, as long as it lets the rest of the jobs complete.

      But it's an emergency measure. Still needs to fixed ASAP by whoever is responsible for that.

    2. Anonymous Custard
      Headmaster

      Re: Sensible to a point

      TEMPORARY FIX - NEEDS FOLLOW-UP

      As is often repeated around here, there's nothing as permanent as a temporary fix, except perhaps a temporary tax increase.

      1. Anonymous Coward
        Anonymous Coward

        Re: Sensible to a point

        temporary tax increase

        I have heard of tax increases, in fairy tale books

  10. Already?

    Email. That's the answer

    Having been in a similar situation, this time needing to get a production system running the day before going on hols, I sent an email to my boss and myself stating something like 'working, will be okay but needs tidying and improving. Priority on return.'

    Everybody’s happy, it’s in the open and two weeks later I’m not struggling to remember what was going on last time I was here. Thinking about it I did a similar email to myself every holiday, reminding me of what was going on.

  11. Jedit Silver badge
    Joke

    Surely the problem was obvious?

    I mean, it was Mal-ware.

    1. Anonymous IV
      Unhappy

      Re: Surely the problem was obvious?

      > I mean, it was Mal-ware.

      I had to go through all the comments available at the time of reading to discover that the last one was the one I would otherwise have typed...

      Grrrh.

  12. Sceptic Tank Silver badge
    Headmaster

    try ... catch AI error handling

    I get so excited when I see code like this:

    try

    {

    // Yadda

    ...

    }

    catch (...) {}

    1. Anonymous Coward
      Anonymous Coward

      Re: try ... catch AI error handling

      I've seen the equivalent of that in PL/SQL and complained we need to at least emit SQLERRM or somesuch.

      I was then told "we can't display database internal information to the users or they will hack us"

      Seriously. The users that can't find the "any" key if it had a red light and siren ... are going to "hack us".

      I then closed every subsequent ticket on that system with "no diagnostic information available".

      1. John Brown (no body) Silver badge

        Re: try ... catch AI error handling

        Seriously. The users that can't find the "any" key if it had a red light and siren ... are going to "hack us".

        When we were testing the new version of some software we use daily, I sent feedback stating that when instructed to "press any key to continue" that it failed to respond and the problem was repeatable. The key I pressed? The Fn key on my laptop. It also "failed" on pressing the Pause/Break key too. I'm not sure what the system was written in or how they interrogated the keyboard, but asking a user to press "any" key is asking for trouble if you don't do it properly :-)

        I've also seen that many systems fail to respond to "any" key when the laptop function key are set to default to the "icon shortcut" (brightness/volume/WiFi etc) mode and require the Fn key be pressed to make them work as normal function keys.

  13. Eclectic Man Silver badge
    FAIL

    Three Mile Island Nuclear Accident

    The three Mile Island nuclear incident was caused partially by some bad programming and partially by a missed warning light:

    "The nuclear power plant had a control panel with a light that showed the status of the relief valve which prevented the reactor from overheating. If the light was on, the valve was open. If the light was off, the valve was closed. Or at least that was what was thought.

    Actually, the person who designed the control panel programmed the light to go off once the system had sent the signal to close the valve – which didn’t guarantee that the valve had been closed. Simply put, the system had sent a signal to close the valve, but a malfunction occurred and the valve did not close, while at the same time the control panel light went off, misleading the staff to believe that the valve had been closed. Everything because of a light."

    From https://safety.productions/2018/11/25/the-three-mile-island-accident-caused-by-a-light/

    1. John Brown (no body) Silver badge

      Re: Three Mile Island Nuclear Accident

      I wonder if they also had a warning sensor/lamp to indicate that the valve closed/lamp off wasn't actually valve open/lamp failed?

      1. An_Old_Dog Silver badge

        Re: Three Mile Island Nuclear Accident

        When I was in uni, our Radiation Center had a green light on top of the building, visible from all directions. If the light went out, security would come running. The lamp was changed well ahead of its expected failure date. It was a nicely fail-safe system.

        1. Jou (Mxyzptlk) Silver badge

          Re: Three Mile Island Nuclear Accident

          Did they have a RAIL? A Redundant Array of Inexpensivedependent Lights? And light of that importance should be, but beancounters are beancounters, so maybe...

        2. Anonymous Coward
          Anonymous Coward

          Re: Three Mile Island Nuclear Accident

          One site I visited had a "chirp", if the chirp stopped you ran to a shelter because it was highly likely to be replaced by radiation leak alarms.

  14. aerogems Silver badge
    Joke

    So, it wasn't like when Homer stopped the plant from melting down in The Simpsons?

    https://www.youtube.com/watch?v=v5MOatu5-DE

  15. Sam not the Viking Silver badge
    Pint

    Alarms and Incidents

    Before mobile phones were commonly available, and as remote control and telemetry were becoming more widespread, we had one customer who was very keen to have all the 'bells and whistles' transferred to his home phone so that he could 'keep in touch'. Previously, the only way to check a plant was operating was to look for a red beacon on the roof of the distant building. He was very self-important and could be irritating, so we complied...

    This lasted about a week before he implored us to "turn all the alarms off" or at least stop the telemetry signals. It turned out that every change of state from 'Start No. 1' and 'No. 1 Started' plus 'No. 1 Running', 'No. 1 Normal' then the stopping, shutdowns, not running, available, you get the drift...... A nightmare even with a small number of machines. If he failed to answer, he received repeated calls until he acknowledged.

    He didn't include this requirement on future specifications.

    1. John Brown (no body) Silver badge

      Re: Alarms and Incidents

      AKA "malicious compliance" :-)

  16. jake Silver badge

    The Story of Mal ...

    ... doesn't quite have the right ring to it.

    CLOAS, or Computations Life Office Administration System, where Computations is the name of the Australian company which inflicted brought the software into an unsuspecting world. I've run across it once or twice, and I believe it's still in use in a few places in the UK and the US. I must admit that I never heard the "Complete Load ... " expansion, but it fits admirably. Some of the worst spaghetti COBOL that I've ever had the misfortune to deal with in a production environment. But it DID (does?) do the job! If I remember correctly, it worked only with non-relational databases under CICS.

    As for "This story comes from the time before remote access", nah. CLOAS is from the early or mid 1970s. The Teletype Model 33 came out in 1963, and was in common(ish) use by top-tier remote mainframe support staff by the late 1960s. From home. Dad had one installed in '67 or thereabouts ... the company even brought in a second POTS line for it, their dime. I soon used the same system to access ORVYL, the Stanford timeshare system, with the blessing of Dad's company. (I also later used it to access a new-fangled thingie that DARPA had Stanford and Berkeley working on ... but that's another story for another day.)

    1. Andy A
      Pint

      Re: The Story of Mal ...

      The worst spaghetti COBOL that I've ever had the misfortune to deal with was a system which was supposed to be doing its last ever run.

      The system was being retired, but, as always, there had been a delay and it unexpectedly had to be kept going into the next financial year. The documentation as to what input was needed to do this was nowhere to be found, so I was brought over to decode the source listing.

      In the first ten pages I found eight occurrences of "ALTER <label> TO PROCEED TO <other label>" statements. Some GOTOs were ALTERed more than once.

      It was horrible. I gave up. I ran off to get some anaesthetic. ====>

      The woman who had previously done this admin task was paid some ridiculous sum to return and generate the single punched card needed.

      1. Jou (Mxyzptlk) Silver badge

        Re: The Story of Mal ...

        If the C-Suite fails to see the priorities early enough and act accordingly they should pay that sum out of their private pocket. You and she both did the right thing.

      2. An_Old_Dog Silver badge
        Flame

        Re: The Story of Mal ...

        Whomsoever writes code with an ALTERS/GOTO (or equivalent features in non-COBOL programming languages) requires a vigorous crotch-kicking. There are better, clearer ways to code things, and those ways should be used instead. (Yes, I can imagine the enjoyment of writing clever things with ALTERS/GOTO, or, in PL/I, gods forfend, using label arrays. But such clever things are not easily maintainable.)

        1. Diogenes

          Re: The Story of Mal ...

          My 3rd ever real life programming assignment was to take an old ca 1975 PL/1 program and basically rewrite it was written in the organisations - 'standards yeah we've heard of them' phase - no proper loops , all GOTOs, no functions or procedures, all GOTOs. It was the 'side job' I was given whilst waiting for the compiles and test runs of my 2nd, 4th and 5th real life programming assignments.

          My first was the one and only time I used coding sheets and punched cards, I had almost finished that work, when 3270 terminals were rolled out, great excitement when my last run was submitted from my desk, rather than trekking down to the computer room with my deck.

          Six months or so later, after the company dropped the word 'trainee' and added analyst/' to my title, I was put into the "maintenance team", it would not be unusual for me to come into work and find a program listing and core dump on my desk with a big note left by my team leader if they had been oncall - "job failed for account zzzzzzzz. removed account from batch, rest of batch executed ok - original batch file is at yyyyyy, - Fix before going on with anything else."

  17. Anonymous Coward
    Anonymous Coward

    Junior devs

    Had more than one junior developer that believed the instruction to fix this error meant comment out the code behind the action was an appropriate fix.

  18. Cliffwilliams44 Bronze badge

    Alert overload!

    It seems every place I've worked I've been the guy who either implements or manages the alerting system.

    One of my primary goals is to only alert on things that are of high or critical importance and define what that actually means. The rest you log into the Alerting system and examine the lower-level stuff to see if it may have some importance. Because if you are alerting on every niggling thing, the alerts will get ignored!

    I recently had lunch with a friend I'd worked with in years past who now works for Bank of (name withheld to protect the guilty). He told me this global multi-national bank's alerting system spit out over 100 alerts per hour and that no one paid any attention to them!

  19. Jyotindra

    The Statement is incorrect - I worked on a financial package written in COBOL called 'CLOAS,' which the staff told me stood for a 'Complete Load of Australian … Something'," Mal opened.

    CLOAS is mainframe software package meant for Comprehensive life office administration system.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like