back to article A lightbulb moment comes too late to save a mainframe engineer's blushes

The weekend is almost upon us – a time for adult beverages and ill-judged foodstuffs. Unless, that is, you're one of the unfortunates on the other end of the phone. Welcome to On Call. Our story today comes from "Will", who told of his time as a City of London mainframe engineer in the swinging sixties and slightly more sombre …

  1. Red Sceptic
    Pint

    Thank you …

    for Douglas Adams reference - that cheered me up!

    1. Anonymous Coward
      Anonymous Coward

      Re: Thank you …

      He was originally going to use a different local estate agent's name but for some reason decided otherwise...

      So 'Gascoigne Pees' became 'Hotblack Desiato'

    2. The Oncoming Scorn Silver badge
      Pint

      Re: Thank you …

      Happily my up vote, makes the up votes Forty-two," he said with infinite majesty and calm.

    3. Anonymous Coward
      Anonymous Coward

      Re: Thank you …

      Would love to upvote you, but as has been pointed out, you're at the magic number 42 right now.

      1. John Brown (no body) Silver badge

        Re: Thank you …

        I felt no guilt in increasing the upvote count since it was already on an unremarkable 47 by the time I got here.

  2. Adam Trickett

    Cupboard light

    A colleague once told me about an ICL mainframe with a fault which confused the engineers all day until they discovered it was the light in the paper cupboard under the printer that wasn't working...

  3. Pascal Monett Silver badge

    I had a somewhat similar problem

    It was at the beginning of my career in IT. I was working for a self-employed engineer who had set up a test bench for industrial pumps. I was in charge of creating the UI software (in Visual Basic - sorry) that would allow the users to start or stop the bench, and see what the test status was. Everything was supposed to be automatic.

    It was an impressive machine, I must say. The pumps would arrive by overhead rail. Upon reaching the proper position, a pump would be lowered for pressure testing. Tubes would be brought into place and cover the proper points on the pump. Once that was done, water was made available and the pump was set to work to ensure proper functioning. When the test was over, the system would drain, tubes would be removed, and the pump went on its merry way.

    My UI was supposed to show all the steps, which I am happy to say that, after testing it for every possible scenario, it did.

    Once everything was ready, we went to do the live demo. Everything went fine until a few minutes before we were supposed to finished, when the test bench froze.

    I looked at the engineer. He looked at me. We hadn't moved a muscle. What had happened ?

    I'll spare you the description of the 90 minutes that followed, what it boils down to is that, somehow, the big red Emergency Stop button had been pressed on the side of the test bench.

    When I got back home (late) that evening, I promised myself I was going to integrate the status of that button in my UI the next day.

    1. pavel.petrman

      Re: I had a somewhat similar problem

      That was acutally a happy ending, from safety perspective. I once pressed the big red button on a machine in integration testing, after which the machine proceeded merrily with its doings and a warning dialog came on its HMI screen, saying "TODO connect big red button procedure".

      1. Paul Herber Silver badge

        Re: I had a somewhat similar problem

        Sorry, emergency stop procedure not yet implemented.

        1. Aussie Doc
          Pint

          Re: I had a somewhat similar problem

          The complete message was probably "Sorry, emergency stop procedure not yet implemented. Please have your emergency elsewhere."

          Cider all round. ----->

    2. PerlyKing
      Facepalm

      Re: I had a somewhat similar problem

      I had a somewhat similar problem in a non-IT setting, which I may have related here before.

      Riding to work on a new (to me) motorbike I stopped at a red light and the bike died. No sign of life, not even the click of a dying battery when I tried to start it up. After a few minutes of checking what I could, I called the AA. Eventually a man showed up in a little van, checked the things I'd already checked, couldn't start it. And didn't have the equipment to move a bike, so he called another man in a bigger van. The second AA man checked the things we'd already checked, couldn't start it, put the bike on his trailer and took me to the mechanic of my choice. As soon as the bike was unloaded, said mechanic looked at the bike, turned the kill switch back on and the bike started first time X-(

      For those not in the know, motorbikes tend to have a big red kill switch on the handlebars to turn off the electrics in case of an accident. The one on this bike had a fairly light action and was in just the right position to catch on the wing mirror of a van as I squeezed past it in stationary traffic.

      1. Emir Al Weeq

        Re: I had a somewhat similar problem

        >turn off the electrics in case of an accident.

        Is that what it's for? I thought it was to flick off and on when riding a carburettored bike through a tunnel.

        1. Fr. Ted Crilly Silver badge

          Re: I had a somewhat similar problem

          you forgot flicking the throttle a couple of times quick to accellerator jet some petrol in the exhaust ;-)

          My old Xj 750 was very good at those...

        2. Screepy

          Re: I had a somewhat similar problem

          Not just carb'ed bikes, that trick worked on my fuel injected Yamaha R6 as well.

          On the topic of 'annoying ways for a bike to cut out' - a sightly coroded safety switch on the kickstand is a real doozy. Switch doesn't deactivate when you put the stand up. Bike will start, but as soon as you put it in gear, bike cuts out.

          Better yet, the switch activates at the traffic lights cos it's raining (I assume some sort of short?), sitting there idling in neutral, green light, pop it in gear... bike cuts out. Cue much honking and beeping of horns behind me.

          1. Doctor Syntax Silver badge

            Re: I had a somewhat similar problem

            On the topic of 'annoying ways for a bike to cut out'

            Back in the really old days we rode BSA Bantams (yes, you could go up hill on them). One day Dad's cut out when he touched the horn switch. Why was the battery completely flat? The generator on the Bantam was actually AC and had a rectifier hidden behind the tool box. Somebody'd nicked it in the works car park. He'd noticed the nuts & washers on the floor a few days ago but assumed they were just stray rubbish.

          2. John Brown (no body) Silver badge

            Re: I had a somewhat similar problem

            "Cue much honking and beeping of horns behind me."

            Possibly even more livid drivers if you'd just "queue jumped" to front and THEN held them up :-)

      2. WhereAmI?
        Facepalm

        Re: I had a somewhat similar problem

        Ah. In ye olden days when the killswitch was off, all the idiot lights stayed on. Result - many flat batteries when the bike would crank over but not fire, becauce the killswitch only killed the ignition. Still see that on some modern bikes. When I rewire a bike from scratch (I'm a masochist - shoot me :-D ) and I don't have to stick rigidly to the OEM diagram, I always hang the idiot lights and the starter circuit off the killswitch. No killswitch - black panel - obvious.

        Until the day I rewired a bike and the owner called me saying it wouldn't start...

        1. Gene Cash Silver badge

          Re: I had a somewhat similar problem

          In ye olden days when the killswitch was off, all the idiot lights stayed on

          In that case, it's still ye olden days, with the addition of a large LCD dash that consumes even more power.

          I've had several bikes where you could accidentally flip the switch off and past "lock" into "park" which did nothing but leave the rear tail light on.

          I'm pretty good now at disassembling ignition switches and cutting out that particular "feature"

          1. myootnt

            Re: I had a somewhat similar problem

            Don't forget about the part that it lets you pull the key out so you can walk away and come back to a dead battery. Oh, and alternator and fuel injection so no possibility of push start. Nothing but expletives for that feature.

      3. Tom 7

        Re: I had a somewhat similar problem

        That reminds me of the time where I borrowed a friend's prized Pirrelli moped that would do 60mph and drove several miles, stopped for a was and set off again not realising I'd kicked the key in the steering lock which prevented it turning past a certain point on one side but didnt defeat the ignition. I managed to lean it over at the next bend after a long straight but clipped the hawthorn hedge and rolled along it for a hundred yards of so before spinning round on the helmet. I was pulling Hawthorne spikes out of bits of my body for months after!

      4. Stumpy

        Re: I had a somewhat similar problem

        Ahhh ... the 'fun button'. Always a good prank when group riding and pulled up at the lights. Get someone to the left of the rider to distract them whilst someone on the right of the rider flicks the kill switch to the off setting just as the lights change to green.

      5. Loyal Commenter Silver badge

        Re: I had a somewhat similar problem

        I used to have a Yamaha FZS-600 which at one point stopped randomly and wouldn't restart. The fuel gauge was showing a small amount left in the tank, but the starter just turned the engine with no result. After ten minutes of huffing and puffing pushing the thing quarter of a mile up the hill to the nearest garage, I popped open the filler cap on the tank, and there was a loud *clank* sound as the tank buckled back into shape. The pressure release valves in the fuel tank (the mechanic later assured me there were three of them) were all somehow blocked, and the engine had been trying to burn vacuum!

        1. martinusher Silver badge

          Re: I had a somewhat similar problem

          Old bikes didn't have the emission control stuff we have now. The tank would just be vented using a tiny hole in the filler cap. A tiny, easily bunged up hole. The effect on the bike is subtle -- it will lose power, then maybe get it back and then maybe it will stop for no reason. Looking in the tank for fuel will, of course, magically cure it. For a bit.

          1. James Wilson

            Re: I had a somewhat similar problem

            I had that. Rode back from deity-knows-where in small bursts (of incredibly boring but very fuel-efficient riding) between stopping to open the fuel cap and allow some air in. I can't remember if the bike had some safety feature where I couldn't get the key out of the cap without it being closed or I was just too young and dumb to think of riding along with it open.

            1. Loyal Commenter Silver badge

              Re: I had a somewhat similar problem

              ...or I was just too young and dumb to think of riding along with it open.

              Not riding with the filler cap open doesn't sound dumb to me. If you drop the bike for whatever reason, you don't want to be covered in fuel on top of everything else.

              1. jake Silver badge

                Re: I had a somewhat similar problem

                To say nothing of the sloshing out under acceleration, braking, turning, hitting bumps/dips ... in fact, any movement at all will slosh out fuel.

                Right on to your crotch ... with the excess dripping on any one of several ignition sources residing about a foot below.

      6. Gene Cash Silver badge

        Re: I had a somewhat similar problem

        I had a brand new Suzuki DL-650 with the fancy new fuel injection stuff.

        I park it at work, and at the end of the day, I go to leave and it's flashing fuel injection controller warnings and won't start.

        I had a colleague take me home, and pick me up the next morning armed with test tools and the service manual.

        Turns out the kill switch was on.

        And the kill switch functioned by killing the power to the fuel injection. So the dash computer couldn't talk to the fuel injection computer and spewed a ton of bogus messages.

        I hoped whoever flipped the switch had a good laugh.

      7. Johan-Kristian Wold 1

        Re: I had a somewhat similar problem

        A common problem with newer Can Am Quads and 6x6 bikes is the anti theft system. There’s a chipin the key and a pickup in the ignition switch that reads said chip.

        Unless you have gotten moisture in the ignition switch, at which point the quad will refuse to start. And moisture in the ignition switch is fairly common, as the bike will often be left outside.

        If you leave the key in the ignition, the key has a rubber lip around the edge that will prevent this, but leaving the key in kind of negates the useability of the anti theft system.

        Got a new Can Am in June - already had to change the ignition lock once…

      8. the Jim bloke

        Re: I had a somewhat similar problem

        In my distant and receding youth, I used to have a small motorbike to get me around town and to whatever work I had. This would be less powerful than most scooters, and with indicators brighter than its headlight.

        At this occasion, I was riding to work before dawn in winter, and the local council had apparently decided streetlights werent required at that time of day.. the engine died, the headlight died.. so i was kicking down through the gears to get power to the headlight while fumbling under the fuel tank to switch to the reserve position..

        Travelling home after work was during peak hour (such as it was back then), i used to see how much traffic I could get banked up behind my bike on a major highway.

      9. Nick Pettefar

        Re: I had a somewhat similar problem

        In my yoof I had a fallow period so decided to fix bikes to make a few bob. I was called by a posh solicitor who'd bought a trail bike for his son and it had stopped working. He'd replaced the coil and spark plug and HT lead and battery and then seen my advert in the local paper. I came round, switched on the ignition, checked the fuel tap, checked it was in neutral, put the choke on, flipped the kill switch to run and kick started it. He looked at me totally amazed. And then we realised what I'd done and he went bright red. I did more work for him and he was always referred to as the Off Switch man.

    3. Anonymous Coward
      Anonymous Coward

      Re: I had a somewhat similar problem

      Similar problem, but only a 36 million dollar oil refinery off line intermittently, when I was doing site tests. Emergency shut off button was highly visible, but on folk passing it in narrow space lane with a bend their rumps touched it. Took ages to find, as when pressed there was no indicator light to say it had been pressed!

    4. Potty Professor
      Facepalm

      Re: I had a somewhat similar problem

      When I was working as a Technical Author for a large Electrical Engineering firm, I went into the Test Department to take some photographs of an enormous suite of control gear that had just been erected in order to do a heat run. I was squirming around in the constricted space at one end of the suite in order to get the equipment in the frame of the camera when I felt something jab me in the back, followed by the whine of the machine swooping down the scale into silence, followed immediately by the howls of anguish and anger from the testers. I had inadvertently backed into one of the Emergency Stop buttons dotted around the Test Area. As this was supposed to be a heat run, and start from cold, the whole day had to be scrubbed, and the assembled scrambled egg had to reconvene next day to witness it. I was never allowed into Test while an equipment was being tested from that day on, I had to grab my chance while it was being assembled or dismantled before or after the test.

      1. Pascal Monett Silver badge

        And yet, transparent plastic covers are cheap as chips, right ?

    5. Scene it all

      Re: I had a somewhat similar problem

      In the only large compuuter room I worked in for years, the Big Red Buttons were in recessed boxes mounted about 6 feet off the floor so it was just about impossible to hit them by accident. IBM had also learned this trick and on the Sytem 360 line the Emergency Off was a knob at the very top of the cabinet and you had to PULL it, not PUSH it.

      It is odd that designers keep making this same mistake.

      1. jake Silver badge

        Re: I had a somewhat similar problem

        "so it was just about impossible to hit them by accident."

        So naturally they were accidentally hit a couple times per month, right?

  4. Disk0
    Facepalm

    How far we’ve come

    Nowadays it’s Windows Dozen locking up because it can’t find the printer. For backwards compatibility…

    1. jake Silver badge

      Re: How far we’ve come

      Now with WSL you might get an error dialog stating "lp0 on fire".

      Ran across this around six years ago when trying to convince a very old bit of kit (IBM 1403) to cooperate with Linux. I jumped about a foot. It had been probably three decades since I had last seen that error message.

      1. Paul Kinsler

        Re: "lp0 on fire".

        I recall they removed that from the kernel quite a few years back.

        I made a point of putting it back in for at least the next couple of self-compiled kernels, but eventually stopped bothering...

  5. tip pc Silver badge

    I’m sure a lesser techie would have sussed that in a flash

    Powerful minds would dismiss the mundane easy stuff in the hunt for something more interesting and challenging, if you don’t know about those more challenging things you’ve a smaller pool of possibilities and can find these mundane problems far quicker.

    I assume that’s why airliners have problem checklists for cockpits, calmer heads have already checked through the easy and obvious so crew can get them out the way, if problem persists they have to come up with something but hopefully the prescripted check lists buys time or provides inspiration

    IBM included a tester button for a reason though.

    1. PM from Hell
      Facepalm

      Re: I’m sure a lesser techie would have sussed that in a flash

      I'm sure there was a documented IBM monthly process to test the lamps which had been disregarded as they always worked

      1. keith_w

        Re: I’m sure a lesser techie would have sussed that in a flash

        but the IBM CE probably tested the bulb last month after which it burned out as he released the test switch

      2. Anonymous Coward
        Anonymous Coward

        Re: I’m sure a lesser techie would have sussed that in a flash

        (dating myself by posting this but...)

        RTFM in the FE prints. The MAPs (maintenance access procedures) covered this kind of stuff back in those days. Also, in those days, machine rentals included maintenance, so why were you messing with it instead of calling the FE? You didn't own it, IBM did, and for certain, the FE knew about the paper out switch (second day of training for 360/40 FEs,BTW)

        .

    2. Robert Carnegie Silver badge

      Re: I’m sure a lesser techie would have sussed that in a flash

      We had an issue today... some software I may have written in let's say 2016 stopped working. A database grew, not mine guv, filled up the server disk drive. Shouldn't have happened, shouldn't have been made my problem, should have been spotted earlier... and I'm pretty sure that the same system failed before, pretty much the same way.

  6. Evil Auditor Silver badge

    ...why airliners have problem checklists for cockpits...

    In case of an emergency situation we can do a lot of things. Well, mostly three: freeze, run, fight. What we cannot do is think. That's where checklists come into play.

    1. Anonymous Coward
      Anonymous Coward

      You don't follow checklist in an emergency only. You follow checklist at every stage of flight when it is required to do so. Because it's far too easy to forger or skip something "because I've done it thousands times" or "hey, it's obvious", when it may not always be.

      1. Anonymous Coward
        Anonymous Coward

        I think that is similar to the (railway) track workers rule of "never step on _any_ rail" - that way you don't ever have to think "am I about to step on the third rail?"

        I got to walk through an underground tunnel once (the "Thames Tunnel" when it was being converted to the ON line) and the guides were clearly all properly trained and you could see them looking very uncomfortable as we all happily stood all over the tracks and third (and fourth) rails...

        1. nintendoeats Silver badge

          Another example, if you are properly trained in firearms (as are all firearms owners in Canada) you learn to always point it in a safe direction and always keep your finger off the trigger unless you intend to shoot. This is true even if you have just confirmed that the firearm is clear of ammunition.

          If you always point in a safe direction, you can't make a mistake about whether the firearm was loaded and do something bad.

          1. J.G.Harston Silver badge

            The two rules of firearms:

            A gun is always loaded, even when it's unloaded

            Never point a gun at anything you don't intend to kill

            1. Paul Hovnanian Silver badge

              "Never point a gun at anything you don't intend to kill"

              Except that guns are always pointing somewhere. If you think in terms of the negative, you'll drive yourself nuts. Instead, pick some safe direction or object you are not particularly attached to and point it there.

              Our local gun store has some targets on the wall, up near the ceiling that say "Aim guns here".

              1. J. Cook Silver badge
                Go

                There's two more rules, actually...

                Ah, Jeff Cooper and his four rules of firearm safety:

                1. ASSUME the firearm is loaded until you have personally verified that it is not.

                2. Do NOT point the firearm at anything you aren't willing to destroy.

                3. Keep the booger hook (finger) OFF the bang switch (trigger) until you are ready to fire.

                4. Know what your target is, and what is BEHIND it.

                Proper firearm owners in the US also know these four and follow them. I can't tell you how many times I've seen rules 2 and 3 violated by people who really should know better. (Newbies get ONE FREE PASS from me, because I WILL call them out on it politely- the second time? I break out the Book of the Profane and start selecting appropriate phrases from it, especially if it's me they've swept!)

                1. Marty McFly Silver badge
                  FAIL

                  Re: There's two more rules, actually...

                  You need to find different folks to hang out with if you keep seeing rules 2 & 3 violated.

                  And instead of savoring the opportunity to read from the Book of Profane, maybe do a range safety review with everyone before the live fire begins.

                  I stopped going to public ranges because of the egos who are just searching for the opportunity to yell and scream instead of focusing on educating the other shooters. Most of them are already embarrassed enough that they broke a rule without having someone go ballistic on them.

                2. John Brown (no body) Silver badge

                  Re: There's two more rules, actually...

                  "1. ASSUME the firearm is loaded until you have personally verified that it is not."

                  Clearly not followed on the Rust film set.

                  1. Robert Carnegie Silver badge

                    Re: There's two more rules, actually...

                    An explanation offered to me was on lines that it's more complicated with actors. An actor typically isn't a gun expert and therefore isn't given any responsibility or freedom to check whether the thing in their hand may go bang. They have to trust the specialist, and the specialist, of course, must not trust the actor. A corollary is that almost always, the thing in an actor's hand must not be an actual go bang thing.

                3. Alan Brown Silver badge

                  Re: There's two more rules, actually...

                  "I break out the Book of the Profane and start selecting appropriate phrases from it, especially if it's me they've swept!)"

                  I used to get particularly pissed off when supposed "firearms experts" would sweep a group whilst explaining how to verify the breech was empty

                  I don't care if you think it's unloaded. the subtext of #1 is "even if you've verified it's unloaded, it's still loaded" - I've seen too many instances of "verified" weapons subsequently discharging to trust anyone

              2. Anonymous Coward
                Anonymous Coward

                Ah, HR occupies the mezzanine office, perchance?

            2. Anonymous Coward
              Anonymous Coward

              Going by Jeff Cooper there's four:

              1 - All guns are always loaded. Even if they are not, treat them as if they are.

              2 - Never let the muzzle cover anything you are not willing to destroy. (For those who insist that this particular gun is unloaded, see Rule 1.)

              3 - Keep your finger off the trigger till your sights are on the target. This is the Golden Rule.

              4 - Identify your target, and what is behind it. Never shoot at anything that you have not positively identified.

              These interleave nicely, so that a failure to follow any one rule should be covered by the others either preventing a negligent discharge, or at least mitigating the outcome of one. There's a slew of secondary rules which help like always check a weapon is clear with a finger as well as by eye, or always hand over a firearm unloaded and clear anything handed to you first, but they all come back to those four.

            3. bpfh

              3rd rule

              Finger off the trigger (and out of the trigger guard) until you are ready to fire.

          2. cosmodrome
            Megaphone

            "Clear of ammunition"

            I've seen an assault rifle that had just been checked "clear of ammunition" go off half a meter besides me. Three meters fifty after I touched ground again, that is. Luckily the gun was pointed in the safe direction of a sand box that didn't stand there for no reason. Wearing no ear protection and not excpecting anyone to fire a live round in my vincinity it was quite a surprise...

            1. phuzz Silver badge

              Re: "Clear of ammunition"

              When I was a cadet on an exercise the kid next to me *thought* he had cleared the chamber. He had not, fortunately they'd only given us blanks and he was pointed downrange, but you can imagine the bollocking the Sgt gave him. Poor kid was crying after about ten seconds.

        2. Anonymous Coward
          Anonymous Coward

          Surgical Safety Checklist

          Most hospitals now use a checklist to follow for operations. This site

          https://www.england.nhs.uk/2019/01/surgical-safety-checklist/

          states

          "...pilots found that use of the checklist reduced the rate of deaths and surgical complications by more than one-third across all eight pilot hospitals."

          "The checklist is a simple 19-item tool which addresses serious and avoidable surgical complications, by ensuring that critical steps outlined in the guidelines are done in every surgery, every time, everywhere. It also serves as a critical communication tool for the operating theatre team."

          1. ChrisC Silver badge
            Joke

            Re: Surgical Safety Checklist

            "pilots found that use of the checklist reduced the rate of deaths and surgical complications by more than one-third"

            I know the NHS has staffing issues, but if they're having to resort to filling in surgical roles with former aircrew then things are even worse than I thought...

            Yes, yes, I know what "pilot" is referring to in *this* context, it just seemed too good to pass over given the context in which "pilot" *had* been used earlier on in this particular branch of the comment tree.

            1. Denarius Silver badge

              Re: Surgical Safety Checklist

              AFAIK, It did come from aviation. Aviation has had incident analysis for decades to improve practices. Simple checklists create a known reliable process. I believe a USA doctor noted the aviation procedures and tried it to manage surgery in his hospital. It showed a dramatic drop in hospital caused injuries and deaths so idea spread.

              1. Scene it all

                Re: Surgical Safety Checklist

                The saying about checklists in airplanes goes "Two kinds of pilots always use checklists: beginners and professionals."

                Once I was undergoing a rather complicated surgical procedure and I was lying in the pre-op area. The entire surgical team gathered around me and the head surgeon stood at the foot of the bed with an inch-thick three-ring binder. He was flipping pages and checkng things off as they talked. I was feeling a bit woozy from the pre-op relaxing pills but managed to ask, "Is all that for me?" He answered, "Yes, it is."

          2. Loyal Commenter Silver badge

            Re: Surgical Safety Checklist

            This is noticeable if you ever haver to attend a hospital as a patient for any reason. You'll find that every person you interact with in a clinical setting will first confirm your name and date of birth. They don't want to be taking the gall bladder out from the wrong patient by mistake.

            This is also why X-Ray transparencies normally have 'L' or 'R' printed on them. Don't want to be looking at them upside-down by mistake and taking the wrong kidney out...

            1. Neil Barnes Silver badge

              Re: Surgical Safety Checklist

              In my case, I was asked all the questions while lying on the operating table... including 'point to the side you're expecting us to operate' :)

              1. Anonymous Coward
                Anonymous Coward

                Re: Surgical Safety Checklist

                Ha ha - yes! I've had that too. "wiggle the thumb" and "point to the eye". (not at the same time though)

            2. Gene Cash Silver badge

              Re: Surgical Safety Checklist

              Eh, I had a nurse inform me that my broken ankle was fine... when I had a broken clavicle. So I'm all for double/triple/quadruple-checking.

            3. eldel

              Re: Surgical Safety Checklist

              Heh - I had a neurologist who was on the point of booking neck surgery for me due to the state of my neck (many decades old injury) contributing to the pains in my left arm. Until I pointed out that the problem was in my right arm.

            4. TRT Silver badge

              Re: Surgical Safety Checklist

              L&R for Left and Right?

              Genius!

              I always wondered why the wife's knickers had C&A on them.

            5. J.G.Harston Silver badge

              Re: Surgical Safety Checklist

              I also asked *them* to confirm what I'm in for. If *they* answer correctly I let them put me under.

            6. Jean Le PHARMACIEN

              Re: Surgical Safety Checklist

              Sadly, I have worked in a major UK teaching hospital where the Renal Transplant Team removed the wrong (functioning) kidney.

              No checklists on double checking in place then (early 2000s)

              1. Richard Jones 1

                Re: Surgical Safety Checklist

                Yes, I saw a Dr once, and only the once, who told me my scan showed that my pain was due to terminal kidney cancer. The problem was that I had not yet had a scan, he also told me that he was too busy to follow things up but that a colleague would see me. It was only when he paused for breath, that I was able to tell him my scan was booked for the following week. I also pointed out that he might need to check the name on the patient's record... It was not clear if he was concerned about trying to scare me, or about breaking the other patient's confidentiality.

                1. Potty Professor
                  WTF?

                  Re: Surgical Safety Checklist

                  I had a similar experience, the consultant commented that my lung capacity had almost doubled in the twelve months since my last visit, but as this was my first visit, I was somewhat mystified. I then caught sight of the front cover of the file he was reading from, and when I could get a word in edgeways, I asked him to look at the date of birth of the patient he was examining. Turned out he was reading my father's file, his initials were the same as mine, but he was nearly 40 years older than me.

                  1. Snapper

                    Re: Surgical Safety Checklist

                    You want to have a twin sister like I do, with the same three initials.

                    As I'm male I'm surprised when they are surprised, but it got a little complex when I was donating a kidney to her....

          3. Ignazio

            Re: Surgical Safety Checklist

            Same on trains. Conductors point at things while going down the checklist, fewer incidents happen. Literally, pointing with their fingers, simple as that. It works.

        3. Anonymous Coward
          Anonymous Coward

          Oh behave....

          .... as long as you don't touch the ground, you can walk ALONG the third rail bare-foot.

          I wouldn't recommend it though.

      2. Anonymous Coward
        Anonymous Coward

        I'm reminded of the time I visited an offshore oil platform to review the drilling and well management operations. I suggested that, for rigging up one particular set of kit (IIRC, a coiled tubing intervention) they develop a checklist as there were several steps that could easily be missed and not noticed until the unmentionable hit the rotating blades. My suggestion was declined as they were all experts and checklists were for the inexperienced! Fine, was my reply. When you board the helicopter for your flight back to shore, tap the pilot on the shoulder and tell him to put away his checklist - after all, he's not a beginner. The company supervisor's view changed immediately - he was ex-forces and had become very acquainted with flying operations and the value of checklists...

        1. Anonymous Coward
          Anonymous Coward

          I honestly don't ever remember seeing a chopper pilot use a check list when leaving an installation. Maybe I only ever flew with cowboys...

          1. fredds

            IIRC they use mnemonics to remember some of their checklists, saves being distracted by pieces of paper. I think they use paper when the checklist is too long.

            1. swm

              CIGAR - controls, instruments, gas, altimeter, radio if I remember from my private pilot days.

            2. khjohansen

              When the check list is too long

              - they use iPads - no joke!!

              1. phuzz Silver badge

                Re: When the check list is too long

                My friend is a helicopter pilot, he's found that an iPad mini fits nicely into the knee pocket in his flight coveralls (which have a transparent window so you can stick a map, or a checklist, in there and have it easily visible). There's plenty of useful flight software that can display maps, checklists, all sorts.

                They still keep paper copies in the cockpit though of course.

        2. gnasher729 Silver badge

          I once looked at a garden shed that was self assembly. According to the manager it’s easy to do. Unless you hire the professionals to do it, because they don’t read instructions. Worst case was a guy who couldn’t fit the roof. He was asked to send in a photo and was told “you built it upside down”..

      3. Fr. Ted Crilly Silver badge

        Or the verbal interrruption at just the wrong moment = missed step.

        Been there too many times...

      4. Alex Brett

        It sounds like it would be a boring read, but on this front "The Checklist Manifesto" by Atul Gawande is a great read, and actually improved my approach to various tasks after reading it, as it helped me understand what sort of mistakes we commonly make as humans and how to avoid them...

        1. J.G.Harston Silver badge

          Almost every contract I've been on I've spent the first couple of days rewriting the scribbled incomprehensible nonsense that passes for documentation into a checklist, aiming for a single A4 max. Forcing youself to actually write down on a sheet of paper and tick off Asset Tag Number, Machine ID, User ID, User email, Join Domain, Run Office Install, Map R: to \\etc, Map S to \\etc gets things done so much more effectively, and documents where you might have missed something, or have been forced to skip a step and come back later.

          1. J. Cook Silver badge

            THIS.

            When I was actively writing documentation for stuff at [RedactedCo], I aimed to write it in such a way that my technophobe mother could follow, if she had access to the app.

            1. Inventor of the Marmite Laser Silver badge

              AKA the Mrs Miggins test.

          2. The Oncoming Scorn Silver badge
            Pint

            & that is why I script things.

            Quicker, easier with one click & every machine is the same (So if theres a fuckup/missed item in the early stages it's at least consistent.

            1. phuzz Silver badge

              The benefit of a tick list is if there's a problem later, you can look at the sheet and verify that you did actually do X before you shipped it out.

              Or more often for me, reminding me what the asset number is, after I've already packaged it all up.

      5. Anonymous Coward
        Anonymous Coward

        I have come to despise checklists, if only because of a bad experience of every dimension of a matrix organization creating their own checklists for the process of fixing a single software bug or feature.

        With the checklists continuously being expanded, with no notice (you were supposed to always check what was the latest). With limited documentation and always incomplete or out of date references to other checklists.

        All the responsibility of the software development team, all others just "audit" you or deny approval to release.

        GAAAH!

        1. Is It Me

          That is less an issue with checklists and more an issue with poor management/organisation

    2. eionmac

      Brilliant reply to comment. Brain freeze!

    3. Intractable Potsherd

      I've recently seen the range of responses to emergency situation summarised as "fight, flight, freeze, facilitate", which I quite like.

  7. FeepingCreature Bronze badge

    It can take a remarkably long time to notice that the alert mailer has stopped working.

    (Quote unattributed - I can't remember where I saw it.)

    1. GlenP Silver badge

      Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

      I have a number of overnight job audit emails running. A while back I must have subconsciously realised, in my sleep, that one hadn't arrived at the expected time (the phone goes ping) as I woke up with the feeling something was wrong.

      Useful as I was able to resolve the problem before any users noticed.

      1. Anonymous Custard
        Headmaster

        Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

        Oh yes, one of those times when an unexpected silence screams louder than any alarm or siren...

      2. Mast1

        Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

        Allegedly, when Rolls Royce were testing Merlin engines to see what went bang first, (and hence find the weak points) the technicians used to doze while sitting next to the rigs running the engines...... they used to wake up when the engine sound changed.

        1. Noram

          Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

          I can quite imagine that being true, I've heard something similar about pilots and aircraft engineers who might notice something is wrong even when asleep in the rest area (long haul) because they're so used to how the aircraft sounds/feels that when something changes it hits the unconscious brain and starts screaming.

          I used to sleep with a quiet computer in the room, and I would always wake up if there was a power cut because the nice reassuring white noise from the case fans would stop, it would invariably take me a few seconds to work out why.

          I suspect there is some deep primal instinct that meant our ancestors would notice the lack of wildlife noise as a warning something (or someone) had caused the local animals to try not to be noticed, those that didn't wake up then possibly never did again.

          1. adam 40 Silver badge

            Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

            This is why snoring is a thing, when the snoring stops, your tribe be dead.

            1. Snapper

              Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

              Yet I have been threatened with death many times foe snoring!

          2. Old Used Programmer

            Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

            I don't have that problem. The only computers in the room that run overnight are Raspberry Pis and none of them have fans.

            What wakes me up is when the UPS alarms go off...

          3. Jean Le PHARMACIEN

            Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

            RAS

            Reticular Activating System

          4. John Brown (no body) Silver badge

            Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

            "I can quite imagine that being true, I've heard something similar about pilots and aircraft engineers who might notice something is wrong even when asleep in the rest area (long haul) because they're so used to how the aircraft sounds/feels that when something changes it hits the unconscious brain and starts screaming"

            My brother was in the Royal Navy as a "lifer". He said he could tell what his ship was doing just by the sounds of the engines/feel of the vibrations through the deck and bulkheads and it was obvious if there was anything unusual in the sound or feel.

            1. Alan Brown Silver badge

              Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

              I got to the point of setting up a "canary checker". If things don't check in regularly then it flags the fact for attention

        2. DoctorPaul

          Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

          Based on my experiences at Ricardo Engineering in the late 70s you *never* stand alongside an engine on a test bed - ALWAYS stand in line with the axis of the crankshaft.

          Reason is obvious - when things go bang, large bits of metal exit the engine at very high speed and physics dictates the direction in which they travel.

          Even then take care - these were Chinese lorry engines (based on a 40s design) running at full chat and maximum load, the exhausts were running cherry red. Then one of the radiator hoses ruptured and the resulting superheated water coming out under pressure would have broiled anyone in the vicinity.

          1. Mast1

            Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

            Many moons back, a work colleague narrated a story of a site where they tested generator sets by attaching them to an electric motor and spinning them up. When the bearings seized, the set pulled itself out of the floor bolts and rolled through the wall, ... and down an embankment onto a railway track. Made a change from the usual "Please can I have my ball back ? "

            1. Alan Brown Silver badge

              Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

              There are a couple of stories of hydroelectric generator bearing seizures where this happened and the rotating assembly was found over half a mile down-valley after exiting the building at speed

              It was only at this point that people considered what might have happened if the assembly had gone in the OTHER direction and went whiter shades of pale

              Bear in mind that these were "small" generators (well under 1MW) and modern hydropower ones are easily 50MW with a small one being 2MW). More recently (1970s) a 2MW unit in Australia's Blue mountains had lubrication turned off before it stopped spinning during an emergency shutdown procedure and by all accounts the end was glowing cherry red within seconds

              For obvious reasons, bearings and lubrication systems on $LARGE rotating assemblies are treated as critical assemblies with multiple redundancies - and if they're not, they need to be

              1. Johan-Kristian Wold 1

                Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

                Check the Sayano-Shushenskaya power station accident.

                The accident lifted a 920 tonne rotor out of its seat and flooded the power station. The devastation is incredible.

        3. Bogbody

          Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

          In the late '70 I worked for a company that made dot matrix printers.

          We had a test room with upwards of a 100 printers on soak test - hammering away in dot matix style.

          If one printer had a faulty printhead you could hear it over the clatter or the rest.

          They were based in Crewe in a wonderfull ex-railway office that curved round with the track. Alas it was destryed in a fire some years later.

    2. yetanotheraoc Silver badge

      Re: It can take a remarkably long time to notice that the alert mailer has stopped working.

      A "success" error (or success "error", if you will) sent at the start and end of a job helps there.

  8. ColinPa

    Some things are meant not to work.

    I half remember the operator-less shuttle on tracks which take you from A to B, the door opens, you get in, the door closes, and off you go.

    Until the queen came to open the facility.

    She got in, and the doors stayed open.... they had to reset the shuttle before it would move.

    There was lots of bad press, until they found the cause.

    What happened was

    The shuttle arrived, doors open, the queen says (quite) a few) words, queen boards shuttle.

    The software had some code which says if the doors do not close within 1 minute, it looks like a problem. Do not let the shuttle leave the station, but send someone to check it.

  9. Ol'Peculier

    Hotblack Desiato

    Hotblack Desiato Will wasn't

    He wasn't an estate agent? Well, that comes as a relief...

    (and I know DNA asked them first, which I seem to think is even mentioned on their website)

    1. The Oncoming Scorn Silver badge
      Pint

      Re: Hotblack Desiato

      Hotblack Desiato appears by kind permission of Hotblack Desiato on the LP sleeve of The Restaurant AT The End Of The Universe.

      Baffled me for years, I just thought it was some kind of meta joke.

      1. Snapper

        Re: Hotblack Desiato

        Is he still dead?

  10. Phil O'Sophical Silver badge

    Positive signals

    There is something to be said for alerts where the light is on to confirm normal operation, and goes out on a fault condition.

    1. John McCallum
      Angel

      Re: Positive signals

      Until the idiot light becomes the problem.

      1. Arthur the cat Silver badge

        Re: Positive signals

        This is why safety critical things have three lights. Even then it doesn't always work. A friend of mine once managed to land his aircraft having forgotten to put the landing gear down.

        1. Down not across

          Re: Positive signals

          Although, in that case the three lights are because there is one light for each wheel to indicate if its locked up, locked down, or something in between.

    2. John Robson Silver badge

      Re: Positive signals

      And preferably one that actually monitors the condition, not the signal sent to make that condition happen (3MI)

    3. Anonymous Coward Silver badge
      Big Brother

      Re: Positive signals

      Relays can fail in both states - contacts welding together, or coil burning out. Similarly for transistors, just not so easy to visualise.

      Yes, I've seen lights stay on when the control signal was turned off. And vice versa.

      Never trust what you can see. Or rather: trust but verify.

      1. Alan Brown Silver badge

        Re: Positive signals

        "Yes, I've seen lights stay on when the control signal was turned off. And vice versa."

        This is more or less that happened at TMI: The lights on the control panels indicated what the control switches were set to, not what the equipment was actually doing

    4. Anonymous Coward
      Anonymous Coward

      Re: Positive signals

      many year ago we used to have some ancient analogue valve equipment that was forever bringing up alarms because it had drifted out of alignment.

      One day someone noticed that there hadn't been an alarm from it for over a week, so wandered over 'on a hunch'. The equipment was showing multiple alarms, but for some reason the alarm at the end of that bay and the remote alarm were not lit, so no one had been alerted.

      Turned out one of the night men had had his fill of being woken almost hourly by it one night and had removed the alarm link, meaning to replace it before that day shift took over, but had overslept and forgotten

      1. Roger Greenwood

        Re: Positive signals

        On an unmanned site I know of the operators were fed up of false alarms from the methane detectors. None was ever found when they visited so eventually they disconnected the remote alarm. Of course in reality when they opened the doors the added ventilation had cleared the gas and the fault.

        This was discovered when one day the methane reached a critical mixture and met a spark, causing the roof and doors to depart the building.

        1. Anonymous Coward
          Anonymous Coward

          Re: Positive signals

          On an unnamed site where an accident would involve the ordnance survey doing some serious rework.

          There was a bing-bong alarm that sounded continually, if it stopped = run

          (officially remain in your office with the doors and windows closed. Unofficialy get in a car and drive as fast as possible towards the fence, do not stop for fence)

          1. Rob Daglish

            Re: Positive signals

            There's a number of buildings like this at Sellafield, and it's quite disconcerting to wander around site when the wind is not in the normal direction and not hear the noise!

          2. KBeee

            Re: Positive signals

            A bit like the V Force nuclear bombers.

            Officially "Fly east, drop your nuclear bombs, return to base."

            Unofficially "Fly east, drop your nuclear bombs, keep flying east cos there'll be fuck all to return to".

            1. Dog11
              Mushroom

              Re: Positive signals

              Much the same for the US F102 interceptors in the late 1950s. Their task was to fly west on afterburner (very fast, fuel hog) to meet the Russian bombers, shoot them down until they ran out of missles. At that point they wouldn't have enough fuel to return home, so they were to end their mission by ramming a final bomber.

          3. Anonymous Coward
            Anonymous Coward

            Re: Positive signals

            There was a bing-bong alarm that sounded continually, if it stopped = run

            I remember being in a GPO manager's office (it was while ago) in a key city exchange, and there was a steady background tick-tock. He commented that while it ticked there was nothing to worry about, but if it ever stopped - that was the start of the "4 minute warning". Not much point in running, though.

            1. Anonymous Coward
              Anonymous Coward

              Re: Positive signals

              My father showed me the equivalent device in a police station. Saying something like "if you leave the volume up it gets on your wick so we turn it off"

              1. Anonymous Coward
                Anonymous Coward

                Re: Positive signals

                They were normally in the 'off' position as the 'tick' was just the test signal. If the alarm was triggered it would switch itself on so the commands/orders could be heard.

                Once a month (or so) they were supposed to turn the volume up to confirm they could hear the tick and confirm the circuit was working... if not they would report a fault.

                I once knew a guy who was asked to take a replacement battery round to one police station... he handed it over and the sergeant promptly put it under the counter with all the other replacement batteries!

          4. Hans van Bemmelen

            Re: Positive signals

            thanks for mentioning the bing bong alarmlast year. I used that as thefinal sound effect in a short film https://vimeo.com/644440323 (featuresthe Numbers Stations and vintage electronics, so Vulture friendly).

    5. Anonymous Coward
      Anonymous Coward

      Re: Positive signals

      IT bod that w**ks at a hospital.....

      Walking along corridor, passed by a cleaner on one of those big sit-on floor scrubbers and he calls out to me and asks "What does error 28 mean? It's come up on the display". Did explain that just because that uses electrons doesn't make it IT kit, looked it up on googly - low battery if you're interested.

      Anyway, looking through the manual, there is also a light labeled "SAFE". However, that light will only light up when there is a fault that can potential cause injury - seems the completely opposite of safe to me, more like "not safe".....

      1. yetanotheraoc Silver badge

        Re: Positive signals

        Page one of the manual says "Do not operate the machinery without reading the manual."

        Technical person: Reads manuals for equipment they don't use.

        Non-technical person: Never reads manuals.

        1. Throatwarbler Mangrove Silver badge
          Alert

          Re: Positive signals

          Sysadmin: reads manual as a last resort.

          1. jtaylor

            Re: Positive signals

            Best manual I ever saw was for Retrospect backup software in 1996.

            First page said something like If you're just opening this because you need to restore data, turn to "Restoring files" on page 72. Otherwise, turn page for Table of Contents.

            Whoever wrote that, you are a hell of a tech writer. Thank you for the help, and the brilliant example!

    6. Anonymous Coward
      Anonymous Coward

      Re: Positive signals

      Had a VW that always had niggly faults - for example the electric windows won't open or the radio will cut out and just display "LEARN", usually about once every six months. Pulling over, switching the car off and on again and it will be fine again.

      Anyway, no surprise that the Check Engine light always came on and despite several trips to the garage and buying my own ODBC thing, I just ignored it and renamed it the "Engine Running Light". Had a hell of a shock driving on the dual carriage way one day when the light went out, until I realised that was a good thing....

      1. Rob Daglish

        Re: Positive signals

        I'd put money on an earthing fault - I had something very similar with my Mito until a pleasant young chap at the garage figured it out. Admittedly, he was trying to figure out why the steering had stopped working, but still...

        1. Alan Brown Silver badge

          Re: Positive signals

          I'd have said the same thing - and expanded by saying that it's most likely to be water under the carpets

          Germans have a tendency to take the attitude "This shouldn't happen, so we don't need to design for the possiblity", whilst Japanese designers take the position "It HAS happened more than once in the past, so we MUST take the possiblity into account"

          Of course if you have a sunroof, the risks go up exponentially because cleaning the drains out is never part of the manufacturer documented regular maintenance procedure, therefore never gets done - and they DO block up

    7. Unicornpiss
      Alert

      Re: Positive signals

      A lot of older cars fed the field winding in the alternator through the warning light in the dash. If this lamp burned out your alternator wasn't charging and you had no indication of it. The only way you'd know is if you happened to notice the lamp didn't come on before you started the car.

      1. jake Silver badge

        Re: Positive signals

        Some heavy equipment uses the alternator lamp trick, too. If you keep your wits about you, it;s often possible to buy used equipment "with unknown electrical problems" for pennies on the dollar, replace the bulb & make a couple-ten thousand bucks in an afternoon.

  11. Sam not the Viking Silver badge
    Pint

    Warning Lights

    A very long time ago, when electronic variable-speed motor drives were the new 'thing', I was sent to commission a job out in the wilds somewhere, transferring water from a reservoir to a treatment works via a long pipeline. The two inverters were protected at each stage by a multitude of fuses; there must have been 50 fuses per machine. The makers recognised that finding a blown fuse might be time-consuming and helpfully included a warning light for every fuse. I was sent with boxes and boxes of spare fuses. After a couple of hours, we realised this was going to be a long haul. The panel kept lighting up like a Christmas tree.

    The 'Lamp Test' button revealed a display to rival Piccadilly Circus.

    Late in the evening, a warning light came on, 'Low Pressure'..... A genuine fault or wonky protection? At first we rolled our eyes but on looking out of the window the true nature was revealed: the pipeline had burst and the adjacent field had become a lake. Made a very pretty sunset as the sun dipped over the horizon. Our day was done.

    1. Neil Barnes Silver badge

      Re: Warning Lights

      In the BBC, up to at least the 1980s, we used butterfly fuses: the fuse wire was soldered between a couple of spring contacts, which sprung out fore and aft when the fuse blew. One of the wings would contact a bus bar which powered a fuse alarm to make a horrible noise.

      As the fuse alarm was electrically powered of course it had to have its own fuse - the fuse alarm fuse. And because you had to know if the fuse alarm fuse had blown, it had its own alarm, which had its own fuse... the fuse alarm fuse alarm fuse. After two levels of fuse alarms, I think they just got bored.

      1. Commswonk

        Re: Warning Lights

        In the BBC, up to at least the 1980s, we used butterfly fuses...

        Ah the nostalgia... those fuses were used on anything operating from the - 50 Volt Station Battery...

        Too many years ago I worked in the (then) BH Ext in London, more or less overlooking Duchess Street. What I was doing at the time is long forgotten but on one occasion I was working on something on a temporary feed from one of those fuse panels, and in simple terms things weren't performing as they should.

        Switch off; check supply voltage; nothing wrong there. Switch on again; problem still present. Check voltage with equipment connected... aha! That's a bit low.

        Some comedian (longer in the tooth than I was) has replaced the fusible link with a resistor, rendering the "fuse" somewhat useless.

        For the uninitiated the fuse wire (or resistor, as the case may be) was actually invisible as the butterfly's wings concealed it when the wire (or resistor) was soldered in.

        Oh happy, innocent days... mostly anyway.

    2. Yet Another Anonymous coward Silver badge

      Re: Warning Lights

      >but on looking out of the window the true nature was revealed: the pipeline had burst

      A good lesson from Apollo13. Not much mentioned in the films and 'failure is not an option' accounts. But they wasted 20mins checking for computer errors AFTER the crew said they felt an explosion because an explosion would be bad so it couldn't be real

  12. Ozumo

    An even more extreme example of what a faulty bulb can lead to is Eastern Airlines flight 401:

    https://en.wikipedia.org/wiki/Eastern_Air_Lines_Flight_401

    1. Pascal Monett Silver badge

      Now that is a horrible story.

    2. Gene Cash Silver badge

      Yeah, there was a US TV movie about that in the late '70s

      Never forget to FLY THE PLANE.

  13. PghMike

    Thanks for a great story!

  14. Mr Ballstein

    Still Happends

    Merely a month ago our z14 mainframe had to be IPLed from the physycal console because the remote operator console buffer run out of space. Set your systems to report just about what you need folks.....

  15. Stuart Castle Silver badge

    I used to work in a University computing lab. One day, we had an open day for the local school kids, and were running activities in several groups on the lab. Everything was going well, until just before lunch, everyone lost their network connection. Because of the nature of the lab, it was on it's own isolated network. This made it easier to track down the problem.

    The lecturer in charge of the day was furious, but agreed to send the kids out to lunch to give us a chance to fix the problem. We looked at the single switch, and it was locked up. We rebooted it, and it locked. We tried to connect to the console port, and rebooted the switch. It locked before we were able to log into the console.

    So, we started dropping connections. After disconnecting a few computers, we found the problem. A couple of PCs were flooding their network connections, so we went to look at those PCs. One of the kids was running a denial of service attack on the switch's IP. When the lecturer's running the groups got back from lunch, we talked to the lecturer running the group using these PCs. She admitted she had shown the pupils the tools used for DOS attacks, and made them promise not to run one on any machine within the Uni. She also apologised, and shut down the application running the DoS attack, so we reconnected the PCs..

    This isn't the same lecturer who was in charge: She was teaching a group elsewhere in the lab.

    1. GlenP Silver badge

      I last worked in education in much simpler times when we had a couple of RM Nimbus networks. Initially the students used to just unplug connections randomly at the end of their session, annoying for the next class but easy enough to diagnose and fix. They then found it was possible to short a network connector which took a lot longer to find. I don't think DOS attacks had been invented then!

      1. Anonymous Coward
        Anonymous Coward

        As someone who currently works in education I can confirm the "unplug connections randomly at the end of their session" trick is still going strong, although now it's usually with the trick of "leave the cable just in the port enough to look like it's connected" or maybe rarely "unplug the Cat5 from the wall socket and plug it into the network port of the next PC along".

        Once had a student have the bright idea cut through the power cable on his PC with a pair of scissors instead. Thankfully unharmed, but that was a fun reaction from the teacher, and soon after the site manager when he came in to flip the breaker back on.

      2. Rob Daglish

        Having been in a school with a Nimbus network, how did you tell when it wasn't working, as they were pretty naff to start with?

    2. Anonymous Coward
      Anonymous Coward

      Saw something very similar one day when visiting Clemson University IT dept.

      Major Incident, all network connectivity knackered, F5 firewalls shitting the bed.

      Techs convinced they were being DDOS'd from outside, went on for some time, until they isolated part of the network, and realsied it was internal.

      Some TA and some students had kicked off a demo DDOS script on a lab machine at 3am and wandered off to bed.

      Me, I was just a visitor so sat and watched with a smile on my face that I wasn't having to work that crap out.

    3. Anonymous Coward
      Anonymous Coward

      Off Course

      A curriculum designed to prepare them for the needs of industry?

    4. Crypto Monad Silver badge

      One of the kids was running a denial of service attack on the switch's IP

      And that is why you should put switch management IPs on a dedicated management VLAN, and firewall it off from the rest of the network.

  16. TeeCee Gold badge

    Lamp test.

    Turned up to work early one day to find the duty operator fast asleep, on a high chair with his feet propped on the System/38 and said chair rocked back on its rear legs.

    The word "precarious" seemed inadequate. Too good an opportunity to miss, so what to do.

    Enter computer suite using keycode that I don't know. Tiptoe over to console. Dial 1 to enable, dial 2 to Lamp Test and hit load. Wind console alert volume to max. Tiptoe out.

    SNDBRKMSG MSG('WAKEY WAKEY') TOMSGQ(QCONSOLE)

    BEEPBEEPBEEPBEEP "Wuh......ARRRGGHHH" <CRASH>

  17. a_yank_lurker

    Watchen das blinken Lichts

    I remember seeing a couple minicomputers with 'das blinken Lichts'. Nice when the bulbs were not burnt out but....

    1. Mark 85

      Re: Watchen das blinken Lichts

      As I recall, there was a sign to that effect which also spoke of "Das machine is nicht fer fingerpoken and eyeballen mit das peepers." Or words to that effect as there were many variants.

      1. Down not across

        Re: Watchen das blinken Lichts

        I've seen quite a few variations of it. I believe the original goes like this:

        Achtung! Alles turisten und nonteknischen lookenpeepers! Das komputermaschine ist nicht für der gefingerpoken und mittengraben! Oderwise ist easy to schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparksen. Ist nicht für gewerken bei dummkopfen. Der rubbernecken sightseeren keepen das cottonpicken händer in das pockets muss. Zo relaxen und watschen der blinkenlichten.

        1. jake Silver badge

          Re: Watchen das blinken Lichts

          Pretty much the definitive history of blinkinlights here.

          Here's a plaintext address for those of you who quite sensibly refuse to simply click on links without having at least an inkling of where they might take you:

          http://catb.org/~esr/jargon/html/B/blinkenlights.html

          Note: The above link contains nazi symbolism in a historical context. Might not be safe for work in some jurisdictions.

  18. Boris the Cockroach Silver badge
    Pint

    Wish my

    operators had lamp test buttons.

    Could easily see which ones are the bright ones, or the dim ones or the not working need replacing ones...

    Beer time... praise whatever gods/goddesses you believe in

    1. Yet Another Anonymous coward Silver badge

      Re: Wish my

      I believe Mr PATT tester is your tool of choice here

  19. adam 40 Silver badge
    Flame

    Ahh the IBM 360

    I think my formative Uni computing years were influenced twice.

    First by attending the observation gallery at Birmingham uni which surveyed a football-field sized computer room full of IBM blue boxes.

    Seemed impressive, until I learned that all the input was by punched card, no terminals. So I passed on Birmingham.

    Then later at Durham, the IBM 360 was supposed to be unhaltable (so the console paper story was interesting).

    Anyhow, some wag wrote an assembler program with a couple of tight loops toggling some specific memory locations. He had worked out that these core memory locations we under the two temperature sensors, that shut the whole machine off in case of fire. The core memory heated up, and shut down the machine.

    The IBM techs were called out and were perplexed to say the least!

  20. Gene Cash Silver badge

    Error 47

    Got a brand new electric bike and took it out for a spin. Stopped for lunch, and when I got back, the bike was flashing "Error 47"

    No idea. Dragged out the owner's manual. No table of errors. Stood with my phone Googling. Nothing.

    Finally decided to push the bike over to the shade, and flipped up the kickstand. Error 47 disappeared.

    I bitched BITTERLY at the company, and they included a table of errors in the next revision of the manual and on their support page.

    1. Yet Another Anonymous coward Silver badge

      Re: Error 47

      Two types of bikers.

      1, Those who have set off with the kickstand slightly down and had the bike switch off followed by them slowly falling over

      2, Those who haven't yet set off with the kickstand .......

  21. DS999 Silver badge

    They need a fault light

    To let you know when one of the other fault lights is out.

    And a fault light for that fault light.

    1. vincent himpe

      Re: They need a fault light

      Error : there are errors.

      That's why the lunar lander had a master alarm. if the software fucked up that would trigger.

    2. The Oncoming Scorn Silver badge

      Re: They need a fault light

      DOCTOR: I, er, had hoped to reach your planet Earth. Skaro was in the future and I used the fast return switch.

      IAN: The fast return switch? You've sent us back too far. Doctor, show me. Show me that switch. Where is it?

      DOCTOR: Well, I can't very well see it without a light, can I?

      SUSAN: It's near the scanner switch.

      BARBARA: Really? But that's the part of the control that's safe.

      SUSAN: Yes.

      DOCTOR: Strange.

      IAN: Doctor, we haven't got very much time left.

      DOCTOR: Yes, I see. Here it is. (pulls a small torch from his pocket) Here, you see? Now, look, there's the switch. You see?

      IAN: Yes, well how does it work?

      DOCTOR: Well, you merely press it down, and. It's stuck. It hasn't released itself!

      IAN: What? You mean it's been on all this time?

      DOCTOR: Yes, it must have been.

      IAN: Well, come on, Doctor. Let's get it unstuck.

      DOCTOR: Hold that. Yes, just a minute now. Yes, there you are, you see?

      IAN: What's wrong?

      DOCTOR: The spring's not connecting. It's come off the base.

      IAN: Hurry, Doctor, hurry.

      DOCTOR: There we are. Take it out. Now, luckily we can turn it over and now it should work. There. Ah, that's all right.

      (The lights come back and the Tardis returns to normal)

      SUSAN: We're safe now.

      BARBARA: Are you sure?

      DOCTOR: Yes, we can all relax. We're quite safe now. But it was a narrow squeak.

      SUSAN: Grandfather?

      DOCTOR: Yes, my child?

      SUSAN: What happened?

      DOCTOR: What happened? It was the switch. It was still in place. You see, there's a little spring inside it and it was stuck. It hadn't released itself.

      SUSAN: But why didn't the fault locator tell us?

      DOCTOR: Well, the switch hadn't broken down, therefore the fault locator couldn't give us any recognition.

      1. The Oncoming Scorn Silver badge
        Boffin

        Re: They need a fault light

        In the real world, my truck & other vehicles, failure of a cam/contacts means the brake lights keep functioning as designed, but cruise control fails to engage as it thinks the brake pedal is permanently engaged.

      2. Hazmoid

        Re: They need a fault light

        Very appropriate for Dr Who day :)

  22. Will Godfrey Silver badge
    Happy

    A comment from the REAL Will

    On our last day before being kicked out of school permanently, our class had some really great 'lessons'. (the teachers all suddenly became humans) The physics teacher set us a puzzle. It seemed simple enough. Design a failsafe doorbell.

    It was great fun, but need I say, we failed.

    1. John Brown (no body) Silver badge
      Boffin

      Re: A comment from the REAL Will

      "Design a failsafe doorbell.

      It was great fun, but need I say, we failed."

      Need more info. Is it more important to not open the door if no one is there than it is to not let someone in who might be dying? ie, does the bell fail "safe" on or off? Or was failsafe used incorrectly such that the teacher wanted a door bell that couldn't fail or at least would guarantee an accurate indicator of working/failed?

      1. Will Godfrey Silver badge

        Re: A comment from the REAL Will

        Yes a rather woolly description I agree. The person operating the bell had to reliably be informed if it had failed to actually sound. Bear in mind this was for a bunch of 16 year olds in the mid 1960s. Eventually someone (not me) was bright enough to ask if the bell not being set a signal counted as the bell not working!

        1. jake Silver badge

          Re: A comment from the REAL Will

          I was assigned a similar assignment. I went home and forged a doorknocker out of a railroad spike. Received an A- grade ... The A was for meeting all the criteria. The - was for being a smartass.

  23. sokolnik
    Holmes

    "Is it plugged in?"

    My first question when users called me, long ago when I was an I-T support tech.

    1. John Brown (no body) Silver badge

      Re: "Is it plugged in?"

      And, of course, they always said yes because they'd never admit to having not done the simple and basic checks that even a non-tech ought to be able to carry out. So the correct first step is, "Maybe the connector has come loose, can you try unplugging it and plugging it back in, (at both ends if relevant)

      1. Is It Me

        Re: "Is it plugged in?"

        another "correct" reply is to ask them to check where it is plugged in (what colour the port is etc.)

      2. Andy A
        Thumb Up

        Re: "Is it plugged in?"

        I always found it useful to give them a chance to push the blame onto someone else. "Maybe the cleaners have dislodged it" often saved a field engineer trip, though you know that office last saw a cleaner around 1937.

  24. Anonymous Coward
    Anonymous Coward

    A certain power station I visited quite often in the 70's ran 3 off, 600 MW or thereabouts steam turbine generators. I was introduced to one entry in their unofficial photo album of Things That Had Happened.

    You have to know that the exhaust from the steam turbine went to a condenser, mounted like a saddle over the turbine. The condensed water went to a small tank, from whence a steam turbine pump whizzed it back to the boiler. In case the steam pump failed, there were electric pumps to protect the turbine, as water washing back into the turbine was considered A Bad Thing - a few buckets of water hitting the blades of an 8 foot diameter low pressure turbine do make a considerable impression - those blades are moving at almost supersonic speed.

    Right. Scene set.

    Apparently the operators were getting a little annoyed at repeated false alarms from the catching tank high level alarm. So annoyed in fact, that the alarm had been bypassed.

    Predictably the 1,000,000 to one chance happed (as they do 9 times out of 10) and the turbine pump failed. The electric standby pump just happened to be on maintenance at the time and didn't kick in.

    The water level rose, unnoticed.

    And kept on rising, unnoticed

    It eventually washed back into the low pressure turbine, where it did an awful lot of not good. People did rather notice though.

    I gather most of the LP turbine removed itself from the building. Via the roof. The main shaft, a substantial piece of chrome moly steel some 22 inches in diameter, simply bent. Some main bearing cap bolts, 30mm or thereabouts diameter, pulled apart in a text-book tensile fractures.

    A guy nearby, working up a ladder, slid down, broke both ankles and ran out of the building..

    It was eventually revealed that the root cause was the bypassed alarm. Further investigation discovered the cause of the repeated false alarms: the level switch was a pressure sensor mounted remotely from the tank and connected to it by what was described to me as a copper pipe not unlike why is in a central heating system. The pipe ran from the tank, along a wall and round a corner to the pressure switch. When it had been installed, the pipe had never been adequately supported. It had sagged and cracked a simple joint.

    By bye turbine. They had to call in the National Spare

    1. Potty Professor
      FAIL

      Levitating Mainshaft

      Sounds like Hinkley Point "B". It wiped out a lot of our diagnostic equipment on its way out through the roof.

      1. Anonymous Coward
        Anonymous Coward

        Re: Levitating Mainshaft

        Possibly Aberthaw B

    2. Alan Brown Silver badge

      > Further investigation discovered the cause of the repeated false alarms

      Quality British Engineering

  25. vincent himpe

    early compiler days

    after 5 minutes of compiling :

    error in line 1 : missing semicolon ...

    The problem is the compiler compiles all the subroutines and other 'includes' first. They are all fine as they are part of a tested and tried library.

    once it built all those object files it starts on the main file... which has a missing semicolon in line 1.

    Those were the days i acquired an innate hatred of any language that uses semicolons for end of statement. use the damn <cr> or <cr/lf> that sits in the source file.

    If a statement is too large to fit on 80 columns , use a continuation character. you will need very little of those.

    And code the compiler so it understands when = means assign , and when it means compare. The logic is not that hard : when you are in a logic test ( if / while/ do until / case statement) it means compare' , otherwise it means assign. and don't give me that ' i want to assign in the logic yadda'. That is cryptic to read ( and even then the compiler should be able to figure it out . the logic is recursive after all). Code should be easy to read and write without requiring a degree in literature.

    1. Will Godfrey Silver badge
      Unhappy

      Re: early compiler days

      Personally I'm convinced that Murphy has a 'thing' about semicolons - I'm sure I put them where they are needed, but come compile time there are always a couple missing.

      1. John Brown (no body) Silver badge

        Re: early compiler days

        The semi-colon is probably the most underused piece of punctuation in the written language. All hail the programming language designers for lifting this obscure mark back up it's rightful and most well used position in the punctuation hierarchy. It's just a shame those language designers don't actually know what the semi-colon is for, but they do think it looks pretty; it's a "safe" character to use since it will never appear in user input.

        1. Yet Another Hierachial Anonynmous Coward

          Re: early compiler days

          Semi-colon? I raise you a glass to the humble @

          Very much underused and obscure until about 30 years back, and since then it's rise to stardom has been meteoric.

          Now, what about that little sinewave thingy ~

          1. John Brown (no body) Silver badge

            Re: early compiler days

            "Now, what about that little sinewave thingy ~"

            I was taught to use the Tilde as symbol to denote an approximation in maths and physics at school ~50 years ago :-)

  26. Kev99 Silver badge

    Reminds me of the time I was taking a Basic programming course, around 1982. The computer was a TI midi connected to a greenbar wide dot matrix. The operator started a program another student had written and then went to lunch. When he got back he went into a panic because the printer had almost gone thru an entire case of greenbar. He killed the printer then the TI. As in hit the power switch. Strongly NOT recommended procedure. When they were done checking everything else, someone thought to check the code. The student forgot a single semi-colon putting the program into an endless loop.

  27. sanwin

    Back in the 70s worked at a place which still had a very old telephone exchange where the operator plugged cables in to connect 'phones to the outside world. Internal calls could be dialled.

    Whenever we tried on our shared office phone to get an outside call, the operator never answered us. The managers in the offices on either side had no problem. One day a colleague got fed up and used a manager's phone to ask the operator why we were ignored.

    The response was "Oh! Your bulb has blown so your line never lit up. I'll change it"

    We never had any further trouble - until a modern exchange was installed!

    1. Slx

      I’m reminded of a situation in a house I moved into in Ireland back in the days when landlines were still something people knew the numbers of and still plugged phones and modems into.

      I ended up stumbling across the specs for the Irish phone network by trail and error deduction.

      We got the phone line activated, but there was no dial tone in any of the extension sockets.

      I went around checking each one and used a multimeter and there were definitely no wiring issues. Simple stuff, two wires, very similar to the US.

      I checked everything on the first socket and found no faults, but it had a dial tone on the RJ11 socket on the front. It just stopped there.

      I then accidentally noticed that a screw I had dropped was magnetically struck to a device in the back of the front cover of this socket. It was only then I noticed there were two magnetically operated micro switches that cut off all the extension wiring when the Telecom Éireann branded “NTU” had its access cover open.

      It was a neat demarcation system for testing the service and eliminating the extension wiring, useful but if unfamiliar with it, it wasn’t at all obvious.

      All ripped out, never to be seen again and replaced with Ethernet and XPON FTTH.

  28. Stuart Halliday
    Megaphone

    If you really need an emergency button mounted somewhere, always make sure it's out of reach of the stupidest person employed there.

    1. Yet Another Anonymous coward Silver badge

      It's actually an OSHA requirement that the emergency stop button (if not adequately disabled) is hidden behind a pile of cardboard boxes with a stack of forklift pallets in front of them and preferably surrounded by a crocodile infested moat

      1. OssianScotland

        Shirley in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.” ?

        1. jake Silver badge

          That's been used so many times that nobody pays attention to it anymore.

  29. Unicornpiss
    Happy

    The universe just likes to mess with you

    A few years ago, was working on several laptops on my desk, which due to the lack of desk space, were stacked on top of each other. (our version of vertical integration, I suppose) Suddenly the machine on top went black, just like someone pulling the plug, I played with the power button, eventually getting it to reboot, only to have it go black again. I resolved to look at it later and started working on the second machine in the pile. It too, went 'off' suddenly. Frustration soaring, I picked it up, only to have it come back on, set it down, it shut off again.

    Finally, it dawned on me that the permanent magnet in one of the laptop speakers on the machine immediately below was lining up perfectly with the 'lid closed' sensor and turning off all illumination on the top machine. After that, I had to show my coworkers a "magic trick" where I concealed a powerful neodymium magnet in my fist and passed it over their machines, making them suddenly go dark.

    1. Ribfeast

      Re: The universe just likes to mess with you

      I had similar, my mobile phone has a magnetic back in its case that attaches to a magnetic mount in the car. When I sit my phone near my laptop, the laptop goes to sleep, as it triggers the lid closed sensor.

  30. Don Casey

    Glorified Selectric typewriter

    My first IT job was with a county government that had 360-series gear; initially a 360-40 and 360-30, then the 40 got upgraded to a 360-50 (by the time I 'graduated' from programmer to PFY systems programmer).

    Our single most time-critical application was the election tally system... if things ran slow we had the media (bad) and the politicians (fatal) all over us. The weak spot in the chain was the 1052 console, coupled with the fact that the application was 'console chatty' (for no good reason).

    We lived in constant fear the app traffic to the 1052 would overrun the console buffer and crash DOS (sic!).

    This was mid-1970s. By the time I left in the late 1970s (as BOFH) we'd migrated to a 370-155 running OS/MVT with a partition or two running the DOS emulator. All this in 384K of memory, as this was pre-virtual (needed a 370-158 or better).

    Good times.

  31. Burt Weedon's brother

    Customers like drama & Flash Gordon solutions

    Gosh! “Explaining to the customer why a printer running out of stationery had taken a football-squad of techies and a large chunk of the day to resolve,” should be avoided at all costs. The customer would be much happier with a drama and an explanation that you first thought it was a tricky problem with the “x794interconnector” running hot that had shut down the whole show to avoid loss of data. The lead techy then has to suck air through his teeth and explain that a colleague had seen this sort of thing once before, he’d run “the fix” and it won’t happen again.

    The guy will be dining out on the “x794interconnector” problem they’ve been having all week.

  32. Diez66

    Big wall, small red button

    So, we are bad people, big test lab set up to run rests throughout the weekend but it was ok for us to be there as our work did not clash.

    So finished up, empty building and a football.

    No, we were very careful. Just bounced it along the corridors and all was well until it was time to break for lunch, luxury.

    Chuck us the ball, clunk, silence.

    1. jake Silver badge

      Re: Big wall, small red button

      I was just putting the finishing touches on a small cluster of vaxen at SLAC one fine Friday afternoon. The annual Big Game between Stanford & Berkeley was to be the following day. A couple of grad students started passing a football (American version) between themselves. In the glass room. Just as I was threatening mayhem if they didn't knock it off, the ball hit the Big Red Button. Needless to say, a bunch of pissed off people couldn't attend the game the following day. The grad students computer privileges were suspended for the rest of the academic year. Personally, I'd have hung them by the thumbs in the Quad as a warning ...

      As an alum in good standing of both schools, all I can add to the above is "Go Bears!"

      1. Alter Hase

        Re: Big wall, small red button

        I was going to upvote your comment until I saw the "Go Bears"..... especially after this weekend!

        1. jake Silver badge
          Pint

          Re: Big wall, small red button

          I grew up in Palo Alto, call me a rebel :-)

          And to be fair, neither team is exactly doing well this year. Covid fallout.

          Have a cold one of your choice.

          1. Don Casey

            Re: Big wall, small red button

            Missed watching the Big Game this year... thought the Randy Rainbow concert at The Masonic would be more fun.

            Given the waxing The Bears gave The Cardinal I may have been wrong in my choices.

  33. Burt Weedon's brother
    Thumb Up

    Customers like drama & Flash Gordon solutions

    Gosh! “Explaining to the customer why a printer running out of stationery had taken a football-squad of techies and a large chunk of the day to resolve,” should be avoided at all costs. The customer would be much happier with a drama and an explanation that you first thought it was a tricky problem with the “x794interconnector” running hot that had shut down the whole show to avoid loss of data. The lead techy then has to suck air through his teeth and explain that a colleague had seen this sort of thing once before, he’d run “the fix” and it won’t happen again.

    The guy will be dining out on the “x794interconnector” problem they’ve been having all week.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like