A lightbulb moment comes too late to save a mainframe engineer's blushes The weekend is almost upon us – a time for adult beverages and ill-judged foodstuffs. Unless, that is, you're one of the unfortunates on the other end of the phone. Welcome to On Call. Our story today comes from "Will", who told of his time as a City of London mainframe engineer in the swinging sixties and slightly more sombre …
It was at the beginning of my career in IT. I was working for a self-employed engineer who had set up a test bench for industrial pumps. I was in charge of creating the UI software (in Visual Basic - sorry) that would allow the users to start or stop the bench, and see what the test status was. Everything was supposed to be automatic.
It was an impressive machine, I must say. The pumps would arrive by overhead rail. Upon reaching the proper position, a pump would be lowered for pressure testing. Tubes would be brought into place and cover the proper points on the pump. Once that was done, water was made available and the pump was set to work to ensure proper functioning. When the test was over, the system would drain, tubes would be removed, and the pump went on its merry way.
My UI was supposed to show all the steps, which I am happy to say that, after testing it for every possible scenario, it did.
Once everything was ready, we went to do the live demo. Everything went fine until a few minutes before we were supposed to finished, when the test bench froze.
I looked at the engineer. He looked at me. We hadn't moved a muscle. What had happened ?
I'll spare you the description of the 90 minutes that followed, what it boils down to is that, somehow, the big red Emergency Stop button had been pressed on the side of the test bench.
When I got back home (late) that evening, I promised myself I was going to integrate the status of that button in my UI the next day.
That was acutally a happy ending, from safety perspective. I once pressed the big red button on a machine in integration testing, after which the machine proceeded merrily with its doings and a warning dialog came on its HMI screen, saying "TODO connect big red button procedure".
I had a somewhat similar problem in a non-IT setting, which I may have related here before.
Riding to work on a new (to me) motorbike I stopped at a red light and the bike died. No sign of life, not even the click of a dying battery when I tried to start it up. After a few minutes of checking what I could, I called the AA. Eventually a man showed up in a little van, checked the things I'd already checked, couldn't start it. And didn't have the equipment to move a bike, so he called another man in a bigger van. The second AA man checked the things we'd already checked, couldn't start it, put the bike on his trailer and took me to the mechanic of my choice. As soon as the bike was unloaded, said mechanic looked at the bike, turned the kill switch back on and the bike started first time X-(
For those not in the know, motorbikes tend to have a big red kill switch on the handlebars to turn off the electrics in case of an accident. The one on this bike had a fairly light action and was in just the right position to catch on the wing mirror of a van as I squeezed past it in stationary traffic.
Not just carb'ed bikes, that trick worked on my fuel injected Yamaha R6 as well.
On the topic of 'annoying ways for a bike to cut out' - a sightly coroded safety switch on the kickstand is a real doozy. Switch doesn't deactivate when you put the stand up. Bike will start, but as soon as you put it in gear, bike cuts out.
Better yet, the switch activates at the traffic lights cos it's raining (I assume some sort of short?), sitting there idling in neutral, green light, pop it in gear... bike cuts out. Cue much honking and beeping of horns behind me.
On the topic of 'annoying ways for a bike to cut out'
Back in the really old days we rode BSA Bantams (yes, you could go up hill on them). One day Dad's cut out when he touched the horn switch. Why was the battery completely flat? The generator on the Bantam was actually AC and had a rectifier hidden behind the tool box. Somebody'd nicked it in the works car park. He'd noticed the nuts & washers on the floor a few days ago but assumed they were just stray rubbish.
Ah. In ye olden days when the killswitch was off, all the idiot lights stayed on. Result - many flat batteries when the bike would crank over but not fire, becauce the killswitch only killed the ignition. Still see that on some modern bikes. When I rewire a bike from scratch (I'm a masochist - shoot me :-D ) and I don't have to stick rigidly to the OEM diagram, I always hang the idiot lights and the starter circuit off the killswitch. No killswitch - black panel - obvious.
Until the day I rewired a bike and the owner called me saying it wouldn't start...
In ye olden days when the killswitch was off, all the idiot lights stayed on
In that case, it's still ye olden days, with the addition of a large LCD dash that consumes even more power.
I've had several bikes where you could accidentally flip the switch off and past "lock" into "park" which did nothing but leave the rear tail light on.
I'm pretty good now at disassembling ignition switches and cutting out that particular "feature"
That reminds me of the time where I borrowed a friend's prized Pirrelli moped that would do 60mph and drove several miles, stopped for a was and set off again not realising I'd kicked the key in the steering lock which prevented it turning past a certain point on one side but didnt defeat the ignition. I managed to lean it over at the next bend after a long straight but clipped the hawthorn hedge and rolled along it for a hundred yards of so before spinning round on the helmet. I was pulling Hawthorne spikes out of bits of my body for months after!
Ahhh ... the 'fun button'. Always a good prank when group riding and pulled up at the lights. Get someone to the left of the rider to distract them whilst someone on the right of the rider flicks the kill switch to the off setting just as the lights change to green.
I used to have a Yamaha FZS-600 which at one point stopped randomly and wouldn't restart. The fuel gauge was showing a small amount left in the tank, but the starter just turned the engine with no result. After ten minutes of huffing and puffing pushing the thing quarter of a mile up the hill to the nearest garage, I popped open the filler cap on the tank, and there was a loud *clank* sound as the tank buckled back into shape. The pressure release valves in the fuel tank (the mechanic later assured me there were three of them) were all somehow blocked, and the engine had been trying to burn vacuum!
Old bikes didn't have the emission control stuff we have now. The tank would just be vented using a tiny hole in the filler cap. A tiny, easily bunged up hole. The effect on the bike is subtle -- it will lose power, then maybe get it back and then maybe it will stop for no reason. Looking in the tank for fuel will, of course, magically cure it. For a bit.
I had that. Rode back from deity-knows-where in small bursts (of incredibly boring but very fuel-efficient riding) between stopping to open the fuel cap and allow some air in. I can't remember if the bike had some safety feature where I couldn't get the key out of the cap without it being closed or I was just too young and dumb to think of riding along with it open.
To say nothing of the sloshing out under acceleration, braking, turning, hitting bumps/dips ... in fact, any movement at all will slosh out fuel.
Right on to your crotch ... with the excess dripping on any one of several ignition sources residing about a foot below.
I had a brand new Suzuki DL-650 with the fancy new fuel injection stuff.
I park it at work, and at the end of the day, I go to leave and it's flashing fuel injection controller warnings and won't start.
I had a colleague take me home, and pick me up the next morning armed with test tools and the service manual.
Turns out the kill switch was on.
And the kill switch functioned by killing the power to the fuel injection. So the dash computer couldn't talk to the fuel injection computer and spewed a ton of bogus messages.
I hoped whoever flipped the switch had a good laugh.
A common problem with newer Can Am Quads and 6x6 bikes is the anti theft system. There’s a chipin the key and a pickup in the ignition switch that reads said chip.
Unless you have gotten moisture in the ignition switch, at which point the quad will refuse to start. And moisture in the ignition switch is fairly common, as the bike will often be left outside.
If you leave the key in the ignition, the key has a rubber lip around the edge that will prevent this, but leaving the key in kind of negates the useability of the anti theft system.
Got a new Can Am in June - already had to change the ignition lock once…
In my distant and receding youth, I used to have a small motorbike to get me around town and to whatever work I had. This would be less powerful than most scooters, and with indicators brighter than its headlight.
At this occasion, I was riding to work before dawn in winter, and the local council had apparently decided streetlights werent required at that time of day.. the engine died, the headlight died.. so i was kicking down through the gears to get power to the headlight while fumbling under the fuel tank to switch to the reserve position..
Travelling home after work was during peak hour (such as it was back then), i used to see how much traffic I could get banked up behind my bike on a major highway.
In my yoof I had a fallow period so decided to fix bikes to make a few bob. I was called by a posh solicitor who'd bought a trail bike for his son and it had stopped working. He'd replaced the coil and spark plug and HT lead and battery and then seen my advert in the local paper. I came round, switched on the ignition, checked the fuel tap, checked it was in neutral, put the choke on, flipped the kill switch to run and kick started it. He looked at me totally amazed. And then we realised what I'd done and he went bright red. I did more work for him and he was always referred to as the Off Switch man.
Similar problem, but only a 36 million dollar oil refinery off line intermittently, when I was doing site tests. Emergency shut off button was highly visible, but on folk passing it in narrow space lane with a bend their rumps touched it. Took ages to find, as when pressed there was no indicator light to say it had been pressed!
When I was working as a Technical Author for a large Electrical Engineering firm, I went into the Test Department to take some photographs of an enormous suite of control gear that had just been erected in order to do a heat run. I was squirming around in the constricted space at one end of the suite in order to get the equipment in the frame of the camera when I felt something jab me in the back, followed by the whine of the machine swooping down the scale into silence, followed immediately by the howls of anguish and anger from the testers. I had inadvertently backed into one of the Emergency Stop buttons dotted around the Test Area. As this was supposed to be a heat run, and start from cold, the whole day had to be scrubbed, and the assembled scrambled egg had to reconvene next day to witness it. I was never allowed into Test while an equipment was being tested from that day on, I had to grab my chance while it was being assembled or dismantled before or after the test.
In the only large compuuter room I worked in for years, the Big Red Buttons were in recessed boxes mounted about 6 feet off the floor so it was just about impossible to hit them by accident. IBM had also learned this trick and on the Sytem 360 line the Emergency Off was a knob at the very top of the cabinet and you had to PULL it, not PUSH it.
It is odd that designers keep making this same mistake.
Now with WSL you might get an error dialog stating "lp0 on fire".
Ran across this around six years ago when trying to convince a very old bit of kit (IBM 1403) to cooperate with Linux. I jumped about a foot. It had been probably three decades since I had last seen that error message.
Powerful minds would dismiss the mundane easy stuff in the hunt for something more interesting and challenging, if you don’t know about those more challenging things you’ve a smaller pool of possibilities and can find these mundane problems far quicker.
I assume that’s why airliners have problem checklists for cockpits, calmer heads have already checked through the easy and obvious so crew can get them out the way, if problem persists they have to come up with something but hopefully the prescripted check lists buys time or provides inspiration
IBM included a tester button for a reason though.
(dating myself by posting this but...)
RTFM in the FE prints. The MAPs (maintenance access procedures) covered this kind of stuff back in those days. Also, in those days, machine rentals included maintenance, so why were you messing with it instead of calling the FE? You didn't own it, IBM did, and for certain, the FE knew about the paper out switch (second day of training for 360/40 FEs,BTW)
.
We had an issue today... some software I may have written in let's say 2016 stopped working. A database grew, not mine guv, filled up the server disk drive. Shouldn't have happened, shouldn't have been made my problem, should have been spotted earlier... and I'm pretty sure that the same system failed before, pretty much the same way.
I think that is similar to the (railway) track workers rule of "never step on _any_ rail" - that way you don't ever have to think "am I about to step on the third rail?"
I got to walk through an underground tunnel once (the "Thames Tunnel" when it was being converted to the ON line) and the guides were clearly all properly trained and you could see them looking very uncomfortable as we all happily stood all over the tracks and third (and fourth) rails...
Another example, if you are properly trained in firearms (as are all firearms owners in Canada) you learn to always point it in a safe direction and always keep your finger off the trigger unless you intend to shoot. This is true even if you have just confirmed that the firearm is clear of ammunition.
If you always point in a safe direction, you can't make a mistake about whether the firearm was loaded and do something bad.
"Never point a gun at anything you don't intend to kill"
Except that guns are always pointing somewhere. If you think in terms of the negative, you'll drive yourself nuts. Instead, pick some safe direction or object you are not particularly attached to and point it there.
Our local gun store has some targets on the wall, up near the ceiling that say "Aim guns here".
Ah, Jeff Cooper and his four rules of firearm safety:
1. ASSUME the firearm is loaded until you have personally verified that it is not.
2. Do NOT point the firearm at anything you aren't willing to destroy.
3. Keep the booger hook (finger) OFF the bang switch (trigger) until you are ready to fire.
4. Know what your target is, and what is BEHIND it.
Proper firearm owners in the US also know these four and follow them. I can't tell you how many times I've seen rules 2 and 3 violated by people who really should know better. (Newbies get ONE FREE PASS from me, because I WILL call them out on it politely- the second time? I break out the Book of the Profane and start selecting appropriate phrases from it, especially if it's me they've swept!)
You need to find different folks to hang out with if you keep seeing rules 2 & 3 violated.
And instead of savoring the opportunity to read from the Book of Profane, maybe do a range safety review with everyone before the live fire begins.
I stopped going to public ranges because of the egos who are just searching for the opportunity to yell and scream instead of focusing on educating the other shooters. Most of them are already embarrassed enough that they broke a rule without having someone go ballistic on them.
An explanation offered to me was on lines that it's more complicated with actors. An actor typically isn't a gun expert and therefore isn't given any responsibility or freedom to check whether the thing in their hand may go bang. They have to trust the specialist, and the specialist, of course, must not trust the actor. A corollary is that almost always, the thing in an actor's hand must not be an actual go bang thing.
"I break out the Book of the Profane and start selecting appropriate phrases from it, especially if it's me they've swept!)"
I used to get particularly pissed off when supposed "firearms experts" would sweep a group whilst explaining how to verify the breech was empty
I don't care if you think it's unloaded. the subtext of #1 is "even if you've verified it's unloaded, it's still loaded" - I've seen too many instances of "verified" weapons subsequently discharging to trust anyone
Going by Jeff Cooper there's four:
1 - All guns are always loaded. Even if they are not, treat them as if they are.
2 - Never let the muzzle cover anything you are not willing to destroy. (For those who insist that this particular gun is unloaded, see Rule 1.)
3 - Keep your finger off the trigger till your sights are on the target. This is the Golden Rule.
4 - Identify your target, and what is behind it. Never shoot at anything that you have not positively identified.
These interleave nicely, so that a failure to follow any one rule should be covered by the others either preventing a negligent discharge, or at least mitigating the outcome of one. There's a slew of secondary rules which help like always check a weapon is clear with a finger as well as by eye, or always hand over a firearm unloaded and clear anything handed to you first, but they all come back to those four.
I've seen an assault rifle that had just been checked "clear of ammunition" go off half a meter besides me. Three meters fifty after I touched ground again, that is. Luckily the gun was pointed in the safe direction of a sand box that didn't stand there for no reason. Wearing no ear protection and not excpecting anyone to fire a live round in my vincinity it was quite a surprise...
When I was a cadet on an exercise the kid next to me *thought* he had cleared the chamber. He had not, fortunately they'd only given us blanks and he was pointed downrange, but you can imagine the bollocking the Sgt gave him. Poor kid was crying after about ten seconds.
Most hospitals now use a checklist to follow for operations. This site
https://www.england.nhs.uk/2019/01/surgical-safety-checklist/
states
"...pilots found that use of the checklist reduced the rate of deaths and surgical complications by more than one-third across all eight pilot hospitals."
"The checklist is a simple 19-item tool which addresses serious and avoidable surgical complications, by ensuring that critical steps outlined in the guidelines are done in every surgery, every time, everywhere. It also serves as a critical communication tool for the operating theatre team."
"pilots found that use of the checklist reduced the rate of deaths and surgical complications by more than one-third"
I know the NHS has staffing issues, but if they're having to resort to filling in surgical roles with former aircrew then things are even worse than I thought...
Yes, yes, I know what "pilot" is referring to in *this* context, it just seemed too good to pass over given the context in which "pilot" *had* been used earlier on in this particular branch of the comment tree.
AFAIK, It did come from aviation. Aviation has had incident analysis for decades to improve practices. Simple checklists create a known reliable process. I believe a USA doctor noted the aviation procedures and tried it to manage surgery in his hospital. It showed a dramatic drop in hospital caused injuries and deaths so idea spread.
The saying about checklists in airplanes goes "Two kinds of pilots always use checklists: beginners and professionals."
Once I was undergoing a rather complicated surgical procedure and I was lying in the pre-op area. The entire surgical team gathered around me and the head surgeon stood at the foot of the bed with an inch-thick three-ring binder. He was flipping pages and checkng things off as they talked. I was feeling a bit woozy from the pre-op relaxing pills but managed to ask, "Is all that for me?" He answered, "Yes, it is."
This is noticeable if you ever haver to attend a hospital as a patient for any reason. You'll find that every person you interact with in a clinical setting will first confirm your name and date of birth. They don't want to be taking the gall bladder out from the wrong patient by mistake.
This is also why X-Ray transparencies normally have 'L' or 'R' printed on them. Don't want to be looking at them upside-down by mistake and taking the wrong kidney out...
Yes, I saw a Dr once, and only the once, who told me my scan showed that my pain was due to terminal kidney cancer. The problem was that I had not yet had a scan, he also told me that he was too busy to follow things up but that a colleague would see me. It was only when he paused for breath, that I was able to tell him my scan was booked for the following week. I also pointed out that he might need to check the name on the patient's record... It was not clear if he was concerned about trying to scare me, or about breaking the other patient's confidentiality.
I had a similar experience, the consultant commented that my lung capacity had almost doubled in the twelve months since my last visit, but as this was my first visit, I was somewhat mystified. I then caught sight of the front cover of the file he was reading from, and when I could get a word in edgeways, I asked him to look at the date of birth of the patient he was examining. Turned out he was reading my father's file, his initials were the same as mine, but he was nearly 40 years older than me.
I'm reminded of the time I visited an offshore oil platform to review the drilling and well management operations. I suggested that, for rigging up one particular set of kit (IIRC, a coiled tubing intervention) they develop a checklist as there were several steps that could easily be missed and not noticed until the unmentionable hit the rotating blades. My suggestion was declined as they were all experts and checklists were for the inexperienced! Fine, was my reply. When you board the helicopter for your flight back to shore, tap the pilot on the shoulder and tell him to put away his checklist - after all, he's not a beginner. The company supervisor's view changed immediately - he was ex-forces and had become very acquainted with flying operations and the value of checklists...
My friend is a helicopter pilot, he's found that an iPad mini fits nicely into the knee pocket in his flight coveralls (which have a transparent window so you can stick a map, or a checklist, in there and have it easily visible). There's plenty of useful flight software that can display maps, checklists, all sorts.
They still keep paper copies in the cockpit though of course.
I once looked at a garden shed that was self assembly. According to the manager it’s easy to do. Unless you hire the professionals to do it, because they don’t read instructions. Worst case was a guy who couldn’t fit the roof. He was asked to send in a photo and was told “you built it upside down”..
Almost every contract I've been on I've spent the first couple of days rewriting the scribbled incomprehensible nonsense that passes for documentation into a checklist, aiming for a single A4 max. Forcing youself to actually write down on a sheet of paper and tick off Asset Tag Number, Machine ID, User ID, User email, Join Domain, Run Office Install, Map R: to \\etc, Map S to \\etc gets things done so much more effectively, and documents where you might have missed something, or have been forced to skip a step and come back later.
I have come to despise checklists, if only because of a bad experience of every dimension of a matrix organization creating their own checklists for the process of fixing a single software bug or feature.
With the checklists continuously being expanded, with no notice (you were supposed to always check what was the latest). With limited documentation and always incomplete or out of date references to other checklists.
All the responsibility of the software development team, all others just "audit" you or deny approval to release.
GAAAH!
I have a number of overnight job audit emails running. A while back I must have subconsciously realised, in my sleep, that one hadn't arrived at the expected time (the phone goes ping) as I woke up with the feeling something was wrong.
Useful as I was able to resolve the problem before any users noticed.
Allegedly, when Rolls Royce were testing Merlin engines to see what went bang first, (and hence find the weak points) the technicians used to doze while sitting next to the rigs running the engines...... they used to wake up when the engine sound changed.
I can quite imagine that being true, I've heard something similar about pilots and aircraft engineers who might notice something is wrong even when asleep in the rest area (long haul) because they're so used to how the aircraft sounds/feels that when something changes it hits the unconscious brain and starts screaming.
I used to sleep with a quiet computer in the room, and I would always wake up if there was a power cut because the nice reassuring white noise from the case fans would stop, it would invariably take me a few seconds to work out why.
I suspect there is some deep primal instinct that meant our ancestors would notice the lack of wildlife noise as a warning something (or someone) had caused the local animals to try not to be noticed, those that didn't wake up then possibly never did again.
"I can quite imagine that being true, I've heard something similar about pilots and aircraft engineers who might notice something is wrong even when asleep in the rest area (long haul) because they're so used to how the aircraft sounds/feels that when something changes it hits the unconscious brain and starts screaming"
My brother was in the Royal Navy as a "lifer". He said he could tell what his ship was doing just by the sounds of the engines/feel of the vibrations through the deck and bulkheads and it was obvious if there was anything unusual in the sound or feel.
Based on my experiences at Ricardo Engineering in the late 70s you *never* stand alongside an engine on a test bed - ALWAYS stand in line with the axis of the crankshaft.
Reason is obvious - when things go bang, large bits of metal exit the engine at very high speed and physics dictates the direction in which they travel.
Even then take care - these were Chinese lorry engines (based on a 40s design) running at full chat and maximum load, the exhausts were running cherry red. Then one of the radiator hoses ruptured and the resulting superheated water coming out under pressure would have broiled anyone in the vicinity.
Many moons back, a work colleague narrated a story of a site where they tested generator sets by attaching them to an electric motor and spinning them up. When the bearings seized, the set pulled itself out of the floor bolts and rolled through the wall, ... and down an embankment onto a railway track. Made a change from the usual "Please can I have my ball back ? "
There are a couple of stories of hydroelectric generator bearing seizures where this happened and the rotating assembly was found over half a mile down-valley after exiting the building at speed
It was only at this point that people considered what might have happened if the assembly had gone in the OTHER direction and went whiter shades of pale
Bear in mind that these were "small" generators (well under 1MW) and modern hydropower ones are easily 50MW with a small one being 2MW). More recently (1970s) a 2MW unit in Australia's Blue mountains had lubrication turned off before it stopped spinning during an emergency shutdown procedure and by all accounts the end was glowing cherry red within seconds
For obvious reasons, bearings and lubrication systems on $LARGE rotating assemblies are treated as critical assemblies with multiple redundancies - and if they're not, they need to be
In the late '70 I worked for a company that made dot matrix printers.
We had a test room with upwards of a 100 printers on soak test - hammering away in dot matix style.
If one printer had a faulty printhead you could hear it over the clatter or the rest.
They were based in Crewe in a wonderfull ex-railway office that curved round with the track. Alas it was destryed in a fire some years later.
I half remember the operator-less shuttle on tracks which take you from A to B, the door opens, you get in, the door closes, and off you go.
Until the queen came to open the facility.
She got in, and the doors stayed open.... they had to reset the shuttle before it would move.
There was lots of bad press, until they found the cause.
What happened was
The shuttle arrived, doors open, the queen says (quite) a few) words, queen boards shuttle.
The software had some code which says if the doors do not close within 1 minute, it looks like a problem. Do not let the shuttle leave the station, but send someone to check it.
Relays can fail in both states - contacts welding together, or coil burning out. Similarly for transistors, just not so easy to visualise.
Yes, I've seen lights stay on when the control signal was turned off. And vice versa.
Never trust what you can see. Or rather: trust but verify.
many year ago we used to have some ancient analogue valve equipment that was forever bringing up alarms because it had drifted out of alignment.
One day someone noticed that there hadn't been an alarm from it for over a week, so wandered over 'on a hunch'. The equipment was showing multiple alarms, but for some reason the alarm at the end of that bay and the remote alarm were not lit, so no one had been alerted.
Turned out one of the night men had had his fill of being woken almost hourly by it one night and had removed the alarm link, meaning to replace it before that day shift took over, but had overslept and forgotten
On an unmanned site I know of the operators were fed up of false alarms from the methane detectors. None was ever found when they visited so eventually they disconnected the remote alarm. Of course in reality when they opened the doors the added ventilation had cleared the gas and the fault.
This was discovered when one day the methane reached a critical mixture and met a spark, causing the roof and doors to depart the building.
On an unnamed site where an accident would involve the ordnance survey doing some serious rework.
There was a bing-bong alarm that sounded continually, if it stopped = run
(officially remain in your office with the doors and windows closed. Unofficialy get in a car and drive as fast as possible towards the fence, do not stop for fence)
Much the same for the US F102 interceptors in the late 1950s. Their task was to fly west on afterburner (very fast, fuel hog) to meet the Russian bombers, shoot them down until they ran out of missles. At that point they wouldn't have enough fuel to return home, so they were to end their mission by ramming a final bomber.
There was a bing-bong alarm that sounded continually, if it stopped = run
I remember being in a GPO manager's office (it was while ago) in a key city exchange, and there was a steady background tick-tock. He commented that while it ticked there was nothing to worry about, but if it ever stopped - that was the start of the "4 minute warning". Not much point in running, though.
They were normally in the 'off' position as the 'tick' was just the test signal. If the alarm was triggered it would switch itself on so the commands/orders could be heard.
Once a month (or so) they were supposed to turn the volume up to confirm they could hear the tick and confirm the circuit was working... if not they would report a fault.
I once knew a guy who was asked to take a replacement battery round to one police station... he handed it over and the sergeant promptly put it under the counter with all the other replacement batteries!
IT bod that w**ks at a hospital.....
Walking along corridor, passed by a cleaner on one of those big sit-on floor scrubbers and he calls out to me and asks "What does error 28 mean? It's come up on the display". Did explain that just because that uses electrons doesn't make it IT kit, looked it up on googly - low battery if you're interested.
Anyway, looking through the manual, there is also a light labeled "SAFE". However, that light will only light up when there is a fault that can potential cause injury - seems the completely opposite of safe to me, more like "not safe".....
Best manual I ever saw was for Retrospect backup software in 1996.
First page said something like If you're just opening this because you need to restore data, turn to "Restoring files" on page 72. Otherwise, turn page for Table of Contents.
Whoever wrote that, you are a hell of a tech writer. Thank you for the help, and the brilliant example!
Had a VW that always had niggly faults - for example the electric windows won't open or the radio will cut out and just display "LEARN", usually about once every six months. Pulling over, switching the car off and on again and it will be fine again.
Anyway, no surprise that the Check Engine light always came on and despite several trips to the garage and buying my own ODBC thing, I just ignored it and renamed it the "Engine Running Light". Had a hell of a shock driving on the dual carriage way one day when the light went out, until I realised that was a good thing....
I'd have said the same thing - and expanded by saying that it's most likely to be water under the carpets
Germans have a tendency to take the attitude "This shouldn't happen, so we don't need to design for the possiblity", whilst Japanese designers take the position "It HAS happened more than once in the past, so we MUST take the possiblity into account"
Of course if you have a sunroof, the risks go up exponentially because cleaning the drains out is never part of the manufacturer documented regular maintenance procedure, therefore never gets done - and they DO block up
A lot of older cars fed the field winding in the alternator through the warning light in the dash. If this lamp burned out your alternator wasn't charging and you had no indication of it. The only way you'd know is if you happened to notice the lamp didn't come on before you started the car.
A very long time ago, when electronic variable-speed motor drives were the new 'thing', I was sent to commission a job out in the wilds somewhere, transferring water from a reservoir to a treatment works via a long pipeline. The two inverters were protected at each stage by a multitude of fuses; there must have been 50 fuses per machine. The makers recognised that finding a blown fuse might be time-consuming and helpfully included a warning light for every fuse. I was sent with boxes and boxes of spare fuses. After a couple of hours, we realised this was going to be a long haul. The panel kept lighting up like a Christmas tree.
The 'Lamp Test' button revealed a display to rival Piccadilly Circus.
Late in the evening, a warning light came on, 'Low Pressure'..... A genuine fault or wonky protection? At first we rolled our eyes but on looking out of the window the true nature was revealed: the pipeline had burst and the adjacent field had become a lake. Made a very pretty sunset as the sun dipped over the horizon. Our day was done.
In the BBC, up to at least the 1980s, we used butterfly fuses: the fuse wire was soldered between a couple of spring contacts, which sprung out fore and aft when the fuse blew. One of the wings would contact a bus bar which powered a fuse alarm to make a horrible noise.
As the fuse alarm was electrically powered of course it had to have its own fuse - the fuse alarm fuse. And because you had to know if the fuse alarm fuse had blown, it had its own alarm, which had its own fuse... the fuse alarm fuse alarm fuse. After two levels of fuse alarms, I think they just got bored.
In the BBC, up to at least the 1980s, we used butterfly fuses...
Ah the nostalgia... those fuses were used on anything operating from the - 50 Volt Station Battery...
Too many years ago I worked in the (then) BH Ext in London, more or less overlooking Duchess Street. What I was doing at the time is long forgotten but on one occasion I was working on something on a temporary feed from one of those fuse panels, and in simple terms things weren't performing as they should.
Switch off; check supply voltage; nothing wrong there. Switch on again; problem still present. Check voltage with equipment connected... aha! That's a bit low.
Some comedian (longer in the tooth than I was) has replaced the fusible link with a resistor, rendering the "fuse" somewhat useless.
For the uninitiated the fuse wire (or resistor, as the case may be) was actually invisible as the butterfly's wings concealed it when the wire (or resistor) was soldered in.
Oh happy, innocent days... mostly anyway.
>but on looking out of the window the true nature was revealed: the pipeline had burst
A good lesson from Apollo13. Not much mentioned in the films and 'failure is not an option' accounts. But they wasted 20mins checking for computer errors AFTER the crew said they felt an explosion because an explosion would be bad so it couldn't be real
I used to work in a University computing lab. One day, we had an open day for the local school kids, and were running activities in several groups on the lab. Everything was going well, until just before lunch, everyone lost their network connection. Because of the nature of the lab, it was on it's own isolated network. This made it easier to track down the problem.
The lecturer in charge of the day was furious, but agreed to send the kids out to lunch to give us a chance to fix the problem. We looked at the single switch, and it was locked up. We rebooted it, and it locked. We tried to connect to the console port, and rebooted the switch. It locked before we were able to log into the console.
So, we started dropping connections. After disconnecting a few computers, we found the problem. A couple of PCs were flooding their network connections, so we went to look at those PCs. One of the kids was running a denial of service attack on the switch's IP. When the lecturer's running the groups got back from lunch, we talked to the lecturer running the group using these PCs. She admitted she had shown the pupils the tools used for DOS attacks, and made them promise not to run one on any machine within the Uni. She also apologised, and shut down the application running the DoS attack, so we reconnected the PCs..
This isn't the same lecturer who was in charge: She was teaching a group elsewhere in the lab.
I last worked in education in much simpler times when we had a couple of RM Nimbus networks. Initially the students used to just unplug connections randomly at the end of their session, annoying for the next class but easy enough to diagnose and fix. They then found it was possible to short a network connector which took a lot longer to find. I don't think DOS attacks had been invented then!
As someone who currently works in education I can confirm the "unplug connections randomly at the end of their session" trick is still going strong, although now it's usually with the trick of "leave the cable just in the port enough to look like it's connected" or maybe rarely "unplug the Cat5 from the wall socket and plug it into the network port of the next PC along".
Once had a student have the bright idea cut through the power cable on his PC with a pair of scissors instead. Thankfully unharmed, but that was a fun reaction from the teacher, and soon after the site manager when he came in to flip the breaker back on.
Saw something very similar one day when visiting Clemson University IT dept.
Major Incident, all network connectivity knackered, F5 firewalls shitting the bed.
Techs convinced they were being DDOS'd from outside, went on for some time, until they isolated part of the network, and realsied it was internal.
Some TA and some students had kicked off a demo DDOS script on a lab machine at 3am and wandered off to bed.
Me, I was just a visitor so sat and watched with a smile on my face that I wasn't having to work that crap out.
Turned up to work early one day to find the duty operator fast asleep, on a high chair with his feet propped on the System/38 and said chair rocked back on its rear legs.
The word "precarious" seemed inadequate. Too good an opportunity to miss, so what to do.
Enter computer suite using keycode that I don't know. Tiptoe over to console. Dial 1 to enable, dial 2 to Lamp Test and hit load. Wind console alert volume to max. Tiptoe out.
SNDBRKMSG MSG('WAKEY WAKEY') TOMSGQ(QCONSOLE)
BEEPBEEPBEEPBEEP "Wuh......ARRRGGHHH" <CRASH>
I've seen quite a few variations of it. I believe the original goes like this:
Achtung! Alles turisten und nonteknischen lookenpeepers! Das komputermaschine ist nicht für der gefingerpoken und mittengraben! Oderwise ist easy to schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparksen. Ist nicht für gewerken bei dummkopfen. Der rubbernecken sightseeren keepen das cottonpicken händer in das pockets muss. Zo relaxen und watschen der blinkenlichten.
Pretty much the definitive history of blinkinlights here.
Here's a plaintext address for those of you who quite sensibly refuse to simply click on links without having at least an inkling of where they might take you:
http://catb.org/~esr/jargon/html/B/blinkenlights.html
Note: The above link contains nazi symbolism in a historical context. Might not be safe for work in some jurisdictions.
I think my formative Uni computing years were influenced twice.
First by attending the observation gallery at Birmingham uni which surveyed a football-field sized computer room full of IBM blue boxes.
Seemed impressive, until I learned that all the input was by punched card, no terminals. So I passed on Birmingham.
Then later at Durham, the IBM 360 was supposed to be unhaltable (so the console paper story was interesting).
Anyhow, some wag wrote an assembler program with a couple of tight loops toggling some specific memory locations. He had worked out that these core memory locations we under the two temperature sensors, that shut the whole machine off in case of fire. The core memory heated up, and shut down the machine.
The IBM techs were called out and were perplexed to say the least!
Got a brand new electric bike and took it out for a spin. Stopped for lunch, and when I got back, the bike was flashing "Error 47"
No idea. Dragged out the owner's manual. No table of errors. Stood with my phone Googling. Nothing.
Finally decided to push the bike over to the shade, and flipped up the kickstand. Error 47 disappeared.
I bitched BITTERLY at the company, and they included a table of errors in the next revision of the manual and on their support page.
DOCTOR: I, er, had hoped to reach your planet Earth. Skaro was in the future and I used the fast return switch.
IAN: The fast return switch? You've sent us back too far. Doctor, show me. Show me that switch. Where is it?
DOCTOR: Well, I can't very well see it without a light, can I?
SUSAN: It's near the scanner switch.
BARBARA: Really? But that's the part of the control that's safe.
SUSAN: Yes.
DOCTOR: Strange.
IAN: Doctor, we haven't got very much time left.
DOCTOR: Yes, I see. Here it is. (pulls a small torch from his pocket) Here, you see? Now, look, there's the switch. You see?
IAN: Yes, well how does it work?
DOCTOR: Well, you merely press it down, and. It's stuck. It hasn't released itself!
IAN: What? You mean it's been on all this time?
DOCTOR: Yes, it must have been.
IAN: Well, come on, Doctor. Let's get it unstuck.
DOCTOR: Hold that. Yes, just a minute now. Yes, there you are, you see?
IAN: What's wrong?
DOCTOR: The spring's not connecting. It's come off the base.
IAN: Hurry, Doctor, hurry.
DOCTOR: There we are. Take it out. Now, luckily we can turn it over and now it should work. There. Ah, that's all right.
(The lights come back and the Tardis returns to normal)
SUSAN: We're safe now.
BARBARA: Are you sure?
DOCTOR: Yes, we can all relax. We're quite safe now. But it was a narrow squeak.
SUSAN: Grandfather?
DOCTOR: Yes, my child?
SUSAN: What happened?
DOCTOR: What happened? It was the switch. It was still in place. You see, there's a little spring inside it and it was stuck. It hadn't released itself.
SUSAN: But why didn't the fault locator tell us?
DOCTOR: Well, the switch hadn't broken down, therefore the fault locator couldn't give us any recognition.
On our last day before being kicked out of school permanently, our class had some really great 'lessons'. (the teachers all suddenly became humans) The physics teacher set us a puzzle. It seemed simple enough. Design a failsafe doorbell.
It was great fun, but need I say, we failed.
"Design a failsafe doorbell.
It was great fun, but need I say, we failed."
Need more info. Is it more important to not open the door if no one is there than it is to not let someone in who might be dying? ie, does the bell fail "safe" on or off? Or was failsafe used incorrectly such that the teacher wanted a door bell that couldn't fail or at least would guarantee an accurate indicator of working/failed?
Yes a rather woolly description I agree. The person operating the bell had to reliably be informed if it had failed to actually sound. Bear in mind this was for a bunch of 16 year olds in the mid 1960s. Eventually someone (not me) was bright enough to ask if the bell not being set a signal counted as the bell not working!
And, of course, they always said yes because they'd never admit to having not done the simple and basic checks that even a non-tech ought to be able to carry out. So the correct first step is, "Maybe the connector has come loose, can you try unplugging it and plugging it back in, (at both ends if relevant)
A certain power station I visited quite often in the 70's ran 3 off, 600 MW or thereabouts steam turbine generators. I was introduced to one entry in their unofficial photo album of Things That Had Happened.
You have to know that the exhaust from the steam turbine went to a condenser, mounted like a saddle over the turbine. The condensed water went to a small tank, from whence a steam turbine pump whizzed it back to the boiler. In case the steam pump failed, there were electric pumps to protect the turbine, as water washing back into the turbine was considered A Bad Thing - a few buckets of water hitting the blades of an 8 foot diameter low pressure turbine do make a considerable impression - those blades are moving at almost supersonic speed.
Right. Scene set.
Apparently the operators were getting a little annoyed at repeated false alarms from the catching tank high level alarm. So annoyed in fact, that the alarm had been bypassed.
Predictably the 1,000,000 to one chance happed (as they do 9 times out of 10) and the turbine pump failed. The electric standby pump just happened to be on maintenance at the time and didn't kick in.
The water level rose, unnoticed.
And kept on rising, unnoticed
It eventually washed back into the low pressure turbine, where it did an awful lot of not good. People did rather notice though.
I gather most of the LP turbine removed itself from the building. Via the roof. The main shaft, a substantial piece of chrome moly steel some 22 inches in diameter, simply bent. Some main bearing cap bolts, 30mm or thereabouts diameter, pulled apart in a text-book tensile fractures.
A guy nearby, working up a ladder, slid down, broke both ankles and ran out of the building..
It was eventually revealed that the root cause was the bypassed alarm. Further investigation discovered the cause of the repeated false alarms: the level switch was a pressure sensor mounted remotely from the tank and connected to it by what was described to me as a copper pipe not unlike why is in a central heating system. The pipe ran from the tank, along a wall and round a corner to the pressure switch. When it had been installed, the pipe had never been adequately supported. It had sagged and cracked a simple joint.
By bye turbine. They had to call in the National Spare
after 5 minutes of compiling :
error in line 1 : missing semicolon ...
The problem is the compiler compiles all the subroutines and other 'includes' first. They are all fine as they are part of a tested and tried library.
once it built all those object files it starts on the main file... which has a missing semicolon in line 1.
Those were the days i acquired an innate hatred of any language that uses semicolons for end of statement. use the damn <cr> or <cr/lf> that sits in the source file.
If a statement is too large to fit on 80 columns , use a continuation character. you will need very little of those.
And code the compiler so it understands when = means assign , and when it means compare. The logic is not that hard : when you are in a logic test ( if / while/ do until / case statement) it means compare' , otherwise it means assign. and don't give me that ' i want to assign in the logic yadda'. That is cryptic to read ( and even then the compiler should be able to figure it out . the logic is recursive after all). Code should be easy to read and write without requiring a degree in literature.
The semi-colon is probably the most underused piece of punctuation in the written language. All hail the programming language designers for lifting this obscure mark back up it's rightful and most well used position in the punctuation hierarchy. It's just a shame those language designers don't actually know what the semi-colon is for, but they do think it looks pretty; it's a "safe" character to use since it will never appear in user input.
Reminds me of the time I was taking a Basic programming course, around 1982. The computer was a TI midi connected to a greenbar wide dot matrix. The operator started a program another student had written and then went to lunch. When he got back he went into a panic because the printer had almost gone thru an entire case of greenbar. He killed the printer then the TI. As in hit the power switch. Strongly NOT recommended procedure. When they were done checking everything else, someone thought to check the code. The student forgot a single semi-colon putting the program into an endless loop.
Back in the 70s worked at a place which still had a very old telephone exchange where the operator plugged cables in to connect 'phones to the outside world. Internal calls could be dialled.
Whenever we tried on our shared office phone to get an outside call, the operator never answered us. The managers in the offices on either side had no problem. One day a colleague got fed up and used a manager's phone to ask the operator why we were ignored.
The response was "Oh! Your bulb has blown so your line never lit up. I'll change it"
We never had any further trouble - until a modern exchange was installed!
I’m reminded of a situation in a house I moved into in Ireland back in the days when landlines were still something people knew the numbers of and still plugged phones and modems into.
I ended up stumbling across the specs for the Irish phone network by trail and error deduction.
We got the phone line activated, but there was no dial tone in any of the extension sockets.
I went around checking each one and used a multimeter and there were definitely no wiring issues. Simple stuff, two wires, very similar to the US.
I checked everything on the first socket and found no faults, but it had a dial tone on the RJ11 socket on the front. It just stopped there.
I then accidentally noticed that a screw I had dropped was magnetically struck to a device in the back of the front cover of this socket. It was only then I noticed there were two magnetically operated micro switches that cut off all the extension wiring when the Telecom Éireann branded “NTU” had its access cover open.
It was a neat demarcation system for testing the service and eliminating the extension wiring, useful but if unfamiliar with it, it wasn’t at all obvious.
All ripped out, never to be seen again and replaced with Ethernet and XPON FTTH.
A few years ago, was working on several laptops on my desk, which due to the lack of desk space, were stacked on top of each other. (our version of vertical integration, I suppose) Suddenly the machine on top went black, just like someone pulling the plug, I played with the power button, eventually getting it to reboot, only to have it go black again. I resolved to look at it later and started working on the second machine in the pile. It too, went 'off' suddenly. Frustration soaring, I picked it up, only to have it come back on, set it down, it shut off again.
Finally, it dawned on me that the permanent magnet in one of the laptop speakers on the machine immediately below was lining up perfectly with the 'lid closed' sensor and turning off all illumination on the top machine. After that, I had to show my coworkers a "magic trick" where I concealed a powerful neodymium magnet in my fist and passed it over their machines, making them suddenly go dark.
My first IT job was with a county government that had 360-series gear; initially a 360-40 and 360-30, then the 40 got upgraded to a 360-50 (by the time I 'graduated' from programmer to PFY systems programmer).
Our single most time-critical application was the election tally system... if things ran slow we had the media (bad) and the politicians (fatal) all over us. The weak spot in the chain was the 1052 console, coupled with the fact that the application was 'console chatty' (for no good reason).
We lived in constant fear the app traffic to the 1052 would overrun the console buffer and crash DOS (sic!).
This was mid-1970s. By the time I left in the late 1970s (as BOFH) we'd migrated to a 370-155 running OS/MVT with a partition or two running the DOS emulator. All this in 384K of memory, as this was pre-virtual (needed a 370-158 or better).
Good times.
Gosh! “Explaining to the customer why a printer running out of stationery had taken a football-squad of techies and a large chunk of the day to resolve,” should be avoided at all costs. The customer would be much happier with a drama and an explanation that you first thought it was a tricky problem with the “x794interconnector” running hot that had shut down the whole show to avoid loss of data. The lead techy then has to suck air through his teeth and explain that a colleague had seen this sort of thing once before, he’d run “the fix” and it won’t happen again.
The guy will be dining out on the “x794interconnector” problem they’ve been having all week.
So, we are bad people, big test lab set up to run rests throughout the weekend but it was ok for us to be there as our work did not clash.
So finished up, empty building and a football.
No, we were very careful. Just bounced it along the corridors and all was well until it was time to break for lunch, luxury.
Chuck us the ball, clunk, silence.
I was just putting the finishing touches on a small cluster of vaxen at SLAC one fine Friday afternoon. The annual Big Game between Stanford & Berkeley was to be the following day. A couple of grad students started passing a football (American version) between themselves. In the glass room. Just as I was threatening mayhem if they didn't knock it off, the ball hit the Big Red Button. Needless to say, a bunch of pissed off people couldn't attend the game the following day. The grad students computer privileges were suspended for the rest of the academic year. Personally, I'd have hung them by the thumbs in the Quad as a warning ...
As an alum in good standing of both schools, all I can add to the above is "Go Bears!"
Gosh! “Explaining to the customer why a printer running out of stationery had taken a football-squad of techies and a large chunk of the day to resolve,” should be avoided at all costs. The customer would be much happier with a drama and an explanation that you first thought it was a tricky problem with the “x794interconnector” running hot that had shut down the whole show to avoid loss of data. The lead techy then has to suck air through his teeth and explain that a colleague had seen this sort of thing once before, he’d run “the fix” and it won’t happen again.
The guy will be dining out on the “x794interconnector” problem they’ve been having all week.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
