When we asked how you crashed the system we wanted an explanation not a demonstration Once again, gentle readerfolk, it is time for the Reg's weekly confessional – Who, Me? in which we recount readers' technical anti-triumphs. This one follows on from recent tales of big red buttons, and goes one better. This week meet someone we'll Regomize as "Hal", who recounted for us a tale from the epoch of the mainframe …
seriously not required?
I could totally see me doing the same procedure, even if the powers that be frown upon it nowadays. Spoilsports.
I once had a sandal chucked at me (deserved it, every bit). I learned from the best, so to say. Said guy would have flying-tackled me, should I had tried to push the BRB...
about to put the third (and last) copy
My employer had been a contractor at a business where, when the day shift came in, they found that the (IBM) hardware had self-destructed, the back-up copy had been destroyed in turn, and the night-shift operator was trying to get into the safe ...
I will admit to doing something similar.
I have also typed the [command that must never be typed] as root. Oh, well, the system needed a new OS install anyway.
...and, as a "young learner"*, I have had more than one chalkboard eraser chucked at me when I wasn't paying attention (Latin and Maths classes, I believe)
* a term my teacher wife and I use to describe students who would have been referred to as "little bastards" in earlier, less sensitive, days. See also: "our valued client"
We had one teacher who could watch the class through the reflection in his spectacles. If he saw you mucking about, or even just not paying attention, he would whirl around and throw the chalk with which he was writing on the board, and unerringly hit you on the forehead. If the chalk was a small dog end, it didn't hurt too much, but a new stick of chalk, well, that was a trip to Matron for a sticking plaster.
Reminds me of the time someone on the front bench in the physics class was mucking about with his fountain pen and managed to flick an elegant arc of ink across the lab. It landed, mostly, on the whiteboard on which the teacher was writing.
Despite not having the benefit of spectacles in whose reflection to observe the class he did realise that Misbehaviour had been perpetrated as he looked at the whiteboard to his left, bearing a line of blue dots. And the whiteboard to his right, bearing a line of blue dots. And then he removed his previously pristine white labcoat and held it up, observing the line of blue dots which linked the other two lines of blue dots...
Was it 1968 when the first eraser was launched in my direction?
Ah...school in the good old days.
The age of Aquarius hadn't quite taken hold with my generation.
Our social strata still resembled "The Outsiders". https://www.imdb.com/title/tt0086066/
The very fact that the intern is blamed for this mess just underscores the complete failure of the management.
The intern was not properly trained - Management Failure
The intern was alone with a critical system when they should not ever be - Management Failure
The "walk us through" exercise was performed in a non-step-by-step fashion where each step is discussed before executed - Management Failure
The intern was physically assaulted ( at least under US Law) by Management - Management Failure
As an ex support engineer and tech support manager I would have booked a 7 am meeting, that's the earliest I can expect some of the guys to get in in the morning and have a handover from the poor sods who have been up all night. In reality this means they could probably get to go home at about 9 am. I've done thew whole 'work the problem until it's resolved' thing too many times in thee past, my longest stint was from 8 am Friday to 2 PM Sunday afternoon I don't put my teams through that its much better to cycle people through and make sure they are rested and can actually think.
I did drive home on the sunday afternoon as we only had one car and the weekend marked the end of a long week with a few rest days to follow but it was not a safe drive.
25-30 years ago as a junior doctor in the NHS, I worked an "on call rota": 0800-1800 daily, and every 3rd or 4th day through the night as well. Every 3rd or 4th weekend was either Friday evening or Saturday morning through to Monday.
Never a day off, even if you had been up all night, or only had one hour sleep in the last 48.
Safe? No!
Eventually the European working time directive forced changes to full shift systems.
Safer? Not really. Doctors are still exhausted as the work intensity has increased massively.
Deploying a new PC (That's covered in other oft repeated tales).......
Background Guy One: Ohh just got a email from Sanjay (For that was his name), telling me where he is now working...... I don't know why he bothers as invariably hes got the sack 1 week after sending it.
Background Guy Two: Great [/sarcasm]
Background Guy Three: Who what eh? [/Confused]
Background Guy One: OK we had a chemical spill incident, down to him, all those that got splashed had to have cold water showers outside & sent home in paper overalls, our clothing destroyed.
Background Guy Two: I remember all to well [/anger]
Background Guy One: So we had the Elfin Safety autopsy & Sanjay was asked to show us what he did leading up to the incident.....
Background Guy Two: I remember that all to well as well [/real_anger]
Background Guy One: So Sanjay went through the steps right to the point where he was supposed to carefully lower a weight into the chemical vat...... & for the second time he just heavily dropped it in, splashing us & this time the entire Elfin Safety team who were dutifully watching up close & personal, after a repeat of the cold showers & paperwork....he was unsurprisingly sacked!
Icon - WTF's just splashed into me eye!
> But senior people needed to know exactly how this happened so it could't happen again.
Just disconnect the big red button.
Or if that is too easy, rig it so that it has to be pressed twice. And that after the first time, flashes up a message saying Please do not press this button again
>Just disconnect the big red button.
I used to work in a room with a Big Red Button.. The room had housed a 70's IBM originally, and when it was replaced by a rack of Suns it was re-wired to shut them down in an orderly manner.
By the time I was in the space the computers had been moved across the building. Only the Big Red Button remained. Myself (and a few of my colleagues) took to occasionally punching it when we wandered past. What we didn't know was that every time we did a monitoring server in the new server room saw an event and alerted an admin that a dozen servers were about to be powered down.
The admin quickly realized that the alerts were associated with servers that no longer existed and reconfigured things so they weren't being paged, but still kept them logged to figure out where they were coming from.Took him a month, as he reviewed the monitoring software config, tracked the message source to an unlabelled metal box in a closet, and then toned the wiring out to our Big Red Button.
Happily, he didn't make us remove it. He did, however, stand there and watch as one of our engineers opened it up and physically cut the wiring.
This one came up on Scott Adams' mailing list many moons ago.
In a large industrial company that had been operating for nearly a century, the IT lads were moved into a newly refurbed office in a converted factory building. One of the features in their shiny, new home was a very large, cast metal box set into the wall next to the door, upon which was a very large red button, appropriately labelled "SHUTDOWN".
Being smart lads they brought this up with the management and were told that it was so deeply embedded in the fabric of the old building that it couldn't be easily removed, but it wasn't connected to anything. All ok then and a nice feature to have on the wall, reminding everyone of the firm's industrial heritage.
One of the correspondent's colleagues took to smashing the button firmly in passing at the end of the day on Friday and loudly stating; "I'm out of here".
While all the lads were painfully aware of this (such things tend to wear a bit thin after a few repetitions) it took the rest of the company some considerable time to work out why it was that around 5 on a Friday afternoon the entire production facility ground to a halt.
Someone told me about an incident where their father worked. It was metal press factory, where they had big sheets of steel and made them into boxes. The machines had safety equipment, but one person managed to get the end of his middle finger sliced off! Lots of blood etc, and the very tip of a finger.
There was a H&S review, and people came to find out why the safe guards didn't work. They asked him to go through the same process as he did before. There were no problems - until they said they were going when he cut the end of the middle finger on the other hand!
Apparently the injured person had very narrow fingers which went through the mesh of the protective cage. With the audience watching he was extra careful; as they were going he went back to normal operational procedures.
The mesh was changed to have even smaller holes.
"Which is why abolishing [standards] is only ever proposed by MPs like the Minister for Workhouses, Rees-Mogg."
Well, and the sorts of people who back their campaigns, because they care more about cost cutting than ... pretty much anything else.
My father worked in a fish processing plant as a student. One day a filleting machine jammed and the operator was going to shut it down to clear the innards, at which point the supervisor said "no" and argued it was easy to get it out and proceeded to demonstrate how to get the back of his hand filleted as it resumed motion. Clearly should have let the operator do what they knew about.
Almost Darwin in action.
I have this on unimpeachable authority that this tale is true.
There was a case like this near me ages ago.
Said operator managed to crush his fingers in a press brake... all nicely broken and bleeding...... so pack him off to the hospital for fixing
He came back 8 weeks later to a manager and health and safety inquiry about how that accident happened...
So he loads up the press brake with a part , slides his hand in where its not supposed to go and pushes the treadle..... crunch.
Loads of blue flashing lights as he goes back to the hospital to have his fingers unmangled again......
Unsafe safety demonstrations also happen in the US, but they really aren't comedy:
https://youtu.be/vfONckOPyaI?t=38
Not really on topic, I guess, but here's an idea to promote gun safety (just off the top of my head): don't have so many fricking guns. Maybe then the US wouldn't have more than 75 times the number of firearm-related deaths per head of population as the UK.
(At least one person apparently still thinks that "guns don't kill people*".)
(*Admittedly, rappers do.)
I'm fine about it. You're the one who seems to be terrified that without stupid numbers of guns in the hands of ordinary idiots, the king of England could come and start shoving you around.
Yes indeed it would have been really fascinating to see what kind of message would prompt this action..
In fact if I had to guess I bet the message would be the result of a command the intern typed in that resulted in thousands of rows of text scrolling by starting with the word "Deleting \root\stuff\"
Reminds me of an operator running some trainees through some basic commands on the console of our Sperry/Unisys mainframe. But of course not actually hitting Enter for those commands which could cause an issue. Finally he got round to showing them the command to reboot by keying in $! (Dollar-Bang), forgetting that this was the one command that did *not* need you to hit Enter for it to be actioned! :o
We were replacing the three generating sets used under emergency to charge batteries at a power station. Each diesel-alternator set was about 1MW. I asked why the old sets were being replaced, because after all, they hadn't done much work over the years and the technology had not advanced significantly.
One of the operators took me into the 'old' equipment room where the three engines were open but with tarpaulin covers over the crankcase. Each engine had a broken crank...... The first broke due to an auto-synchronisation fault, allegedly. The second broke during a demonstration of manual-synchronisation with the grid. The third broke when the expert took over to show 'how it should be done'.
The replacement sets were a good order for us so ----> all round.
The problem is the power behind it. Even if you are 1% off you might see a bolted down generator, weighting a few tons, jump off ground. If you are lucky it will be the only thing broken. You don't want to do that manually when serious power is involved. Especially your light-bulb method is not precise enough.
Well, despite retaking my electrical engineering exams every year during my uni course, I managed to qualify as an Electrical & Electronics engineer. A dozen years later and I'm working as the guy in charge of the design team at a very well known and rather conservative Instrumentation company.
They produced *mechanical* synchroscopes (in the UK, no less) which were a work of art, and even had jewel bearings. These things would rotate 360 degrees when the supply and generator output were out of synch, but could be made to stop at 0 degrees by tweaking the genny speed. A separate box with a clever transformer with two primaries could close a relay when both the voltage and phase were in acceptable limits. Result: No broken crankshaft.
Someone decided that they wanted an electronic version as a competitor had introduced one with a ring of LEDs to show the phase angle, and a couple of go/no-go lets monitoring the voltage. That sounded like an ideal job for one of those new 10 bit PICs I had been reading about, so off I went to design something that did everything in a single unit. And it worked, very well and brought in vast fortunes for the company.
However, the marketing wonks wanted to sell this as an automatic synchronising system, and I cannot convey how hard I had to fight to convince them that an 80p microprocessor was not adequate for such a task. Sync Check, yes. Control No. Thankfully I won.
Fast forward a few years at a German trade show and this guy is showing off his container mounted generator rig, and there is my bit of kit on the front. As instructed, there was a big switch in series with my relay output, the purpose being that the operator had to use their eyes and ears to confirm all was well before pressing the button. To my horror he simply started the engine and pressed the button even though the display was whizzing around at an alarming rate ... and as soon as it hit 0 degrees for a number of cycles the breakers connected the generator to the grid. No bangs, no bent crankshaft. His English and my German weren't compatible enough for me to explain why that was such a bad idea. All I got out of him was "It's the best device on the market".
An £0.80 microprocessor controlling a £10k+ generator. Madness. Mind you, we never got any warranty claims despite selling thousands of the things.
But you didn't explicitly ask for an explanation rather than a demonstration did you? Your insufficiently detailed specification for the job left the newbie free to interpret the task however he wanted. And we all know how that can sometimes go.
Obviously you are NOT a programmer - none of them would let you get away with such a stupid comment - solving problems within code streams is hardly robotic on the other hand remembering you should have/not pressed a particular button requires mere attention & memory !
Two points
1. If that was the worst thing a media company ever did, I wouldn't need to drink.
2. Most news articles are written with the "inverted pyramid" style, meaning that the newest info is at the top, with the article becoming more about context and detail as you go. So, halfway through the article, the reader may be ready to move on to something else because they've got the gist of the current article. Obviously this doesn't apply to all article styles, but if they have one layout for all articles well...you can fill it in for yourself.
First thing that strikes me is why are the links to other stories, infuriatingly dumped in the middle of the article like a clickbait site singles advert, headed "More context" ? To further ensure that people take longer to figure out they are not part of the story?
I've seen this in a lot places but usually with a heading like "In case you missed this". Usually though, it has recent articles.
Before working in IT i worked in a laboratory where we had the only 2 testing machines for a particular test required on a very unique product. One of the machines was being serviced by a PFY and mangaed to break it when he stuck a screwdriver in it. bearing in mind that the only other machine like it stood 2 feet away you can iumagine our sheer horror when the PFY's supervisor asked him to demostrate what he had done.
We were doing a computer room shutdown so some heavy duty electrical work could be done. Being the Solaris guy I had cronned a init 0 so when I got in all my babies were at the OK prompt. I got to the console on the first one and entered the command power-off just exactly as the facilities guy hit the big breaker. For a few seconds I had a blind panic that I had somehow managed to connect my sun box to something much more serious. The Windows guys were swearing copiously as they had only just got half way through their complicate and order shutdown process.
I staved off brown underpants but only just.
"...Fixed something only to break something else?..."
Yes. Literally on Saturday night (well Sunday morning as it was gone midnight.
Not tech related but it was noticed that there was water coming from under the door of one of our outbuildings.
Uhoh... sub-zero weather...water.
But it wasn't a burst pipe. The connector to a tap to a sink had popped (presumably never fully tighened + cold weather).
Turned off the stop cock in the outside toilet...only for the connector to the pipe to pop off!
WTAF?
Happily I had the tools, spares and basic know how to be able to fix it quickly and without even shutting off the water but not what you want at that time in those conditions!
Just about qualifies as computer/user interface error...
As we know, all modern cars are in fact a number of computers surrounded by pretty metal and some leather. My old jalopy (2006) still young enough to have about 30 of the damned things. One of them is supposed to do something, which indeed it did on cue.
So, I'd had the old jalopy's headlights refreshed by a Man who does - fine paste grind-back of the plastic lenses and polish, followed by sealant and UV barrier coating. Whilst we were wrapping up after the last stage which, Mr Man reminded me, needed to be allowed to fully dry (I was even thinking about garaging it as rain seemed to be due), we started discussing his other old-jalopy-skills, namely sealing chips in the windscreen. We were looking for the chips and I said "it's easier to spot them if I run the wash-wipers", which I proceeded to do (3 x unstoppable 2 second bursts). Whereupon the module responsible for washing the, er., headlights on the first then every seventh windscreen wash, sprayed wash water all over the headlights. Doh. At this point we discovered that 5 minutes in direct sun turned out to be as good as the quoted hour in less friendly weather for the coating to dry.
Got away with it, apart from the embarrassment.
i once had a wiper fly off after the first swipe during a cleaning.
The uncoated metal stump then proceeded to scrape the hell out of my windshield for the remaining two passes.
Decades later I am still apprehensive about initiating that unstoppable process.
Thirty years ago, I worked at a company where the systems all shut down one afternoon. It turns out that a janitor was cleaning the server room, noticed that a "thermostat" seemed to be set higher than he thought it should be, so he set it lower. It was the "if the temps ever go over this value, shit everything down" thermostat. So, of course, they did and it did.
The following week... the systems all shit down again. Why? Because the sam janitor decided that there should be a wire cage over that thermostat so no one could adjust it. And he took it upon himself to install one.
In so doing, he bumped the setting.
I'm told the head of the computer department had tp be forcibly restrained.
Afraid to say that in my younger days, I was the only one that caters for fixing performance problems.
There were surely associated with servers that had Broadcom nw cards trying to auto negotiate with HP Procurve switches.
Never worked. Reached a point where is go through all servers and change the speed explicitly. Controlled of course
I kept having a fault in network switches only to find someone diagnosing a failed UPS by changing its settings, repeatedly throwing it into self test and watching it not recover. Sure, it's a great mystery why the ups is broken but bypass it before causing hundreds of users to freak out over their PCs.
I upset a German bank a long time ago, by replicating 'an error'.
For a while I lived in deepest darkest south Germany (so south it is technically Switzerland, but the German's nicked it), and returned some years later for a holiday - and attempted to use my NatWest Switch card (that dates it doesn't it?) in a Sparkasse ATM.
Removed card from wallet, inserted card into machine, machine immediately reboots itself.
Once rebooted, of course, there is no sign of my card...and the bank was closed - and the phone numbers went to answer services services.
Next morning, waited outside the bank until they opened, and explained (in halting German) that their pesky machine had b0rked and could I have my card back please?
The attendant's response was (gesturing at the machines) "all the machines are working, if your card has been taken, it's because your bank has reported it stolen". I protested.
Eventually, someone checked the machine, and found the card. I could see them looking at it, but still refusing to give it back. I protested a bit more, saying I'd been using it happily in other places and it was fine, and offered to ring my bank to have them explain.
After a few minutes I was handed my card back, and asked to show them what I did.
So I put the card back in the same machine. And it rebooted again.
They apologised, retrieved my card again, and I toddled over the Deutsche Bank and I withdrew some cash from their machines quite happily.
No idea what combination of info was on the magstripe that clearly b0rked Sparkasse machines, but it did tend to mean I used "larger" banks after that :)
Okay, admit it - it is YOU, little Bobby Tables, isn't it...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
