To err is human. To really screw things up requires a wayward screwdriver Sometimes a shortcut can result in a short circuit. And occasionally a shortened career if one lacks an understanding boss. Welcome to another Register reader confession from the Who, Me? archives. The latest tale takes us back to the 1970s, "when computers really were computers and a megabyte was a LOT of data," recalled …
Maybe... but not always.
Ask the operator in South Africa's Koeberg nuclear power station who dropped a nut into the one generator (of two) under maintenance, and didn't tell anyone. When the units were brought back online, that nut caused massive generator damage, causing a shutdown, and because the second unit was in maintenance, caused ESKOM, the South African power utility, to have to deal a massive power shortage.
There was lots of finger pointing, although I understand that the French company, Framatome, walked away from that unscathed.
Any South African Reg readers will know this story well because it led to so-called a long period of 'load shedding' in the Western Cape region.
Ohhhhh, that one is very well-known in safety circles. That the nuclear warhead landed without going bang was a bit of a surprise (they really were expecting it to vaporise much of the immediate area), and sparked long and hard discussions between Sandia Labs, Los Alamos and the Department of Defence about nuclear safety.
Yep, shallow learning curve.
Me troubleshooting an ol' terminal. It didn't seem necessary to unplug the beast before opening its case. After all, I switched its power switch off. All went fine until I started opening the inner case of the power supply. And my muscles' sudden contractions sent me flying back into the nearby wall. The dizziness lasted for another 10 to 15 minutes.
As fun as that was, a little while later my boss tasked me with troubleshooting a malfunctioning printer. It didn't seem necessary to unplug..... No flying backwards and the dizziness lasted for a mere few seconds.
Learnt: 230 V AC tolerance developed, I guess?
I had a friend whose screwdriver "slipped" while he was repairing a PC and managed to hit him in the eye. That was the account his employer got, at any rate. The facts of the matter may have been that he was practicing the essential tech skill of spinning his screwdriver in the air and catching it like some kind of crosshead throwing knife.
My favourite is when working in underfloor ducts. In order to prevent too much knee an back creaking a pile of recently used tools collects within arms reach. Frequently near a desk in such a way as anyone approaching cannot see them, but fortunately in a place where someone could not step on them. Without approaching you first and then backing away from you so they stand on the one tool that rests on another and all the others rest on the other end of it and so are launched in an arc of steel that would make a good Roman helmet topper. And strangely they all come down (or go up) point first except for the network crimping took which never comes down anywhere!
Come on, we can be honest with each other here. This is more "I ... had a friend", isn't it?
(I already confessed my own most heinous cock-up on The Reg, but it's been a few years so: once, I mistook the 110/230V switch on a PSU for the on-off switch. That's a noise I don't want to be within ten feet of again.)
Heh, that's much like when I was tasked with setting up a PC in a rack for temporary monitoring something or other, and completely forgot to check the switch, since everything else in it was either sensing or 230V by default. I didn't know that PSUs had the blue smoke in them! (But it did its job admirably, and all was fine once swapped.)
This is a for real friend, I keep out of computers since the early days of pentiums when my brother helped me build a tower unit (I think it was a P166, so pretty powerful!) and after it refused to start we had to get actual expert help. That's how we learnt that anti-static bracelets exist for a reason and also that my brother apparently generates way more static than most people.
Reminds me of a conversation I once had with a visiting Injection moulding contractor (we were wrestling with some complicated relay logic ladder)
Al do you know anything about relay ladders? he pauses to reflect for a moment... 'well I know how to get my welded on screwdriver off the cabinet'.
Wise words indeed.
When I was contracting at Bayer Leverkusen, in northern Germany, we were installing experimental switchgear onto live busbars. Only 6 volts per switch, but the whole string was across a 400 Volt DC supply, which was fully floating. Ok as long as no-one else had grounded their particular section of the string, but if they had, then your part could be anywhere from plus 400 Volts to minus 400 Volts with reference to the projecting steel rebar sticking out of the concrete structure. And to add spice, the bars were running with 25000 Amps, at 80°C, and about 15 feet above the concrete floor. We had to use wooden ladders because of the voltage, and they were very flimsy and wobbly too. The magnetic field was immense, so steel socket wrenches and ring spanners tended to either jump up and stick to the bars, or if too heavy, one end would rise and the tool would sit there waving gently back and forth. The german fitters were all issued with bronze tools, but we had brought our own steel ones. The money in my pockets felt strange, the british coins in one pocket were not magnetic (late 1970s) and simply lay in the bottom of the pocket, but the Deutschmarks in the other pocket clumped together and squirmed about like a demented mouse as I moved - most disquieting. I visited that site three times over the two year period we were involved, and I was extremely glad when the contract finished. As a footnote, ten years later, Bayer asked for someone (ie ME) to go and supervise moving the 12 switches from one site to another and recommissioning them, but Manglement said that we were no longer in that business (thank heavens).
PS. While I was there, Bayer assigned two fitters and a foreman to help me. The two fitters were both called Uwe, and one of them spoke with a lisp. The foreman, nice chap, was called Herman Adler (German for eagle), and none of them spoke a word of English, so my schoolboy German was stretched to the limit. I also picked up a tummy bug from eating fast food at a Belgian motorway services on the way there, and was taken to the site infirmary. I was talking with the nurse, who was writing my symptoms on a clipboard, when the doctor came in to examine me, he was surprised to read that I was from England, he said he thought I was from Holland because I spoke German like a Dutchman. (At that time, there was no love lost between the north Germans and the Dutch).
Working as a field engineer in the City of London back in the late 80s, one of our customers was Barings Bank (remember them before Nick Leeson?). I was on their dealing room floor replacing a faulty fusing assembly on an HP LaserJet II. The fusing assembly was a mains powered Halogen bulb which heated a metal roller that fused the toner powder onto the paper. Step one of any repair was to make sure the power lead was out of any equipment before working on anything - especially the fusing assembly of a LaserJet.
I was stood with my back to the office, screwdriver in hand, deep in the printer chassis removing the fusing assembly screws when there was a bright flash, a loud pop and from behind me the sound of a couple of dozen PCs spinning down and the crackle of freshly turned off CRT monitors. Yep - the printer was on, I'd shorted the halogen bulb to ground and popped the earth leakage breaker on the circuit. It took down half the dealing floor and I learned my lesson. I still have the screwdriver which bears a blackened pit on the driver shaft and blobs of molten tempered steel running down it.
Dropping a chrome vanadium spanner across a car battery's terminals also is a way to get a bend where there shouldn't be one. A butterfingered friend of mine did that. There was a bright flash, we both ducked and there was a loud clang from the other side of the garage. When we went over there the spanner was glowing red hot and bent through about 120 degrees. The battery was a write off as well.
Reminds me of the time I was checking the mis-running of my father's Talbot Alpine (yes, one of THEM).
Plugs out to check for a spark, (ie plug removed, affixed back in cap and held loosely against top of engine to visually see sparks): all good.
Fuel pipe off carb to check for flow when engine on and turned over? (pump only ran when engine was turning):
BANG!
Cue scorched eyebrows and very loud swearing.
Note to self.... REPLACE high voltage plug caps and plugs BEFORE checking if fuel will spurt out of feed pipe! Electrical gremlins are one thing.... when mixed with highly volatile liquids in gaseous suspension they are something else!
When changing the exhaust manifold/cat assembly on my old car I had to take the alternator off. Guess who forgot to pull the battery negative and shorted the main alternator feed wire against the 3/8" drive extension?
It very tidily welded the extension, 13mm socket and nut together! Luckily they were persuaded apart and I learned a valuable lesson
Two events spring to mind - one computing related, one not.
The first was where a Newbury Labs terminal engineer used snipe-nosed pliers to slide sleeving over a mains transformer terminal - with the PSU connected to the mains and switched on. He attempted to claim for replacing his glasses which had a sputtered metallic coating... This was declined, but he was let off the cost of replacing his now snub-nosed pliers.
The second was an Irishman who dropped his metal mallet into the uncovered, ceramic fusebox in the roadway in Covent Garden. Each 'bang' relaunched the mallet into the air again, accompanied with what sounded like 'Feck'. There were many iterations, and people scattering in all directions.
As a very junior Operator working shifts on an ICL 2980 at Bath University, I heard a very persistent rumour of an ICL Engineer caught "earning" overtime by judicious use of a 9v battery dragged across the backplane of said mainframe.
Apparently, this used to happen close to the end of his shift, because, of course, once he started the "repair", he had to stay and finish the job. And it happened too often to be just bad luck.
It seems he got caught when, after suspicions were aroused, his manager hid in an empty cabinet and caught him red-handed.
As I said, just a rumour, perhaps an urban myth, I have no confirmed facts, but it was a very popular tale at the time.
One of my users decided to add a BIOS password to the laptop we had given them. By the time it came back, they could no longer remember the password, and there's not really any way around that.
My boss opened up the laptop, and dragged a 9V battery across the motherboard until it wouldn't turn on any more. A quick call to the manufacturer's warranty department later, and we had a new motherboard fitted, sans-BIOS password.
Lots more, especially if there's a testing certificate attached to it that says it's safe past x kV, where x can be anything from 1000 to 10,000.
Most of us, if in need of an insulated screwdriver, grab a roll of good old Scotch 33 black tape and wind it around the shaft of a screwdriver. :D
(I managed to stumble on a sale of insulated drivers, and those are my daily drivers if I have to deal with angry pixies.)
You can buy insulated screwdrivers, but in general a) there's no need and b) it makes the shaft fatter - oo-err, etc - and that makes it less useful in other situations that are about as common as ones where insulation is good.
Come to think of it, I've seen both situations combined - insulated screwdriver required, but space was too tight to get anything in but an uninsulated one. Ended up wrapping my narrowest screwdriver shaft in insulting tape.
My late father, after a bit of an organising session, had an old ice cream tub containing all the various rolls of insulating tape we'd gathered. The first few times I saw it I didn't quite twig what was nagging at the back of my mind - he'd labelled it "Insult Tape" which I found a bit amusing.
Used to work with an old school BT engineer. He used to say that the first thing they did when they got a new pair of cutters or a long bladed screwdriver was to get a short length of scrap cable, pull the wires out then slide the outer sheath over the handles/blade and use a lighter to shrink it in place.
For some reason no one in store ordering seemed to think it would be an idea to buy them ready insulated... pliers, yes... cutters and long bladed screwdrivers, no
I once went to change a broken light fitting in the toilet at home. I told everybody what I was going to do and switched off the mains at the fuse box. Switching on another light switch confirmed the power was off.
I went to undo one of the screws and there was an almighty flash, with soot marks across the ceiling, a slightly shortened screwdriver and a tingling in the arm!
I went to double check the fuse box... it was still off! I then noticed another rusty box in the gloom about 12ft off the ground... yes, it's sole purpose was to serve this one light fitting. (I think it may have dated back to when the outside privy had been converted to an inside toilet by the simple application of a sledgehammer)
Those are actually really quite dangerous, the only thing preventing you getting a shock is a tiny resistor, that could quite easily fail short circuit, plus if you haven't got a particularly good connection to ground yourself (e.g. well insulated shoes or flooring) they may not light up when there's voltage present and make you think something is safe when it isn't).
What you really want is a GS38 approved voltage tester, or failing that a multimeter and the knowledge of how to use one is still better than one of those screwdrivers...
plus if you haven't got a particularly good connection to ground yourself (e.g. well insulated shoes or flooring) they may not light up when there's voltage present and make you think something is safe when it isn't).
In my experience, they still light up while wearing well insulated shoes on a well insulated flooring (rubber soles on a massive wooden floor on wooden beams).
That small resistor is very high resistance. Probably 270k. For it to fail short circuit, it'd need a massive voltage across it with decent current available. Like 10s of KV at least. At that point the neon would go pop (which is quite fun to witness in itself)
A mechanical failure of the plastic body is far more likely.
Also don't forget to allow for a bulb failure in the test kit. It happens.
Knowing how to use a multimeter is a better option regardless.
The resistor is unlikely to fail and cause the short circuit directly. But it's quite possible something inside the plastic barrel can contact something else it shouldn't, and then the end of the tester is live.
Can anyone find any links to actual accident reports? I'm sure I've seen them in the past, but can't find them now.
It's not a replacement for the regular checks, and it's not to be taken as a test that something is safe. If it lights up when, for all other reasons, you believe the circuit to be dead, then you've gained an extra life. I used mine last week as a quick check of which line was live live and which was switched live. Works through rubber gloves.
"And I would think the MTBF of the complex voltage-field-sensing system of some modern meters is far worse than a simple neon and resistor."
Don't forget the MTBF of the test leads.
That's why at $dayjob if we do HV electrical work, we use an approved meter and a 3 step process:
1) "proof" the meter by touching the leads to the meter test box (demonstrate it can detect voltage in the appropriate range).
2) test the equipment to be repaired.
3) proof the meter again.
I know a guy who got 277V from a lighting circuit due to skipping step #3 (cheap meter failed between steps 1&2).
I had something similar in my aunt's house. Went to install a ceiling fan, lights on then flipped the MCBs in the basement one by one until the light went out and my assistant yelled down to me.
Flipped all but the last one back on, confident I had found the circuit (they were all unlabelled of course), went to change the fitting and as I was working the hairs on the back of my hand stood up. Checked the circuit... still live. Some utter muppet had taken the ring out of one MCB and brought it back into a DIFFERENT MCB. It needed BOTH off to kill the lighting ring. I figured that the lights still being on in another part of the kitchen was down to the kitchen having two rings or something weird like that (it was a HUGE house, built to house a 70's pop sensation, so it had had a recording studio in it at one point in time - one reason my aunt bought the house was that it had a soundproofed bunker room and she has a fair number of special needs children so it made an ideal playroom for the noisy ones / isolation room for the autistic ones - the distribution board was about 2 metres long and had three separate metered supplies, presumably for business and tax reasons).
I had to tell my aunt to have the place checked and rewired professionally - that cost a good few grand even back in the day.
Some utter muppet had taken the ring out of one MCB and brought it back into a DIFFERENT MCB
There are rather more of those muppets than we really ought to have. And of course, normally lighting circuits aren't even in a ring anyway (so doubly qualified muppet).
And it's why checking for dead is a key part of any work.
Of course, none of us more technically literate people here would ever skip that step when it's a quick job and the tester isn't right to hand would we ... walks away whistling innocently.
I've come across many lighting rings. I think they must have been in fashion some time in the early to mid 70s. No idea why. My aunt's house as I said is very big, somewhere around 13 bedrooms - it's more children's home than house - so they might have been trying to avoid voltage drops on a long radial. Who knows what goes through the minds of some people?
The ring on the lighting confused the heck out of my stepfather in our old house. He'd put in some wall lights in the bathroom and somehow managed to get the pull switch to either turn the main light on full or all on at half brightness. I had to rewire it all for him and I was only 13 at the time. He's a good builder but electrics leaves him in a daze.
A few years ago I was rewiring the 4' tube light fixtures in my parents' kitchen (bypassing the ballast) so they could accept LED tubes. I turned on the light using the switch next to the kitchen, flipped breakers on/off (all but a few are helpfully unmarked) until I found the one that turned off the lights, then figured I was good to go.
First cut I made I was rewarded with a brief flash of light, loud pop and a small divot in the cutters of my wire stripping tool. I had neglected the fact there was a second switch able to turn on those lights (which no ever uses which is why I didn't think about it) in the dining room on the other side of the kitchen from where the switch I tested was. That was controlled by another breaker, though this time I didn't need to search for it as it had tripped.
I still use that stripping tool, figuring the little divot in the cutters will serve as a reminder. At least the insulated grip saved me from an electric shock to accompany my shock of surprise at the light show and loud pop.
I moved into a new (to me) house and the breaker box was labeled in Egyptian hieroglyphics so I figured I'd update it by turning on all the lights, turning off all the breakers, and flipping them one by one, then going into the house and seeing what lit up.
There was a lot of "HUH, that's INTERESTING" said that day, especially seeing when one room lit up, as well as the closet in an unconnected room. And when one breaker turned on the light and one outlet in the "family room" but all the other outlets were on another breaker.
Granted, this is a house old enough for a NEMA 10-30 for the dryer, which has a neutral but not a ground (which is why it's no longer code)
Granted, this is a house old enough for a NEMA 10-30 for the dryer, which has a neutral but not a ground (which is why it's no longer code)
Heh. I can relate to that. The previous owner of my house was a radio ham, which is why there's a 240V outlet box coming up through the floor under my desk, and why there are grounding rods all round the house in various places for - I presume - his antenna array.
The wiring on this 40-year-old house is interesting. The office and garage are on the same breaker - I keep meaning to split them, especially as it's only 15A, with 14/2 NMB cabling.
The dryer had a NEMA 10-30 outlet; when I remodeled that part of the house last year I inquired about moving/extending the (aluminum cable!) dryer outlet, only to be firmly told "nope, don't even think about it". Which meant I had to rip and replace the aluminum cabling, running new cable from the breaker panel, and a 14-30 4-prong outlet. Oh well. As I was already running a bunch of new cables to the breaker for a tankless water heater it wasn't quite as much of a hassle as it could have been.
One house I've been in had both the basement outlets, and the kitchen, on the same circuit. Found out during a LAN party when we brought in a bunch of computers to the basement, and then someone used the microwave and/or toaster at the same time.
Mine was the only computer still on, thanks to the UPS that I'd been laughed at for bringing...
Ho ho ho. Me too.
Many years sinceupon I began work replacing the central heating system in our old house. First to go was the elderly Baxi room heater fire (yech) with back boiler and its attendant central heating pump. Power was fed from a dedicated switched fused wall plate in the kitchen, which was on the other side of the wall.
I very carefully dropped out the fusebox breaker and returned to the kitchen, armed with a nice pair of wirecutters and applied them to the feed cable (you can see where this is going).
Cue dirty great flash and the sudden need for a change of underwear. Apparently the central heating wasnt on THAT fusebox position. I do still have those cutters somewhere, some 45 years or so later. I have had new underwear sine then, though.
I live in an ancient farmhouse with what they call "farmhouse wiring". It's a three phase setup, with sockets and lights tapped off all over the place at random. And damn near impossible to figure out as a lot of the wiring that fabric coated rubber stuff that crumbles if you look at it askance.
When I'm doing anything more complicated than changing a light bulb, I shutdown the server and press the big switch that trips out everything. Not taking any chances.
Actually, the lead sheathed stuff is blooming marvellous compared to some other types. Unless damaged, air and moisture can't get in and the paper or rubber insulation doesn't degrade.
Of course, that's the case except where the ends are open to the atmosphere and the last inch or so does degrade. A family member spend some time as a volunteer on a ship that was 50 years old - most cable faults merely needed a few inches off the old lead covered rubber insulated cable and it was fine even after half a century at sea.
And a significant proportion of house power supplies are delivered with PILC (paper insulated lead covered) cable - but the end is potted with pitch. Apparently a very quick way (as in a few hours at most) way to get a DNO engineer to your door is to report hot pitch dripping from your old cast iron cut-out (a.k.a. main fuse) which it can do if there's a fault (bad joint) and the connections are getting hot.
Another tale from my apprentice TV engineering days.
I was sent out with an engineer called Mack to help collect a TV that was no longer required by the family renting it. The TV was on one side of the fireplace, the double mains socket on the other. At some point in the past the mains lead had been extended (using a now-illegal taped up connection) by someone in the family. The extension part of the lead was pinned to the skirting board by cable clips before it ran underneath the carpet and around the fireplace to disappear behind a cabinet before reaching the wall socket. Mack asked if they wanted to keep their extension wire - they did - so he asked them to unplug the other end, which they did. He got out his side cutters and chopped through the wire next to where the taped up joint was. There was a loud bang, Mack was suddenly wearing a very surpised expression and the side cutters, whose handles were thankfully insulated, now had a hole where their blades had previously met.
"Oh," said the member of the family who had done the unplugging, "I think it might have been the other plug."
Ah, a TV story.
Boarding school, late 80s, rainy weekend. The big telly in the dining room was not working, showing a white line across the screen. So being a group of boys, the logical thing to do was to take it down (massive CRT, surprised we didn't drop it) and remove the cardboard back cover and look to see if anything had worked loose.
Loads of dust, and scary warnings about CHASSIS LIVE, but this wasn't a problem as it wasn't plugged in.
The wiring arrangement was a bit odd, so to get a better look at where all the coils on the back of the tube went, I thought it would be simpler to pull off that big thick wire attached to the tube itself, and put it aside.
Apparently I backflipped clean over one set of tables and crashed into, and through, the ones behind - I didn't weigh much as a 12 year old.
I don't remember. I woke up in the sanatorium aching like you wouldn't believe, and matron so white and huge eyed it was like she'd been out fighting ghosts.
It hurt for weeks.
Even now, the insides of cathode ray devices are a "don't touch" zone for me. Thankfully my current devices are flat screen LCDs. Less fixable, but not punting tens of thousands of volts around.
We really need a "shit my pants" icon for these sorts of stories.
It was in the second half of the last century, but only just, and my father built our first TV from a kit; I think it was a 'Cossor'. This was some feat as he was colour-blind and had to get someone else to 'read' the resistors.
On the first use ('commissioning'), i.e. switching on power, all personnel and the dog were commanded to perch underneath the table holding the open-chassis assembly. Cathode ray tubes were known to have undesirable failure modes. After the immediate danger was considered to have passed, I still recall the eerie glow from valves and that hot-dusty aroma as things reached temperature. We had a working TV and we soon got used to tuning to the other channel with a long screwdriver. I don't know what happened to the red-handled pock-marked screwdriver in the long run.
For some reason, I have never lost a fear of cathode tubes.
I almost had this happen when I was 8 years old back in the early 2000s, but got away with it. Popped the back off a CRT and poked around a little. "It can't shock me, It's not plugged in...". It had been recently though! Very fortunately, I didn't get a shock, I was more interested in the area around the faulty aerial socket so didn't go too near the flyback thankfully. I may then have plugged it in for the hell of it which might actually have saved me, I knew not to touch anything with it like that.
I still shudder when I think about it. At least it wasn't a Microwave, the caps in those are even worse.
Later I had an 80s CRT which had the same fault yours had. White line. Give it a whack and you'd get the picture back for a while, but it kept getting worse.
A long time ago, in a Big Broadcasting Company...
It was nearing the end of my shift as a videotape engineer, my jobs were all completed and I was awaiting the nod to depart from the shift supervisor. Suddenly, he bursts into the control room, with a 90 minute, 2" tape in his hands and a wild look in his eyes. Naturally, his eyes landed on me!
"Presentation forgot to book a machine for the trailers! They are due on the air in 5 minutes! Get this tape on VT 13, and ready for transmission as fast as you can!", thrusting the tape into my hands.
Now back in those days, all the videotape machines were 2" quadruplex behemoths that normally required around 30 minutes to prepare for transmission! No time for proper checks or alignment, just throw the tape on, put everything in "auto" and put your faith in the system!
Now VT 13 was unique in the department in that, at the time, it was the sole machine made by RCA. All the rest were Ampex machines. The RCA had a number of quirks, one of which was quite weak spooling motors. It took quite a bit longer to spool through the huge 2" tapes than the equivalent Ampex machine, and naturally the wanted trailer was at the tail end of the tape!
Another quirk was that the transport controls for the machine were at the end of this giant monstrosity, conveniently out of reach of the internal 'phone! Because of the panic, the control room was still plugging up all the cables necessary to connect VT13 to the presentation studio, so there I was trying to ring the studio on the phone, whilst trying to find the right spot on the tape, and not having arms long enough to do both simultaneously!
I managed to get the machine on cue with seconds to spare, and hit the "play" button on time. Fortunately - or maybe not - the colour monitor in the booth had been left looking at the "off-air" signal rather than the output from the machine, and it was soon obvious that something was severely wrong! All the skin tones were bright green, instead of the usual fleshy pink! That was the point at which I realised that the machine was on the wrong pulse train (used to synchronise the machines to the various studios). After some terse exchanges with the studio, I decided the only option was to change pulses live, on-air!
I hit the pulse selector to the correct one for the studio. The VT gave an almighty hiccup and lurch before stabilising again. Now all the skin tones were cyan instead of pink...!
There was nothing else to do but to let it run out the rest of the trailer - mercifully only a minute or so - and let the presentation guys apologise to viewers afterwards.
Naturally, there was an inquest immediately afterwards, and I was asked why I hadn't noticed the machine was on the wrong pulse train. Well, the indicator bulb had blown on the pulse selector, so there was nothing to draw my attention to it. Whether I would have noticed in the panic, I don't know, but at least I was not held responsible for the fiasco. However, it was deemed that this fault had to be rectified NOW, before it could catch anyone else out!
Cue the entry of a very disgruntled maintenance engineer, who, like me, had been about to depart for home. Changing the bulb required a major dismantling of a very fiddly row of selector buttons, which he achieved in record time!
Having replaced the bulb, he started punching the buttons in sequence to make sure all the others worked. At this precise moment, the doors to both transmission suites (network 1 and network 2) flew open and the senior engineers ran out shouting "Who is f*!&ing about with the pulses?!?"!
Yes, both networks were crashing about and reframing on air!
It turned out that this pulse selector in VT 13 had yet another issue, in that when switching pulse trains, it momentarily shorted two pulse trains together! Not enough to cause any electrical damage, but certainly enough to seriously disturb the transmission machines!
The tale of woe in the fault log the next day made for very interesting reading....!
--
Pete
Ah, the days of quality in broadcast and massive enquiries as to what went wrong, even if it were for just a few seconds.
Nowadays, a nostril webcam via zoom, with aperture and colour correction on auto and constantly self-adjusting is considered broadcast quality and suitable for prime time.
Maybe it's just me, but I recall in years gone by anything that was not perfect had the "amateur footage" caption emblazoned across the bottom of the image. And that was when we were watching at home on soft/fuzzy CRT's and HD and UHD were unheard of.
.... I fatally short-circuited the Dell T20 mainboard in my home/home office server while trying to reseat a card that I unjustifiably assumed was causing problems. Since then I do remember well what I knew in theory but had always ignored—that a "poweroff" command doesn't really turn anything off, without me actually switching off the PSU...
Of course that happened on Friday afternoon.
I was lucky enough, though, to find someone selling an identical box within a two-hour drive on that same weekend, so the downtime remained bearable. And a few weeks later I even found a ridiculously cheap replacement mainboard to revive the old box, which then became my new desktop PC.
While I (so far) never repeated that specific mistake, every time I get off cheaply I wonder whether that might be detrimental to the learning effect.
I do recall seeing a scrap PC motherboard at our local enthusiasts shop, many years ago. There was a huge scorch mark cross the bottom, radiating in a manner reminiscent ot Tyco crater on the moon, away from the spot where had lain the discarded screw, down onto which said motherboard had been tightened. And then switched on....
Oddly I have been a little cautious over dropped screws since seeing that.
Ah - one of my colleagues discovered this many years ago. We'd not long gone from AT to ATX boards, and he'd got a new Dell OptiPlex (E or G)1. To improve the gaming ability, he wanted to fit a graphics card, and those Dell's had a riser card that you plugged your cards into, which was in a frame that lifted out. PC was powered down, so he removed the riser, fitted the new graphics card, and then put the riser back in - which somehow managed to short somehow and power the PC on. Luckily for him, the only damage turned out to be the new graphics card. We were all a bit more careful about unplugging after that!
While doing my engineering degree we tried to learn Fortran IV on the uni ICL 1902 (IIRC). I lived 2 miles from it and in trying to do my thesis which involved programming I'd sometime walk there and back 4 times in a day, arriving to be greeted by a large sign on the wall lit up with 'HARDWARE FAULT'. After a term of this I was allowed into a small office where there was a PDP11 and Fortran and rather than writing a bit and then coming back later to fix the errors I could do it within a minute. No physical explosions but my brain exploded after that!
One Christmas Eve a few years back at the in-laws ... me and the Mrs turned up to discover the inside of their house was something like the Sahara Desert at high-noon. (This was winter in the UK.)
We queried why it was so hot - and were told "because the hot water isn't working". I should have left it there - but the engineer in me had to ask. Evidently the hot water tank had become a cold water tank, so they ignorantly figured "If we turn the heating up to 11, maybe the hot water will work". (I guess, if given enough time, the hot water tank would eventually find equilibrium with the now 30+ degree hot house...)
After some grub, I offered to go take a look. I quickly determined the cause was likely an electronic valve wasn't actuating. (Turning it manually with the physical override made hopeful gurgling noises inside the tank.) Figuring it might just be the 7-day controller thing was unhappy, I popped it off and blew the dust off the back.
It was when I came to reinstall the 7-day controller thingumy that it all went wrong. A similar bright flash occurred, and the ominous sound of lots of valves clacking shut and the gas boiler starting to wind down...
I instantly realised what had happened. The designer of the 7-day thingumyjig hadn't engineered the removable front (which contained the brains of the operation) such that it cannot possibly misalign the 240v rear connector with the wall-mounted (and electrically live) back half.
Evidently a pair of Normally-Closed/common pins were "out by one" when I reinstalled the front, causing an open short across a +ve and -ve contact on the wall.
I checked the back of the unit, confirmed it was now "very black", reinstalled it correctly and considered how to proceed.
A quick calculation in my head suggested that if the well-insulated house lost even as much as 3 degrees Celsius an hour, it would be well past bed time before anyone noticed it was getting chilly. So I calmly walked downstairs, explained "It looks like one of the electronically controlled valves isn't working ... I'd keep an eye out for any blown fuses etc", conversed for another hour and went on home.
Next day, Mrs was rang to be told "now the entire heating system isn't working, but a plumber is on the way" - who replaced the blown fuse (oops) and diagnosed a faulty electronically controlled valve. Replaced the valve, and everything was golden.
Thank you. That's one of the best ones I have heard for a long time.
I also learnt long ago not to offer to fix others peoples domestic problems while visiting. Nowadays it's a bit like "while you are here could you just have a look at my printer...."
>>. I should have left it there - but the engineer in me had to ask
Ah, the ancient cult of Ikabai-Sital. More dangerous than Sciencilogy!
Someone told me that they were involved with "moving" some disks - 2 ft to the left - while they were still spinning - these were the huge spinning mainframe disks - in cupboards 6ft high and 3 ft square.
They put there backs into it and pushed with their legs and moved the disk. When they came to stand up, someone's belt caught in the Emergency Power Off pull lever - and there was a sudden silence as the whole bank of disks were EPOd.
These days he would have been gone since his manager would have never done anything wrong in his life.
That same manager would never have done anything right in his life either.
As for never having done anything wrong, that is a matter of opinion and knowledge (in case of that manager lack of the latter).
I distinctly remember working for a "fruity" firm prior to any of that infernal i business.
We were testing a rather large monitor (a BARCO CRT monolith) prior to installation for an offsite presentation.
Ever diligent (this was likely to earn us a tidy crust) we checked the guns and alignment.
My Colleague at the time commented on the convergence and decided to pop the back panel off to make some adjustments.
I dutifully strolled off to the lab for a pot adjuster only to return to a rather loud crack and a sheepish hand waving a kitchen knife in the air.
Yep, you guessed it. Pop goes the weasel, er, BARCO.
One fried monitor, no demo and one sheepish Colleague having to fess up.
Reminds me of when I was on hardware installation detail in a QC lab in a company involved in making Lubricant (For cars) installing a PC when a YTS engineer went was fixing a piece of lab ewuipment and somehow inserted the screwdriver into the machine and it went bang. His supervisor came over and asked the guy. "Show me on this machine exactly what you did". Bearing in mind that the 2 machines in the lab where the only two in existance somebody was best not happy.
Yes, but they are useless for many of the tasks people have mentioned for the simple reason that, as another commentard has mentioned, they are weak and/or brittle. And unless you both own, and carry all the time, both sets - sooner or later you'll have a job for an insulated variety which you don't have (hence the oopsies described), or a job (more common) for a proper metal type and your expensive non-metal version you try to use becomes a useless ornament.
A previous boss of mine back in the late 80s was working on a powered-up machine. He bent over the machine, and his tie clip came loose from his dress shirt. Tie swung forward, clip touched something, and one <KZZZRT> later the root issue of his initial call-out was moot.
More than once, in my younger years, I did not unplug the power to swap out a PCI/PCI-X Card or to change RAM. Quite a few times the machine would turn on when swapping things. It required a fried RAM module, with a tiny bit of magic smoke, to make me really learn. Since the damage was not visible we could RMA the RAM as DOA. Luckily the Mainboard survived.
A friend of mine had problems with her computer. So, she reached behind it hoping to find a reset switch. She found a switch without any problem and managed to switch the PSU from the normal 220 volts to 110 volts. I managed to rescue some of the data off the hard drive - the motherboard, CPU and RAM were all fried.
"For about 10 hours in 1980, the United States faced a nuclear threat of its own making after an airman performing maintenance on a Titan II missile dropped a 9-pound socket 70 feet, ripping a hole in a fuel tank and leading to an explosion that propelled a 9-megaton warhead out of the ground."
https://www.seattletimes.com/entertainment/pbs-explores-1980-nuclear-threat-triggered-by-socket-wrench/
Years ago I was in the computer room of the company where I worked and saw the operator drop her steel bodied pen through a hole in the top of the mainframe situated right beside the operators keyboard.
It was an accident waiting to happen really.
In a truly magnificent display of reflexes she whacked the power off before the pen hit anything inside the CPU.
Amazingly there was no damage and no impact other than the couple of hours it took to bring the mainframe up again.
The next time that I was in the computer room there was a spanking new grommet blocking the hole.
My first job (many many years ago) was in the Post Office telephone section. A young tech. was working on a fault in a relay rack. Turned off the 50V DC and went off to lunch, at which time one of his helpful colleagues wired a packet of flash cubes (remember them?) across the 50V supply. Tech came back from lunch, sat down on his bench, and turned on the 50V. Lit up the whole floor, and took years off his life. Can't do that stuff nowadays :(
Ahh. The old ICL 2900 series.
My work experience whilst still at school was with a company with a 2976 - someone caught the reboot switch with their suit jacket sleeve on the OP10 operating station (the reboot switch stuck out on a front panel and was a simple toggle switch). The 2976 was down for 2 days just because of that! ICL engineers all stood around with scopes and all sorts. I later got a job with the same company, and it was one of the reasons I never wore a suit jacket in the machine room and always had my shirt sleeves rolled up.
OLTS used to sound like R2D2 when it came to the end of the test sequence. We used to run that once a month with the ICL engineers. It got to the point where I could tell if the damned thing was crashing by just listening to it (remember the HOOT button on the panel? Turn it up and listen to the machine do it's thing).
Now I am really feeling nostalgic. Those were the days you really had to know your systems in order to get the work done. 640Mb disk drives the size of washing machines.
Biggest bang I ever saw though was down to one of the aircon units splitting a pipe. Lifting a floor tile changed the pressure under the floor and took the water over the top of a power outlet powering the printers....
So about 13 years ago while working at a university in Australia we were putting in big blade chassis' that due to various other 'design' choices made throughout the life of the data centre, we had to upgrade circuits and power rails from single phase (1P / 32amp) to three phase (3P / 95amp) for the rack densite needed for our kit (lots of kit). Not a problem, just a bit time consuming to find three 32amp circuits next to each other and re-arrange connections.
We went off, did our research and ordered a stock of APC 3P power rails. Upon receving them we realise that their plugs are not Clipsal and thus not compatible with our brand new 3P outlets. So we call the in-house electrician to come around and change the plugs on the power rails.
Electrician removes the shipped plug and installs the compatible Clipsal plug on the first rail. We (myself and the in-house electrician) take the rail into the data centre, install it in place, I attach a 10amp device (KVM switch) to the power rail and turn it on (the on switch was 2cm from the inlet on the device and heavy duty plastic). BANG and silence. The data centre is suddenly silent (event the air-con units went off) and it is 10:30 on weekday, which is extremely unnerving. Then everything starts powering on. All 50 racks.
Rush out of the data centre to the foyer, get on the phone and call the other 11 engineers (our office was offsite at this stage) and the line mgr to get to the data centre asap as we've just had a massive failure. Through the emmotional shock I managed to recover the 120 servers I have sole responsibility for and the applications on them.
The university is pissed as we were central IT and hosted all the crown jewels, but 5 hours later we're back online and we'd only lost a few hard drives, no actual servers.
Third party electrican is brought in to do OHS inspection and to remove the offending power rail. I was told that the in-house electrician had made an catastrophic error (a near miss fatality (me)) - the APC power rail used european wire colour coding internally which doesn't match Australia. This had resulted in 32amp positive being wired to the neutral, which was on the 10amp and 32amp outlets.
I remove the KVM a few weeks later and opened its cover. The entire PCB was black from the cooked electronics - 32amp into 10amp rated device. I was extremely fortunate that I didn't receive a shock.
Thankfully I had two awesome managers and the blame for the outage was put onto the in-house electricians (who worked for facilities management). From that day onwards, only the trusted third party electricians were allowed into the data centre.
"blame for the outage was put onto the in-house electricians (who worked for facilities management). From that day onwards, only the trusted third party electricians were allowed into the data centre."
Yes. Bear in mind that no half-decent electrician is going to take the in-house job. It's for trainees and those so inept no-one will work with them. You can let them change plugs on non-critical 230v stuff, but that's about the limit.
At my second most recent former employer, we had a medium sized data center at a university. At one time it had held mainframes, now it was racks and one IBM Z9. Lots of racks. Redundant power was from batteries in the basement and a generator outside. One of my coworkers from another department was checking the battery status lights when a single piece of paper fell off of his clipboard and got perfectly sucked into an air vent. It didn't stop immediately, but there was evidently no safe way to get to the paper without shutting everything down. The entire data center had to be shut down for an hour just to remove that one piece of paper.
...can really be a problem. Imagine the scene: me, about 17 years old, just completed building a short wave receiver, mainly from spare parts from old TVs. 1960s, so that meant valves ("tubes") and a 250 v DC line at the back of the aluminium chassis, also recycled from a TV. Needed to tune the little ferrite core in a coil (the only component I'd bought new). Picked up the screwdriver I'd made in metalwork at school: neatly milled brass handle, long thin steel business end perfect for the job. Unless you missed the target and hit the DC line. Fizz. Pop. Twitch! Jump!! I was definitely out for the count for a minute. Fortunately I twitched enough to break the contact. More fortunately, there was no RCD in those days so the power didn't go out and my parents never knew. Switched to transistors ASAP.
About 20 years ago now, I was installing some new data points in the office area of a client's factory. Cheap job, nothing hard, bit of cat5 from a panel at the other side of the office across a false ceiling and down some mini trunking to a new socket at skirting height. It was late in the day, all the staff had gone home bar the client's Financial Director, whom we were doing the work for. I was working on the last socket, crouched on the floor, simultaneously texting my girlfriend (now wife) as to why I was going to be late home, when there was a bang, a sharp pain on the top of my head, and some bright flashing lights behind my eyes.
It transpired that the FD, keen to help, or maybe just wanting to get home quicker, had hopped on a desk and tried to reposition the ceiling tiles that I'd moved to string the cable. Unfortunately, these were old, large and heavy, possibly asbestos or some sort of fireproof cement things, rather than the more usual soft sort, and he promptly dropped one. Right onto my head.
He ended up taking me to the local hospital where they applied stitches and checked me out for concussion. My boss turned up to collect me and take me home; I can't now remember who collected the van. My gf was worried sick because I'd suddenly stopped replying to her texts, and didn't arrive home for hours..
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
