Crash, bang, wallop: What a power-down. But what hit the kill switch? Well done, Reg readers, you’ve once again triumphed in the battle with the 9-5 and made it to Friday. Celebrate with a dose of On Call, our weekly instalment of tech conundrums and calamities caused by other people. This week, we meet “Clark”, who wrote in to tell us about a very particular kind of fault, which is probably …
Probably similar timeframe - VAX 11/780s in late 1970s. Kill switch next to a phone on the wall... Engineer makes call on phone, leans against wall,.... lights out!
Same installation. Telecomms 'electricians' removing a cabinet. Not sure if power is off. Large screwdriver between active and earth shows, momentarily, that power WAS on, lights out once again.
Some air-conditioning fun from same location if On-Call is interested in that sort of thing.
Sysadmin putting some old tapes back on a high shelf, which needed a stepladder to reach. He overstretched, the ladder wobbled, and he instinctively reached out to keep his balance. He said that even as he felt the main breaker handle start to move under his hand he realised that it wasn't the best thing to have grabbed. Too late...
Problem is, EPOs are pretty much required to be somewhere quickly accessible near the space they protect, and obviously you want access to it protected to the same access level as the equipment. Then door releases obviously should be located near the door, and in places where people turn the lights off, that probably should be there. However, putting these switches all together at least shows the user there is a choice to make - the alternative it an exit, and an action to perform (release door/turn off lights) - if the user glances at one lonely switch, they're going to press it, so if their gaze falls on the EPO rather than the one placed not quite close enough, they're just going to hit it, with autopilot engaged.
Molly guards, and putting EPOs out of direct sight line are options in some locations, but other times, local or national regulations prohibit them
This placement makes a lot of sense when you are leaving and turning the light on. When you're coming in and groping for the switch to turn the light on, line of sight is not that relevant. Motion sensors for lights people... At least this story has made me feel better about taking out all the customer support gateways....
"Motion sensors for lights people..."
As long as the timing's generous. Place I used to work there was barely time to read a Reg article or play a level of Plants vs Zombies before you'd have to do the penguin walk to turn the lights back on to ... conclude your business.
This post has been deleted by its author
Working with a v tall (almost 7 foot) colleague in the server room and he leant back from the rack as I took a look at the KMS screen only for his head to whack the kill switch and turn almost everything off. upside was that a couple of v old systems didn't go down as they were plugged into non protected sockets so we got those properly plugged in as soon as we bought everything else back up.
Big red kill button, box of paper on floor, operator bends down* to pick up paper and nuts kill switch.
*If he'd had manual handling training of course this wouldn't have happened!
I have also had a Unix server where the power button and CD Drive eject were very close together, that one got a card flap over the power button.
Remember those old AT towers with the Reset and Turbo buttons side by side? AARGH!
I also remember that the old AT towers had a power switch on the front that was at ${mains}V across the terminals, as this was the good old days before ATX and soft-power on.
The first time I ever had 240V was when I was digging under the IDE cables and the hand that wasn't holding the well-earthed case came upon the bare terminals of the switch. Certainly took me by surprise...
No problem. You just hold the Reset button down for as long as the PC takes to complete its job...
A colleague set the Turbo button in on his 486-66 for months without noticing. As I understand it, the Turbo button set the ISA bus clock speed into a compatibility mode for legacy ISA cards. My colleague's PC had a VL Bus graphics card and a 3Com ISA Ethernet adapter, neither of which seemed to be affected when running Windows 3.1. He noticed something after installing Linux and pressing the wrong button at some point.
Ah yes, the reset button (or power button in the pre-ACPI days) which only engages when you take your finger off the button.
I remember taping someone's finger to a switch after they pressed it at the wrong time, and as long as the finger was taped in place, the computer stayed up.
I got my work computer, G3 Power Mac, back after it went dead. Got called by the tech guy you had the side down (wonderfully accessible machines) and was looking at the board.
'The only thing will be to hit the reset switch' he says (it was on the board) obviously not wishing to take responsibility for doing it. At which point my finger snaked out and pressed it, machine whirrs into life and I'm a happy scientist.
It was still working well when I left that lab. I did personal surgery to that box, swapping a drive bay with another box to enable the installation of an internal Zip drive. Very easy to do. Compared to iMac surgery the G3/4 towers were a dream to work on.
Remember the Minuteman rackmount UPSes that had their sexy round power button on the front of the faceplate, sticking out proud(ly) in the center? A button that only needed the merest brush to immediately and ungracefully shutdown the UPS. Only took twice before I decided to act. Went to the local hardware store, bought a cheap can of bright yellow spray paint. Threw the spray can away, duct-taped the yellow plastic cap over the power button. Problem solved.
"I have also had a Unix server where the power button and CD Drive eject were very close together, that one got a card flap over the power button."
Yup, had to make a flap for the UPS on the home PC. It was a low flat box with the power switch on top and it was almost on the floor. A real problem with cats who walk anywhere they want to.
But mice are a problem too. I had a mouse fall off the keyboard tray and it hit the power strip power button. For that one it worked to rout the power cords over the switch and velcro them together.
Bit off-topic, but what the heck, it's Sunday...and we were talking about cats dealing with mice.
Many MANY moons ago when I was about six or seven, we had a toy cupboard with rather a lot of old paper in the bottom. Judging by the rustling, we were pretty sure that there was a mouse in there. So we got the cat up, who looked very interested at the rustling. We then poked around with a broomstick, and after a while a mouse shot out of the toy cupboard and across the room. On spotting the mouse, the cat howled piteously, jumped into the air and shot off down the stairs and out of the house. Didn't see him for hours.
No idea what happened to the mouse.
When I was a kid, we moved house. The previous occupant used to keep parrots, in a purpose-built aviary at the bottom of the garden. Our cat, Mickey (long story), was extremely interested in the aviary, running back and forth across the front with his nose pressed against the wire mesh. We opened the door to let him in, and didn't see him again for three days. Every morning there was a row of dead mice lined up outside the back door, heads towards the door, eleven the first morning, seven the second morning, and five the third. Mickey then strutted back into the house with a smug air about him, but didn't bother with the food we put down, so how many of his victims had not made it to the line-ups we never found out.
Hah, my current desktop system (Thermaltake case), the power button is next to and shaped exactly like a USB port. I've stopped using the front panel USB ports.
See also: laptops with a headphone jack directly next to the USB port, where if you're fumbling around the side and insert your headphone plug into the UB and short out the 5V supply, it kills the power supply and hard resets the system.
Brand new Dell Precision Xeon optioned notebook
Headphone jack on right side of machine. USB A plugs 4mm from it.
Honking huge vent above those.
Plugs are in the way of external mouse for right handed people.
Left side has a smartcard reader and an SD card slot, then dual USB-C.
No interference.
This was designed by a southpaw.
I have not yet triggered such an emergency switch. However at my new workplace, all the doors have magnetic locks. They have two buttons, the normal door release button, and the "emergency door release" button, which are always right next to each other. They are also both green (except the emergency button has "EMERGENCY" written in small green text above it).
Both also do the same thing (release the door). but the emergency one will also sound the alarm (forcing a building evacuation) and call the fire and police. Multiple times already I found myself almost pressing the emergency button by accident, instead of the normal button when wanting to release a door.
I really wonder who thought it would be a good idea to make both buttons the same colour and put them side by side like that. I have seen others in the office make the same mistake as well, but realise at the last minute before pressing. One of these days, while someone is in a big rush, I suspect the inevitable will happen.
Building codes changed a bit over the year, magnetic lock doors here now can't work like that anymore.
You can get the usual ID mechanism (card reader or biometric or whatever on the side of the door) for normal access, but emergency access has to be the "normal door way" i.e. turn the handle / press the release bar in the middle of the door, except secure doors are allowed to have a timer of up to (15 seconds I think) with an audio warning before the emergency release happens / alarms are triggered.
The scene is the late 70s - mainframes, fanfold paper and lengthy batch processing. Senior analyst called in to fix a problem with a run over the weekend. His wife is out so he has to bring the kids with him. It's raining hard, and the kids end up playing football in the computer room, until a perfect kick makes contact with the big red button ...
I was just putting the finishing touches on a small cluster of vaxen at SLAC one fine Friday afternoon. The annual Big Game between Stanford & Berkeley was to be the following day. A couple of grad students started passing a football (American version) between themselves. In the glass room (that wasn't glass). Just as I was threatening mayhem if they didn't knock it off, the ball hit the Big Red Button, Needless to say, a bunch of pissed off people couldn't attend the game the following day. The grad student's computer privileges were suspended for the rest of the academic year. Personally, I'd have hung them by the thumbs in the Quad as a warning ...
(As an alum in good standing of both schools, all I can add to the above is "Go Bears!")
Yes, I do keep that story on hand. It's called a memory. I might post it again, eventually. Not everybody reading here today was here last time I posted it; It's new to them. And that would include yourself. Nice to see that you find my drivel important enough to read my backlog, though.
Do you have anything on-topic to contribute, or are you just here to bitch?
Used to work for a large investment bank.
Went from my floor to the dealer floor to help with an issue, when I got there, there were people standing up looking round like meerkats.
Went back to my floor, people still standing up
Turns out, the power was out in the computer room, 1000 cabinets worth of stuff.
There was an electrician in the computer room, who was moving an electrical point, he had followed the cabling under the floor in direct line to a pair of power sockets on a pillar.
Except......
That cable, went round the pillar to the other side and another cable came round the pillar to the socket. On the other side was an emergency power off button.
Back in history when the DC was built, all the EPO's were wired up with orange cable labelled EPO every 6 inches, the installers ran out of EPO cable so rather than getting more, they used what was there which was reels of power cable in grey. The poor electrician, couldn't get the final tile up as something was sitting on it so assumed that he had the correct cable.
Outcome, a day of carnage for an investment bank :)
The electrician was escorted from the premises never to return.
"For something that wasn't his fault."
Only it was his fault - his actions directly caused the power outage and I am assuming it was made very clear to him that powering down the building was bad.
The correct course of action would have been to clearly identify what he was working on and if he was unable to do that, advise who he was working for that there was an issue and come up with a way that the risks involved could be reduced (i.e. carry out the work out of hours or determine how to investigate further or even discover the incorrect cabling and plan to replace it all). i,e, perform the task in a professional manner rather than bodge it.
Instead, the electrician committed the same mistake as his predecessors who decided not to do things properly. The difference is they got away with it. And I suspect the investment bank likely had a history of working to JFDI with the unwritten rule being don't get caught.
If you substitute the electrician for an IT contractor, does your view of the actions/consequences change?
This post has been deleted by its author
> We all know Big Red Buttons are irresistible!
> At CERN there's even a spoof one for visitors to press
Ah, yes; I was manning a mock-up of a portion of the LHC at a Norwich science fair a couple of years ago. There was a faux Emergency Switch near one of the exits. Several times I had to encourage young kids to press it while they dithered; nearly as many didn't need urging.
For my sins, I had two air-rifle pellets in my window on the way back to Cambridge; two other windows and the driver's side window were also hit. There must be SFA to do for young peeps in the Fens on a Sunday night...
@NBCanuk: I have seen one that was effective. A little card with "Do not press this button. Severe environmental degradation will result" That card was hanging from cruise missile launch console. The console was fully secured and powered down. No-one in the visitors group was even tempted to disobey. Anon for safety of tour guide.
When Cormorant came to be tested by the Army there was 2 emergency stop buttons. 1 by the PDUs where the operator would sit and 1 by the door. The one by the door was perfectly located so that when you reached the top of the steps and steadied yourself to go through the door you could quite easily hit the button. Most didn't, but some did. A modification was soon built from a Pringles tube and black nasty.
Not my story, but told to me years ago...
RAF station had a power outage, while Top Top Top Brass was visiting & the UPS\Battery backup failed to kick in.
TTTB demanded a full investigation & wanted the report on his desk ASAP, from someone who wasn't stationed at the base.
That report highlighted two major things:
It was full of crumbs & other flaky pastry detritus, where it had been used to heat then keep pies & the like hot & ready to eat, along with tea & coffee spills inside, on the sides & at the rear (with mug rings on the top of the unit).
That the staff sergeant had signed off on, but not checked & topped up the batteries electrolyte levels, in fact It was almost empty and hence no charge, unlike the sergeant who promptly found himself on one & demoted to corporal.
Same thing at a large RAF base in Cyprus in the 1960s. Major storm knocked out power for a couple of days. The telephone exchange had an enormous battery which should have provided power for at least 36 hours but died after about 30 minutes. It transpired that a sergeant in the Royal Corps of Signals (who looked after all RAF telephones overseas) was responsible for ensuring that the electrolyte was topped up and regular voltage tests were carried out; he hadn't checked for ages and there was hardly any electrolyte in any of the cells.
After that same storm and flooding there was a signaller (private) in the Royal Corps of Signals who was using a canvas bucket to bail out a cable pit that was full of water. An officer in the Royal Corps of Signals drove up in his LandRover and asked the signaller how he was getting on. When the lad replied "Not very well, sir" he was told to put his back into it and bail more quickly. Shortly after that the SNCO from the Royal Corps of Signals arranged for an RAF fire engine to pump out the cable pit - a job that took about 4 hours because all the cable ducts were acting as drains and feeding into the cable pit.
Data centre for a telco. Needs No-break installed. Generator shed gets built, stonking big diesel genny gets carted in and the guys from Perkins (or whoever manufactured that beast) gets the thing chugging. They listen, probe, do a few adjustments, listen some more, consider it Fine, let it run some more and figure it's time to push the Engine Stop button, sign off on the job and hand it over to the sparkies to let them do their bit. Alas, the Engine Stop button didn't., as it apparently wasn't yet hooked up. "OK, maybe it's that other button".
That didnt't stop the engine either, but it did stop the computer room.
During WW2, when the first prototype Wright Tornado engine was started up, someone had arranged the throttle linkage incorrectly so it was unintentionally set to full throttle. The unloaded engine accelerated abruptly and, in a few seconds reached its ultimate limits and destroyed itself.
On a more hardware side, some of our older kit (semiconductor manufacturing machines) shipped with their bright red EMO mushroom buttons entirely exposed, as it was mandatory for CE marking (being able to whack them from any angle).
Of course this lead to the inevitable misoperation, from being smacked by various operator body parts (the front panel buttons were at roughly arse level for example, and the ones at the rear usually around chest height).
Needless to say the first thing most of our customers did after the tools were installed (sometimes before they were even powered on) was glue perspex collars around all the buttons to try and prevent this.
Forget lost program runs, try losing 150 wafers worth of chips-in-production. Not cheap...
We wouldn't be allowed in the cleanroom with all that make-up. And they (sadly) don't make black bunny suits..
Of course the fact that the EMO buttons didn't actually kill the power to everything was only recently realised, and indeed on the odd occasion didn't actually do anything at all. But that's a whole other story.
Ahhh yes, the great EMO button that doesn't actually do anything.
Reminds me of the time I was working late in the cleanroom, far away from anyone else, getting sprayed in the face by 40 liters a second of water, desperately trying to keep the flow contained, just barely able to press the nearby EMO on the water conditioning cabinet, only to realise the flow doesn't stop and the pump is still running full blast...
Then you're left with 2 options. Release the broken L-joint, duck left, sprint to the nearest console and enter the required commands for a remote shutdown or option 2, release the broken L-joint, duck right, sprint to the main shutoffs in the control cabinets at the end of the lane and just close the valves, letting the spike in water pressure force the pumps into an auto shutdown.
I'm glad I'm not the one that had to report a 40 000 gallon water spill at a later time into the sub-fab basement at a customers fab though... That must have been a tough talk.
A long, long time ago, I can still remember.. when the "cool" kids used to set off the fire alarm at school.
This, of course, didn't only interrupt any teaching going on, but also triggered the fire department. Not really all fun anymore, and quite costly, or so they claimed.
So the brilliant solution? Replace the glass cover for the alert button with a metal one.
A guy I occasionally chatted in ye olden days of FidoNet used to be a sysadmin at "an intelligence facility" where the mainframes had been fitted LITERAL kill switches - as in, thermite charges attached to various critical components. Don't know how those charges were supposed to be triggered but hopefully (from the point of view of people working there) not by an unprotected and unlabelled button on the wall.
A/C because it's an ex-colleague of an old friend - not my story to tell, really!
Anyway, this chap was sent around the British Consulates and Embassies to perform upgrade work in their server rooms. On the back of the (very strong indeed) server room doors were metal hammers. If the building is invaded by unfriendly forces, the idea was to haul the hot-plug servers out and hammer the hell out of them, ideally before they stopped spinning.
I understand more modern devices take a SCSI command and detonate a vial of solvent within the drive to destroy them. Maybe others know more.
I won't type it out again, but I am sure that everyone has worked with a Father Dougal at some point in their career. There's something so seductive about a Big Red Button.
Was in a client's comms room with my boss, sometime during the 80s. Suddenly the power goes dead and we all turn round and there's my boss at the kill switch. His response was along the lines of "What? I just wanted to know what it did." Needless to say, he was barred from the comms room and that ban still exists today.
There was the story of a shift operator showing some newbies some basic op console commands for a Unisys mainframe where I worked. All harmless until he decided to show them the $! command which did *not* need you to then hit Enter to action it to reboot the computer (think CRL+ALT+DEL)!
Dollar, Bang by name and Dollar Bang by nature! :)
"The emergency button was enclosed in a new hinged box of clear plastic to avoid accidental shutdowns "
Having experienced a similar accidental shutdown the IT "Problem Management" team were put on the case. Their recommendation was to shield the emergency power buttons with the metal shields or cages often seen covering fire call points, thus preventing accidental activation. Their smugness was short lived as I pointed out that the typical use of the Emergency shutdown button would be in the event of someone being electrocuted, and that perhaps a conductive material wasn't the best solution.
I used to work at a site that had a lot of mathematical modellers. One of the Cray Supercomputer models we used was just the right height for a raised elbow. The designers hadn't considered this but did consider the perfect place for an emergency stop button was on the top of the box. We put a plastic cover over ours so no one ever accidentally shut the thing down. Seeing this a Cray engineer tells he'd been to a site that had used half a coconut shell for the cover.
Must've been a cray for help?
Sorry. Like other commentards, I'm sure I've read this story at on-call (or maybe who-me) too recently before. Mine's the one with ... oh, they all are.
Edit to add: Here's a much more dramatic rendition.
I once had a role that evolved into looking after a 60 rack Data Centre. It was located in the shell of a former 1960s computer hall although most of the vintage services had been stripped out or abandoned.I spent a year or so reverse engineering all of the services as the electrician who had worked on it over the years died in unfortunate circumstances.
Whilst we didn't have an ESD switch, the room did have an FM200 gas suppression system which was disarmed whenever staff lifted floor tiles.
One weekend, I received an urgent phone call from a colleague with the immortal words "There's been a bit of a cock-up". The gas system had gone off, two thirds of the cooling had shut down and Servers were gradually shutting themselves down due to localised overheating. Not knowing how the interlocks worked, I eventually got the Air Con up again by pulling the battery leads off in the alarm panel. (There was a discharge detection sensor that required a manual reset, something that went into the documentation once we were told about it). Fortunately, the cooling interlock wiring had not been extended to unit 3 (and standby unit 4) so we were able to avoid compromising the really important stuff.
It turns out the staff member got confused when re-enabling the fire panel and got it into his head somehow that it would reset if he pressed the button marked RELEASE under the hinged flap with the seal on.
Getting the FM200 replaced was another saga, it took weeks...
I received an urgent phone call from a colleague with the immortal words "There's been a bit of a cock-up".
If there is a silver lining to this kind of incident, it's the opportunity to come up with a truly memorable understatement. Make it good people!
Sir PTerry knew ages back...
“Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”
― Terry Pratchett, Thief of Time
> Sir PTerry knew ages back...
> “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”
> ― Terry Pratchett, Thief of Time
When I were a lad, we used to sometimes go into the Big Smoke (for us, that was Byron Bay) to do our shopping. There was a big department store on the corner of Jonson and Lawson where you were served individually at the counter and your bill and cash whizzed up on a cable to the cashiers on the second floor who then let the receipt and change fall down on the same cable. Next to the main counter was a small wooden sign:
IN CASE OF FIRE, PLEASE RAISE THIS FLAP
Underneath it said:
NOT NOW, STUPID -- IN CASE OF FIRE!
As a very small child in London for the first time, I obviously pushed the emergency stop button on one of the long zone 1 tube station escalators. Not sure everyone was amused as I was.
Got to do again it many years later as an adult when someone fell over behind me while getting off one. Just lucky I'd been waiting for a second opportunity...
Two incidents come to mind. Once, at the local authority where I worked, we had building work going on in our computer room. I was working upstairs when all went dead. I rushed down with others to find that the builders had been hammering so hard on the door that they had triggered the emergency kill switch near it. Half an hour of doing the headless chicken dance and all was back with the world. In those pre outsourced, pre SLA, pre critical indident days there was a bit of a meeting which involved reading the riot act to the builders and all went back to normal. I dread to think what would happen now.
Second was at the same authority but a different computer room. We had to shut down for weekend electrical work. I was doing the Sun boxes while colleagues were doing the Windows stuff. The facilities guy was there to turn off the power when we were done. I had just got the payroll DB server to the OK prompt and typed the words power off when the facilities guy hit the off button at the exact same moment that I hit return. Cue silence and a lot of swearing from the Windows people who had only got half way through their shutdowns.
Beer cos it is Friday.
We've covered this before, however...
The company I worked for had a large computer room, basically long and narrow. The doorway was partway along one of the long sides.
One day, one of the senior bosses was showing some prospective clients around. Took them into the doorway of the computer room.
Spread his arms out emphasise how big it was.
But you're wrong - that was OK.
Then, to let them get a better view, he stepped to one side of the door, and leaned back...
It's the early-mid 1990's we are an outsource supplier to a very large UK bank stock brokering operation - They have invested heavily in a clustered VAX infrastructure that has eliminated most of the single points of failure, except one big one in that the cluster (And shadowed disks) were in the same physical data centre and in fact in the same large bank customer only data center hall/room.
This clustered VAX infrastructure worked extremely well and the bank was very happy with the high availability, much improved speed and the service we were delivering, so much so, they wanted to bring a bunch of their senior executives to our facility to show them what the banks money had been spent on.
Cue to day of the visit, everything was going really well, the final part was an escorted visit to the actual data room where all their kit was housed - Some questions asked by executives but suddenly the chairman announced that he understood that it was impossible to shutdown the computers (Great selling of high availability by the account manager) and suddenly the chairman reached out to the big red button on the wall. Literally mill-seconds from disaster I managed to intercept the chairman's had (with quite a bit of force) and stop him hitting the EPO.
However the aftermath was me apologising for the "assault", the bank chairman thinking we had something to hide and overall 12 months to rebuild the customer confidence in us - Even though we did stop a complete powerdown which would have cost them hundreds of thousands.
Lesson learnt, never ever take customers into the data center even if they own it - So next step was a couple of windows installed so customers could see but not touch.
Perhaps a communication failure by the marketing team, or those who fed them their spin?
When talking about disaster recovery, getting back to normal performance within X hours, etc it is imperative to stress that recovery is VERY expensive. And a disaster is different from somebody pressing the Big Red Button.
I worked on a local government ICL 1900 site and the system kept crashing. Over time we noticed a coincidence between the crashes and the presence of a particular operator in the line printer room.
It transpired that this guy had a chip on his shoulder and found he could crash the system by switching the circuit breaker at the rear of the printer off and on.
I did once flip the reset switch on a client's machine. I had been planning to do so later on, to give them some practice in running the equivalent of fsck, but not quite then. This was during the installation and training phase, not production, so nothing was lost but time (ten minutes?) and face.
On the other hand, I was at a tech day when my phone informed me that certain of our servers were down. This proved to be so, and I hopped the Metro back to the office. It turned out that the SAN tech, while doing I know not what preventive maintenance, had flipped the power switches with his shoulder. Everything started up again nicely, to our and his relief.
..used to have the big red buttons at same height as the head rests of the chairs. Fortunately the control room kit was just effectively remote control panels - all the actual kit was a floor below and ran happily without control panels being active. After a few brown trouser moments big flip covers were fitted.
Not a DC but a whole college campus. Back in the late 80's I did electrical engineering at college. As part of an assignment we had to go in to the main electrical switch room and draw and lable what we saw. Anyway there was a very large airbrake style breaker the ones that look like a one arm bandit fruit machine, now I knew what it was and NOT to touch the large handle. BUT something in my head said I wonder what that does and I flicked the small flapper swich on the side of the air breaks case. CLUNK air break tripped and the entire college lost power! It was the FECKING breakers test button that I pressed!
My mates all helpped out and said my A4 ring binder folder had accidently knocked the switch. For the rest of the term the college clock and bell was out by about 10mins as thats how long it took to get the power back!
At my previous employer we had the same problem. This was a few years ago though
EPO. Upton under the door exit one and someone pressed the wrong one, replaced with a bigger button that you had to hit hard to trigger it.
The same person (I think) was repsonsbile for unplugging a generator for the building (somebody stole the cables to the substation) from the main supply to run his envelope inserting machine. It took an hour before the ups ran out and the site died....
The better one was the day that the power overloaded, entire room down (nobody was in there), went in with torches and the breakers in the room were fine. Went outside and we found that the 100A board in the comms room was fed by a 40A breaker outside....
When I was working for a very large electrical engineering company, building motor control gear for a well known maritime organisation, I also caused mayhem with part of my anatomy. One of the units was in Test, and the testers were doing a heat run, running the equipment at full chat whilst the observers from the customer looked on. I had gone into Test to take some photographs for the Instruction Manual that I was preparing (I was in Technical Manuals Department at the time) and I had to scrunch myself up into one corner of the roped-off area in order to get all of the cabinets in shot. Suddenly everything went dark, and the high pitched whine of the invertors wound down the scale to inaudibility. Cue furious shouts from the Test Engineers, I had inadvertently backed onto one of the emergency shutdown buttons that were located at various points around the department, and that had shut off all power to the Test area and surrounding parts of the building. A complete morning's heat run ruined, and the customer's observers were distinctly unimpressed. The heat run had to be rescheduled for the next morning as it had to start from cold. Needless to say, I was NOT allowed into Test whist a heat run was being performed on that or any further equipments.
Some years ago, a past workplace of mine got one of those systems installed which could deoxygenate the server room in the event of a fire. The control panel was mounted right by the door, and unfortunately was very sensitive to jostling, like the vibrations from the door slamming because someone was carrying something bulky in and didn't have a hand free to close the door gently.
Luckily, this system would wait a few seconds before actually activating, giving someone the chance to stop it first. So those of us non-admins who worked nearby remained blissfully unaware of the hazard until the inevitable day when someone didn't manage to deactivate it in time. A lazy summer afternoon was interrupted by the almighty FFFFWRRRMMM of the system activating, followed by a couple panicky minutes of realizing what the sound must have been, trying to locate someone with server room access who was not actually in the server room at the time, visualizing colleagues passed out from hypoxia... (It was not until after this incident that we were told it didn't lower the oxygen level so far as to become totally unbreatheable.)
In the computer center the chief operator was an ex marine gunnery sergeant. In New Hampshire we always complained about the low humidity. The HVAC folks said they could fix this. The next moment the machine room started filling up from the bottom with fog. The operator calmly went to the EPO button and pressed it. It worked and he probably saved a lot of spinning rust.
When designing the operating system I had to debug the power off shutdown fault (meaning that there was 160 usec before CPU power dropped). There was just this time to save all of the essential state in core (yes, real core). So I sat powering down and up the main cpu - a huge monster - to test this. The field engineers were not happy as they said that the CPU wasn't designed for repeated power cycling but never had any problems.
So there are some legitimate uses for these buttons.
i heard an awesome story about a guy in the dell factory at limerick. He was working on the production line when he pressed the emergency stop button by accident, and caused the entire line to shut down. It caused a huge rumpus as it cost $$$, so much so that someone from Dell HQ flew over to see what had happened, to try to see if there were workarounds they could implement. So this big shot was talking to the operative
'Walk me through exactly what you did' the big shot guy said
'I accidentally pressed the emergency stop button, like this', the guy said. Pressing it again to show him
I was once helping out by clearing a backlog of fault calls at a railway depot. Suddenly the PC I was working on went dark and all went quiet.
Enquiries revealed that someone up the road had chopped the BIG cable with an excavator, so it was not going to be a quick fix. I went to the server room and gave things a graceful shutdown before the UPS gave out. This was late afternoon after the office workers had left, so only the engineering staff were around. I headed over to them to inform them that I was heading off too, and commiserated with them that they would now be unable to brew any tea. "No problem" they said. "This buffet car gets its power from the overhead line".
I work for a large privately owned IT services company in Canada, and one of our clients is a large energy corp with petrol stations across the country.
I have lost track of how many times I've been sent to a site to troubleshoot why the paypoints can't communicate with the petrol pumps, only to find someone had accidentally pressed the big red emergency shutdown button for the pumps.
Never saw a colleague pull a "Fred's Head," but once, with all the staffing happily doing staff things, the building suddenly went dark. The UPS started to scream and the server shut down, though by then all the other work stations were already dark. The lights were out and the phone system dead. The boss had us checking the breakers to the building, but they were fine. The city utility said, "it wasn't us!!" An electrician was called in and found that we had power to the pole and down the descender to where it entered a conduit under the drive way into the parking lot. At the riser though, where the line entered the building, there was no power. The enter undergroun line had to be exposed and the conduit dismantled before the "cultprit" - a tree squirrel - who had for some squirrelly reason mades it way into the conduit and half way to the far end before settling in to chew through the power line with fatal results.
Back in the day I was a computer operator on an IBM System 370 box, which required something called a motor-generator unit to supply 3-phase power to the system. The unit itself was a plain looking box about the size of a wide freezer, with doors on the long sides for access to the guts. Toward the top of one of the doors was a cut-out through which the EPO button could be accessed. Now some clever engineer just knew you didn't want to place a push button at waist level that would kill everything in the room, so they smartly made it a "pull" to activate switch.
Unfortunately, one day the IBM CE was enjoying some free coffee, leaning against the unit at just the right spot, and when he moved forward a belt loop on his pants managed to snag the rim of the switch and Bam! down goes the system. Okay, that's embarrassing enough, but as luck would have it the wall panel relay also died in the incident, and it took a full day to find a replacement for the part.
Shortly after that the versatility of the 80-column punch card was put to use -- one was taped over the hole in the motor-generator's door panel.
We had nice sticky out kill switches and after being activated several times, ok I did it once simply by stacking boxes on a table handily put under a switch, management came up with a missive to prevent accidental activation.
This turned out to be a load of plastic cups and me trotting around the place covering the red bit with a plastic cup. Seemed to work as I can't recall them being set off again.
Oh yes and we had the halon gas stuff and raised floors. One data centre had an electrical fire that caused the halon to release.
The whole computer room was wrecked as the injectors were under the raised floor and several floor tiles went into action and became part of the PDP11's.
On the plus side room was refurbished and had the injectors sticking out of the cieling tiles.
I remember that halon system going off in the taperoom for the Cambridge university mainframe during the late 80s. Minor fire went out, but the level of dust raised meant all the tapes had to go through a special wash/cleaning machine (actual liquid bath with sponge pads). Extremely disruptive.
Worked with an outfit supporting small medical office in the UK. We were still using XP (haha) and I had to manually patch each machine with the latest updates. We had one particular office that was unable to be contacted, and we had to call in for the office manager to check the server. She was compliant and the server came back on but only for a few minutes then it was not responding.
I picked the short straw and had to go down to the facility to see what was going on. Log story short, the office manager replaced the server power cable with the kettle cable as the other one was longer, and every time she went into the server room and plugged it in and left, the door closing would pull the plug out. Even better, the server was moved next to the door as the kettle plug didn't reach and damaged the patch panel.
Around 1980 I was the site engineer for a large 24x7 data centre in the midlands which was full of IBM and Amdahl CPUs and rows and rows of IBM and StorageTek disk and tape running all the usual mission critical work. It was a new, purpose build data centre, wide corridors everywhere, double doors into the dc itself one of which was usually kept locked and the other side was opened with a card key for staff to enter.
Everything was running fine, all the kit was up and running, lots of sounds of spinning things, until some new kit was being delivered during one busy day and the other half of the double doors was opened to allow it in. Yup - the (very new, recently installed) shiny red EPO button was not quite far enough along the wall and the second door hit it...cue sound of lots of spinning things slowing down.
Rather than silence I can speak from personal experience about what happens if you hit the kill/field dump switch on an NMR/MRI machine. All the stored energy in the coil vaporises the liquid helium coolant, followed.by a very loud bang/woosh as several thousand pounds of now gaseous helium, with about x1000 volume goes up the vent pipe. Had to be done as some idiot had come in with a steel gas cylinder which was literally dragged off the trolley into the coil. Not an experience to be repeated. Try leaning near the kill switch if you go for an MRI and see the techs.reaction...
at least your MRI did shut down. In this report the kill switch had been disconnected so the two staff in question had to suffer entrapment for several hours until a GE engineer came to shut the system down
https://medicaldialogues.in/ge-pays-rs-1-crore-settlement-to-technician-who-got-stuck-in-mri-at-tata-memorial/
One of my customers installed 3 PC's to run a taxi booking/dispatch system. They decided to do things properly, so they got an electrician to install an independent ring with sockets for the computers, the screens, printer, router, radio TNC, and some spare sockets. The ring was run off a smallish UPS.
So far so good.
Since the ups would only provide a short duration in the event of a power failure they bought a small generator, so if mains went, the ups would take over for long enough for them to get the generator out of the cupboard and connect it to the input of the ups.
Seemed very good.
Some years later a digger in the street severed the mains. Remember those 'spare' sockets.... two electric heaters and a kettle had found their way onto the circuit so the UPS threw a tantrum. They couldn't find the key for the cupboard containing the generator. The fuel in the generator was several years old and no-one new where the starting instructions were. An arguing mob of irate drivers soon had sore arms from pulling on the starter handle. then some bright spark noticed that the shop two doors away had power. And they had a long extension lead. Soon they had a working computer, screen, router, and TNC.
Now an even brighter spark reasoned that if they removed the socket from the extension lead and replaced it with a plug, and plugged that into one of their wall sockets, it would power their 'ups' ring and all their computers would work. I suspect that their degree of familiarity with H&S laws was about as advanced as their knowledge of three phase mains distribution (yes the shop down the road was on a different phase)...
A friend of mine was once working on a PDP-11 machine, typing in a lot of code using TECO.
Someone else in the office came over and started playing with the front switches on the PDP-11. For no reason, you understand - he was just fiddling for the sake of it. It probably wasn't going to do any harm, but not unreasonably my mate asked him to stop playing with the switches in case something went wrong. "Oh, look, it won't do any harm", says the other guy, "but if it makes you happy, I'll turn the keyswitch to lock so that the keys CAN'T do anything."
So he reached for the keyswitch, and turned it the wrong way - which switched off the PDP-11....
Service call goes out for a mainframe tape deck, engineer turns up later on & phones to be left in.
Someone walks down to main door and lets him and another man waiting next to him in.
A minute later second guy said "oh big red button" and presses it.
Operator and engineer look at each other, engineer said "why did he do that? "
Turns out it was some random nutter they let in.
I was powering on an HP EVA Array after maintenance, and as I flipped the main breakers on the main PDUs, my co-worker snuck up behind me and clapped really loudly. Laughs all around from him, until I start clutching my chest and reaching for my medication (I had a severe heart attack a few years before), at which point his expression changed to "uh oh, bags not doing CPR". Fortunately the chest pains subsided after a few mins and we had a good laugh about it later.
In case anyone is still reading, the regional control centres that the UK government commissioned for the FireControl project (go look it up) had their equipment room EPO buttons next to the magnetic locked exit door AND they didn't have an exit button for the door so it was the only button present. The situation was in fact even worse than that in that their official process at the time I came across it was to allow third party staff into the room with no means of exit and expect them to phone security to get out, a scheme that didn't even work reliably as during the delivery phase the security office wasn't reliably manned. All in all a rather impressive breach of health and safety and lone working arrangements.
Back in the eighties, a friend worked at at '38 shop as lone operator. Handily placed behind the two band printers was a wide, highly polished and almost frictionless marble windowledge. His method of getting reports off the printers for distribution was to flick the perfs on the fanfold at the appropriate point, stack the reports on the ledge and once he had a decent stack, give the stack a shove along the smooth, polished surface where it would stop at the end. Repeat until all reports are sat in a series of stacks along the ledge. Easy and efficient.
One day he took the day off. The developers were busy with a software release required that day so it fell to The Boss to sort the reports out. He'd observed the operator's process and decided that it was the best idea, so that's what he'd do.
Turned out that knowing the maximum permissible height of the first stack, so it wouldn't smack into the EPO button at the end of the ledge when slid that way, was a crucial part of the process that he'd missed.
We were experienced IT staff, so the Emergency Power Off was a large twistable lever located in the middle of the data centre complex (so that people would hurry the least distance on average whilst their coworker was in trouble). Not such a great position when the computer room started to fill with water and a volunteer was needed to wade across the sparking computer room to turn the EPO switch. Nowdays there's a second EPO switch inside a cabinet on the outside wall of the building -- that's got some theoretical issues, but none of those issues put people's lives at risk.
A related story, wherein the Big Red Button is A Secret and A Joke:
A Story About 'Magic' : ESR/catb.org
Some years ago, I (GLS) was snooping around in the cabinets that housed the MIT AI Lab's PDP-10, and noticed a little switch glued to the frame of one cabinet. It was obviously a homebrew job, added by one of the lab's hardware hackers (no one knows who).You don't touch an unknown switch on a computer without knowing what it does, because you might crash the computer. The switch was labeled in a most unhelpful way. It had two positions, and scrawled in pencil on the metal switch body were the words ‘magic' and ‘more magic'. The switch was in the ‘more magic' position.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
