Techies take turns at shut-down top trumps Welcome once more to Who, Me?, our weekly trip down memory lane for Reg readers who have cringeworthy yet humorous stories to share with the rest of us. This week, we've collated a few of our favourite accidental power-down stories, prompted by a recent tale in our companion column, On Call, which is dedicated to tech support …
All of these stories demonstrate that we had to learn the hard way about visual identification and accidental triggering.
Not one of these stories would happen today because it is blindingly obvious that you enclose general shutdown buttons in protective covering, and they should be red and not any other color. Harsh lessons are the ones best remembered.
"they should be red and not any other color"
Remember that some people are red/green colour-blind. Maybe that explains the green EPO button. Perhaps a distinctive shape would be a better distinguishing feature - and better to identify if the room's filling with smoke.
Remember that some people are red/green colour-blind.
Vile creatures, mutants from the very pits of hell. Red, green, it doesn't matter what shade of grey we make the buttons, this fifth column lurking hidden in plain sight will be a constant menace to the chromaware unless we adopt the very sensible "special shape" idea Doctor Syntax hints at - but doesn't dare to elucidate.
I suggest a skull and crossed-bones shaped button, with prominent comic eyeballs over-etched with big Xs, and make this the universal sign of a killswitch by ISO code.
Just to be sure people only press it when they mean to shut down the computer, we should cover it with a shroud and put 50 000 inescapable volts on it. Also, the lifting of the shroud should trigger a three-second delay followed by the opening of the trapdoors over the pit o' snakes positioned in front of the button.
I must say I have never seen a red EPO button (although clearly they have existed) and also I have also never seen a red door release, although green does seem to be common. Having said that, door releases also seem to come in a baffling array of other disguises by being next to the fire alarm, looking like a light switch, on a wall panel behind you as you face the door etc.
in some places it looks like they were deliberately vague just for the amusement of the team sitting behind them so they can point and laugh as you bounce off it like a bumblebee in a greenhouse...
This post has been deleted by its author
"Having said that, door releases also seem to come in a baffling array of other disguises by being next to the fire alarm, looking like a light switch, on a wall panel behind you as you face the door etc."
There's one place I've been to where the after-market door openers are so slow at opening the heavy, Victorian wooden doors in the listed building that they have now placed the door opening button about 20' before you get to the door. The switch looks like a light switch. But it does have a big sign next to it saying "Door switch". But naturally, no one is paying attention that far away from the door, so there's also a wooden stand with another sign on it saying "Door Access Switch" and a big arrow point at the switch. Just in case you miss both of those, there's another sign at the door saying "Door Access Switch 20' behind you". A remarkable bit of foresight and humour for a local council Town Hall :-)
At the end of February, I was staying in a gated residence. For some reason the release button for the gate was located in the entry hall of the last building on the way to the gate. A good 10 meters away from the gate. No sign, no indication, just a switch looking like light switch.
If my landlord had not shown me that, I would have been trapped the next morning.
I think the idea was to prevent from passing the arm thought the gate and activate the switch from outside.
Err.... I think they will still happen. Management wants to save a few dollars. Server rooms get re-configured (but the switches aren't moved). And the mindset that "no one will ever press a red button on the wall unless it's an emergency". In one place I know of, there's something like 8 switches. Four of them have had the cabling removed, the other four haven't. The catch is, none are marked as to "active" or "inactive" and none are covered. Bump or touch one at your peril.
The only real solution is to label the darn switches... "lights", "Emergency Power Kill', etc. Even a a bit of paper and taped to wall will work.
What's "touch this button and die, coward!" translated into Klingon anyway? Something short I'm sure.
Because as any real geek knows, to use the button correctly would be cowardly and worthy of death. To use the button incorrectly means being blown out of an airlock and then some for both sheer incompetence and crippling the commander's vessel.
Klingon computer rooms are most definitely not Elf 'n Safety compliant!
I dont know if there is such a thing as sudden-onset colourblindness, but I believe most colour blind people will have lived with the condition all of their lives, and entered the adult workforce with management techniques developed from living with the condition.
This would probably include reading labels. Which does mean adequate labeling is essential.
Just because something is strange and outside your own experience, doesnt mean it isnt routine and everyday for others.
I am not saying its easy, but people cope.
In my youth I did some youth hosteling in Wales. After spending a week in Corris, climbing slate faces and hillwalking, it would take me a day back in city life to be able to see colours other than green and grey with any vividness. All there was in Corris was grass and slate. Everything was made of slate and if you left anything alone for a few days, grass and moss grew on it on account of the damp. Anywhere you let your eyes rest, grey and/or green.
Which is why the red and green you use are distinctive and not "real" red and green. Just look at a "green" traffic light. We all call it green, but it's not actually really green is it? Because if it was then 10%-15% of the population would not be able to tell it apart from red, other than by its position.
It always puzzles me why lawnmower power cables are orange, when evolution has shown that orange stripes are the perfect camoflage in dappled light/shadowed grass - just look at a tiger! After mowing over my lawnmower's power lead, I immediately changed it for a white one.
Just look at a "green" traffic light. We all call it green, but it's not actually really green is it? Because if it was then 10%-15% of the population would not be able to tell it apart from red, other than by its position.
I remember a Usenet discussion on colour blindness and traffic lights. One contributor who was colour blind said that to him traffic lights were grey, yellow and white, and it was the position he went by.
Unless you're really dedicated, Usenet dates your comment.
Over the last decade or so green traffic lights seem to have got a lot bluer than I remember them being, probably as a result of this exact situation. (I hate them because - for some reason - I find it hard to focus on blue things).
Locally we had (up until the early 90s) traffic lights that dated back to at least the 60s (or possibly earlier) with the black and white painted pole, red, amber and blue/green with stop written on the red and go written on the blue/green
So perhaps reverting back?
I don't know if that's intentional, or even true, but having looked into it for the purposes of producing figures, blue and red are distinguishable for more types of impaired colour vision than green and red are. In fact, plug them into simulators for the different types of colour blindness and it's the most reliably distinguishable pair, which is strange when you consider for people with full colour vision it doesn't appear much different to the infamous red/green combo. (Complete colour blindness is rare, and the conditions that cause it tend to lead to additional problems.)
When my dad was in Seattle post-war, they still used those traffic lights hanging down in the centre of the intersection. A single set of bulbs illuminated RYG on two sides, and GYR on the intersecting sides. His friend judged by what the rest of the traffic was doing, and after one incident which brought the problem to their attention, another friend sitting next to him would quietly murmur Red... Green... as they approached intersections.
The epic, well-known discussion of relativity and absoluteness.
My take is as thus:
What is color? After all, colors in all languages are adjectives. We describe something to be blue, green, red, whatever, because we perceive them to be thus.
Back to the example of the traffic light:
To a colorblind person, the light's colors are different.
Yet they are still sources of light that emit waves with a distinct wavelength that falls somewhere on the visible light spectrum.
The color might be red, but it's still light, and one with a wavelength of 700 nm.
TL;DR: Some properties are relative, others are absolute.
P.S. If a relative quantity is seen to be the same by everybody within the universe, does that make it absolute?
As I recall, it's the other way round. Blue was usually the last name to be given, resulting in a few interesting things. Try this for a source. It's kind of long and they'll be talking about other things that are unrelated, but it's never a bad time to listen to radiolab.
I can see it now:
Consultant: "All done Sir. Black, White, Red, Pink, Orange, Green, Yellow, Brown and Purple. All you'll ever need."
King / Emperor / Chief / God / Whatever: "Good work. What colour's the sky?"
Consultant exits stage left muttering about picky clients...
You mean "Radioblab". The presenters never let their experts explain, but immediately start talking over the recorded soundtrack, often just blabbing the same stuff the guest is trying to say over the guests voice.
And the "blue" show was the stupidest steaming pile of stupid I ever heard.
Thesis: "Until the Egyptians invented blue dyes, no one could see the color"
Evidence: "Homer talks about wine dark seas and skies, not blue ones."
My refutation: "I guess no-one in those days saw clover flowers, irises, lupins or lapis lazuli. Odd that the tin and copper miners of the bronze age never mentioned Copper Sulphate too."
Stupid with a capital stupe.
Nowt wrong with NPR. It has some of the best programming you can get over the air in your car.
Radioblab just in't one of those programs. Even when they have interesting and believable stuff as the theme, they fuck it up by treading all over the people they've invited to speak on the subject. I would understand if they were covering for inarticulate glooks, buy most of their guests are very articulate and gifted with talking science to the masses. The value of the Two Amigos blabbing over the answers they asked the expert of the day for is dubious at best and annoying all the time.
Rancid Orange,
Sorry, I don't speak ancient Greek. But they were even weirder. Not only was the sky green, rather than the normal blue. But also colour words sometimes included texture as well - just to be extra confusing. So the Iliad and Odyssey are full of references to the wine-dark sea.
When the sea starts looking like wine, you've either been drinking some very odd wine - or rather too much of the stuff...
The RN medical included a lengthy visit to the optician, where instead of doing 1 or 2 pages from the book with numbers in patterns to check for colour blindness, instead I had to do the whole book, which includes several pages that you can only see something if you have some rare eye disorder (kinda worrying the first time as you can't see anything, to then be reassured by the optician that not seeing anything is a good thing)
There are many people who are red/green colourblind
About 1 in 7 males IIRC, though there's a range from "barely noticeable" through "quite pronounced but I can still mostly manage" (like myself) and onto the extreme of can't tell the difference at all. Interestingly, when I took my first aviation medical, it wasn't the red-green I almost failed on, it was the yellow-white. I blame the fact that the pigmy bulb in the lantern was yellow to start with - so I had to tell the difference between "supposed to be white but is actually yellow" bulb and "same bulb but through a yellow filter" :-/
Before that the AME got out the Ishihara book and I commented "this'll be interesting". As expected, I couldn't see numbers on many of the plates - but was OK when he turned to the back where they were almost black & yellow. Then he got the lantern out, mumbling about the hassle as he untangled the power cord - I suspect he didn't have to use it often.
I'm red/green colour-blind, and like most other colour-blind people, I can still easily tell the difference between red and green.
The two situations that I've not been able to clearly see a difference between red and green are are:
1) In an Ishihara test
2) In very low light, when it's so dark that even fully sighted people are having to squint to distinguish colours.
It's generally a much less severe condition than most people think.
To be honest, having a shutdown button right next to the door is always going to ask for trouble.
At my previous workplace, each wing was controlled by mag-locked doors and the release button to get out of a wing was to the left of the door - right next to the light switch. Its quite impressive how many people head to the door, reach out for the release switch and hit the wrong button. I remember the lights going off then back on a good 3-4 times a day when I worked there.
Thankfully, these were just office lights. Emergency shutdown buttons though should be well away from doors and light switches, bright red and with a shroud over them.
This means they have to be somewhere accessible in the dark and that means near the door.
But why your data center would be in the dark before you press the EPO button? Especially if the button is inside the data center and you have to be inside the data center too to press the button.
"But why your data center would be in the dark"
Dark as in "so full of smoke that you can't see anything" or potentially "chemical leak caused blindness" or "manager did something so dumb that I smacked my head against the desk so hard that I've got blood trickling into my eyes"
Isn't it standard now to also fit signs that glow in the dark?
They have a slightly green sheen to them, and given that such areas tend to be permanently lit there's plenty of "charge" for them (I know from experience that they do not work at all in hallways and stairwells with on demand lighting :) ).
As for being colour blind, my colour vision is so acute I have worked in a plastics factory for a while, developing colour pigment recipes. I am one of those annoying exceptions to the rule that women see colours generally better than men, although I'm rubbish at naming them (also because it's usually the customer who classifies it, colour blind or not :) ).
Indeed, in a previous life it was my job to administer the Farnsworth-Munsell 100 hue tests to patients. The professor demonstrated how he wanted it doing by administering it to me. I scored zero. The professor had never seen any untrained subject score perfectly before.
After decorating and repainting the doors, the other half decided we needed new door handles. So, after buying new handles, I took the old ones off, filled and sanded, and repainted where the old handle was. Only to discover the "white gloss" the door had been painted with two months ago was a different white to the "white gloss" I'd patched with. Ended up have to repaint the whole door. All seven of the damn things.
Oh yes - my first week in the colour matching job was filled with being amazed at how many different versions of "white" and "black" we actually had, and that was not counting the versions of white that needed a UV convertor to be that bright (which, incidentally, was not a lasting feature when I did this work, maybe those additives have improved since).
"To be honest, having a shutdown button right next to the door is always going to ask for trouble."
Indeed. If one wants to prevent accidental or malicious activation then design the shutoff like the nuke missile silos. Two keys required and separated by enough distance that one person cannot turn both at the same time. Perhaps one at each data center door. Problem Solved.
While this may cause other problems, my job was to fix the accidental or malicious activation problem.
My current office is built into a hill, so you come out of the "basement" on one side, through an L-shaped room with a service-lift at one end and a set of bins at the other.
It's also a secured building, so there's two sets of doors you have unlock by pressing a button. And for some reason best known to the building's architect, there's a short corridor between the basement and these two doors, so you have to pass through three sets of doors when leaving via the lift. And there's a light switch next to the first door (which isn't secured).
After a hard day's hacking at the coalface, I took the lift down and ambled towards the first door, where I absent-mindedly reached out and slapped the button. Except it was the aforementioned light switch, and one of the office cleaners was working around the corner in the bin area.
So the room went pitch black, and I was further startled by a shriek of terror, seemingly from out of nowhere.
Thankfully, she saw the funny side after I'd flipped the light back on - and after she'd recovered from the near heart attack!
There are low-power green led-lights embedded in the exit signs and the occasional ceiling tile, but these are very dim and it takes your eyes a few seconds to adjust enough to see again. And as a "basement" room, there's no windows and two sets of doors between you and the exit to the next room, which has power-saving lights and is also dark.
So to all intents and purposes, if you flick the switch, it's pitch-black...
I know those EPO buttons were all the rage in former times, did anyone ever use them as intended?
We tripped ours with a ladder (about the same way as Ned), then got a shroud and tripped it again with a soccer ball - through the shroud which got a nice dent in the process. It then got a stronger shroud - the matches continued, obviously.
"How would you get a horse in the data centre?"
I rode one from Portola Valley Equestrian Center to SLAC once. Got her into one of the buildings, and down the heavy goods lift into the basement, hitched her to my office door and got to work. I stopped there, but I could have taken her into the actual computer room down the hall :-)
Why? To prove a point about lax security ... Neither the horse, nor myself, had our badge with us. Things are much better at SLAC now. No, I wasn't arrested. Today, I'd probably get life without for even suggesting such a stunt ... Prior to this, I had been riding my panhead down the same path for about a week without anybody batting an eye.
I rode one from Portola Valley Equestrian Center to SLAC once. Got her into one of the buildings, and down the heavy goods lift into the basement, hitched her to my office door and got to work. I stopped there, but I could have taken her into the actual computer room down the hall :-)
Hope you were nice enough to shovel when you took her out, a horse just standing around will take the opportunity to lighten the load. BTW, a mare with a good mind is a lovely way of getting around, and if yours rode the lift without demur, she was a good 'un.
Many years ago I was taking a short-cut through the data centre and spotted smoke coming from one of the (washing-machine sized) disk drives. Naturally I pushed the Big Red Button. My boss was not entirely happy; he pointed out that the drive wasn't loaded at the time so just powering it down would have been sufficient. Perhaps.
"did anyone ever use these buttons the way they were intended to?"
At our computer center at college we had a computer who was a retired marine gunnery sargent. One day someone complained about the low humidity. The HVAC people made some adjustments and the computer room filled up with a dense fog. The operator justifiably hit the emergency power off button on the way out.
Back in the nineties we had a "server room" in the middle of the office, for development and test servers (mini-computer sized). For some reason they decided to fit the red emergency power off button right next to the green door release button and it didn't matter about putting a shroud around it, a bit sign up to say which was which, etc. there would be regular cases of visiting engineers pressing the wrong button to leave the room. I think this even happened once after the shroud was fitted with a cover :-(
I think the human mind has an infinite ability to go through with an act, despite all countermeasures and protections, even if we are screaming at ourselves inside to NOT DO THAT THING (*).
And we do it anyway, and then we can only shake our heads at ourselves in disbelief that we actually did it.
* : ask the Air Force about how many ways they have tried to alert a pilot that his landing gear was not down while in the process of landing and how efficient the methods were - if I'm not mistaken, they still don't have foolproof method
@Pascal Monett
TP The Last Continent:
Any true wizard, faced with a sign like 'Do not open this door. Really. We mean it. We're not kidding. Opening this door will mean the end of the universe,' would automatically open the door in order to see what all the fuss is about. This made signs rather a waste of time, but at least it meant that when you handed what was left of the wizard to his grieving relatives you could say, as they grasped the jar, 'We told him not to.
A friend of mine had the job of testing all the fire alarm call points (break glass type), in somewhere rather large, with a map of the location of them all he set off and all went well until he found one that was not on the map. He penciled in the missing red break glass call point onto the map and released the glass to test it, people started rushing out of the door along side it, it was the haylon release for the server room !
(it was stationary). Unfortunately the previous user had not put the hand brake on and it started to roll back. I was puling anything with a T shape or an angled lever, all the while putting my full weight on the conventional brake pedals (no manifold pressure). Someone else had to climb in the passenger side and pull the actual hand-brake. It was a SQUARE! wire handle painted YELLOW! I'm ex airforce, so square wire handles is a cut-away shape for parachutes, and yellow is a danger colour for something that can't be undone when pulled (I thought it would cut away the box body from the chassis or something). I have a lot to say about Kraut engineering these days, not a lot of it complimentary!
"the wrong serial cable on a UPS"
I remember an odd instance with a serial-connected UPS on a SCO box. At some low level it must have tapped into the network stack because doing something - I can't remember what - on the network started what appeared to be a UPS-iinitiated count-down to power-off. It took a couple of repeats to be sure it wasn't the UPS.
yes they wee PAC's we had several hundred and they would be moved around as teams moved and were created / disbanded, this meant we often had 10-20 UPS's awaiting reuse. It was a pain because our SOP ended up keeping the cables plugged into the UPS then taping them to the case rather than just storing the UPS's in rack with a box of cables beside them. even then we'd end up having to check the pinouts of a cable occasionally to make sure it was the right one for a particular UPS.
I have a vague recollection from many years ago of an oddity involving an APC (of course) UPS where, if the serial cable was connected while the server was starting up, it would cut the power during boot at the point where Windows NT initialised the serial ports.
The workaround was to plug the cable in after the system had started up. I can't remember what I did to fix it properly, or whether the problem simply went away after a service-pack or something.
I arrived at a customer to help with a mainframe problem to be told it was down, because someone had moved some disks. The physical disk unit needed to be moved 6 inches to the left to make space for some newer disks. They decided to do it while the disks were in use, so a couple of them put their backs against it, and carefully moved it backwards (using their legs, for good health and safety). This was fine, until one of them moved away, and found his belt was caught in the Emergency Power Off button, which you had to pull to activate. I was told "the sounds went from the hum of the disks happily spinning to the deafening silence of someone filling their pants"
This was the same company who had two machines close to each other. They were joined by a cable which had to be no more than 4 meters long. Instead of running it down to the floor, under the floor and up to the socket; The machines were too far away, so had the cable coming out at waist height, over the back of a chair, some disks and into the other machine. When someone complained, they wrapped it in black and yellow tape.
The machines were too far away, so had the cable coming out at waist height, over the back of a chair, some disks and into the other machine. When someone complained, they wrapped it in black and yellow tape.
- What if someone trips on it?
- Who would want to do that? It sounds unpleasant.
"The machines were too far away, so had the cable coming out at waist height, over the back of a chair, some disks and into the other machine."
I remember a job where a phone guy was given EXPLICIT INSTRUCTIONS that a line was to be terminated no more than 2 metres from the router. He mounted the socket half way up the wall opposite the rack. He was about 6ft and if he stretched his arms out he could touch the router and the socket with his fingertips... job done!
(we found it was less hassle to specify the 2m gap for these jobs and order 5m cables to connect them)
I once worked, many years ago on a system, I think it called an FT 640. Filled a rack with monster hard drives of tens of megabytes and a tape drive. Ctrl-X in the shell shutdown the system. One Friday morning I managed to press Ctrl-X. Being the professional I am, I went for lunch. Came back to some really major panic that a fault tolerant (FT) system had shut itself down.
At a small firm I used to work for, the boss was away one week so we decided to play office cricket with of a paper+tape ball and a bat made from a broken chair back.
The game was more like rounders than cricket, but we're not girls or American, so we called it cricket.
Anyway, when the boss came back, he had a cunning plan - lets play office cricket.
For me, I was doing a guided tour of the computer room to some company bigwigs (both international and local).
The room housed a Digital C-350, an IBM 4341 and - the latest acquisition - an IBM S/38
While showing off the 8-inch floppy drive on the S/38, I leaned back to allow the assorted gentry to see things more clearly...
… and hit the EPO with my spine.
As someone else has already alluded - the resulting silence is deafening!
Much industrial plant, in general, will be littered with emergency stop buttons. The one I recall was a North Sea offshore oil production platform commissioned at the start of the '90s. The designers had placed emergency shutdown buttons at various strategic locations. These buttons were shrouded and were pointed out to all new arrivals on their induction tour. Their purpose was explained, along with a warning that they would only be needed by the core operating crew who were managing the plant - it was unlikely that others would find themselves in that position. (The full platform complement was around 150 people - the majority not having direct operational responsibilities).
One new contractor pipes up "Why? What would happen?" and hits the nearby button! This is followed by a number of loud bangs (emergency relief valves being triggered), the flare going into overdrive (safely bleeding down gas pressure throughout the plant), muster alarms sounding and the general activity of ~150 people going to their emergency stations, the majority mustering in survival suits ready to evacuate the platform... (Behind the scenes, signals would be going out to divert rescue vessels to the area, getting helicopters diverted, emergency support teams readied, etc.)
The emergency situation was quickly stood down and the platform back in production by the next day - and that new contractor was off the platform on the first helicopter available. I never knew the full cost of that incident but the lost production was probably in the low millions of $$.
On my offshore survival training they showed a picture of a new crew being transferred to the rig by a "witches hat" cargo net from a ship. They were actually taking this picture for use in safety training, when one of the new crew members, decided to hold on by one hand & wave at the camera for posterity.
Posterity was the right word as the OIM saw him do that & kicked him straight back off the rig for the 1-2 trip back to shore by boat, adding an amusing anecdote to tell the trainees & reinforce the Godlike status of the OIM.
That final anecdote in the article is remarkably similar to a recent (couple months ago) occurrence here. We have swipe cards to come in through the doors in our facility and a push button to get out of the main doors. The Father Ted / Father Dougal Award goes to the in-a-hurry delivery courier who dropped off a parcel at reception then rushed out, pressing the big square green alarm button by the main doors instead of the obviously-labelled Press To Exit button. Naturally when the fire alarm sounded he legged it as fast as possible, closely followed by the personnel evacuating the 9 storey complex.
Following a planned power off for electrical work, 06fartooearly on a Monday, I come in, fire up all the network kit, lights come on in the right order, everything goes green as it should, lovely. Go to the server cabinet, start them all up, all the usual whirry noises, fans, lights, etc. Fantastic. Then my ears pick up a regular beeping noise, one not normally heard. WTF?? While my half awake brain processes this, all the servers shut down and there is silence again. Mild panic ensues. Press power again, nothing. Beeping continues. Finally look at UPS at the bottom of the rack....oh-oh, red lights all round. No power. Phone electrician on call...…
After the inspection one of the RCDs hadn't been replaced correctly, so the entire rack had no power. All the servers fired up on UPS power, which then promptly shut them all down gracefully before it ran out of battery power.
Lesson learned, check everything for red lights. Even the stuff right at the bottom...….
UPS at the bottom of the rack
I recall hearing about how one of our clients at my last place who had a server that needed a "BRS reset" - ie it had hung and needed a power cycle. It was the bottom server in the rack and the helldesk guy casually told the customer to power off "the bottom server in the rack".
I think you can guess the rest, never assume people know the difference between "server" and "UPS" !
Back in mainframe operating days, my first tour of the machine room as a junior operator finished with being shown the emergency shutdown buttons which were right next to the emergency breathing apparatus for whoever hadn't made it out of the room before the doors dead-locked and the gas came down. There was only one set of the oxygen masks so presumably it would have got interesting if more than one operator failed to get out within the 30 second warning.
Most of my gainful employment has been with SME accounts systems, where the "server room" was at most a broom cupboard with an NT based box in it. But many of these companies were in manufacturing, so a walk round the shop would offer many opportunities to see the Big Red Buttons. However the only time I ever encountered one pressed intentionally was before all that, when I was a lowly apprentice at Ferranti, during stint in the machine shop when one of the other kids decided to faint while stood at a lathe. (Thankfully he didn't fall into it, but the instructor wasn't taking chances.)
1st, not me but a colleague, went to fix a multiplexer in the computer room (we didn't call them data centres back then) of a large catalogue firm which he duly did. He noticed a faulty power supply in another mux in the same rack so swapped it for a good one. Plugged it in and pressed the power button and BANG, everything went dark and the familiar whirring sound slowly went quiet.
2nd, same comms room, someone had lifted a floor tile and instead of laying it flat stood it on end next to the hole. Another engineer laid a power cable across the hole and inevitably the tile toppled and severed the cable, same BANG and result.
3rd, me this time, tidying up a real rat's nest of cables in the bottom of a rack one power cable was particularly tangled. It was coiled up with a cable tie. My mate said "cut it" so I did, again a massive BANG and a flash but this time just the rack went off. We decided the best course of action was to leg it. At this point he said "the cable tie, not the cable"...returned a few hours later to find another engineer on his hands and knees peering through the rack at the PDUs looking confused. I've still got the wire cutters with a piece missing out the blades.
In the computer room at college they decided to put in an elevator from the basement to the computer room to make it easier for the operators to bring up paper supplies etc. The workmen used a carbide cutter and lots of water to cut through the floor of the computer room. Sparks flew when they hit the main power cable for the computer room. Their reaction: keep cutting.
This one is legend around these parts and I was told about it as an apprentice nearly 40 years ago. A gang of telecomms engineers were trying to install a drop wire (overhead telephone cable) in a small village where the power was provided by three uninsulated cables strung between power poles. They needed to get the drop wire over the power cables and weren't sure how to do it. The T1 (bloke in charge who normally did nothing but bark instructions) said he'd sort it and grabbed the starting handle out of the cab (I said it was a long time ago), tied it to a piece of rope and lobbed it over the power cables. Unfortunately he didn't quite get the trajectory right and the starting handle started to rotate around the power cables bringing them closer and closer together until there was a huge explosion and no power to the village. The explosion also severed the rope and the starting handle ended up flying through a butcher's window covering his meat display in shards of glass. I was told it was true so it must be, happy days...
The building power was known to be on the edge of reliability and I wanted all the servers out of my office (!). So while looking at the circuit breakers I found the one labled "Board Room" was too hot to touch. Checked in board room and there was nothing even plugged in. So I pulled the breaker to see what would go out.
All the servers.
Seven hours later I had them back, and a few days later the MD's permission to have the building rewiring I had been requesting for months. And a separate, air-conditioned space for the servers.
She didn't fire me for that but for saying that Linux was the way to go for movie render farms not NT4.0 (yes that long ago).
One of them, of course, swiped him out – and told him that the green button he had been repeatedly pressing was, in fact, the emergency shutdown switch.
"The problem was soon addressed by changing it to a red button."
Why on earth would a shutdown switch be green? Who thought that was a good idea?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
