‘I crashed a rack full of servers with my butt’ Welcome for the third time to Who, me? The Register’s new column in which readers ‘fess up to messes of their own making. This week, meet “Digby” and his powerful rear. Digby once worked for a business that started life as a division of another company. He arrived just as the new outfit was getting to his feet and found “the …
I was doing some development on a test server and had not saved my work. An associated program would only run if "Turbo Mode" was turned off (this dropped the CPU clock rate down to match an original 8088 chip). Some tower PCs had the turbo button next to their push-button power switch. I pressed the turbo button to start the other program, but hit the power button by mistake, immediately experiencing the well known "How stupid am I?" feeling. I managed to keep the button in, so the power stayed on. Unfortunately, I had used my dominant right hand when I had bent down to touch the switch. Unable to see they keyboard, I managed to find the keys I needed to save my work, and then type in the shutdown command. Then I drank coffee and taped a cardboard flap over the power switch to stop the stupid person doing it again.
Edit: That is one reason why I really liked the large red power switch at the back of original IBM PCs, it was almost impossible to turn it off by mistake.
Hah, I played around with Smoothwall back then.
The tower case (which was the Smoothwall) was conveniently next to my work desk.
Daughter was crawling around all over the place, she saw the tall, white box with blinkenlights, and she did a gefingerpoken at the reset button.
Daddy was NOT amused :)
I disconnected the actual reset and power button cables from the motherboard, then set the motherboard's BIOS settings to turn itself on when power was restored.
Then it was simple - shut it down, it'll power off, and switch off at the wall. When I want to use it again, power on at the wall and the PC'll start up again.
Little fingers still poked the buttons afterwards, but nothing happened. :)
I've got you beat on the little fingers department. Way back when, my son was around 2 years old and the wife's computer was in the basement next to my lair of untidyness. Which she did not appreciate at all! Anyway, I had an old PC (gateway? HP?) where the case slid forward to add/remove stuff inside it. I had canibalized it for some parts and left it on the floor *mostly* closed up. Wife is working with the nipper playing under the desk. Then all hell breaks loose because he pushed in the power switch and the flap of spring steel caught his finger and wouldn't let go. Normally he could have pushed it all day until the cows (or I) came home. But the button popped out and he got trapped.
I got a hysterical call from her and since I was 30+ minutes away and she's not really the techie type, esp under pressure like this when the first born is wailing his head off... I told her to call the local police for help. They came, got the finger out without loss and I got a royal reaming when I showed up just after they had left.
Consequently, her computer was moved upstairs, and I had to do a major cleanup of my mess.
Now the kid is as tall as I am... memories.
and taped a cardboard flap over the power switch
This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs. I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee.
"I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."
There was a horizontal desktop PC case, monitor on top, that had the power switch on the front - low down and projecting. If you pushed the bulky keyboard too far back on the desk - it hit the switch.
"I think the worst I saw was a Tiny midi-tower which, when kept underneath the desk, could be accidentally switched off with a knee."
@Rich11
Much more recently that there is the Compaq 500B MT about 2010 protruding power switch on the side knee height. At least you can change the function of a soft power button to something other than shutdown.
Anon because they are we still our main PCs
a cardboard flap over the power switch
A "molly-guard". Don't they have the Jargon File where y'all work?
This was a standard part of the installation procedure for some types of cases through most of the 90s. There were some very poor designs.
Dell, a consistent producer of systems with crap design, had a number of these. I had a Dell tower system with a hardware reset button positioned right below the CD-ROM eject button.
"I managed to keep the button in, so the power stayed on"
I did this a few times, back when computers had Turbo buttons and switches were Real Switches instead of these noncy soft-switches they are today. The trick was to very quickly let go of the button, then instantly slam it back in again. Usually, enough residual power resided in the capacitors of the PSU to carry it through, and the computer would stay on.
My main PC is one of those with a power button on the top of the case, which is fine until the cat jumps up and lands her heel on the power switch before sitting down and turning things off with me in mid-work! A strategically placed book seems to have fixed the "problem" now.
I've done the "inadvertently press the power button, realise I'm an idiot" thing as well - I was supposed to press the CD-ROM eject. I didn't immediately release the button, and was able to reach a piece of paper that I folded one handed then wedge into side of the power button until I could cleanly shut the machine down.
Our work PCs have one of those nice setups with a bunch of USB ports, headphone and microphone jacks on the top edge of the case. Very convenient. Except the power button is directly below the headphone jacks so when you're poking around under the desk to get the cable in there's a decent chance you're going to have windows asking if you want to start saving things before shutting down.
Yep. Sounds all in order. Elementary commonsense: you don't leave an important switch where someone might accidentally hit it, regardless of their girth, clumsiness. Or indeed if they're wearing big loose clothes that swish. And in a server room, you *also* give good care to your cables. Hence some of those neat little inventions like recessed switches. Didn't you learn the principle when you were little and your parents told you not to put your glass right at the edge of the table?
Once again, the protagonist seems to be innocent. When are we going to get someone owning up to a proper f***up?
Or taking ownership of a new server room, finding that the door release button was right nest to the emergency power down button.
Despite pointing this was a bad idea to the customer, it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons.
When I worked in the server support team for my local health board about 25 years ago, an engineer was called out to fix a faulty open reel tape drive for a development server that sat on a table in the data centre. It was quite a heavy old bit of kit and the table was next to a wall, so space was fairly limited. The engineer picked up the drive and, apparently without thinking through the manoeuvre he was about to make, swivelled round and bumped the emergency power off button on the wall with the back of one of his hands. The data centre was then a hive of busy people, trying to bring up the ICL mainframe, comms kit and multitude of midrange servers that lived there. Once he realised the chaos that was unfolding, said engineer started to sweat profusely as he continued his task of repairing the rarely-used tape drive. Some time later, once services were restored and things reasonably back to normal, a wooden box was placed around the power off button, with a hole cut in it so that it could be pressed when actually required.
Similar situation happened to me at the Home Counties-based headquarters site of a well-known British aerospace company in the late 80s - we had to have the electricians in to do some work on one of the PDUs and, as they went to leave, one of them says "Is this the exit button?" and before we could yell "NOOOOO!!!!!!!!!" he pressed the big red emergency power off button...
We weren't sure whether we should have been relieved that we didn't need to spend the next few hours rebooting various mainframes (the IBM and VAXs wouldn't have been much of a problem, the biggest headache would have been the ICL 2966s...) or extremely p***ed-off that the Emergency Power Off - intended to save our lives if anything went wrong like someone connecting themselves to 415v AC, remember - did absolutely nothing.
Nada.
Not a gorram thing.
We went with p***-ed off, and the Data Centre Manager made sure scalps were collected from the muppets at fault...
"it was only after the second time the room was unexpectedly powered down did they agree to spend about £150 to move one of the buttons."
Vs about £6 for a flip cover?
http://www.boshisafety.com/a/products/4/2015/1015/275.html
Lots of other examples around.
Schnieder even make a transparent one: https://www.schneider-electric.ae/en/home/house-electrical-products/full-time-weatherproof-protection/weatherproof-socket-cover-transparent.jsp
How many times would I told an user to "keep on saving your work, F2 (Turbo Pascal's shortcut to save) will do it"... only to have said user ignore my advice blithely, carry on typing a beautiful and perfectly working program, only to have the PC crash.... (It was my experience with programs and PC's to keep on saving frequently, especially when doing Turbo Assembler stuff. Back then there was no virtual machines).
v2 of the same program was buggy and full of errors. Shame. NOT.
I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.
For most people it takes a while before paranoia becomes second nature. My friends laugh at me for having at least an idea of the potential "Plan B" (C,D,....) when I do things. They are learning the hard way....
From Schlock Mercenary:
Commander Kevyn: So we're supposed to evacuate the rest of them?
Commodore Tagon: Sure, that would be a great solution. Let's call that "Plan A."
Captain Murtaugh: Also known as "Ablative armor that probably won't protect Plan B."
Captain Tagon: Plan B always takes the bullet for Plan C. That's why the alphabet has more than three letters.
"They do have a plan B. You."
That's usually their Plan A.
It's like Cassandra and the usual Greek gods' attitudes. Gifted by the gods to warn everyone about a foretold disaster - fated to have to suffer the consequences with them. The Ancient Greeks knew human nature - or at least their story tellers and playwrights did.
I have lost count of the number of people to whom I have given PCs - and usb back up drives which they never used. Muggins was then expected to recover their data when they had a problem.
I can be smug here - I did buy the kids USB disks - and then made a royal pain in the arse of myself b buffing them to use them! Paid dividends in one (of three) who has evangelised ever since...
This post has been deleted by its author
"v2 of the same program was buggy and full of errors."
VBA in Excel 2010 has an interesting quirk. If you Ctrl-Z a few times to undo an edit - it regularly corrupts whole swathes of the module. It deletes unrelated chunks of statements in a totally different part of the module - which you only discover when you try to compile.
After any significant change I make copies of the project directory with a yyymmdd_hhmm suffix.
Using the same technique on VB.Net hit an unpredicted problem. Norton security*** regularly quarantines one of the project files that is a .dll type - because it is "new" and has no "reputation". You then have to do the arcane ritual to reinstate it and make it "safe" for the future. According to Norton support that only protects for that specific version's directory name.
This doesn't happen all the time. It was horrifying the first time to see it purging the e-sata back up disk of all instances of that file going back years.
***Norton works for me - and often cleans up people's disks who have used a different AV. YMMV
Me: Can I please have git installed on the Project Millstone server?
My Boss: What's git?
Me: It's a version control and software repository tool. Our vendors use it. I could use it to comply with your order to "stop using .old files to save stuff that gets changed, dammit!"
My Boss: No. Everyone must use the Microsoft version control tool we have.
Me: Okay. What's it called?
My Boss: Dunno. Microsoft something-or-other.
Me: Where's the documentation for it?
My Boss: Dunno. On a share somewhere. Maybe somewhere on sharepoint.
Me: Who owns it? Who's the SA?
My Boss: Dunno. One of the windows guys.
Me: Is there training for it? Can I have some please?
My Boss: No. The budget is all used up.
Time Passes In Month Chunks
Me: I've written my own version control. I run a script that just appends ".old" to everything of the same name before you edit.
My Boss: That's the opposite of what you were told!
Me: I know. I was faced with insurmountable difficulties in implementing your methodology.
My Boss: How in hell do you tell one so-called version from the rest? what happens if you need to reinstate a previous version?
Me: You figure out how many versions you need to go back and count the ".old"s.
My Boss: Whut?
Me: It works very well.
My Boss: It's stupid! And you've soft-linked this "script" to all the servers and filesystems, I'll bet.
Me: Of course not!
My Boss: Well that's something at least ...
Me: I simply copied the script into every directory on every unix computer we own.
My Boss: WHAT!
Me: Very quick and cheap, if a tad cheerful. Works though.
My Boss: Get rid of it!
Me: With pleasure. Can I have git please?
My Boss: No. Everyone has decided to use Toad's built-in version control. I don't know what it's called before you ask.
Me: I do. It's something called "git".
stretching their legs out under the desk was a frequent cause of problems as toes pulled out power leads, keyboard and mouse cables variously.I had told them that proper cantilever desks with cabling trays were required rather than reusing the old 4-leg plain top wooden tables that had been there since 1953.
Good diagnosis, bad solution. Yes, cabling should be kept clear of the meatware's legroom. Real-life computer desks get that horribly wrong by restricting the legroom: sadly I hadn't heard of "constructive dismissal" when I was forced out of office-based working by desks that forced me into postures that were incompatible with my back. The right solution is to keep the legroom but provide alternative safe spaces for cabling!
On a few days backfill work for a insurance co (When they were still called Norwich Union), a lady wrapped her leg around a power bar\cable & brought down a entire row of computers.
Having rectified the issue, she then went & did it again.
Thus prompting a better solution in cable management.
When I was at Uni, we had diskless Sun SLCs as our workstations. There were two SLCs which had massive 1GB HDDs to serve the other SLCs in the room. Unfortunately, it was very easy (by accident) to knock out the ethernet cable from the SLC with the HDD and bring the room to a standstill. (And I'm ignoring the joys of 10Base-2!)
Well if you're running your racks down a wall, then I suspect getting to the back of them may prove to be a little tricky as a starter for 10! (But in principle, I agree that L shaped configurations of racks in rooms is entirely plausible, feasible and indeed, practised in some areas.
As the rather clumsy protagonist responsible for this submission, I will call your BS claim as the story is indeed true and happened to me roughly 5 years ago. Perhaps a bit more detail would help you visualise things as there is the odd minor inaccuracy in the story as it happens:
The server room in question was a small, square room. As you enter, the main server rack is directly facing you (with a rackmount-capable UPS sitting next to it at the left of the rack). Turn 90 degrees to the left from the door and you have two more racks. The rack in the corner of the L shape was the comms rack and was facing directly towards the UPS and the side of the main server rack. Hence when patching phones in the comms rack, the UPS and main server rack were actually directly behind me (not next to me). Crappy diagram time:
< /\
<
Hence, crouch down in-front of the comms rack, arse protrudes out and neatly catches one of the trip switches at the top-rear of the UPS behind me. The cabling issue then came when I poked my head around the back of the rack to check the power cables running to our Oracle server and the movement of the cables dislodged the poorly-seated feed into one of the distribution units in the rack.
Not my cleverest moment, even if sensible cabling would have prevented an outage...
That's twice in a week or so that you proclaim something that happened to be impossible.
You have clearly never been at the coal face of hardware installation, upgrades and maintenance, so kindly keep your trap shut on those matters because it annoys those of us who do and have done so for decades.
I see what you did there. (I strap mine to my leg so it doesn't wag against anything important or delicate.)
You sound like a most considerate mog/mawg, most wouldn't have thought of such a thing! At least in any of the documentaries that I've studied anyway.
One client had the single server rack installed into a stupidly tight space in a corner. There was barely space to stand in front of the rack for the times when you needed console. And behind the user was a big fuse board with circuit breakers aplenty.
Apparently I wasn't the only one who managed to turn off several of those breakers with my posterior and thus turning off several office computers, lightning, and worst of all, the coffee machine. I felt pretty stupid for doing that.
You couldn't remove or install servers in the rack unless you carefully turned the rack at least 45 degrees or so. Didn't really help that the ethernet patch panels were built into the back of the same rack...
A customer had offices in a row of ancient houses several storeys high - with an internal labyrinth of interconnecting doorways, corridors, and stairs. One comms fixture was in a small old toilet room. The tall rack was backed up against the wall with the cables at the front.
To get to anything on the rear of the cabinets the lower section of the rack was empty. You sat on the floor and rested the back of your head against the wall in the small gap behind the rack. That was the day I found I would now need close-up spectacles to read some essential label just beyond my nose.
"A customer had offices in a row of ancient houses several storeys high - with an internal labyrinth of interconnecting doorways, corridors, and stairs. One comms fixture was in a small old toilet room. The tall rack was backed up against the wall with the cables at the front."
That sounds remarkably like a certain brewery in Tadcaster.
Apparently I wasn't the only one
Now we're into the territory of everyday life. Like when the meeting room is overcrowded and someone trips a switch just by squeezing in. Those light switches at around shoulder height by the door (elbow height or bum height work too), and sometimes switches that operate something more entertaining.
Indeed - we have a large, well-equipped meeting hall with powered window blinds, powered projector screens, various spot-lighting arrangements, etc ... all controlled by a cosmos of switch panels strategically placed at hip-height about the perimeter of the room ... rarely does an all-hands meeting pass without blinds, screens, and/or lights being triggered at intervals by folks leaning on the wall ... endless fun.
Should I say "bums and arses against the lightning!" now, or read down to see if anyone else has, or, not bother?
It is an obscure reference. And bums and arses are the same thing, anyway - I claim poetic licence. And it clearly should be "Bums and arses against the lighting", anyway.
And it may not even be funny. And yet aren't bums always funny?
Many years ago, when computers filled a room, we had lots of important visitors being shown around. I can't remember anything being powered down, but I do remember the "write protect" button on a washing machine sized disk drive (RP06 perhaps?) being pressed during one such visit.
Here are a couple of photos of our visitors. One had to be very circumspect at such times - no rushing into the room shouting "which blithering idiot?".
@Alan J. Wylie
Suggested caption for the second photo:
"Hahaha I've found the shutdown button!"
Saaaay...the terminal the bearded chap is typing on in photo #2 -- that wouldn't by any chance be a Tektronix storage tube graphics terminal (4010?), would it?
Wrote a graphics package for one of those in uni.
RK06? http://www.psych.usyd.edu.au/pdp-11/Images/11_60.jpeg
Buttons right out there on the front, easy to back in to.
// VIP visitors are always "fun"
Tektronix storage tube graphics terminal (4010?)
Yes - indeed it was.
[I] wrote a graphics package
In that case you might be interested to know that the man standing to Prince Philip's right in photo 1, in the striped suit, tie, finger pointing, is Peter Woodsford, author of GINO
Peter Woodsford - I thought for a second it was Dinsdale Landen as Doctor Judson (Doctor Who - The Curse of Fenric), which also featured early computer development as part of the plot to break the Enigma machine.
However I thought first initially & going for Herr Lipp, but Dinsdale seemed more appropriate.
Not me, but you know how old Shuttle PCs had sticky-out power and reset buttons?
I was typing, someone came to talk to me, etcetera.
Thankfully, Shuttle learned and later generations didn't have them.
... and noticed a youngster at the next pump getting out to go in to pay in advance. He left the car out of gear, and no parking brake. It started rolling back, towards my truck. I yelled, and lunged for his vehicle .... and caught my toe in the fuel line. Landed face first, with only the thought to get up and stop the car crashing into my truck, which I did. But not before the nozzle popped out of the tank unnoticed by me, and proceeded to pump diesel all over the station's forecourt. Failed auto-shutoff.
The attendant said that it was only about three gallons lost between the time he he noticed and managed to hit the emergency shut off, but have you ever spilled three gallons of diesel under pressure? What a freaking mess. That stuff gets EVERYWHERE! Fortunately, it's a small town, so we called the kids father ... who made him clean up the mess. In foggy weather the forecourt was slicker than owl shit for months afterwards.
And that's why in civilised countries we don't let the pumps operate without a person holding the valve open.
Oh, and we dispense fuel then go and pay - rather than guessing how much fuel might or might not be needed. Multiple trips in to 'pay' for fuel is a crazy way of doing things...
Ahhh, @John Robson, but that's why those civilised countries have a higher incidence of fuel theft and have to resort to ANPR camera usage to prevent that.
At least in this instance, it actually makes sense to pre-pay (although I've never liked this weird way of doing things). :-)
Many years ago I was doing a long run from Denmark to Luxembourg via a short stop in The Netherlands. In the middle of the night I discovered that Dutch petrol stations on major routes weren't open 24 hours.
The pre-payment machine only took a limited range of Dutch bank notes. I had only a few of them - and then discovered it only liked the couple that were in mint condition. Just enough to get home.
"and have to resort to ANPR camera usage to prevent that."
Many years ago I pulled into the pumps behind a car that was just finishing filling. I noticed that his rear number plate appeared to be covered with tape - and then he drove off at speed without paying. The till operator was immediately on the phone to the police. Apparently he always noted the front number plates of cars arriving late at night.
> Ahhh, @John Robson, but that's why those civilised countries have a higher incidence of fuel theft and have to resort to ANPR camera usage to prevent that.
I prefer pay at pump myself (at which point pre-auth is sufficiently easy to make sense - particularly since the 'refund' doesn't then need the card a second time) - but fuel theft is no different to theft in other shops (except that ANPR makes it really easy to ID the culprit).
prefer pay at pump myself (at which point pre-auth is sufficiently easy to make sense - particularly since the 'refund' doesn't then need the card a second time)
The newer way of Pay-at-pump systems (for card payment) make a lot more sense... no need to ever go and pay someone in person.
Pre-pay became compulsory a few years back in Alberta, after a gas station attendent was unfortunately killed after chasing non paying clients.
The latch to allow the nozzle to dispense gas without needing a human is bloody useful at -27C (Especially if driving a truck that takes approximately 120 litres).
> The latch to allow the nozzle to dispense gas without needing a human is bloody useful at -27C (Especially if driving a truck that takes approximately 120 litres).
So you're not dressing appropriately for the conditions?
Some issues which are resolved:
- Can't drive off with the hose still in.
- Remaining grounded so you can't ignite the vapour with a static discharge (and then pull the nozzle out in a panic and spray (now ignited) fuel over the forecourt...)
John, you are being a tw*t.
In Alberta -27C doesn't include the windchill, and no you won't be standing outside in any amount of "appropriate dress" in that because even breathing is uncomfortable if you've ever actually done it at those temps. Which you obviously never have.
The hose-in drive-off is still depressingly possible with a manual-only pump gun as five minute browsing YouTube will show. I'll grant it is slightly more likely with a pump latch.
And in "Civilized Countries" (you started it) the pumps are designed to recover the vapor (no "u" because we are being snotty) so there is nothing to ignite with the static electricity you feel will be coursing around the forecourt.
Why you feel that a person holding the metal trigger with gloves graded for -27C and wearing waterproof rubber-soled boots good for -27C would provide any sort of reliable ground anyway is a bit of a puzzler. Which I suppose could be why the pumps themselves are grounded. Long shot I know, but that sorta works for me.
What the flock are you talking about JR?
You either say "I want X amount of gas" or "I want X dollars worth" to the attendant (EXACTLY like it used to be in the UK back in the late 70s when "pay before you pump" was first initiated) or you use the almost ubiquitous pay-at-the-pump credit/debit card technique.
Where do the "multiple trips to pay" come in?
I've also brought down a full rack. A colleague and I were having trouble getting a network card on an older UPS communicating with the outside world. We used an incorrect serial cable to connect to the UPS. Now when a new UPS comes, the serial cable is tagged and paired with its owner.
The room gets extremely quiet when one rack goes dark! We still laugh about that day.
OK then, FINE. I *may* have brought down a big U-shaped robotic tape storage vault by feeding it a cassette which I'd just dropped on the floor. In all fairness, no-one actually told me why the vault had to be shut off the next day so someone could go inside and cut out a wad of tape which was jamming one of the readers.
A former colleague of mine, we'll call him Suhail,. because that was his name, did exactly this. We worked a kind of shift, ish, some of us would come in at 08:00hrs, so we had time to rectify any minor niggles before the muggles clocked on, and then others would work until 18:00hrs, so someone was available, allowing whoever was on call to get home. So muggins is staying until 18:00hrs, when said colleague called at 17:00hrs, and fesses up to knocking the power lead out of a disk array down in the DC, which for some reason only had one feed. He said he'd love to stay, but he had a pressing engagement (not the buttock against the kettle lead this time) and would I do him a solid and make sure everything came back OK. Luckily, it did.
Mumble years ago, we had a small department network (Netware 2.x), thin Ethernet[1]. This had been set up with floor ports under desks. Well, I say floor ports, but I mean angled plates in an inverted V formation sticking out of the floor, with a connector on each side, to which a cable was connected.
For reasons of not spending much money, the end users had what were known at the time as diskless workstations - PCs which booted their OS (DOS) from an image on the server. So, in the event of no network, these were slightly useless.
We had a user who, in his non-working hours, was a drummer. And he used to tap his feet a lot while sitting at his desk.
And one day, he managed to move his tapping a bit further under the desk and CRUNCH. One totally destroyed BNC connector and no network in his department....
[1] Coaxial cables joining one computer to the next with a terminator plug thingy at each end. Had the slight drawback that one broken or disconnected cable would drop the whole network
Ten years ago I ran a small infrastructure team for a large news/data organisation. We had a bunch of Sun and HP kit in a small comms room in St. Katherine's Dock. One day we get a fire alarm, so we all troop out then back in only for another one to kick off and we all troop out again.
Third time it happens, as we're heading back in my Unix guy calls me saying "Can you nip to the comms room, some of the boxes are reporting over temperature." You can no doubt imagine the conclusion I drew so my response was "Very funny" and I hung up.
He's back on the phone 30s later "No, I'm not taking the piss...". Turns out the outflow from the de-humidifier had blocked so the air con had shut down, or some similar cause. 30 odd boxes had to be shutdown to cries of "I have a demo" and "How can we develop" and me responding to the effect that they should go and get a coffee. Ot something similar related to travel.
Having said that, there was one evening I was manually patching a remote windows box and selected shutdown rather than reboot. It would have been OK but for the fact that box was in a remote data centre and we had no details of the rack. Or in fact an understanding of which of the many remote sites it happened to be in... Fortunately we narrowed it down to one, small, site and a local ops guy was kind enough to push the on button on the one HP DL 380 in the room that had no lights flashing on it.
Happy times.
I was supposed to be refreshing the test products database from our live database. Both databases were on the same host. I accidentally mixed up the database names and refreshed the live database with the test data. At least the backup system worked to restore the data when I realized what I'd done. Fortunately this was 15 years ago and most of our workload was batch with a few screens for data maintenance. If I did this now it would of affected online websites and whatnot. The databases were moved on to separate hosts soon after.
Another time I was coding a database trigger so that on delete of one record it would delete some child records in another table. Unfortunately I missed the where clause from the child delete and it emptied the entire table of records. Furthermore the tester didn't catch the issue (it had deleted the record they were expecting and they didn't notice that all the rest of them had gone also). However it was quickly noticed when the code ran on the live database for the first time...
Well, the worst situation I ever encountered wasn't in a purpose-built computer room. This was back in about 1983, and I had developed a system to log radar data from an airborne ice-penetrating radar for use in Svalbard. State of the art at the time - an S100 bus Z80-based single card computer! No OS, of course; everything had to be programmed in Assembler, and my development system was an Osborne 1 with Wordstar, asm and link - remember them? All was well in the office in Cambridge, but of course, it wasn't quite as we wanted it when we went off to the polar regions, so we took the development system with us. No problem - the Osborne 1 was (just about!) portable.
Unfortunately, the room we worked in had a nylon carpet, and of course, the humidity in the polar regions is extremely low. It was so bad that every time we brushed against a metal filing cabinet next to the door, we got a serious spark - enough to sting! It took a few goes for me to remember to touch metal BEFORE touching the keyboard of the Osborne 1; if you didn't the poor little thing just died. Fortunately, the whole setup was such that I always saved stuff (to floppy discs!) at regular intervals.
An unnamed BT/OpenReach engineer once used his belly to hit the emergency stop button in our server room.
He couldnt be bothered to use his hands to move his security pass from his belly to the pass scanner to get out of the room so belly flopped it onto the pass reader (and hit the emergency stop button located 30 cm's underneath)
Think of "Fat B$$stard from the Austin Power movies....
Been there. Done that.
Some years ago I was involved in an EU funded project to put learning centres throughout SE Scotland in various small business, libraries and learning establishments. This involved installing a server, workstations and (if required) structured cabling and network kit.
In one Uni in Edinburgh I was shown the cabinets in the server room and pointed to the space where the new server was to go. Except in this case read Dexion racking for cabinets and basement for server room.
Now, I am not a particularly tubby guy but I struggled to manhandle an IBM Netfinity server down the gap between the wall and the back of the "cabinets" as they hadn't left a huge amount of space. Part way down my journey along the wall I felt a click on my backside. Looking round, I saw that the geniuses had put a PDU with a honking great rocker switch on the back of the racking which I had managed to switch off with my arse, turning half of their kit off.
I pushed my backside against the switch again to turn things back on, dumped the server and left in a hurry. On my way out I heard one of the admins wondering what was going on with their DNS servers.
When the Sequent engineer came in to register all of the new SCSI disks we had spent a week installing and configuring. This was in the days when 4G SCSI disks were considered the best thing ever and we had a full rack of these, populated front and back. It ran the Customer Care database, the incident logging DB, and most important, the transaction log that we were required to keep by law.
cue the Sequent engineer kneeling down to read the lower disk IDs, over balancing, reaching out, and yes, he hit the main power contactor. "It was only off for a second", he told me, as the DBA's started wailing that their database had gone down. And then wouldn't come up because it seemed we still had the transaction log on the same disk array.
Cue backup taper recall from the safe store, reinstall, and oh? Where are the historical table spaces? It seems that the Sequent backup program for Oracle doesn't backup readonly table spaces at all.
Next thing happening:
1) Me , explaining to the regulator, why we couldn't fulfil their request for information we had to have;
2) The DBA's doing a repeat after me: "A backup you haven't tested is not a backup at all"
3) Sequent issuing a fix: namely a page to insert in to the manual set saying the readonly tables spaces don't backup.
Very expensive and painful lesson learnt, and we too put cardboard flaps over the very exposed main switches!
Installing a new phone system at a previous place and dislodged a wooden panel at the base of the server rack. Panel dropped into the floor space under the rack and managed to land perfectly on the single power switch hidden underneath the rack, resulting in the whole rack going down. Turns out the 2 large UPSes had been disconnected as well by a previous person so there was no UPS backup for the servers. Took 20 minutes before everything came back up again.
Many years ago when I was a system programmer we had a big red ermergency power off button on the wall by the exit. I was working on a communications box which was directly under the button and straightened up and hit the button with my shoulder shutting everything in the computer room down. The second time it happened I suggested that we cover the button with a plastic flip up cover, my boss threatened to fire me if it happened again instead. The third time it happened, I was in my cubicle and popped up and yelled "Not Me! I'm here!". The plastic cover got installed.
To save my repeating myself again here's one I posted earlier honed to its essentials.
A couple from me. One was my mistake, the others I just witnessed..
The one that was my mistake.
It was my first job after getting my degree. We had what we called a "server room" that was actually a bench occupying half my then office. It had a few assorted servers, including the company webserver and our NT domain PDC and BDCs.. All protected by a UPS (or so I was told).
One day, the boss was off. I had a CD walkman I bought in every day, I was doing work on one of the servers,so I plugged my CD walkman in to listen to it. Bang. The circuit breaker tripped, and the UPS lasted just long enough to shut everything down safely. When I queried it with our FM department, they said that the electrical circuit in question was up to it's limit, and my boss had been told there was too much equipment plugged in to it.
The second one was a company I worked at had a relatively new building, with a nice, riverside location (as nice as one can get in Woolwich, anyway). They spent a fortune cabling up the building for a new network and built a state of the art server room in the basement. The problem is, being on the bank of the Thames, the basement of the building was prone to flooding. They moved the server room a couple of years later.
The final one was when I left school. I worked for Network Southeast, in a local office, just doing general admin work. I heard a loud bang come from the lift next door. Had a look, and there was a rather large computer at an odd angle on the floor. I knew they were moving some offices around, and I figured it was just a delivery man dropping a PC. It wasn't. I heard several more bangs throughout the day, I queried it with my boss, he reported it. Apparently, he reported it, and found out that it was actually a removals company who'd been hired to move the servers in one room to another room.
In Africa - a long awaited mainframe memory upgrade was finally delivered from the UK. In those days that was a 6' (1.8m) high cabinet for all of 128KB.
The engineers opened the door - to see the internal rack frame now looking like a no-longer rectangular rhombus. A tell-tale mark near the top of the case showed where it had been abruptly stopped after falling sideways at some point in its journey.
"The engineers opened the door - to see the internal rack frame now looking like a no-longer rectangular rhombus."
Don't you mean parallelogram? The only way it could be a rhombus is if it had been a square originally (as both are defined as being equilateral but the square demands right angles).
I have previously refused to even enter a customers server room after seeing network and power cables running left, right, and centre across the space they expected me to go through to reach the server I needed to work on. Nope, yer not blaming me for downing your servers! Fix your cabling disaster first, then I'll fix your server. Came back 2 weeks later and it was tidy and accessible.
I was in the optician's trying to get a retina photo - nothing on the monitor. Cue much switching the monitor on & off etc. Eventually I looked where the cables were going, and found the PC mounted under the table cunningly positioned so my knee had hit the off switch. Meanwhile the wife was waiting outside thinking I'd got horrendous retina problems. Apparently it was all my fault...
It was the late 90's, and there was a known condition that would arise on one NetWare 4 server. The workaround, from Novell, was to cold-boot the box. So I roll into that data center at 3AM, still somewhat groggy despite a 75 minute drive, walk to the server, open the door and push in the power button. It was at that exact moment that I realized that my finger was on the button of a nearly identically named server (two transposed letters in the middle of the name). Needless to say, I was now fully awake. I then proceeded to do the one-handed-shutdown boogie from the KVM in the next rack.
Pull up chair and let me tell you tail of IT misery and woe.
This the story of the time and MCI data center went down.
During a routine inspections of their sprinkler systems they use a long pole to inspect them and see if the senor worked. Now some cheeped out and did not provide an cover the EPO button and you can guess what happened.
No biggie just turn the power back on. Only if it was that easy.
See kids this is were danger (side kick ) housed their servers. To make things worse this was an enhanced data center meaning the did voice and data.
Just when you think it could not get worse. This is were ATT and MCI did physical peering and was marked off line by a DNS update.
I had a colleague a bunch of years back working in a large Datacentre that was pulling in a bundle of Cat5 cables for a new install through the big floor tile trench they had opened & just as he got to the end of the pull the whole place fell silent ! Turns out his derrière had bumped into an emergency power shutdown button that did NT have a protective cover, Oops ! What me, nah run away :)
Timely article. This happened to me today. I have one of those stupid big Alien PCs at work with the stupidly big stupid alien head as the power button, which stupidly stands proud of the surface of the unit. My colleague once shut my PC down by kneeing the button as she rolled her chair over to my desk to ask me something. So I changed the power button to do nothing. Problem solved or so I thought. Today she managed to hard reset the PC by crossing her legs and resting her boot against the stupid button. Thank the $deity$ for autosave in Excel!
Spacing between bays, hmm.
Simple job, one company that i had a few days per month with regularly hired in stuff for sports or concerts and all the downstream odds, my job was plug it, test it and link it without taking anything running off air. ( or fail and no more work)
The video edit/severs had arrived, there was a vtr +extras loom in the bays in the racks, nothing in the rack behind me, empty bay just a loom. Concentration on job. GigE extras from the stuff for Hd to dangly lead later.
First server was at about knee height (26U up?), so I squatted down and felt a tug on the back of my belt and a sound of ripping, but from above, not behind.
A 6/8 port GigE switch on air with several other servers behind me, was tagged to me after parting from velcromount. Non-latching power from a wallwart, mains via an iec10 to Bs-13A not cable tied in.either.
FROZE!
I worked out the best sequence of movement to stand up with no pull on the cables and did it slowly until I could reach to put it all back. and did so. NOT funny. !!!
Logic says, hooks on the fixed mounts, furry on the wanderer so it is easier to clean and picks up less crud in transit, commonsense. It also stops it latching onto the back of your fleece.
There was a 3-ish metre cat 6 cable terminated in cat5 ends, hoods don`t fit connectors stressed and also this means thelatch hooks are out. and I had been caught. by one as sat down
I do not get why the bodge of cat6 .
10, even 20m of cat 5I work to GigE spec for all but counterfeit cables, Cat 6 3m of installation cable as regular pluggable/? as flexi as made of pasta and no cable grips possible, daft.
I`m losing mine slowly, but there is a world shortage of commonsense IMHO.
... one of the evening sessions at DECUS many years ago where one speaker lamented the fact that the controls for a certain model of disk drive (this was back when disk drives were washing machine sized) were the same height as his rear-end and, with a wallet in your back pocket, it was very easy to take disk drives offline by backing into the drive.
This is from a Rock and Roll Lighting perspective rather than IT, however we do use lots of IT kit these days. This happened one summer about 5 years ago, I had done a UK Stadium tour with a big Alternative Rock band, and during overnight programming sessions I had noticed all the APC UPS's dropping back to Battery power every time the Lighting Designer hit a big look - Lots of strobes and audience blinders - and I realized the sensitivity settings of the UPS's were too high for working on generators. I got the warehouse to send out a USB to serial interface and a serial cable and armed with the DOS based update program I went round all the equipment racks and changed the sensitivity to Low without having to turn anything off or disturb the Lighting Designers programming time.
A year later I was looking after the racks understage a large UK festival (big Egyptian shaped stage). I had noticed when loading in that there was one UPS that was too sensitive and had obviously not been on the tour the year before when I had reset all the others. Again I requested the necessary kit from the warehouse but it did not arrive until the last day of the festival. Undeterred, I had asked for the kit, so I was going to use it - and in my opinion ALL the UPS's the company owned should be set to Low Sensitivity by default as we often run on generators.
I go round the back of the rack, plug in the serial cable and fire up the software. No UPS detected, I try again, same result.
While I'm sat there pondering why this isn't working - I had no problems the year before - another member of crew comes running into underworld shouting that "ALL THE LIGHTS ARE OUT".
Panic Time - Slash is onstage above my head - there are 60,000 people on the other side of a black drape from me and they are looking at a band playing in a dim stage (thank god it wasn't properly dark)
I run round the front of the wall of mains distribution racks and the whole data rack I was working on is off, no reassuring blinky lights to be seen. I turn the UPS back on and a minute later everything is working again.
It was unfortunate that the Intercomm Masterstation which powers the headsets to talk to the LD at Front of House (and has big flashing strobes and sirens to attract attention) was in this data rack. It had taken theOperator several minutes to get hold of somebody at stage over a radio, amid all the noise, to come and find me.
It transpires that the serial cable needed is a 'special' one - with non-standard pinout - with the APC logo embossed on the connector. This time the warehouse had sent out a generic serial cable, liuterally the moment I plugged this cable into the back of the UPS the UPS had turned off - no beeps, no warning just off and I could not tell from the back.
I sent the serial cable back to the warehouse cut into lots of short pieces.
When I was working for a very large electrical engineering company, building motor control gear for a well known maritime organisation, I also caused mayhem with part of my anatomy. One of the units was in Test, and the testers were doing a heat run, running the equipment at full chat whilst the observers from the customer looked on. I had gone into Test to take some photographs for the Instruction Manual that I was preparing (I was in technical Manuals Department at the time) and I had to scrunch myself up into one corner of the roped-off area in order to get all of the cabinets in shot. Suddenly everything went dark, and the high pitched whine of the invertors wound down the scale to inaudibility. Cue furious shouts from the Test Engineers, I had inadvertently backed onto one of the emergency shutdown buttons that were located at various points around the department, and that had shut off all power to the Test area and surrounding parts of the building. A complete morning's heat run ruined, and the customer's observers were distinctly unimpressed. The heat run had to be rescheduled for the next morning as it had to start from cold. Needless to say, I was NOT allowed into Test whist a heat run was being performed on that or any further equipments.
A few years ago I was the sysadmin for a medium sized company. They had two racks in their server room, and had grown to the point where they needed a third. It was gonna be a bit of a tight squeeze, luckily they had the foresight to measure me up at the same time they where measuring up the room. So they could fit in their third rack, and still leave room for me to squeeze between them when I needed to. I joked at the time that if they had not done that, I would have had to swap jobs with one of the programmers, a tall skinny guy.
Much further back in time, when I was a programmer at some other company, I had noticed the hardware guy had a habit of powering up clients computers very quickly after powering them down when he needed to power cycle them. I warned him a few times that he should wait at least five seconds between off and on, which he ignored. Hey, I'm just the software guy, what do I know? Much to my amusement I had the joy of watching him do that to his own computer one day, with a loud bang and smoke just before my " told you so".
Just last week, I had reason to read a manufacturer number off of a part in a pedestal server... needed a flashlight.... oh hey, cell phone! I touched nothing in the case and got what I needed, but when I emerged from the server room/storage closet (small-ish business), I was hearing cries of despair, the main local repository of most of our company's work has suffered, um, Total Inability To Support (insert something funny, I'm not funny, and also American, so help me out here)! (a whee bit of exaggeration there)...
Reboot, everything was working again. Didn't think too much of it at the time. Fast forward to Monday, I go investigating another unrelated issue and notice that the RAID was missing two disks... Seems "something" caused one of the disk controllers in the box to glitch out and lock up, resulting in a soft lockup of the system, and software RAID6 coming back up with two disks missing... survivable, etc. The disks appeared to be fine, and it was clear a controller crap-out was the cause. Anyhow, I re-add the disks (which are all in perfect health), and while it's merrily rebuilding, the controller flakes out again... this time very noticeably, given the wall of crap on the console. After the reboot, the RAID6 is stuck in some silly state where two of the disks seem to have become.. independent .. of the other 4, and unfortunately, those two were part of the formerly working array...
Fast forward to today, we've cloned the disks, made backups, and forcibly reassembled the RAID, fsck'd the filesystems, etc., and the thing is back chugging along with a new disk controller, now with only 2 disks on each.
Could've been a lot worse, as we became acutely aware that our backup target had become completely full (I had been trying to keep up on that, but non-IT workload had become a bit excessive), and the last full backup had occurred some time in mid-December. Lessons learned. Getting fresh NAS hardware, expanding the offsite backup target, and revising our policies/procedures...
Root cause? Dunno, flaky Marvell onboard SATA controller (which had worked perfectly for several years prior) and possibly RFI from the cellphone... anyone here ever encountered that sort of thing?
edit: Actually, my father had suggested, in passing, the possibility of some sort of photosensitivity issue, a la the Raspberry Pis that could be crashed/rebooted with a camera flash. Having operated the thing for many years, and shone many flashlights in it, it seems unlikely... still an interesting thought
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
