Many computer rooms suffered similar accidents. Before the days of online systems, often the ops were the only ones who were aware of them. The simple solution was a cover for the big red power button. But the bigger danger was the fire alarm that dumped halon into the room - which took more time to recover from.
Wham, bam, thank you scram button: Now we have to go all MacGyver on the server room
Monday has arrived, and with it another tale to send administrators scurrying for their event logs, and engineers reaching for the coat hanger. Yes, it's Who, Me? "Jon", for that is not his name, was working for a now long-defunct IT giant back at the end of the 1990s. Fortunately for those depending on the big iron for …
COMMENTS
-
-
Monday 16th December 2019 13:15 GMT Venerable and Fragrant Wind of Change
Isn't molly-guard the usual term for such protection, when this story appears in this column?
The trains have had "in emergency break glass" hammers for as long as I can remember, but they're well-protected against accidental access.
I think my personal nearest to this anecdote is leaning on the wall in an overcrowded conference room.
-
Monday 16th December 2019 08:30 GMT Fabrizio
world-wide DC access
When I left a certain company, back in the beginning of the new millennium, HR had a list of company property they wanted me to return and when they were done with their list I asked them to add a single item to the list:
"Data centre access badges and keys". Just 3 access badges and 7 (or 8) keys opened all global data centres 24/7 and about 95% of all racks.
I got a phone call 2 weeks later by the VP of IT how the hell I amassed so many... Told him: solve lots of critical issues for more than 10 years and people will just tell you: "Keep them for next time you have to perform magic"
-
Monday 16th December 2019 10:00 GMT qwertyuiop
Re: world-wide DC access
In one of my jobs I was home-based, although I spent most of each week travelling to offices around the UK. The company did me proud, setting me up with a decent laptop, docking station, two monitors and a decent printer for my office at home.
On the day I left the organisation I duly returned all of my kit to the desktop support team. Their manager asked why I'd bothered to lug two monitors, a docking station and a printer 100 miles up the motorway to return them. I reminded him that they were company property. He told me that, with the exception of my laptop and mobile phone, he had no record of what kit I'd been issued so I could just have kept it!
When I pointed out that I was the company's InfoSec manager....
-
-
This post has been deleted by its author
-
Monday 16th December 2019 09:04 GMT ColinPa
Dont have your machine room at the top of a building
I was at a IT conference, and we were discussing horror stories. As it was in the bar, some of the details are hazy. One guy said that someone spotted smoke coming from a display, pressed the scram button, there was power down, and the halogen system was activated.
To reactivate the halogen system, they could not get the kit up in a lift, so they had to take out an exterior wall, and have a huge crane lift in some kit (Halogen tank?) to the 13 floor, rebuild the wall, and "clean" the machine room.
There was a 4 month outage all because some insulation on a screen was starting to smoke.
Not all details are accurate, but I hope you get the idea.
-
Monday 16th December 2019 09:41 GMT Chloe Cresswell
Re: Dont have your machine room at the top of a building
Makes me think when I was told about cam.ac.uk having a halon dump on a false alarm.
That's when they found there was plumbers grease in the halon pipework. Well, there was. It was atomised and deposited all over and in their new mainframe.
-
Monday 16th December 2019 12:06 GMT Vulch
Re: Dont have your machine room at the top of a building
In the early 80s the computer centre of what's now University of Bangor was moving from its home in the old MANWEB maintenance base to a shiny new purpose based building with lots of equally shiny new kit. Commissioning started with the aircon being fired up for the first time which proved its effectiveness by coating everything with a fine layer of cement dust from the open bag that had been left in a duct by the builders. I'd graduated by then, but several friends landed summer holiday jobs cleaning it out again.
-
Monday 16th December 2019 14:16 GMT Sequin
Re: Dont have your machine room at the top of a building
I worked in a government department in Bootle, just north of Liverpool. When the building was being put up (you can see it coming down here https://www.youtube.com/watch?v=z3i2baydpCM) it was involved in the longest strike in British history, and the shell was open to the elements for about 5 years.
After the strike ended, they started fitting out the building, putting in windows etc, but found that hundreds of local feral cats had moved in in the interim. Somebody decided it would be impossible to catch all of them, so they decided to seal up the building and pump it full of poison gas.
It proved impossible to find all of the corpses , so they just left them to decompose. We used to come in every day to find our desks and computers covered in dust and gritty stuff - the management tried to insist that it was just dust, but we all knew it was mummified cats!
-
Monday 16th December 2019 20:07 GMT TheFurryCircle
Re: Dont have your machine room at the top of a building
What actually happened is the cats all survived, and by their very nature as cats had to win against mere humans, and proceeded through several generations to dump asbestos through the vents as revenge. Should have just let them colonise the building, it's safer in the long run.
-
-
Tuesday 17th December 2019 00:29 GMT vir
Re: Dont have your machine room at the top of a building
We had central air in one of the houses I lived in, with a duct right over my bed. One night in the early summer, when we first turned on the A/C, I was reading a book in bed and noticed a tiny spider on the page. I looked over and the bed was coated with them; there was a nest in the duct that got blown out into my room. I'll take cement dust any day.
-
-
Monday 16th December 2019 17:11 GMT Stevie
Re: Dont have your machine room at the top of a building
"Plumbers grease?"
When I go a-plumbing I carry a tin box full of stuff including a blowtorch, squeezy flint lighter thingy, solder, emery cloth, wire brushes, a bunch of copper pip and fittings, a tin of flux and a brush for same in a little jar.
There is no grease. I'm not sure in what capacity it would ever be called for. You certainly wouldn't want it contaminating any lines you had just carefully soldered together.
-
Monday 16th December 2019 18:08 GMT DubyaG
Re: Dont have your machine room at the top of a building
Plumber's grease is used to lubricate various rubber parts like O-rings and seals in things like water filters and valves. There must have been some valves in the system that probably had tags on them that said, "Do not lubricate" and were duly ignored.
-
-
-
Monday 16th December 2019 10:07 GMT Adrian Harvey
Re: Dont have your machine room at the top of a building
I seriously hope it was Halon, not Halogen. Halogens (Flourine, Chlorine, Iodene, etc) are highly reactive and would be an “interesting” choice for a fire control system. Good for the BOFH though, very effective for wrapping up stories.
Icon may well appear on Halogen cylinders.
-
Monday 16th December 2019 10:47 GMT Anonymous Coward
Re: Dont have your machine room at the top of a building
You're not kidding. Iodine is solid, Bromine is liquid, so the only way you're going to have a cylinder of halogen is if you're dealing with Chlorine or Fluorine. Chlorine has been used as a war gas - it's nasty stuff. Fluorine isn't used as a war gas as it's far too dangerous to be chucked around like that...
-
Monday 16th December 2019 15:04 GMT DJO
Re: Dont have your machine room at the top of a building
Bromine is fun being one of the few elements that is liquid at room temperature.
When I was a nipper I knocked over a bottle of it but luckily it was in a fume cupboard so no problem, the fact that the fume cupboard discharged over the lower school playground and bromine vapour is a lot heavier than air really didn't concern us in the slightest.
-
Monday 16th December 2019 17:41 GMT Stoneshop
Re: Dont have your machine room at the top of a building
Fluorine isn't used as a war gas as it's far too dangerous to be chucked around like that...
That didn't stop the Germans trying out ClF3 in flamethrowers as it's hypergolic (also with asbestos, sand and test engineers) so you don't have to have an ignition source, but they actually considered it too nasty to work with. Which must have been quite a high bar to clear, considering what they used for powering the Me163.
-
Tuesday 17th December 2019 09:43 GMT Terje
Re: Dont have your machine room at the top of a building
80% hydrogen peroxide (the oxidizer used in the Me163) is a LOT safer and easier to handle compared to ClF3 and there is no way to extinguish it if it starts to burn, and then we have not even started on the lovely combustion products you are likely to get.
-
Tuesday 17th December 2019 15:13 GMT Alan Brown
Re: Dont have your machine room at the top of a building
80% peroxide..... that reminds me.
There was an incident on the M25 some years back involving a shipment of 85-90% peroxide on a lorry full of other stuff. Some might remember it for the reports of blue buckets strewn across the motorway.
http://news.bbc.co.uk/1/hi/england/london/4197500.stm
The peroxide in question was jetpack fuel...
-
Tuesday 17th December 2019 18:10 GMT Stoneshop
Re: Dont have your machine room at the top of a building
Still, both components are quite toxic and prone to cause explosions which is something you'd like to avoid if there's a test pilot with their bum more or less on top of 1500 liters of T-Stoff (the 80% peroxide), with 500 liters of C-Stoff (hydrazine hydrate plus methanol) uncomfortably close.
-
-
-
Wednesday 18th December 2019 19:38 GMT Anonymous Coward
Re: Dont have your machine room at the top of a building
Derek Lowe's article on FOOF is quite amusing, as usual. Quote from the paper he was quoting:
"When a drop of liquid 02F2 was added to liquid methane, cooled at 90°K., a white flame was produced instantaneously, which turned green upon further burning. When 0.2 (mL) of liquid 02F2 was added to 0.5 (mL) of liquid CH4 at 90°K., a violent explosion occurred.”
Note: 90 K is -183 °C / -298 °F.
https://blogs.sciencemag.org/pipeline/archives/2010/02/23/things_i_wont_work_with_dioxygen_difluoride
Of course, the aforementioned ClF3 is ugly, too: (hypergolic = autoigniting)
"It is also hypergolic with such things as cloth, wood, and test engineers, not to mention asbestos, sand, and water-with which it reacts explosively."
https://blogs.sciencemag.org/pipeline/archives/2008/02/26/sand_wont_save_you_this_time
-
-
-
-
-
-
-
Monday 16th December 2019 14:59 GMT stiine
Re: Dont have your machine room at the top of a building
Or you can be enterprising and use the plastic clamshell shipping containers to create a roof over the switch chassis that you located...under the trap in the hvac system above...
FYI, just because the ductwork is wrapped in insulation doesn't mean there isn't a hidden drain plug just above the optimal location for your expensive hardware.
-
Monday 16th December 2019 17:30 GMT TeeCee
Re: Dont have your machine room at the top of a building
The water tank for the building's fire suppression system is bad enough. When a seam splits on that you really know about it.
Also there's nothing like claiming for flood damage on the 1st floor to make an insurance company start asking difficult questions.
-
Monday 16th December 2019 20:23 GMT The Oncoming Scorn
Re: Dont have your machine room at the top of a building
Especially when it happens at a branch of Norwich Union......
One of my earliest contracting jobs & the managers moved printers around to try & get their documents out. (Didn't work, different subnets & IP's). Patching cables & inputting new IP addresses wasn't what I was there for, but by golly its what I did (for the most part) for 2 days.
I think they wanted me back two weeks later to do what I had been booked to do\take over the role on a more regular basis, but by then I had a longer contract elsewhere.
-
-
-
Tuesday 17th December 2019 22:31 GMT Jakester
Re: Dont have your machine room at the top of a building
I worked at a bank where the server room was on the bottom floor and the top floor (single-floor building with a flat roof). Unfortunately, roofers were not able to fix leaks in the roof for more than 2 years (only rains a few times a year). We had to have plastic sheeting over all equipment, supply shelves and desks when it rained, which made working and equipment ventilation difficult during periods of rain or melting snow.
-
-
Tuesday 17th December 2019 15:17 GMT Alan Brown
Re: Dont have your machine room at the top of a building
Not being under the main water tank for the building is also good advice.
And CHECK the build if you can, because even if the original spec is NO WATER PIPES IN THIS AREA WHATSOEVER, you may find the architect has ignored this requirement and every single piece of plumbing passes through the ceiling space of your data store (This happened. The people concerned "no longer work for the company", but that didn't get rid of the unwanted plumbing)
-
Tuesday 17th December 2019 18:21 GMT jake
Re: Dont have your machine room at the top of a building
"even if the original spec is NO WATER PIPES IN THIS AREA WHATSOEVER"
... That's when The Boss decides she needs an Executive Powder Room, and the only place the lowest bidder can figure out to run the fresh, grey and black water pipes is through the ceiling of the data center. Naturally, they all leaked at various times in the ensuing decade. Fortunately, my warranty was made null and void by this bone-headed mistake.
-
-
-
Tuesday 17th December 2019 15:05 GMT Alan Brown
Re: Dont have your machine room at the top of a building
"There was a 4 month outage all because some insulation on a screen was starting to smoke."
Yup. there's got to be some level of "sense" about when you want the gas system to fire.
Inergen systems are reputed to be about $5k a bottle, and that's apart from the other costs(*) - I tend to get quite paranoid about loose crap under the floor - especially drilling debris and insist that all areas worked on are thoroughly vacuumed afterwards. I've had contractors regularly gripe about me being anally retentive and how noone will ever see it, why does it matter, etc
(*) I think we've all seen the "don't shout at your drives" video
-
-
Monday 16th December 2019 09:16 GMT Korev
Some former colleagues had the opposite problem; they were setting up the PKI infrastructure in a "secure area" of the datacentre (effectively a large cage); they managed to get locked in there and had to wait for people to unlock the door to escape...
Probably what they looked like once they'd escaped -->
-
Monday 16th December 2019 09:17 GMT EVP
Not his fault really
You are supposed look around and be careful with your protruding body parts while wheeling a trolley or doing whatever you do in a production environment. Still, I would not blame “Jon” and his unnumbered peers for sudden and uncalled-for electricity losses. Blockhead who insisted placing the button right there where everbody and his butt passes by is to blame. That person, if anyone, should receive all the bollocking, not anyone else >:/
What comes to his actions to hide his deed and not to tell anyone... well, next time he knows not to scratch any buttons.
-
-
-
Tuesday 17th December 2019 15:24 GMT Alan Brown
Re: Not his fault really
"Each one of those datacenters was the source of one of those stories"
EVERY SINGLE DATACENTRE I have had to deal with has been installed with unprotected BRBs
and it usually takes stand up screaming arguments with the beancounters to get that sorted.
(locking them in the server room and pushing the gas discharge button is a last resort but oh so tempting)
-
-
Monday 16th December 2019 23:51 GMT dave 76
Re: Not his fault really
"I am amazed when I hear all these stories about unprotected master shut down buttons. I think I've only ever seen one such button without a cover, in dozen of datacenters I've been in."
In one building where I worked (back in the 80's), the computer room isolate was just *outside* the Ops room and was triggered at least twice by people thinking it was the button to get our attention. It certainly did that!
-
Wednesday 18th December 2019 07:55 GMT DanceMan
Re: unprotected master shut down buttons
A small arena used for rock shows I work in had a new led house light system installed. As a backup for the mini-controller hidden in a room under the seating there were two ordinary wall switches located in a vomitory* just above head height. They could instantly kill or bring up the house lights. Said vom also led to the only men's washroom in the building, They were enclosed in a box maybe a year later, thankfully without incident.
*Definition of vomitory
: an entrance piercing the banks of seats of a theater, amphitheater, or stadium
-
-
Monday 16th December 2019 20:26 GMT The Oncoming Scorn
Re: Not his fault really
"You are supposed look around and be careful with your protruding body parts while wheeling a trolley or doing whatever you do in a production environment. "
A friend of mine while transporting a butchers device at a trade show, managed to stick it in the wrong place during the move & amputated his finger.
Icon - Obvious.
-
Tuesday 17th December 2019 15:22 GMT Alan Brown
Re: Not his fault really
"You are supposed look around and be careful with your protruding body parts while wheeling a trolley"
Lesson one: If it can be whacked by a passing body part or trolley part, it WILL be whacked by a passing body part or trolley part. Plan accordingly.
Lesson two: If whacking said item will cause a claimable injury, then you will get a claim and your insurers will gripe at you. Again - plan accordingly.
Lesson three: Someone will ALWAYS go "what does this button do?" whilst pressing it, unless it is under a protective cover of some sort that can't be broken by simply walking into it (see #1) very clearly labelled what it does, how much pressing it will cost and that there's a camera monitoring it.
-
Tuesday 17th December 2019 18:10 GMT jake
Re: Not his fault really
"Lesson three: Someone will ALWAYS go "what does this button do?" whilst pressing it, especially if it is under a protective cover of some sort that can't be broken by simply walking into it (see #1) very clearly labelled what it does, how much pressing it will cost and that there's a camera monitoring it."
FTFY
-
-
-
Monday 16th December 2019 09:38 GMT KittenHuffer
Re: And this is why...
I have a titanium toothpick and a pliers multitool on my keyring as part of my EDC kit.
Recently a cow-orker was trying to open their desk drawers first thing in the morning because they'd forgotten their key. Depress the tumblers using the toothpick, apply torque using the screwdriver blade from the multitool, and jiggle! To their amazement I opened their desk on the first try, in less than 5 seconds!
Says a lot about the quality of the locks that the average desk is fitted with, and also a lot about how effective your average 'clean desk' policy is in a secure environment!
-
Monday 16th December 2019 09:53 GMT Pascal Monett
Quality ?
Their only quality is that they semi-prevent the drawer from being opened when it is "locked".
A while ago I was consulting in a company that worked with vast open space offices. We're talking a hundred people in the same room, working on a hundred desks. I was obviously told, on my first day, that the desk had to be clean every evening - papers (and valuables) were to be put in the drawers.
I settled in and my neighbor wasted no time in getting me up to speed on the clean desk policy. It would seem that it was common knowledge that there were only three different lock types, meaning the key on my drawer could open at least a third of all desks present. On top of that, he showed me a trick to get the top drawer open : some desks didn't have the desktop strongly fitted to the drawer section, and if you could lift it just a few millimeters, you could just slide the top drawer open like that.
I dutifully put my papers in the desk, but my personal belongings stayed with me for the duration of that contract.
-
Monday 16th December 2019 17:44 GMT Anonymous Coward
Re: And this is why...
Ha! They couldn't be bothered to find me a proper key for the desk I was assigned when I hired in. Instead, I found a key which was "almost" right, and would use it to "pick" the lock on my desk every morning and afternoon. The selected key had all of the tumbler pins low, except for the last one, which was high. So, it was just a matter of pushing it into, and pulling it out of, the lock, while holding a rotating pressure on it. When it cleared each tumbler, the rotating pressure would cause the tumbler to stick in the open position. Once the last tumbler had been strobed, the cylinder rotated, and the desk opened/locked.
-
Monday 16th December 2019 18:49 GMT Chris G
Re: And this is why...
For the average Eurolock, just push any key that will slide in to a point about two thirds in, apply torque to the key and give it a tap in with a hammery type object. The tumblers will bounce to neutral and the lock will open, takes a little practice but once you have the knack.....
Courtesy of Da Toobs!
That's why I spend a little more on my home security!
-
Thursday 19th December 2019 04:50 GMT Kiwi
Re: And this is why...
That's why I spend a little more on my home security!
If you're spending money on "home security" you've already been robbed.
Most burglars would just smash a window or door, don't give a shit about alarms, and don't worry about cameras or cops. Your lock is only as strong as the weakest part of your door or your windows. The alarms will annoy the neighbours who will sit inside griping about your alarm going off, few will ever look. The 'monitored alarm system' guards will phone and politely wait for the phone to be answered before taking action, and if someone does answer the phone and sounds like a drunken homeowner struggling to remember passwords then they'll give the burglar's mates plenty of time to act. That's if they bother responding in the first place, and then their nearest guard could be more than 30 minutes away. Cops won't be called till the guard says they're needed, and won't come unless there's some compelling reason to (a guy holding a gun to your kid's head probably isn't compelling enough!)
A good dog may be the better option. 3 good dogs will keep most pairs of burglars at bay. Really good dogs will have the perps ringing the cops themselves and begging for rescue (exceptionally good dogs will leave the perps completely unchewed yet psychologically messed up for life).
Oh.. And cameras.. They can be ripped out and re-sold to people who're foolish/inexperienced enough to think they make a difference.
(How I know - much comes from living in an area where a lot of places got hit, and watching defences not so much crumble as sublimate. Best thing we did was start our own neighbourhood walking tours at night, and a couple of planted press stories about how some of us were facing charges for being a little too rough with someone caught trying to break into a house (and a big reason why I don't trust the press, I know how easily stories can be planted ;) )
Best thing - watch the 'Adam ruins security' and use his main take-away - "don't worry".
-
-
Monday 16th December 2019 09:40 GMT MJB7
Re: And this is why...
I used to work for a manufacturer of Hardware Security Modules. The procedure for signing firmware was elaborate and involved hardware that was kept locked in a cage next to the meeting room with a hard drive kept in the safe in the accountants office. Some of the other signing procedures were less elaborate, and the associated hardware was less securely protected. The hard drives were kept in a locked filing cabinet in the main office (in open sight). Unfortunately there were seven people who needed access and only six keys.
Not a problem: the chief cryptographer also had an interest in physical security and would unlock the cabinet with a paper clip (about as quickly as those using a key). This had two major advantages: a) it saved having to get another key cut; b) it prevented anyone having any illusions about the security properties of the filing cabinet.
-
-
-
-
Monday 16th December 2019 14:58 GMT Anonymous Coward
Re: Isn't there an old adage that simply states 'locks keep honest people honest'.
That, and it means that if someone did get in, past the lock, (and were caught), that we can assume that it was very unlikely they are an honest person who opened the drawer/cabinet "by mistake" or "looking for a pen", or whatever other excuse might be proffered.
-
-
-
Monday 16th December 2019 17:52 GMT J. Cook
Re: And this is why...
... and if they are from a certain manufacture, they are all keyed the same. Every last one of them, unless you've replaced the cylinder, which is semi-nontrivial. (i.e., our in-house access control people managed to do it, although the replacement cylinder they fitted isn't *quite* the same as the stock one, and the stock one can't be re-keyed.)
There's a DEFCON presentation on the internet from a penetration testing company that lists the keys that he has that will get him into ~80-90% of buildings, enclosures, and at least one type of car.
-
-
Monday 16th December 2019 21:25 GMT Sgt_Oddball
Re: And this is why...
Modern Dacias are the same.
Got told by an auto lock Smith that in his home county they only had 7 keys and 3 immobiliser codes... he found this out after going back to his car drunk to sleep out off and drive in the morning, waking up to find the wrong CD in the stero before realising his car was 3 cars further down the street.
Got him a career out it in the end though.
-
Tuesday 17th December 2019 08:44 GMT Tim99
Re: And this is why...
I had a new light blue Volvo in the 80s (Yes, I know - It was an overreaction from the previous car, a somewhat unreliable FIAT). Mrs Tim99 and I stopped for a lavatory break, and an evening snack at a Little Chef (That dates it) in the depths of rural Lincolnshire. When we returned to the car, unlocked it, and I tried to start it, I realised that this car had a manual gearbox - Ours was an automatic. Our car was parked on the other side of a van parked next to the "wrong" Volvo.
-
-
-
Monday 16th December 2019 19:13 GMT JohnFen
Re: And this is why...
"Those rack locks are trivial to pick"
This is true. Even better is that somewhere north of 80% of all of those racks take one of four different standard keys (all of which are readily available from the rack manufacturers and/or eBay). The expectation is that the customer will replace the locks with something else if they care, but most don't care.
-
Tuesday 17th December 2019 19:35 GMT spold
Re: And this is why...
I travel a lot, so I also carry a lock-pick set, but also a Leatherman Wingman multi-tool, not only deals with all sorts of work crap but it fixes a myriad of things in hotel rooms that would take the hotel an age to get around to fixing, it also has a nice blade I've sharpened that opens equipment boxes in a jiffy.... However I'm on number 6 now as I've way to frequently got to the airport security line and gone to turf out the pockets... oh fuck forgot to put this is back in the suitcase...
-
-
Monday 16th December 2019 09:28 GMT OzBob
Yep, had a very dodgy HP tape array in the noughties
that was only about 8 u high (and was double-stacked). The door release mech never worked properly when pressing the LCD display, so we just used to stick a half-unfolded paper clip into the appropriate hole and release the damn thing manually. Actually taped it to the front on a piece of string so we did not need to recreate it if we lost the old one.
-
Monday 16th December 2019 09:50 GMT GlenP
Re: Yep, had a very dodgy HP tape array in the noughties
a half-unfolded paper clip
Isn't that standard equipment for all IT departments? I've certainly got one in my desk drawer!
Main use has been for retrieving CDs/DVDs that have been left in drives and the system powered down of course.
-
-
Tuesday 17th December 2019 05:37 GMT swm
Re: Yep, had a very dodgy HP tape array in the noughties
In 1965 we were working on a DN-30 with bit buffers that connected to teletypes. The field engineer installed everything but didn't put jumpers in a block to connect the bit buffer to the dataset. It was late at night so I got a bunch of paper clips and started bridging various connections (using a volt meter to find some useful voltage sources). The whole mess was sort of fragile as none of the paper clips could touch. The field engineer came in the next morning and took one look and pulled out all of the paper clips and used proper jumpers etc. It didn't work so we had a discussion to get it right.
-
-
Monday 16th December 2019 13:31 GMT Andytug
Re: Yep, had a very dodgy HP tape array in the noughties
Someone should do an appendix to the BOFH manual, on the documented 101 uses for a bent paper clip.
Mine: to remove 2FA smart cards inserted into the gap between motherboard and CD drive, conveniently placed 3mm below the actual smart card slot (Thank you HP for that inspired piece of design!).
-
Monday 16th December 2019 17:25 GMT Doctor Syntax
HP Tape & bent paperclips
We had a 1/2" reel HP drive. Periodically its loading mechanism would get into a state where it failed to work - much blowing of tape into the slot and re-winding. The solution was to reset it by shorting two pads on a circuit board. The spacing was just right for a bent paper-clip.
-
Monday 16th December 2019 20:33 GMT The Oncoming Scorn
Re: HP Tape & bent paperclips
I have used a paperclip in the reassembly of the paperfeed\seperation* assembly when reassembling HP LJ4000 - 4100 series printers.
*I don't recall exactly what the assembly was after all these years, but there were two nasty springs under tension that needed to be in place during reassembly.
-
-
-
Monday 16th December 2019 09:49 GMT jake
Coathangers are handy.
Here's an old post of mine from 2009, originally titled "That's just the switch, they do that!" ...
The scene: Old house in Mountain View, converted to Vet clinic.
The time: Late 1999.
The job: Convert the Vet's database from non-Y2K compliant PSI/Idexx on HP/UX to Cornerstone on Win98 (I know, I know, but that was what the Vet wanted).
Considering that Cornerstone included the database conversion in the cost of the 8 hour staff training session, my job was basically setting up the Windows boxes, installing software, and pulling wire. Easy. In fact, I did the complete network setup back in my lab, so all I had to do was make space for the computers, printers, label makers etc., plug it all in, and turn it on.
Unfortunately, the house was built in the post-war building boom, and originally had 2 prong plugs for mains power. Some had been converted to three prong, but not all. I had to convert the six locations where the Vet wanted the computers installed. Three new breakers, pull some wire ... Again, pretty easy.
Day of install, the Saturday afternoon before the Sunday the Cornerstone rep was to hold the staff training session. The Vet wanted me to plug a scanner into her personal machine. The floppies that came with the scanner didn't come with Win98 compatible drivers, so I used the new-fangled V.92 modem to dial into my ISP ... But the connection speed reported as 2400 baud instead of the expected 56K (or so). I broke the connection, redialed, same result. And again. Now, I had already used the modem from my lab, just to verify it worked, so I knew it wasn't my hardware ... I picked up the phone. Line noise. 60hz line noise, to be precise.
I turned to the Vet and asked how long the phones had been buzzing ... she brightly answered that "That's just the switch, they do that!" ... WTF? After further questioning, it turned out that instead of having six phone lines coming in, she had a small switch with six extensions. It was installed when she took over the practice about a decade earlier. (You can probably see where this is going ...).
She lead me to a storage and washing machine room in the back, and pointed at the wall. Hanging there was a dusty, cobweb covered, slightly sad looking $TELCO supplied switch. It had a three-prong cord ... plugged into a two prong socket. The $TELCO tech who did the install had cut the ground pin off the power cord to fit the availabe mains power. I asked her to go get me a cordless 'phone and grabbed a wire coathanger. With phone to ear, I grounded the case of the switch to the cold water pipe with the coathanger ... No more AC hum.
I explained the situation to her, and she went ballistic. After she calmed down a little, I asked if she wanted me to call $TELCO for her. (I could have put a new cord on it myself, and put in another 3-prong socket, but if you touch $TELCO gear, they will refuse to fix it if anything goes wrong in the future.) On further questioning, she allowed as to how in the last decade she had had techs out half a dozen times or so to service her telephones, including what sounded like a firmware upgrade to the switch itself.
Long and short of it, we had a $TELCO tech out 45 minutes after I hung up the phone.
-
-
This post has been deleted by its author
-
Wednesday 18th December 2019 00:13 GMT Anonymous Coward
Re: Coathangers are handy.
Brits can only dream of that kind of response from Openreach.
They do it all the time. However, if you're worth £3 a month to them, safety is not an issue and there is no SLA in place you're not exactly front of the queue. Among the grown ups give your service provider a call and they're generally contractually required to be on site within two hours of the referral. The service is there for whom it matters.
-
-
Monday 16th December 2019 11:25 GMT Franco
This is why for years I've been collecting rack and server keys. New racks invariably arrive with lots of keys, which also invariably are kept in a locked cabinet that no one has access to. More than once I've driven an hour to a customer's site to find that the server bezel or cabinet is locked and the only person with the key is the MD who's gone home because the server has crashed and needs a manual reboot (I.e. a hold of the power button or a yank of the cable)
Thankfully much less common now with DRACs, iLOs etc but the keys are still in my bag.
-
-
Monday 16th December 2019 18:50 GMT Nunyabiznes
At A.P.
You would think. I've spent a lot of time on the clock cooling my heels waiting for a key person to come back to the mess they made and then left without a care. Multiple people at my current org have done this. They are all still employed at considerably more base rate than I. Sometimes I have a problem with speaking truth when silence would have been a better choice - so naturally I have a much bigger HR file than any of said people.
-
Monday 16th December 2019 20:23 GMT Kernel
"I've spent a lot of time on the clock cooling my heels waiting for a key person to come back to the mess they made and then left without a care."
Yep - I once spent about 4 hours wandering around a beach at the customer's expense, while waiting for a fibre jointer to turn up to pull in and splice another pigtail on the line fibres of a DWDM system. Due to the analogue nature of the line optics of DWDM systems commissioning has to proceed in sequence from A to Z and then Z to A, it's not possible to miss a site and go back to it later.
The one they'd put in originally was brand new, but had a massive scratch across the core face at the connector which caused it to be rejected by the test software on my inspection scope. Supposedly the original pigtail had been inspected (as per documented work procedures) before being installed.
-
-
Tuesday 17th December 2019 03:25 GMT Unicornpiss
Keys
"This is why for years I've been collecting rack and server keys"
I do the same. I had accumulated a whole box of them at one point, which pretty much covered every situation.. Until.. someone "helped" us clean our messy IT room one weekend and threw away the whole box of them. Which had me angrier I think than all the other little atrocities committed when someone with no clue about your day-to-day work "helps out" This same moron was throwing away brand new CAT-6 cables still in the bags because "too much clutter".
It brightened my day when I heard he had been let go some months later--he apparently couldn't do his job either, much less ours.
-
Tuesday 17th December 2019 16:04 GMT Nunyabiznes
Re: Keys
Let me guess: Facilities guy? Ours insist on having unrestricted access to our data closets and centers.
One day I came in to find a contractor, not escorted, busily installing a (home) ac unit in one of our centers to alleviate the heating issues we were having. The facilities boss insisted we didn't know what we were talking about when he spec'd the original ac unit, and now that was a failure he raided our budget to install a home unit with an el cheapo contractor. Lovely seeing sawdust floating towards the intakes of your equipment. Even better when you realize he has his equipment plugged into your rack UPS.
-
-
-
Monday 16th December 2019 16:53 GMT Marty McFly
Helping out...
Same solution, different story....
I came across a hapless soul who had lost her car keys while swimming in the lake. After spending two hours wandering the shallows looking for keys, I managed to unlock her car with a couple sticks and duct tape. She had left a window down a bit (out swimming, so it was a hot day). Taped the sticks together, slid them through the window, and bumped the unlock button on the other side. She could now get her cell phone and phone a friend to bring the spare key.
Granted had she known their phone number, instead of relying on the iThingy to remember it, she could have simply called them in the first place.
-
Monday 16th December 2019 22:04 GMT John Brown (no body)
Re: Helping out...
"Granted had she known their phone number, instead of relying on the iThingy to remember it, she could have simply called them in the first place."
To be fair, that is so ubiquitous nowadays that I don't blame here for that. I think the only two numbers I know from memory are my own landline and mobile numbers these days. I used to have a veritable phone book stored in my brain, but there's almost zero need for that these days.
-
-
Tuesday 17th December 2019 03:42 GMT doublelayer
Re: Helping out...
But you probably give that number to people or enter it on forms from time to time. If not, you might not need that number. My mental phone book is sadly pathetic. I can give you the phone number of friends I had in 2007 when I placed calls by manually dialing on a land line, but people I've met since then, including those I call or send messages to frequently, are only known by my contacts database. I have that backed up, but maybe I should try memorizing the numbers.
-
-
-
-
Monday 16th December 2019 17:54 GMT Dave 32
Big Red Button
In our case, it was the contractor in charge of changing the burned out light bulbs who caused the first incident, when he sat his ladder next to the door on the way out. One of the rungs hit the Big Red Button perfectly. Ker-CHUNK! Whoopsie.
So, to prevent that from ever happening again, another contractor was hired to put a shield around the Big Red Button. The first thing this contractor did was to bring his ladder into the room and lean it up against the wall by the door. Ker-CHUNK! Whoopsie!
-
Monday 16th December 2019 20:03 GMT PassingStrange
SCRAM
Back in the early 70s the "perceived wisdom" was that SCRAM stood for "Start Cutting Right Away Man" (vide the instructions for the Atari game of the same name). In reality, I suspect that there's a 99.9% chance that the name came first, and every "explanation" is, basically, a backronym..
-
Tuesday 17th December 2019 11:22 GMT PassingStrange
Re: SCRAM
(Or, possibly, "Start Chopping..." - memory is fallible, and it's been nearly half a century. But I agree with the the article linked to below by @Jake - the slang term "scram" seems by far the most likely origin. It was certainly sufficiently in use around the time to have even reached the UK; my father - UK career soldier - used it regularly, when I was under his feet.)
-
-
Monday 16th December 2019 20:29 GMT jake
Etymology of "the scram button", according to the U.S. NRC.
My daughter just sent me this link:
https://public-blog.nrc-gateway.gov/2011/05/17/putting-the-axe-to-the-scram-myth/
And now you know.
-
Monday 16th December 2019 20:50 GMT jake
Scram button, revisited.
As a side note to my fellow IT techs ... If you ever come across a drag racing advocate in your travels, don't assume that the "scram" button, often on the steering wheel, means "power down all systems". In fact, it means quite the opposite, and usually adds between 10 and 15% more power to the motor for a last-second improvement in ET and MPH. It's basically a press to pass button, and can activate one or more of several different methods of pushing a motor past it's normal limits for a second or two.
It's short for "scramble", which some racers call it.
The "All systems off" switch is located at the rear of the car where track personnel can access it in an emergency, and is usually well marked.
-
Monday 16th December 2019 21:34 GMT Nunyabiznes
Re: Scram button, revisited.
I put another master switch within reach of me when I'm belted in. One switch cuts ground, one switch cuts positive. I race off road and sometimes you are your own safety crew. Being able to cut power and release your own fire extinguisher (same for co-driver) may be all that lies between you and a bad day.
You have to make sure the alternator charge line is included in the positive master switch. Luckily, I found out my very basic mistake (late night wiring before the race) during testing and not when it counted.
I've never had a "scamble" button. Would that be like the WW2 fighter planes that had "emergency" power levels available for short periods? I've used nitrous in drag cars, but that's a portion of total power output and is in use every run, subject to tuning.
-
Monday 16th December 2019 22:31 GMT jake
Re: Scram button, revisited.
I usually have a kill switch and fire suppression within arms reach, too. Prudent. The switch itself isn't protected by a molly-guard, but the arming switch for the fire suppression is.
Yes, just like the aircraft emergency power button, but often to the point of destroying the motor just as you cross the finish line. It can activate another level of nitrous (dry or wet), increase the redline (if you normally run a rev limiter), tell the wastegate to increase the boost, adjust the timing, or more usually some combination. It is normally only used in the semi finals or the finals because of the chances of doing internal damage to the engine.
-
-
-
Monday 16th December 2019 23:09 GMT Anne Hunny Mouse
Knew of a UK Gov related data centre which regularly had accidental shutdowns as the shutdown button was where you would expect door release to be.... In addition it was the data centre without resilience as they were too tight to pay for it.
On another note, recently picked the locks on some Telecoms NTE cabinets that nobody had keys to.
One was in the process of being removed, the other needed the coaxes switching and hadn't seen an engineer for years
-
Tuesday 17th December 2019 03:12 GMT Unicornpiss
Similar, but different story..
I didn't take down a server room, but I did find myself in a large, noisy dyno room, standing on top of a control room that I had to reach by climbing a 15-foot ladder. I had inadvertently taken down a locked rack of switch gear and one fairly critical server that was located up there. The equipment was not accessible and I had no idea who had the key, but I suspected it was in the other building down the road, held by a different team. It was too noisy up there to even make some phone calls to try and find out. At the time I was a contractor with less than a year in with the company and the lowest on the totem pole. I didn't think I would get fired, or even yelled at too badly, but the event wasn't going to do much to improve the general opinion of me either.
I stood up there for a while, thoroughly discouraged, listening to the almost painful roar of the screaming dynamometer, feeling my head pounding, and having had a long, fairly rotten day already, didn't have the energy to climb back up there again, much less go on a snipe hunt for the key. Fortunately locks on equipment cabinets are not all they could be in many cases, especially with cheap cabinets. I spotted a snipped off bit of zip tie in the grit where I was standing, and with a little wiggling was able to open the simple wafer lock, start everything back up, and relock the cabinet just like it was.
A half hour later our Helpdesk, consistently a day late and a dollar short, called to inform me of an outage with the equipment, as I was driving home. I told them it had already been taken care of. Beer icon for obvious reasons after a day like that.