Sysadmin bloodied by icicle that overheated airport data centre Why look at that: the calendar says “Friday” which means it's time for another edition of On-Call, The Register's weekly column in which we relate readers' recollections of being called out to fix nasty things under nastier circumstances. This week, meet “Alan” who “used to work for a rather large airport company in England …
It takes a miracle of biblical proportions to make the core drill guy responsible for the install drill at an angle so it self-drains.
In my own home office: "See these are the install instructions - you have to drill at 5 degree angle to ensure that water drains correctly out of the condenser". 2h later, the retard proud, gleaming and explaining with extreme pleasure about how he used the spirit level on his drill to drill it exactly level.
I just sighed, paid him up and used a smaller pipe and some filler so it goes at an angle.
I worked in an out of town office where the numpty that installed the liquid run off pipe (soakaway) from the septic tank was carefully angled up hill. Apparently the installer didn't understand that liquids tend not to flow uphill. The solution was that we had to have the tank emptied regularly otherwise the surface of the car park became rather less than pleasant.
None of this was helped by equally numpty cleaners who, despite being told repeatedly, continued to pour inappropriate cleaning chemicals down the drains.
This post has been deleted by its author
Our system has an uphill outlet but it is pumped to the top of the drainage area which is downward. If that pump trips or the float switch gets caught (which happened after initial install until the cable was shortened and rerouted) you'd get a day or two max before the alarm would trip.
It would make it impossible to live with if the run off wasn't working, the frequency of the tanker would be thrice weekly and cost a fecking fortune
(apart from my typos), I'm afraid that it's a genuine story of an office in Hertfordsire.
The tank was definitely not small and the run off was continuous but as the car park was at angle so when the damn thing did overflow it did at least run off the carpark. Don't forget that as the tank was set underground, the overflow that should have been a last resort was somewhat higher than the top of the tank and therefore much of the runoff seeped straight back down into the ground, it's when the ground was too soaked or otherwise impermeable that problems happened.
The tank was emptied pretty much every two weeks. I beleve that the shit sucking people were paid per visit and they were happy enough (well, as happy as you can be working in such things). The land was probably owned by a member of the council anyway...
This post has been deleted by its author
I'll see your cess-pits and raise you a pig-farm slurry pit.
There were a series of reports to the RUC that a couple had been killed and buried, the theme being that the bodies were dug up and moved. Nobody really believed it but it couldn't be entirely dismissed. (Have I ever mentioned the golf ball episode?)
Anyway, one of the variations was that the bodies had been dumped in a slurry pit in a pig farm. This had to be pumped out so that it could be examined. Even empty it would be done by the underwater team with dry suits and breathing apparatus.
The pit had been used for dumping carcasses of dead pigs and the pump was quite capable of drawing up and discharging bones. Somebody who was considered able to tell the difference between a pig bone and a human bone but cheaper than a regular pathologist had to spend a couple of days of an Irish winter standing at the outlet of the pump checking what came out. That was me. I suppose I should have been concerned about where the effluent was draining to. I wasn't
The subsequent examination was negative and, I heard, very brief. I didn't see that for myself. I'd already left.
This post has been deleted by its author
I know the term 'cesspool' which is probably the same in practice and meaning as cesspit.
We were renting a house (rural community), built but not finished but the price was right.
It was beginning winter but there was still some of the original summer garden growing
which was a gift, given that it was over 50 miles to a decent store with veggies.
There appeared to be a piece of plywood on some of the land (without plants) and I thought
to move it and see if the soil could be worked and seeded with peas for the spring.
What I saw was....undigested (not enough bacterial activity to work on waste) sewage. There had been enough activity to keep it warm (hence the growing plants).
I realized where the seeds for the plants had come from and became sick immediately.
We had not been told anything about this situation and the owner said we should have asked.
He had hoped to sell the property to us. The place stayed vacant until he got a proper permit
from the county and installed a septic system.
What that entailed in a rural area was a large tank which would contain the waste, digest it, and the fluid which rose to the top would flow into a leach field where it would percolate into the surrounding area.
The house was 40 miles from the county seat (NOT the area with the good stores) so the owner/builder had thought he'd get by doing things himself and sell it before someone found out.
In California there is the 'grandfather' rule which is that what was acceptable at the time of construction would continue to be acceptable until alterations or additions were made.
Consequently, those in the outlying areas knowing that building inspectors did not leave the county seat looking for code violations (often off the grid and on poorly maintained forest service roads) so that if violations were found, the owner would claim that it was grandfathered in. Used plumbing & electrical parts were in demand, often sold at flea markets or scavenged from the local dump.
And the cesspools/cesspits? Since there had been more than adequate land available when the original houses had been built (acres), after a period of time, another pit would be dug, the pipes rerouted, the pit topped off with stones, wood waste, etc. and a tree planted in the middle of it.
If you saw a magnificent tree where other plantings weren't so large or healthy, you knew what was there before.
Even way back then (well before our time there) people had understood the grey-water/brown-water
split so a leach field for grey water was only noted so the soil wasn't disturbed.
The Irish have an expression about 'bad cess to you' (ill-wishing someone). Makes me think they might have used the term 'cess' for a pit or pool as well.
"The council in England where I live have something called 'building inspectors'. These folks make sure that everything meets the building control regulations."
Really? In 2017?
In most parts of the UK that I'm aware of, "building regulations" inspection has been outsourced, often to a subsidiary (or friend) of the builders. Bonfire of the red tape and all that. After all, what could possibly go wrong.
Recently constructed PFI-financed Edinburgh and nearby Scottish schools found out what could go wrong a few months ago, fortunately without real casualties:
http://www.edinburgh.gov.uk/news/article/2245/independent_report_into_school_closures_published
"An independent report into the closure of 17 Edinburgh schools has been published today (9 February 2017).
The School Closures Report (PDF 3.8MB) was commissioned by Chief Executive Andrew Kerr last summer after the collapse of a wall at Oxgangs Primary School and subsequent closure of 17 schools.
It was led by respected construction and procurement industry expert, Professor John Cole CBE, who will present his report at a special City of Edinburgh Council meeting today.
[snip]
Some of the key findings from the report, which runs to over 250 pages, include:
* the collapse of the wall was due to poor construction and inadequate supervision;
* insufficient independent quality assurance and poor record keeping by the Council and ESP;
* ineffective quality assurance measures within the construction industry;
* the Council made the correct decision to close the 17 schools;
* the alternative education arrangements put in place for over 8,300 pupils was a ‘remarkable feat’;
the issues identified in Edinburgh are likely to be more widespread"
The report contains many recommendations for the Council and other bodies, both public and private, as well as the construction industry. These relate to areas such as: procurement, construction, training and recruitment, the role of the building standards and independent certifiers and the sharing of information.
[report continues...]
"
More recently and very much more tragically, there's little need for an inquiry to determine the root causes of what happened at Grenfell Tower; they're already well known to anyone who is interested, and some of them are even documented. The particular sequence of unfortunate events that led to Grenfell Tower isn't yet documented; the root causes that didn't stop the collapse in standards from happening are well understood already.
We just need the relevant people, who are generally very well paid because they are supposedly in very responsible positions, to actually be held to account.
LOL.
Trust me, not all councils planning and buildings apartments are as good as yours. Here we've got amazon building a warehouse for 24/7 working, while promising us that the trucks won't have reversing alarms (a fact that'll only be in enforcement on our site, and these alarms generally can't be turned off for obvious reasons), that the overflow that occurs every time it rains, but never occurred before the building work started, is _nothing_ to do with the building of said warehouse, and the fact it's at least 3 small round about's away from any major road meaning the silent lorries are going to have extreme fun navigating around this little village (many small windy roads with 1400's architecture - so 6 foot thick sandstone walls and such - they will win against lorries ;))
Also, septic tanks can be setup with soakaways depending on the amount of waste they're expecting and the proximity to things such as rivers and boreholes.
This post has been deleted by its author
Not just the UK, in my brand new apartment in Brazil all three showers had drainage points in the highest part of the floor. Must be something to do with being in the southern hemisphere and what Brazilians thinks of as down is really up
They did it on the balcony floor too!.
"None of this was helped by equally numpty cleaners who, despite being told repeatedly, continued to pour inappropriate cleaning chemicals down the drains."
They tend to stop when given a choice of continuing to ignore instructions or being given a P45 (in the case of 3rd party cleaners, just threaten to cancel the contract for non-performance)
...you have to drill at 5 degree angle...
I was betting it would be 5 degrees *the wrong way*...
In my teens, when I was working on my first car, the shop manual offered this piece of sage advice:
"Before putting all your weight behind a wrench, stop for a second and consider what would happen if the wrench should slip."
I might not have been smart enough to apply this knowledge to the unfreezing of a pipe, but who knows?
And a quick story from the warmer climes, where ice is not a problem: The wife's sister lives in Florida. Her aircon drain backed up, soaking her wood floor. Her husband investigated and removed a (deceased) large lizard from the drain pipe.
Never had any snow or the such, but with the onset of winter our server room temperature monitor would spam us regularly with SMSes when the temperature dropped too low.
Adjusting the aircon helped a bit (a few degrees upwards) but only for a time.
Simply by adjusting the minimum level on the sensor did we manage to stop the SMS spam.
Interesting to note that it only started to bombard us with SMSes between 18:00 and 20:00 - other times of the day it was all peachy and happy and quiet.
Our office block in Pretoria was in a street of contiguous shops etc. Consequently it was tall and narrow - with offices on the back and front - which were respectively East/West facing.
With sealed windows the temperature was maintained by air conditioning - cloudy days were rare. Unfortunately there was only one thermostat for our floor - and that was on the back of the building. So our office at the front had its temperature adjusted in inverse proportion to the sun shining through our windows. While the boss's back office had a constant pleasant environment - we either froze or fried depending on the time of day.
Years ago we had trouble with the temperature in the server room; the main air-con unit had failed and the backup unit couldn't cope and pissed water down the wall. When the service guys arrived, they found the external unit was no longer on the roof.
It turned out the contractors relaying the roof had hacked through the pipes and cables to the unit and thrown it away - because they'd been told we didn't have any air-con in the building!
The bean counters wouldn't stump up for a replacement, so for the next year or so we had to make do with the backup unit which dripped continuously into a large refuse bin that had to be dragged out and emptied every few days.
Wouldn't they have been tipped off by the fact the AC was running or even if not the fact the electrical wires they were hacking through were LIVE?
Not sure why they'd want to do all the extra work of hauling off an old AC unit even if it was no longer being used. Sure, it is easier to relay the roof without it in the way, but there's no way the time savings is made up for by the cost of removing a roof unit.
"The bean counters wouldn't stump up for a replacement, so for the next year or so we had to make do with the backup unit which dripped continuously into a large refuse bin that had to be dragged out and emptied every few days."
Just have frequent precautionary shutdowns of any systems the beancounters rely on. After all, it's not good for the business to have no backup - which you don't if the intended backup has become the primary.
"I would have just put a condensate pump on the drain line and routed it out a window to drip continuously somewhere appropriate."
UK building regulations for domestic properties insist that the overflow from a cold water cistern in the loft comes out somewhere you can't ignore it, like above the front door.
When the builders have skimped on lagging the cistern, said cistern freezes in winter, rendering the float valve inoperable. If you are unlucky, the water overflows through the ceiling. If you are lucky, the overflow pipe does its job.
Ooh look, sheet ice just outside your front door.
First of all, he had to climb a ladder outside. So the data center's raised floor was above grade. And yet it didn't leak all over the lower level, but rather filled up like a bath? It's difficult to do that intentionally, accidentally is just about impossible. And how on earth could a long plug of ice have formed in a 10mm copper pipe in such a short period of time? If it was THAT cold, thawing it from the outside would have no affect internally. But it wasn't that cold, the data center was 50C, so no long plug was possible. And even if it were possible, the space under a raised floor is nowhere near enough to raise any kind of pressure, so it wouldn't have knocked him off the ladder.
All's well with the story, though, because the facilities guys allowed a data center guy to ascend a ladder with a blow torch ... and we all know that that's going to happen when all kinds of things freeze over.
We have a brand-spanking-new data center, and its floor is indeed designed to fill up like a bathtub so that, should the water cooling spring a leak, the waste water goes where it's supposed to, not to the floor below.
Said data center has so far been severely degraded by a backhoe cutting a fiber that was less redundant than it was believed to be, and put off the air, first by a failure of cooling water supply from a central heating/cooling plant, then by a failure of the incoming 1MVA electricity supply and no fewer than three back-ups. Lucky a) that this is academia, and b) that I'm not responsible for it.
Having worked in an office with a raised floor, and having removed every single floor tile in the place when we moved in to do some rewiring, I can see how this could happen.
The floor was several feet worth of reinforced concrete that didn't have holes drilled through it anywhere for fire safety, so that's not going to leak downwards. There is (obviously) no reason for holes drilled sideways through the building to the central lift shafts/stairs, and there wouldn't be in any case due to fire safety. The single centralised duct between floors was basically in a closet adjacent to the liftshaft, and this wasn't a bit that had a raised floor so cables to it went over the ceiling.
So in my experience, filling up the cavity between the reinforced concrete and the floor tiles is possible, if you had an immensely huge source of water. That said, I wouldn't have thought that the AC units would have been producing *that* amount of water in a data centre, you only have so much humidity that can be sucked out of the air! (unless they were taking external air in at a high rate?) and once you had more than an inch then the underfloor power rails would get submerged, short and the RCD's/breakers would blow so I suspect the flood wasn't quite at biblical levels. ;)
To my astonishment when we had a major AC plumbing failure all the servers and power breakers were working just fine with the under floor sockets and plugs under 6 inches of water.
Without an RCD that can happen. It may even heat the water for you :).
That said, the clock is ticking, don't expect it to last (and keep well away).
You may not know this but not all ac systems are air cooled. Some are water cooled by either cold loops orjust potable water feed throughs.
Heck, some systems are just cooling coils with cold tap water running through them.
If any one of these looses integrity you will collect a lot of water in a short period of time.
This post has been deleted by its author
You quite clearly demonstrate that you know absolutely nothing about how an AC unit works... nor indeed how a data centre is designed. If you have never utilised or understood either why post such drivel?
I know that if I knew sod all about something .. I would just keep my criticisms to myself rather than demonstrating to all and sundry my complete lack of knowledge. Of course if I wished to learn more about said topic, I would read up on data centre designs or even post a nice comment asking others to explain the mechanics of the situation.
Imagine a fiber conduit being routed inside a sewage pipe "because it saved on digging".
Now imagine said fiber getting gnawed apart by sewer rats, & being forced to slog through the frozen waist deep shit to get to it.
Last but not least, imagine all the fun you can have standing in a steaming shower for hours scrubbing off the stench that's permeated into every single one of your pores while standing in said testicle shriveling frozen crap.
While you glory in that mental visual I'm going to take another shower to scrub my brain with mindbleach.
My concept of teamwork ends at a point before where I enter a sewage pipe filled with frozen shit as part of an IT job.
I'm surprised ANYONE here would be OK with it, with all this "No U in team" bullshit being spouted. Are you really that worried about finding another job or love your company so much you'll literally crawl through shit for them like Andy Dufresne?
"If I ever had a job where I was asked to do this, I'd refuse. "
Anthony Burgess (A Clockwork Orange) wrote of his time as a conscript in the UK's post-1945 army.
A requisitioned country house was no longer needed as a barracks and had to to tidied. The outdoor latrines for the many occupants over the years had consisted of effectively a communal long plank over a hole in a hillside.
Burgess's squad were detailed to shovel the accumulated excrement from the bottom of the hill into a truck. As it was the middle of winter it was frozen solid. Fairly quickly they downed shovels - and in spite of threats of a court martial they were eventually allowed to leave. The mountain of excrement was probably still there when he wrote that chapter of his autobiography.
Not IT or cold. During a hot summer we had been away for a few weeks and this had caused the outlet to our cesspitt to dry out and block up. For various reason this was in out neighbours garden and they came to complain about the smell a couple of weeks after we returned. The outlet was accessible next to a bridge where a 2ft wide concrete platform let you access the hole about 6' off the ground and maybe 10' below the cesspit and 30' long, the platform being 3' above the river. My dad, another neighbour who had rods and a house guest who had just given his first public orchestra conduction and had all had drinkies afterwards decided it was now time to sort this out. The blockage was too hard to get anything other than the rods up it. The three of them forced the rods slowly up and eventually the rods reached the far end of the cesspit. They withdrew the rods and scratched their heads over what to do next when they noticed, like very thick black toothpaste, the blockage was creeping out of the hole. They stood there transfixed until it started to noticeably speed up when they reaslised their error and, panicking on the 2' wide platform tried to get out of the way asap at which point the blockage releases and a 6' wide fan of ordure (like the ones the firebrigade use for putting out chip shops) let loose and the observers were damn near hospitalised with hysterics,
Fortunately it was still a roasting hot summer so they weren't too unhappy to stand outside while we passed out drinks and their tuxedos (WTF??) dried out and the rest of us continued to roll about in tears.
"Imagine a fiber conduit being routed inside a sewage pipe "because it saved on digging"."
No, don't imagine it, look up the originally publicised business model for comedy ISP FibreCity, and its various associate companies and subsidiaries and its handful of failed projects in the UK.
FibreCity went bust after a couple of years, and many of the same people and the same business model turned up at an outfit calling itself CityFibre. And there was an investigation by the Serious Fraud Office, and a failed project in Australia's NBN and maybe some other stuff too.
CityFibre is still around. Greg Mesch held senior positions in both companies, as did various others.
https://en.wikipedia.org/wiki/Fibrecity_Holdings
Years ago, working in Academia, I'd not long been in position looking after a bunch of Unix servers. I'd gone into the little server room we had, and it was uncannily warm, so I started shutting down all bar the core servers, intending to call site services to look at the air con, when a student knocked on the door, and said 'I've got a problem in the resource room'. Usually this meant 'can you help me with some syntax', and I said 'I'm a bit busy, the air con has failed', and he said 'Well, there's water coming through the roof'.
That got my attention, so I went out to the resource room, and Lo! The heat exchanger was apparently in the false ceiling, had frozen over, and had been dripping, the ceiling tiles were sodden, and fell onto the desk below. There was a nice little pool of water, and surrounding it were teenage students, and I really didn't want anyone to die, because I'd go to jail. So I tried to unplug the machines. I quickly gave up on that when I got a mild shock, seems the water had got into the power trunking running up the middle of the desks.
So I had no choice but to kick all the students out, and power down the entire room. Still, nobody got fried and none of my servers failed. I just had to listen to the sites and services guy turn the air blue.
seems the water had got into the power trunking
1980, I'm on a training course at Pr1me Computers in Hounslow. Suddenly every terminal starting showing the message "please log off now, there's water coming out of the 13 amp plugs, we're shutting down". Turned out a guy on the top floor had drilled through the main water pipe(*), which was in the same ducting as the power circuits. Fortunately it was about 30 minutes to (lunch) pub o'clock, so a long lunch was had to allow the mopping up.
(*) I've had several incidents of drills through pipes in my career, and have never managed to work out exactly how they always managed to hit dead centre as if laser guided.
(*) I've had several incidents of drills through pipes in my career, and have never managed to work out exactly how they always managed to hit dead centre as if laser guided.
There's a special kind of gravity and magnetic attraction shared between water pipes and main power conduits, anyone doing DIY can tell you that.
Many years ago I reconstructed a bathroom with my dad in our newly bought house. Strip, rerun water and drains, shower, bath, sink all plumbed in, tiling done and grouted, cleaned - the works.
All we had left to do was put some hooks up for clothes, so my dad pre-drills with a 4mm masonry drill. An almighty flash, and power fails. The drill is missing the first inch and looks molten. No power, and the master fuse keeps tripping when we try to restore it (we discover later that was not a bad thing).
We take a chisel to that newly tiled wall, and in minutes it becomes clear that using a tester would have been good - my dad unerringly zoomed in to the one single spot where a master power cable came down - that's a single thick cable in a wall that's 4m long and he hits it dead centre. His drill connected the armoured outside with all 3 phases in the cable. Which was 380V, fused at 40A. Which we exceeded for a brief, violent moment to the point where all 3 phases welded together inside the wall.
We had to make the hole a lot bigger to repair the cable properly, but since then no hole has been drilled without a detector first. Except once. And I promptly hit water - it's like a bloody curse..
"... but since then no hole has been drilled without a detector first."
My approach too. But .... a while ago I decided to be a good boy and fix a bookcase to the wall for safety, in a newly refurbished part of the house. Checked the wall with a detector - all clear. Started drilling and promptly got my second shower of the day. Then realised modern piping is mostly plastic with only a thin aluminium barrier layer (e.g. http://www.screwfix.com/p/jg-speedfit-bpex-barrier-coil-pipe-25m-x-15mm/11027 ), so a metal detector doesn't detect it :(
Perhaps time to get a Cat phone with a FLIR thermal camera in it :)
Have a good weekend.
"And I promptly hit water - it's like a bloody curse.."
The neighbours' kids acquired a swing and it was put on a front lawn. They immediately exceeded its centre of gravity going forwards - fortunately without any injuries. It was not designed for ground fixings - so I drove in lengths of 10mm stud as ground anchors.
Now presumably there are standards for how deep someone might dig their garden without risking hitting the utility pipes or cables. I assumed that there was a shortest direct line between the house and the various service covers in the pavement. So - I avoided those lines by a generous margin. Just to disappoint you all - I didn't (I don't think) hit anything.
Now I live in a different block of the houses - built by the same developers at that time. When my neighbour lost power one day the electricity company had to excavate the pavement to find the break. It turned out to be almost at the junction with the street supply cable. That was lucky as the neighbour's cable did not go by the shortest route to their section of pavement - but was laid diagonally underneath my concrete drive. That was a surprise for everyone - fortunately the repair team could just manage to craft a splice without digging up my drive.
... but was laid diagonally underneath my concrete drive"
That reminds me. Many years ago I rented a Kango hammer (jackhammer) to take up a tile floor. Not working anywhere near cables or pipes - I thought. Happily broke up the screed then noticed some hissing. Stopped and looked at a steel gas pipe which the Kango had punched half a dozen holes through (didn't know it could do that). Ran faster than I've ever done before or after to turn the gas off.
Discovered that decades earlier some $#@!! decided to run the pipe diagonally across the kitchen, in the screed! (The new gas pipe runs under the kitchen cupboards, nice and visible. Briefly considered painting it fluorescent orange but decided that was going too far.)
------
Mate of mine used to look after the construction ops on a medium-sized oil refinery. Local council sends a team to take soil samples to check for pollution. Survey sites discussed, away from underground services, permit to work issued - lovely. Drill operator decides to test his rig at the place he happened to park it and manages to drill straight through a major power cable. Refinery not happy :)
"Discovered that decades earlier some $#@!! decided to run the pipe diagonally across the kitchen, in the screed! "
Did a kitchen redevelopment. When British Gas connected the new hob they laid a new spur pipe in the screed. The old gas pipe feeding the central heating and stove point was also buried in the screed. It runs across the kitchen - with a right-angle dogleg halfway along. The only clues to its direction are the outlets - and the short run of pipe in the garage that then enters the kitchen wall.
This post has been deleted by its author
Aside from the e coli 0157 that I caught while on a job in Africa (completely unique sub strain apparently) One of the worse injuries I have had while in IT wasn't even a work computer, but working on a families computer.
The floppy disk had failed and I had one to swap out. The computer was a cheap nasty one with a badly folded edges.
Pulling the floppy cable out, it was a little stiff, so pull a little harder... and harder. then it cam out and I banged the back of my hand on the back of the case, cutting it down to the tendons that I could see moving.
One trip to the hospital later, one set of butterfly tapes and one big scar on the back of my hand.
Turns out some tw@t had used hot glue to keep the cable from coming out.
The 250 mile drive back with a bandaged hand wasn't fun either.....
"Back in the day when PC cases were made by Gillette."
I once had the good fortune to have a couple of PCs made by a firm who clearly hadn't got that message. The lids wrapped round to form the sides, were hinged at the back and had stays to hold them open. They were just held shut be a couple of catches - press to release and lift. All PCs should be made that way - at least those intended for IT folk and lab users.
"I swear some of those computer cases had insides sharp enough to cut a tank in half and not dull one bit."
Adding the upgrades to a friend's new "working from home" company PC - I suddenly noticed red splodges on the interior. That was my first realisation of interior edges so sharp you didn't feel the cut as it happened.
When I used to build PCs all day for a living, I constantly had a cut or scrape on one hand or the other. No sooner than one had healed, I'd slice myself up on some other poorly built case.
Mind you, the worst injury was when you went to plug in a cable on the motherboard, slipped, and jammed a pin up under your fingernail. Ouch :(
Ah yes, reminded me of my days fiddling around inside PC cases.
As is traditional, I was doing some hardware maintenance late into the evening, and as the ceiling lights were pretty dim, I thought it would be smart to have a work lamp clamped to the case itself. Given that this was probably 20 years ago, the lamp wasn't a nice cool-running LED affair, it was a bare 100W tungsten filament bulb, and I proceeded to burn my forearm on it, probably when attaching the heatsink and fan to the CPU with those dreadful "push so hard you fear you're going to snap the motherboard" fittings.
Wouldn't have been so bad, but I didn't do it just once, or twice, but three or four times, and in exactly the same place on my arm each time. It blistered up like the big version of bubble wrap after the first couple of times and after it burst on the third burning I had to be careful not to let the fluid drip onto the electronics. I still have a nice round scar to show for it.
I think everywhere I've worked has had some sort of Aircon related disaster...
In my first job is was an outlet on a south facing wall. The outlet was in direct sunlight and one Friday evening apparently it got too hot and the system completely shut down. There was no remote monitoring and the first we noticed was on Monday morning coming in to do early morning checks and the server room was too hot for anyone to enter. Mostly because of the portaloo sized AS400 in there (which fortunately stayed up as no one knew where the 8" floppy was which was required to boot the bastard thing) The only casualties were a couple of Compaq towers.
Then there was the stockbrokers I worked for who's domain controllers were desktops sitting running from a 4way sitting under a drippy aircon unit...
Not to mention the major university I worked for which did a failover test of the datacenter (or as management dubbed it "The Computer Room") aircon which failed because some engineer or other had disabled the failover system during previous testing... Suffice to say there were a lot of people on site that weekend...
As for my current employer, our datacenter turned into a swimming pool when the building's chiller pipes sprung a leak. Fortunately in this case everything survived because of decent void space below the units. It could have been so much worse as the DC was is in a completely watertight subbasement and there could have been millions of liters of water!
Yeah not surprised the AS/400 didn't fail. In fact not surprised that most of the computers didn't fail. A lot of servers are rated for ambient temperatures of at least 35°C, and that's usually conservative. So why some people insist on running their server rooms so cold you have to don a coat when you enter, I don't understand.
Our Sun servers are rated at 40 degrees. We don't have aircon, because when the unit failed about 7 years ago, I decided a replacement was not really needed. Servers rated at 40 degrees - hottest day ever: 36 degrees. Who needs an aircon?
However, a few days ago, it was the hottest day since 1976. The server room hit 29 degrees several days in a row. BT "Professional" router failed consistent at about 24 degrees. I phoned BT "professional" helpline and they said I should keep the room cooler. I pointed out that 24 degrees was a "warm office" and "Professional" kit was normally rated for 40 degrees. After a protracted argument, they sent me a replacement. However, in view of the fact that the outside temperature had fallen to 16 degrees before it arrived, I am now using a "competitive" product.
(If it gets any cooler, I may have to re-index a few database tables).
Scrooge McDuck.
I think everywhere I've worked has had some sort of Aircon related disaster...
Not every place, but I think I'd be close.
First job as FS tech: total aircon failure on the site where I was Site Responsible at the time (though not for Facilities, fortunately). Temperature jumped over ten degrees in as many minutes while the sysadmins tried to shut down all not utterly bloody totally essential systems. Lost only 3 RA81's out of a hundred or so.
Another site had the aircon spring a leak, with the glycol-based coolant dissolving the original linoleum floor (they had simply installed a raised floor in one of the offices and put a pair of VAXes there) into a custard-like substance, engulfing a DELNI and what might once have been some thickwire transceivers.
Wooden 'temporary' office, with half the aircon capacity out of order due to a pinhole leal somewhere unfixable. Low tarpaper roof, nice toasty summer days. kept in check (for extremely tolerant values of 'in check') by two garden sprinklers under the heat exchanger.
Computer room, running at about 15% capacity (power and cooling). No problems. Systems are brought in, power and cooling are now at 25..30%. Still no problem. However, an external power failure now demonstrates that the UPS would very much like to have some of that cooling capacity too; it wasn't a problem previously because natural ventilation had been sufficient.
And one I heard from a colleague in the US: watercooled IBM mainframe in a university datacenter gets shut and switched off for the Christmas/New Year holidays, to save electricity. Snow ensues. System is switched on after the holidays and promptly cooks itself due to frozen and blocked heat exchangers on the roof.
"Not to mention the major university I worked for which did a failover test of the datacenter"
"Suffice to say there were a lot of people on site that weekend..."
It's a big enough risk to do "It's a small change - a five minute job" type things on a Friday. Are you saying they did a full on major disaster test on a Friday? Why is there no "Horrified" icon?
"I think everywhere I've worked has had some sort of Aircon related disaster..."
Mainframe operators turned up one morning. They noticed a thin mist of water on all the surfaces - and still powered up the system. The A/C had been overdoing the humidifier treatment. Took the engineers three days to dry every board with hair dryers.
New mainframe computer room in Africa. After a few weeks the designed air conditioning is not keeping the room cool enough. Ship in another A/C unit and problem goes away. These units were very tall against a wall - inlet at the top and cold blow outlet under the false floor.
We go in to do a software upgrade. Operators very friendly - offered us a beer and a slice of watermelon. They lifted a floor tile at the base of the nearest A/C unit - and lined up against the cold outlet were rows of crates of beer and several whole water melons. Mystery solved.
Engaged by my mother to teach YTS oiks the delicate art of computing - 4 sessions on databases, word processing and spreadsheets - we had to trek over to Liverpool from Manchester one very, very cold day. The training centre rooms were on the six floor of some ancient and run down building, with about a quarter of the windows broken. Although the room the computers were in was relatively warm (relative to the blizzard outside), helped by the 12 TRS-80 Model IIs that were in there, the common areas of the building were absolutely frozen; stairwells, corridors and most importantly the toilets. The air temperature was below -10°C, and the urinal was a porcelain wall covered in a thick layer of ice. I'll never forget peeing onto that surface and seeing the trickle freezing before it reached the bottom.
We had to end the session early because it was too cold to continue and the snow was reaching the point where the roads back to Manchester might close. Bless the YTS lot, though, all but one of them turned up for the training.
It was a similar story the next year, although not quite as severe - by then we had the Amstrad PCWs.
"We had to end the session early because it was too cold to continue and the snow was reaching the point where the roads back to Manchester might close. Bless the YTS lot, though, all but one of them turned up for the training."
That would have been covered by the Shops, Offices and Railway Premises Act, which set a minimum working temperature which must be reached within 1 hour of starting work. I know this because we sent all our YTS home a couple times when the heating failed in winter. We staff had to hang around a further couple of hours before the boss finally agreed we might as well go too.
"The air temperature was below -10°C, and the urinal was a porcelain wall covered in a thick layer of ice. I'll never forget peeing onto that surface and seeing the trickle freezing before it reached the bottom."
As someone who grew up in a city where it regularly hit -20c in winter i feel confident in saying that detail of the story is absolute pish.
"As someone who grew up in a city where it regularly hit -20c in winter i feel confident in saying that detail of the story is absolute pish."
As someone who grew up in a city where it rarely ever gets much below freezing and hence the buildings are not designed or insulated for the outliers, especially 30+ years ago, I don't find it odd at all that at temps of -10c in an old building, quite likely 100 years old with broken windows and likely a poor heating system (if it was working at all!) that the conditions were as described.
The only heating in the rooms was by electrical panel convector because the piped system had long since ceased to function - though the glorious cast iron radiators were left in place - they were a work of art.
There was no heating whatsoever in the stairwells, toilets or corridors.
Actually, I was thinking of this sorry affair when I linked that, but both are appropriate.
Mere Culpepper.
The story reminds me of my time in central Italy. Much of it spent in a portakabin, 'cos there were more people than regular office space for them.
For the most part, life in the portakabin was good. It was relaxed. I could sit in a comfy chair, and park my 'puter on a decent table with proper leg room under it, and even put the feet up. No PHB to moan about looking untidy. And for many months in comfortable temperatures of spring and autumn, we could keep the door wide open, and welcome local cats who were good company.
But the Roman summer is hot. And in the latter part, very humid: it progresses from a relatively dry heat in May/June to something unbearable in August. So we had to shut the door and turn the aircond on.
It would regularly ice up. At worst it's over 40 outside, and with the sun beating down mercilessly, the temperature inside would rise rapidly to the point where the wetware comprehensively fried. We started to structure the day to run it for a couple of hours, then turn it off and find things to take us over to the main building while the aircond recovered. Morning coffee, lunch, afternoon tea, and a bit of time spent with whatever cow-orker one could find an excuse to visit.
Having said that, I was sorry to leave the portakabin. As soon as I had an office in the main building, some office manager insisted on standardised desks and would hear no argument. These gave me no legroom and forced me into a posture my back couldn't cope with, so I had to quit the job. Sadly I hadn't heard the term "constructive dismissal" at the time.
Icing window-unit A/C is often the sign of a lack of proper coolant level. Dunno why, but it is a diagnostic that saves a lot of messing around with thermostats.
That said, I had a unit that iced up regularly until I took the front facia off and discovered that someone had "reconfigured" the blind copper pipe part of the thermostat. Once that was bent back into the proper shape and proper place, the A/C gave two more years of sterling service sans icing before quitting.
A story I was told was of a Army Ptarmigan Switch in Kosovo or Bosnia. During a generator change the switch ran on batteries so it didn't need powering down. They gave you a few minutes of up time. After the change you had to turn the ACU back on manually ('hit the tit' I think was the phrase used). This was forgotten and some time later, when the switch was checked water was dripping from all equipment enclosures..
ACU was turned back on and the door locked and not opened for 2 days to allow it to be dehumidified. No interruption to service, no board failures which says something for the robustness of the kit.
I had to do exactly the same thing once when I moved into a place in Glasgow in midwinter. Only it was the mains water supply (unlagged, of course!) which had frozen solid.
And all I had available was a hairdryer. And a long extension lead.
After hours of aiming the hairdryer at the pipe, there was a glugging and a shooshing and a rattling, and the water came back on.
It only takes a whiff of burning paraffin (kerosene) to take me back to my 1950s childhood. The house toilet was outside. After a lead pipe freeze and leak - my parents added a storm lantern as a source of sufficient heat to prevent the problem recurring. The light was also useful for reading the neatly cut pieces of newspaper hung on the nail behind the door.
One of my roles as an engineer many years ago was doing CPK analysis. Rule 1 of CPK analysis is don't analyse something you have no control over. So imagine my frustration when they started doing CPK analysis of the moulding compound store room. Moulding compound should be kept in a maintained environment to ensure it's not too hot or too cold when put into the moulding presses. All sounds okay so far, but...
It turned out the store room was a shed attached to the south facing wall of the factory, with a tin roof and no air conditioning.
In a previous job, we had a room full of servers in the office basement: after an explosive event the UPS manufacturers described as "unprecedented", the decision was made that this should be someone else's problem, so all the machines were duly packed off to a Big Blue warehouse elsewhere in the country. The aged aircon was disabled, and the basement fell silent for the first time in years.
A week later, it was noticed that the whole office was noticeably colder than usual: one stairwell, the one directly above the old server room funnily enough, was down to single-figure Celcius.
It turned out that when the server room had been installed in the early 90s, the waste heat from the server rooms had, rather sensibly, been used to heat the rest of the building... but no-one in IT or facilities remembered that, so the building aircon wasn't adjusted (and was, in part due to being the best part of 30 years old anyway, unable to automatically adjust for the loss of so much free thermal energy). Once I twigged that and made a quick call to a confused sounding Facilities phone-jockey, the right people were informed and it was corrected for.
I don't know if my department ended up footing the bill for the increased heating costs or if it worked out a net gain once the server room's power bill was taken out.
an explosive event the UPS manufacturers described as "unprecedented"
Alternate translations:
"A UPS hasn't exploded in this room before"
Or maybe:
"This UPS serial number has never exploded before"
The explosive nature of hydrogen gas is one reason why I really don't like centralised battery systems for emergency lighting. The other being that they tend not to be maintained very well.
An office where I used to work had a small room that housed the servers, network gear and phone system, and had a wall-mounted AC unit.
One day, we came in to find no servers and no phones. During the night the AC had died because the heat exchanger was a solid block of ice. The dead AC meant the servers rapidly heated the room up, the ice melted, dripping water into the phone system's main box which was mounted directly under the AC.
When I started my current role the computer cupboard (not big enough to be a room and located off the main office) had two heat pump units instead of a proper aircon. The "external" heat exchangers were actually mounted on the other side of the wall, in the factory building, under a steel clad roof. Drippage was a frequent occurrence but I'd replaced the plastic sheeting over much of the kit with a strategically placed plant trough.
Bring on the hot weather, they couldn't cope and after running continuously one of them iced up and died. Cue lots of moaning about leaving the door open with fans running to try and keep the temperature down.
The replacement was a 7.5KVA unit, properly plumbed in, which could get the room down to arctic temperatures if required. It got moved when we moved offices and is still doing it's job.
When I first moved to the UK I found it strange how you have all those bare copper tubes going in and out of the houses... And of course one day that cold winter a couple years ago our water froze and I had to crawl following said pipes with a hair dryer or the kettle, pointless exercise, just had to weather to let up..
The correct method to deal with a frozen overflow pipe, as demonstrated by my father some years ago, is:
1) Boil expensive electric kettle
2) Stand on rim of bathtub in slippered feet and pour boiling water from expensive electric kettle out of window over pipe
3) Lose footing and slip from rim of bathtub
4) Grab for window and lose grip on expensive electric kettle
5) Fall backwards into bathtub, injuring back and cracking skull, spilling commemorative toothbrush mug into tub so it can shatter and inflict small wounds on body and feet as bathtub exited
6) Recover twisted, mangled but still expensive kettle from bushes where it bounced after striking the back garden path. Never find lid
7) Yell at all the children in the house, accusing them of complicity in frozen pipe fiasco by risible chains of logic
8) Hit any child who laughs
...our transmitter site at the end of a nine mile trip up a snow covered 4000 foot mountain, so the power company wasn't too anxious to send fuse replacement crews up there. Besides, everybody up there has a backup generator. Until we didn't.
Temperatures close to zero for more than a week had gelled the diesel fuel in the brand, spankin' new above-ground armored fuel tank. Once the generator had run through the contents of its day-tank, it was lights out. A thousand gallons of fuel, which never needed anti-gel when it was located underground, obviously did now. Two UHF TV stations, three FMs, and four AM stations (utilized SAP from one of the TV stations for programming) were all off the air.
The company that had installed the new fuel tank was summoned, and promised that they would not only get liquid fuel to our site, but the fuel would be in the day-tank of a trailered generator that they would connect to our transfer panel that very afternoon. That very afternoon arrived, but the generator didn't. Slid off of the aforementioned nine mile long, snow covered road about two-hundred yards from our transmitter site.
At this point, power had been off in the building for more than eight hours. The temperature inside the transmitter building was hovering in the mid-thirties F, and the sun was going down soon. The high power UHF klystron tubes were cooled by distilled water boiling in the base of the tube, the steam being condensed back to water by an enormous radiator system (3 - 8x12 foot radiators). That would all freeze as we dropped below 32 F. As darkness set in, we started to light our field expedient smudge pots (metal garbage cans with ventilated lids containing old t-shirts and thawed diesel fuel) to keep the radiators from freezing solid when, eureka!, the power came back on. I can't imagine how much soot would've coated everything in that building if it hadn't. As it was, five broadcast engineers and three generator mechanics spent more than three days in eighteen inches of snow, with the resultant wet clothes, bad attitudes, and imaginative cursing.
Two lessons learned...
1. Don't cheap out and save twenty bucks just because the fuel hasn't needed anti-gel before.
2. Have an "extraordinary circumstances" clause in your contract, so you'll be properly compensated by the company that decides to cheap out on the anti-gel.
Here in the great white north its surprising to a certain large retailer based in the US that yes indeed, we have summers. Big hot wet ones in the nations capital.
Working for a big three letter company tasked with maintaining the IT for said large retailer, we had the joy of supporting their in-store servers. These were very well set up, and pretty well fitted in special server rooms, with excellent top-mounted AC units to cool the cabinets.
A pattern started to develop on the weekends where we would get late night calls for server outages to these sites, with a 2 hour SLA, things are tense to say the least.
Each time it would be the same scenario, somehow the AC unit had exceeded the draining capacity of the water pan, and leaked down the side of the servers, taking out 2/4 redundant power supplies, and since they way they work is in an X formation, 2 on one side means DEAD server. As well as the management cards.
The server could be brought up with a bit of work. An electrician had to be called in to give the OK that the water was clear, and things were safe etc.
To say the 2 hour SLA was doable, was laughable.
This kept happening. All over the place. And it seemed that on particularily hot humid nights too.
Turns out the shipping/receiving desks are in the (not air conditioned) rear warehouse section of the stores.. Right across from the nice cool server room. A server room with a door. That is to be kept shut.
Unless of course you fancy a nice cool breeze.... Then why not prop it open, heck, throw it wide and embrace that lovely cool air!
For hours at a time the AC units went on full overdrive trying to keep things cool. The amount of water they were condensing out of the air far exceeded the specs of the drain pan... and overflowed.
Idiots.
Costs were in the high 6 figures for that. I highly doubt any of those guys lost their jobs too. And cost my company thousands in missed SLAs as well.
Still chuckle remembering the senior dev who knocked a hole in the outside wall of his tiny little office to vent an evaporative cooler. And how quickly it became the hottest room in the building. We all liked the guy but it was funnier not pointing out the reservoir needed refilling ;)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
