Secret weekend office bonk came within inch of killing sysadmin Welcome once more to On-Call, in which The Register celebrates users' many, many failures by sharing your stories of being asked to clean up behind them. This week, meet "Matt" who sent us a story from 2004 when he "worked for a company who specialised in building high speed printers." Matt was the new kid on the block, so …
I worked on the second floor of a building that I labelled prone to flooding. People mocked me at first but every time something flooded I reminded them of that fact and they started agreeing with me. Pipes burst, aircon leaked, you name it.
The lowest electrical equipment I was responsible for was stood on 2cm wooden blocks. The lowest UPS I was responsible for was stood on a glorified metal coffee table.
This post has been deleted by its author
They did exactly what I guessed they'd do - added something to slow down the concrete hardening, and shovelled it out.
That thing was : sugar! Apparently more than 1% sugar in a concrete mix can stop it hardening almost indefinitely.
I definitely wouldn't want to be the person that had to clean all the equipment, though..
"...a small quantity of sugar (about 0.05 per cent of the weight of cement) will act as an acceptable retarder: the delay in setting of concrete is about 4 hours....A large quantity of sugar, say 0.2 to 1 per cent of the weight of the cement, will virtually prevent the setting of cement."
Properties of Concrete (A M Neville). 3rd edition (which probably shows my age, somewhat).
I have a dim and distant memory that the effects of sugar retarding the setting of concrete were first discovered when sacks previously used to transport molasses were then used to transport cement. But that might be me imagining things.
This post has been deleted by its author
This post has been deleted by its author
"Hats off to the people who fixed that one."
AC because I was (very loosely) involved in the recovery operation.
When we saw the photos, the joke was that they wouldn't get it working again for a week. In fact it was up again in time for start of business the next morning. Two reasons:
1) The sugar trick already mentioned; and
2) Within an hour there was a van packed full of spares heading up the motorway.
As it happened, the raised gap designed to protect against (water) flooding limited the damage, and the electronics came back up first time. The bloke in charge of supporting the recovery was later given a well-deserved award from LU.
One of our LAN rooms suffered an aircon leak, with water pouring out of the unit.
Some bright spark decided that regularly emptying a bucket was tedious, and put a large plastic recycling bin under the aircon so they didn't have to visit the room so often.
A bin with no wheels, which was rather heavy and very difficult to move when full of water.
I would. By these easy steps:
1. Submerge whole hose in water, so that no air remains in hose (bucket must be large enough).
2. Squeeze or otherwise block one end of the hose.
3. Pull hose into position (out the nearest window is usually best, as sinks are typically installed too high, relative to bucket).
4. Release blockage from end of hose.
5. Enjoy, while the bucket empties itself.
Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.
"Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.
I do.
Admittedly it's with homebrew."
If my Dad's 'Homebrew' of old is anything to judge by then :
I refer you to the "... Stomach lining. ..." above. :)
Self inflicted does not change anything :)
On Topic, I don't understand why anyone would have a AirCon Unit that drains into a Container when there are quite expensive racks of kit around !!!
Surely, it is worth the cost to 'plumb in' a proper drainage pipe to avoid the potential for flood.
On Topic, I don't understand why anyone would have a AirCon Unit that drains into a Container when there are quite expensive racks of kit around !!!
Surely, it is worth the cost to 'plumb in' a proper drainage pipe to avoid the potential for flood.
I've seen the "drain" get plugged and building maintenance in all their glory can't find the plunger, a long length of wire, etc. to clean it with. So they call a "professional" who can't be there for week or two...
Who primes any kind of hose by sucking on it, these days? Almost no liquid is anywhere near safe for your lungs or stomach lining. In some places, not even tapwater.
I do, most weeks.
If water filtered once by rocks is pure then water filtered thousands of times by Fluval and fish kidneys must be really pure, right? Right?
1. Insert the entire hose in the air-con water. It is preferable the usage of a hose where both sides can be sealed by your thumb or working watertight valves.
2. Fill the entire length of the hose with liquid. Keep both sides submerged in the tank. Avoid bubbles.
3. Seal both sides.
4. Displace one side of hose above suitable drain. Open the other (submerged) side. Nothing should happen, yet.
5. Remove thumb/open valve on the drain side, but keep hose in position aimed at drain while water flows at high speed.
Bonus: No need to contact your mucous membranes with air-con water.
1,2 as stated *
3 seal drain/output side
4 Displace one side of hose above suitable drain.
5 as stated.
* or get fancy:
- Stick tube down into the bin, one end first, make sure to let air escape through the other end. When there is an inch or two left above water, block the dry end. Dry end is drain end. move on from 3. If done right, all that gets wet is your thumb.
This post has been deleted by its author
In actual fact, that'd be perfectly safe. Don't know about anything else living in the water though. Anyways, wer'e talking about condensate from a compressor aircon here, aren't we - Out of the cooled air, into the bucket, not a cooling tower for an evaporative cooler?
Leigonella ingested is harmless. It's only dangerous when it's breathed in as an aerosol. - like the fine spray from a shower rose. It multiplies rapidly in warm water.
I gather that before we knew exactly what it was and exactly how to test for it the standard test was to sniff the culture in the Petri disk. Apparently it has a characteristic odour.
The beast can survive in water up to near 60 degrees C, which is why the greenie advice to turn your water cylinder stat down to 50 or 55 degrees to save energy was quietly dropped. (AFAIK no one from the greenie camp publicised that they'd dropped another Bo**ock, so there's probably lots of water cylinders running dangerously cool.)
My worry is that many condensing boiler flues can run way below 60 degrees, and if the system is not used to generate stored hot water, or the boiler stat’s wrong, or the system is a combi with hot water on demand set to well below 60 degrees (which should be safe, since any legionella in the cold main doesn’t get stored at nice cosy breeding temperatures) that flue could never get to sterilising temperatures.
The water produced when methane or propane burns, which you see as a plume of condensed water droplets from the flue of a condensing boiler, "has a ph similar to tomato juice" and has strict disposal rules - you aren't supposed to run it untreated to a fresh water drain, feed it into a cast iron system that does not have other water regularly running through it or let it contact concrete or brickwork. (The condensate can dissolve more than half an inch of mortar / pointing in a year.)
I wonder what ph is necessary to kill leigonella, and whether the bug can evolve to cope.
I've asked the question at training and assessment courses but never had a satisfactory answer. Sand, meet head.
I hear a variation on this story every couple years. I always question it. Why? Well, I don't know about you, but I've pulled the cap on both fuel tanks and the vent/cleanouts of blackwater tanks. They look absolutely nothing like each other (first clue), they obviously have completely different purposes (second clue) ... and of course their smell is a dead giveaway.
For those of you offering solutions, the facilities people came up with the bright idea of using the bin because it happened over a weekend, and soon after the "LAN room flooded and on fire simultaneously" incident I've mentioned in a previous On Call.
Problem is, they forgot to tell us until the bin overflowed and water started seeping under the door.
How about this one... leaky drain pans in the ceiling, part of the HVAC system. Maybe the drain hoses were clogged. So install large plastic totes (bins) to catch the drips. Installed on the ceiling tiles. Then forget about them. Ticking 'time bombs'.
Eventually, weeks later, it was raining totes in the office area, each brimming with 50+L of icy cold filthy water. Oh, another. And another...
How about installing a bunch of servers in a building that's still (partly) under construction, walking out the fire-escape door for a quick smoke while waiting for the install to finish, only to find out the fire-escape ladder wasn't installed yet...
I almost dropped 18 feet down onto the concrete pavement below...
I was always curious how exactly those can possibly happen; our elevator definitely isn't any sort of fancy, but the doors are literally, physically interlocked to the cabin with a small latch you can see turn and unlock the door each time the cabin arrives. So okay, these are "open by hand" doors not the modern self-opening kind, but still... (why) aren't all doors unable to open without a cabin present?
As I read it they're saying the air-con was left switched off by the installing contractor, but it was turned on to make said bonking more enjoyable/less sweaty. It was then left running when done with said activity. Thus causing the flooding. I presume the bonking was found out because there was a thorough investigation into WHO turned on the air-con unit. Thus access logs were inspected and security camera footage reviewed. One or both involved parties then fessed up to what they were doing when asked.
Where I have worked for the last 25+ years we never had RCDs (or RCBOs) on power circuits for apparatus rooms as we called them - except for test benches. IIRC they were deemed too risky from the point of view of losing service - and also a few mA leakage is a pass, but once you have a handful of kit all running on the same RCD (or RCBO) the combined leakage could cause a trip and not all our kit was dual PSU.
I presume we passed all the regs or else the 600+ bays should not have been live.
My former boss and mentor told the story of fun in the old days.
He was a BOFH for a paper mill which, if you’ve never been in one, involves very large machines, toxic chemicals and energy in the form of steam. He was in charge of the computer systems and had a computer room in the middle of the mill.
During a construction project, they decided the best place to run a high pressure steam line was directly through the computer room. There was no need for that extra five feet of pipe to go around the room, “we can save money by going right through.” So they did.
Everything was fine until one evening when the steam yearned to be free and the pipe burst INSIDE the computer room. Since it was evening and everyone had gone home for the day, no one noticed. That was until the computers went down. The pager went off at 2:00am and after a quick trip to the mill, he walked up to the computer room where the computers were on display. The window was opaque and white. Hot and white. Hot, white and there was a screaming sound coming from inside the computer room.
He was smart enough to NOT open the door but called the plant maintenance team who reluctantly turned off the steam line. After everything cooled down and they could enter the computer room, it was obviously a total loss. They scrambled to replace all the kit while the plant maintenance guys spent a larger chunk of money for enough pipe to route around the room.
And in those days, it came to pass that incompetent staff were summarily fired. And low, there was much rejoicing throughout the land, for they were difficult to work with and made everybody's life a living hell whenever possible.
Well, that's the way it should work, anyway. Gut feeling is they got a raise for figuring out how to re-route the pipe after the fact ...
"And in those days, it came to pass that incompetent staff were summarily fired. And low, there was much rejoicing throughout the land, for they were difficult to work with and made everybody's life a living hell whenever possible".
These days, they'd be more likely to sack the IT guy for insisting that the machine room be located where it was. THEN give the other guy a raise for re-routing the pipe. THEN give themselves a raise for solving the problem of an inconveniently-placed machine room.
I am unsurprised. I worked somewhere where a large plastic hydrogen vent line was routed above a suspended ceiling. Of course, the hydrogen vent system was completely foolproof and could never ignite. Ever. And the company that installed the approved system was so good at it that they went bust.
And the pipe was PVC.
The hydrogen containing system that could never ignite duly ignited and the entire large PVC pipe exploded, sending large fragments of PVC through the insubstantial suspended ceiling. Some of them stuck quite deep in the floor below. Mercifully nobody was hurt.
It also turned out that the person responsible for the installation had been made redundant some time before so there was nobody to blame.
Oh, and did I mention the hydrogen was radioactive?
One company I used to work for had an unwritten rule. In the event of an actual real fire the fire assembly points were to be completely ignored and everyone should just get as far away from the building as possible. Such as the next village. Something to do with the large tank of hydrogen below the factory...
"Real engineers?" They cost money! It's a lot easier to get the local handyman to do it. He'll get it done cheap and won't be asking any troublesome questions, nor bring up those pesky rules and permits that other people seem to care about.
Or, you could get Scott Pruitt to do it...
"Well, to be fair, tritium's a very weak beta emitter. PVC would stop it easily."
I'm glad someone gets it. I left that as an exercise for the reader. I'm not going to say much more because I don't want to doxx myself - except to observe that a lot of places in the developed world used to use tritium, and our own beloved government used to think the best thing to do with the vast amounts of it they created while making bombs was to put it in the North Sea, you can guess where.
But eventually the Irish complained and it was realised that actually 5keV electrons released inside fish were probably quite bad for them, so in a sudden reversal of policy it was decided that the best thing to do with the stuff was to release it into the atmosphere.
Before anybody got worried about it, I visited a plant once where tritium was being used in a manufacturing process. In a corner of a shopfloor there was a lump of vacuum equipment and a small gas bottle of tritium-protium mixture connected to it, along with a fairly basic standard welding type extract.
"I'm glad someone gets it."
It helps to have worked in a carbon dating lab. Carbon 14 is also a weak beta emitter, a bit different in energy but it meant the same counting technology would see both if the material going into the counter was a hydrocarbon.
Tritium in the hydrogen supply was a concern. On the methane counting system there was a supply of "old" hydrogen as the source of the hydrogen in the methane. The later benzene counting system was a different matter. The hydrogen came from water (carbon and lithium heated to produce lithium carbide, add large excess of water to produce acetylene which is then catalytically converted to benzene). The best that could be done was distil the water to reduce the tritium content. That just left the radon problem. When your water supply comes from a granite catchment you have radon in your tap water.
Oh, and did I mention the hydrogen was radioactive?
My last place, we had a wonderful mix of low-flashpoint solvents and alpha-radiation sources on site, in fairly close proximity to each other.
Which is why our fire alarms directly alerted the nearest fire station and we had to pre-clear our fire alarm tests with them. And why we got annualy refresher training in fire extinguisher usage and our Elf'n'safety person allowed us to play^W test with almost expired extinguishers..
(Although they did sometimes use them as ready-alert tests by forgetting to tell the crew on duty..)
"How they hell was that install even legal or passed inspections?"
Don't ask me, I didn't install it.
I will just mention in passing that in Berkshire, not at all far from inhabited areas, are tanks of highly radioactive liquids. At the time I was aware of them people did not know what they contained. Some of them had to be cooled by water passing through internal pipes. Some of them might be leaking.
Such is the insanity of nuclear deterrence.
Military secrecy and the needs of national defence cover a multitude of practices that would probably never be allowed outside.
Toilets blocked - and caustic solutions didn't clear it. Pushing a drain rod hard eventually gave a satisfying feeling of the blockage released. Flush worked - job done.
The comms engineer was called out later on. All the modem racks were covered with a foul mess dripping from the false ceiling. In the ceiling void was a 100mm toilet drainpipe neatly separated at a rubber seal join.
I worked at a large chemical plant where steam is often used for heating - it allows heat to be provided to places that would object rather explosively to open flame.
To transport such steam, it is sent overheated (read: well above 100ºC, it's basically a gas) through pipes at high pressure. They started a re-insolation project of all those pipes along walkways when one of the pipes developed a tiny leak and someone was pretty much cut in half by walking through that jet of invisible steam - the plume of that leak was only visible some 10m further.
I don't envy the emergency staff at these sort of plants - they get to see some rather nasty stuff.
A revist from one of mine last year.
Just came back from a training weekend down south in Orange County (same airport as Harrison Ford "Schmucked up his aircraft landing") & tale of a high pressure steam leakage on a submarine was told to me by one of the attendees (quite a few ex-navy attendees actually).
When a pin prick leak occurs in a high pressure steam pipe, it's visible as a "cloud" in the general vicinity, when that's spotted all in the immediate area freeze & slowly look about for something within arms reach to use as a guide like a spanner or wrench as the North Americans have it, then move it in the general area of the cloud to find where it intersects with the leak, the results are pretty obvious once it does so.
A newbie submariner, forgot all his training on his trip out, spotted the leak & immediately went to point it out to his shipmates (he was also in an area of the boat he wasn't really supposed to be in) inadvertently using his arm as the detection device.
His forearm connected with the high pressure steam jet & a Anakin Skywalker type injury occurred, as the jet sliced through the flesh & bone severing without hope of re-attachment (Both sides of the wound cauterized instantly).
One of the first rules of AC work is "Always have proper drainage for the evaporator." ... It's not an afterthought, or something you have to go back to the shop to get. It is designed into each individual system, just like the ducting and power. It's just something you do, like breathing. It's in the estimate, and/or the bid. The parts for any particular install are on the truck with everything else. As a contractor, if my AC guy didn't have this prominently listed in his/her paperwork, I'd find a new AC guy pronto.
Also, I've got to ask ... How many cubic feet was that one system conditioning? It'd take an awful lot of air to wring out enough water to deposit an inch on a typical data center floor ... more if it's a raised floor, of course. This is especially true seeing as modern AC systems recycle interior air[0], meaning once the water is condensed out, it's out.
[0] Yes, I know, "fresh" air is introduced in larger systems, but that happens at a heat exchanger, typically on the roof where the noise of the air handling system won't annoy anybody inside. That condensation is typically handled by the roof, same as rain water ...
Ahhh beancounters, where would we be without a bunch of cost-obsessed maniacs to get in the way?
Probably in the place of a certain northern English university that I'd better not name (not that I work there, but still).
This university had a sports complex with a flat roof. Not the smartest option ever designed for use in England, and like every flat roof ever, this one developed a leak. The University Estates people were called, and failed to fix the leak to nobody's surprise since University Estates tend to have the collective IQ of yeast.
So, the job was put out to tender, and the Powers That Be unfortunately included the beancounters in the decision. The problem was that there were only a couple of contractors locally big enough and reliable enough to do the job, and they were well aware of this and tended to cartel-like pricing. A third group also tendered, at a much lower cost; let us call them "Del-boy Trotter & co". The beancounters saw only the cost, and insisted on this bunch being hired to do the job.
Thus it was a week or two later that what looked like the contents of several scrap yards, plus exhibits from the Museum of Dodgy Roofers turned up. Shortly after Security tried to turn them away fearing a gypsy invasion, it was determined that these were the new roofers. Quite quickly, they got to work and started messing about with boiling tar, roofing felt and endless cups of tea, and a fortnight or so later pronounced the job done and asked for payment. In cash.
This was refused until after a couple of good rainstorms showed that yes, the leaks were gone, so Del-boy and hist motley crew were paid and duly disappeared, never to return.
Several thunderstorms later the leak re-appeared. The loft space of the sports complex was investigated, and it was found that the large plastic bin placed under the largest leak by the roofers was now full of water and overflowing.
One of the usual suspects for roofing was duly engaged, after it was discovered to nobody's surprise save for the beancounters' that Del-boy & co had vanished.
To this day the beancounters of this university have their every decision questioned with the words "But remember the sports hall!"
Ahhh beancounters, where would we be without a bunch of cost-obsessed maniacs to get in the way?
In one of my previous places, we'd have a server room with a UPS that didn't go in to bypass by the time we'd added about 60% of the kit (we were combining two server rooms and, on the technical side, we'd specced up the UPS to be able to carry twice the combined load because we knew there would be expansion in the new, larger room). Despite telling us that "sufficient budget" had been allocated for our design, it was discovered that the UPS *actually* bought by the purchasing team was only about 40% of the rating required "because of budget" and didn't have sufficient capacity for the existing combined load, let alone any future additions. And the UPS stack was surrounded by fixed racks and so we would have had to delay the project (and cost more money) to fix it. We ran on bypass for quite a long time.
Same with the aircon. Original plan was to have two separate units (so one could be shut down for service), each capable of handling total load+50%. What we actually ended up with was two units, each of which could only carry 60% of total load. Which meant that, if we wanted to do service on one of the units, we had to shut down 40-50% of the servers..
Beancounters - can't live with them and, for some reason, you go to jail for introducing them to the pit of quicklime..
Jake, it may be a rule but who enforces them?
Some years ago my company moved into a managed office suite in a new block. The heating didn't work terribly well, and after a year a consulting HVAC engineer came round to investigate. I got talking to him and he told me this:
The building had been designed with a condensing boiler system. It was designed for 55C, but the pipes were running at 82C.
He had talked to the facilities manager about this and had been told that you couldn't run the system at the design temperature because if you did, the boiler room floor flooded.
Sure enough, the water vents from the boilers had been left to drip on the floor by installers who had presumably not understood them, and not understood what the nearby drains were for.
The system already needed repair due to deposits blocking thermostatic valves, all for the lack of a few £ worth of plastic pipe and a facilities manager who was totally devoid of clue.
I've had an almost identical experience - spent a week walking into a server room to find a soaking wet floor, with no explanation why. Eventually discover, hidden behind a partition, the drain for the aircon in the adjacent post room - the aircon that the new cleaner not just turns off, but cuts the power to every night (no, I don't know why the aircon unit had a massive cutoff switch next to it) and dumps the water onto the server room floor...
The mainframe operators came in one Monday morning. They noticed that there was a thin film of moisture on the console table. They then pressed the master "power on" button to sequence up the mainframe and all its peripherals.....
It took the engineers three days to dry all the circuit boards with hair dryers.
I'm just glad the Halon system where I used to work was never triggered. There were only two breathing masks available, so if it went off it would have been a case of who can remember where the masks were, find them through the clouds of Halon and put them on before you suffocated. Oh, and if anyone else was trying to do the same, well, let's just say survival is not a team sport under such circumstances.
I always made sure I was in there with either
a) people who would have idea where the masks were or
b) smaller than myself so I could get the mask
So yes, very glad it was never triggered. Anon for obvious reasons :)
As a student on work placement many many moons ago, a Halon system was being tested.
The "dry" run out tho the emergency door/hatch proved the 10 second delay in the Halcon flood was still going to be fine after some new kit was installed.
But I asked why does it smell if its a dry run.
Arr.
Was told its just the bit of gas left in the pipes venting.
But I smelt it while the 10 second alarm was going off.
Arrrrrrr.
Yes wired wrong. no delay been like that for 6 years.
You get much the same issue on boats - where the pipework is designed on the Olympic system (the bloke who gets there first runs his 1/4" pipe straight, and the big 18" low pressure steam conduit has to bend round it).
For my sins I started work in a military command and control complex, where keeping the computers and signals equipment up under nuclear war conditions was a business requirement. So we had full replacements and engineering support on site 24hrs (or rather UNDER site, since we were well underground).
The worst outage we had was due to a stationery delivery, when the trolley clipped one of the aircon pipes. You can't get rid of heat easily when you are underground, so that was critical for the mainframes. And of course we ended up with an airlock.....which lowered the defensive capability of the country for nearly 36 hours....
Also had a leak in the server room once.
Now, the servers' located in the basement. Which is Not A Good Idea.
However, the buidling is on a hillside, which allows the western part of the basement to be open. So flooding will not be a problem.
Right?
Wrong.
Seems that the builders neglected to check on the horizontal level of the floor (which is now the server room). I've found out only after the fact that it is NOT truly horizontal, but tend to incline towards the western side, which has a wall. The door is to the east, where you enter and exit.
It should be pretty waterproof, but we had to cut a hole in the wall for a fiber-optic (dang word, I never know if it's fibre or fiber, but anyway) conduit. It was sealed properly and everything.
For the first year things went well. Second year, the conduit piping carrying the fiber inside started to crack due to vehicular traffic. Third year we had a nice water leak outside the premises... and yup, water entered the server room.
Luckily the critical servers was high enough to keep their toesies dry. And I'm glad I did not get electrocuted - I waded into that mess barefoot. Water got warmer at a point, then I noticed that a kettle plug had immersed itself in the water - and I skedaddled out of there at a good pace, and refused to enter again until all the water had been pumped out.
It woke manglement up, and the company had to spend some $$$ in getting proper drainage installed to keep water away from the server room.
So far we've had a couple of rainstorms etc, but the server room remained properly dry.
Only time will tell if we'll have a repeat performance or not. If it does happen again, the servers will be relocated somewhere else... which'll be a major PITA.
And for the record, I will not try this experiment again, with a live electrical connection submerged...
I once worked in a college with several computer rooms. Due to a blocked surface drain in a courtyard, a basement computer room started to fill up like a bathtup through the airbricks in the wall.
The computers were on high benches and the students sat on bar stools. As long as their feet didn't get wet, they were happy with a good 40 cm of water on the floor!
The computers were high and dry so to speak, so no problems :-)
Electricity isn't like a swarm of animals, it doesn't "move" in water. If something like that happens, the bulk of the current flows very close to the points where the life conductors touch the water. In fact if you submerge a typical power supply, you're probably fine even close by. Even if you have some relevant current going somewhere the RCD will trip way before there's any danger.
Safetly precautions are there to keep you away from danger as much as sanely possible. However even without them not everything is highly dangerous.
BTW There's a great German short movie called "Staplerfahrer Klaus" (availiable on Youtube even with english subtitles) However that's perhaps not really SFW.
BTW There's a great German short movie called "Staplerfahrer Klaus" (availiable on Youtube even with english subtitles) However that's perhaps not really SFW.
I've posted it here a couple of times, and nobody has whinged before - but if you're squeamish, there's a lot of fake blood and fake severed appendages towards the end.
The German Bluray release actually includes an interview with the producer in which he tells about safety specialists who, in all seriousness, see this as a safety instruction film. It is actually shown as a safety instruction film in many companies.
Let's be honest though: "Staplerfahrer Klaus" probably contains all the rules typically applied in smaller companies and a clear demonstration of WHY those rules are there. Plus it's a heck of a lot more entertaining than the typical "occupational hazard" instruction material.
I worked briefly at one customer site that had their computer room in the deep basement. At ground level, just in front of the building was a ornamental pond. A pond that needed to have a running water feed into it was it leaked badly.
Guess where the water ended up....
A very famous computer department in Manchester used to have a rooftop garden above their main machine room, complete with a pond. No prizes for guessing what used to happen on a regular basis; the pond leaked.
The only highlight of this was that every year for a couple of decades, one lone duck would build a nest in that rooftop garden, the fledging chicks getting a free ride out into more suitable habitat when the time came; photo-opportunities with the Computer Science duck were a great honour.
These days, the machine room is a big office. The pond is a skylight.
It still leaks.
On similar lines, in BBC TV Centre in London the old VT (Video Tape) rooms were in the basement directly underneath the fountain in the courtyard.
After a few months of occupation the fountain was turned off because the sound of it echoing off the walls of the circular courtyard made some of the female staff members keep visiting the loo every 10 minutes.
This is not the point of the story, however. The point is that the fountain was also found to leak. When the Queen visited the site at some point in the 1970s the management decided to turn the fountain on to impress her, but in order to minimise the risk from the leak they decided to only turn it on whilst the Queen was outside. The Queen was seen to be approaching, so the fountain was turned on. She met the management, went inside and it was turned off. But then they had to turn it back on because the Duke of Edinburgh turned up in a land rover a few minutes later.
The upshot of the story is a large part of the expensive VT equipment underneath the fountain was ruined!
I have also felt the pain of a wet server room.
Went in to help one of the electritians wire in some new telephones. Noticed a pool of water underneath one of the server stacks. The worrying bit was when I mentioned it to said spark he just laughed and said it didn't surprise him with that building.
Though in the end we discovered it was due to a 1st level support chap who couldn't be bothered to take the computers for cleaning away from he building when he air blasted the dust out of them. Which also was where the intake for said air conditioning was. Dust blocked pipe which in turn lead to the overflow water just dripping onto the floor.
Fun times...
Turned up for work one day about, wow 20 years ago, to find no-one in the IT Office, one of the officey types said they had all sprinted down to the basement shortly before.
Turns out there had been a major flood and the basement server room was 8 inches deep with black water that was only made blacker by the lack of lighting. Lots of power cables trailing in the water which I naively agreed to help lift out and place on higher platforms.
In hindsight, I was pretty stupid and thoughtless back then.
Back in the seventies (sigh) the IBM mainframe computer room was 3/4 below street height. The windows were knee height on the street so easily accessible for viewing in or worse. Not 24hr operation then so when a couple of drunks decided to break in there was no water but a few "leaks" discovered next morning. Whole computer room moved at the next major HW upgrade.
...at an NHS trust many years ago where their cowboy fix for the aircon in one of the server rooms was to let it run down made up channels stuck to the wall into a bucket. It obviously relied on people checking the bucket wasn't full. I reported it once, got ignored.
When you work in the NHS, you see why its such a mess.
I once led a team updated the IT systems for a museum in London, somewhat sugar related. One of the sites on the south bank was converted from an old industrial site and the main computer room was in the basement. I got a call one morning from the server installation team saying there was a network issue so duly headed to site. Anyway long story short I wandered into the computer room, one with nice shiny metal floor tiles and agreed that yes for some reason the new network switches installed a few days before were all down. So wondering about the cabling I decided to lift a floor tile, so there I stood with a floor tile in my hand looking at water lapping up against the bottom of the tiles. I looked up and noticed a nice big 3 phase distribution board with all the cables exiting at the bottom and going under the floor. At that point I calmly suggested the team all leave the room (duty of care and all that) then got the hell out myself. It turned out there was a big water tank at ground level outside for the fire system for the buildings old usage and some bright spark decided to empty as it was no longer needed without first checking which way the ground sloped. Luckily it was Friday and we could head off early for the pub based weekly wash up meeting, my hands were only shaking a bit as I slurped my first pint!
I worked at a site which had an incident where a series of problems resulted in a flood, failure of aircon and overheating. Such was the design of the building - open floor-to-roof in the large reception area - that the flood water evaporated then condensed up at roof level and came back down.
Overheard from an engineer on the phone to head office about the problem: "I don't know. I can't see from here - I had to go outside to get out of the rain"
Ah, the joys of an air conditioning system, low on coolant, in a humid environment. The evaporator tends to cause the humidity in the air to freeze into ice. Imagine my surprise that Saturday that I wandered into the machine room, and found a 300 pound block of ice hanging down from the ceiling. Oh, that's gonna hurt!
I had a similar experience, but on a smaller scale. Nowhere near a 300lb block, but enough to seriously injure someone standing under it if it gave way.
The aircon engineers were called in, took a look at the kit on the roof, and handed me a report which consisted of five words...
"SEAGULLS HAVE EATEN THE PIPES !!!"
Well, that explained the lack of coolant.
The unit in the machine room was defrosted, emergency portable aircon brought in, and non-essential systems powered down to reduce the load on the portable kit until the pipework could be renewed.
Then there was the Great Melton Street Flood. A high pressure main in Melton St next to Euston Station ruptured, washing out a huge crater in the road and flooding the underground car park.
It didn't affect our computer rooms. It found something much bigger and electrical: the Tube system. We had an operator on his way in to start a shift. He told us the train stopped. The lights went out. And then the water started coming through the floor....
This post has been deleted by its author
I once worked in an office which roasted in the summer, because there was only one small air vent for the 5-cube office. They eventually put in a "portable" AC unit (about 3' wide, 3' deep, and 5' high), and ran the exhaust hot air line up to the drop ceiling. Nowhere beyond that - it just pushed the hot air above the drop ceiling, causing the (still warm) air to work its way back down into the room. It essentially became a GIANT dehumidifier. (Which was still an improvement, though it was loud and we had to take turns emptying the condensate bucket.)
which lowered the defensive capability of the country for nearly 36 hours....
I'm sure that lowered defensive capability was more than covered by the 19,999 other "courier" deliverable buckets of instant sunshine.
I was also astounded by the wheelie bin and garden hose solutions to the H20 disposal issue from the air conditioner problems. I am hoping they were ALL posted with humour intended behind them because why else would one suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ (or tens of thousands ££) of delicate and mission critical electronic systems that keep the company running and pay packets flowing to those employed below the level of HR, Bean Counters and other Management tosser.
why ... suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ of delicate and mission critical electronic systems
Because B&Q / HomeDepot don't offer the same sort of perks to customers as companies selling mission critical electronic systems
"I am hoping they were ALL posted with humour intended behind them because why else would one suggest a couple of $$ (or ££) of plastic and a manual water control system to protect tens of thousands of $$ (or tens of thousands ££) of delicate and mission critical electronic systems"
At the pay-grade of those who do things like that it's not delicate, mission critical electronic systems, it's just stuff. The water just needs catching and emptying it's the next shift's problem.
I've seen a delivery driver want to drop a heavy, delicate and expensive piece of electromechanical kit off the back of his van because there wasn't a tail lift and they'd not arranged for a fork-lift. To him it was just another crate. There were enough of us to insist he didn't and to slide it down a plank.
Many moons ago, I was traveling (for work actually) with a work companion in of all places Tyler Texas, where the motel rooms had occupancy sensors on the thermostats to "economize". Of course this was totally unacceptable as we wanted to return from our outings to a reasonably cool room rather than sweat it out while the AC attempted to cool things down in glacial time.
Our solution was to open up the thermostat and "bypass" (I believe it involved a screwdriver adjustment) the motion sensor. This meant that we slept comfortably at night and came back to a nicely cool room after the days outing. When we did come back, we noticed that the condensate which drained from the AC onto the exterior walkway looked like a river for our room, but the other rooms had barely a trickle. We just smiled and carried on.
Yes, Tyler Texas in the summer is both hot and humid.
No need to be part of the gunpowder plot, this was back in the 80's and the company is long gone.
Once upon a time, a long time ago, at a small company I used to work for, was a lad named Eustace. Eustace's parents were druggies so he grew up without a moral compass, good role models or a clue. One day his step-mother, taking pity on him, prevailed upon the owner to hire him so he could establish the beginning of a work history, see how normal people behave and maybe get a leg up in life.
Eustace proved quite pedestrian at his work - light industrial and janitorial duties and kept asking me if I wanted him to clean out my office (my "office" was the only room on premises that had a door that could be locked for it was both the R&D lab and server room).I kept telling him not to bother for clutter is my particular idiom.
One Monday the owner told me I might want a new desk chair and that Eustace was fired over the weekend. He then told me how these two things were related.
On Saturday Eustace made himself useful and cleaned up what passed for my office. A bit later the owner showed up and immediately noticed 2 things:
1. His boombox was missing
2. Soft music was playing in the R&D/server room.
Of course, he investigated.
What he discovered in the de-cluttered room was not just soft music playing but cardboard had been taped over the window and candles were burning on the workbench, apparently to set a romantic(?) mood as Eustace about to engage in coitus with his girlfriend on my desk chair. Now I must point out that this was not the girlfriend he was living with and made a baby with and who was going to school so the three of them could have a decent life together. This was his *other* girlfriend, the hot one.
I laughed my ass off upon hearing this. And then, figuring that Eustace lacked both the presence of mind and decency to put at least a sheet of newspaper on the chair, I dragged it around the back of the building to let mold and fungus clean out whatever might be soiling it.
One of my colleagues has a far far far worse story about liquids in a server room.
This server room had been installed in a hospital, and had been in that location for several years. My old mate had just recently installed a couple of new blade centers and storage, and one Monday morning unlocked the door to the room looking forward to a day of playing with shiny new toys.
Unfortunately, over the weekend, there had been a bit of a plumbing leak. What he at the time didn't know, was that in the ceiling above this server room, some bright spark had run a sewerage pipe. From the intensive care unit. Connected to "bed pan disposal unit". This pipe had for some reason popped off. Which meant that for all weekend, all the super disgusting sick person excreta had been dripping through the server room ceiling all over this brand new shiny kit.
Needless to say, while specialists cleaners were called in to clean the room, all the kit was replaced under an insurance claim. Now that's a day you'd never forget.
Another revist...(might have been on El Reg initially) about a shit\urine flooded comms\electrical room in a basement.
Basically......
The building designer\plumbing contractor had routed all the black water into a straight pipe for the entire building (not sure if a office or tower block) with a 90 degree bend at the very bottom.
The inertial mass of free-falling turds with gallons of falling water from the top floors, had over time weakened the joint\pipe & the thing finally gave way. So the pipes contents instead of carrying on the journey to the sewers, the cellar comms room had taken on a new role as a septic tank.
Something not quite similar happened when a "restroom" flooded at a slaughterhouse I used to work at.
Anon as this is probably still there
Previous life as Theatre Electrician - theatre in Scotland with a concrete encased basement electrics room containing three phase incomers and main lighting dimmer racks. Local Fire officer required the water sprinkler system remain functional even when it was pointed out water and serious amounts of electricity didn't mix. Even a hint of fire alarms and lx team was not seen for dust.
Same theatre that due to the foibles of hydro power any matinee show in winter meant that at 5pm when the kettles and ovens went on for tea voltage could drop to around 190 and the lighting control boards would just give up - you could be left with every light in the rig live or complete blackout.
Not saying where, but had a very similar near-disaster caused by water somehow getting backed-up in long gutters on flat roof over several months.
When someone finally got around to unblocking one of them the resulting torrent of filth ended up going through three sets (!) of ceiling tiles and 2.5" of water in the room like a tidal wave.
Evidently the volume of water was underestimated, of course the servers nearly drowned but fortunately the only casualties were a few mains leads and other expendable items.
Shared because it could happen anywhere and MRI scanners/X-ray machines *really really* hate water immersion, replacement cost would have been £4M+ and heads would have rolled for sure.
AC
Yep had that brown momemt,
Due to various tectonic events, we had another company working out of our first-floor boardroom for a year. They had a couple desktops, a nice APC 1.7 kVA UPS on the floor (foreshadow) and all was well.
An aftershock knocked over a water cooler, which was plumbed into the mains, not one of those bottle ones. It piddled away for a day and a half, releasing a lot of water whcih ran downhill. The boardroom was at the low end and was ~50mm deep in water.
On Monday I walked in to see the damage, and became aware their stuff was still powered. So I turn off the wall socket.
>CLONK< goes the UPS, as they do.
The UPS was probably half under the filthy water and had just switched to batteries. I'm aware how a UPS is built inside, with the batteries below and the board along the top, so any waves could short it. Gingerly reach over and hold the power button on the UPS till it powers off and all the PCs go down.
By how much did I avoid a shock? No idea, and I really don't want to know.
TL;DR, earthquakes suck, AND they have aftermaths (afterbaths?) that go on for many years.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
