They wanted to play W:ET!
They just ended up flooding their infrastructure when the servers retaliated with a buffer overflow.
Feeling the burn? Stress getting to you? Today's edition of Who, Me? concerns pressure of a different sort as a Reg reader experiences a most unexpected deluge. Regomised as "Peter", our hero was working in the IT department of what he described as a "medium-sized city" somewhere on the West Coast of the US. It was the early …
I've referred to such a bodge before: a site run by the Ministry of Agriculture was down half its cooling capacity as it had an apparently unfixable pinhole leak in one of its two circuits, and they had already exhausted their Freon quotum refilling it. This was in a temporary barrack, one storey, low, tarpaper roof, and with even just moderate sunshine temperatures in the offices next to the computer room would already be approaching 30C and the computer room would hit 25C with the one chiller running flat out. So they had a tap installed, with two fairly serious sprinklers underneath the working half of the heat exchanger.
So once the computer room temp would pass 25C, someone would open the tap, which then often had to stay running a fair bit into the evening until the temperature had sufficiently dropped again. "Tap duty overtime" must have been a nice little earner for some, hence the reluctance to fit a valve controlled by a thermostat and/or timer. Dropping a few buckets of white paint, or tacking aluminised bubble foil on that section of the roof was similarly declined.
They also suffered from a well over average number of parts failures,
... knows you don't plumb two water sources together. Ever. For lots of reasons.
A cousin of mine decided it would be nice to build a manifold so he could close a valve or two, open a couple others, and thus easily prime either of the two jet-pumps on his irrigation systems ... with the city water supply as the priming water.
Several thousand dollars in fines, and quite the tongue lashing from the raised-rural Judge later, and now he understands why most of us have a pickle/olive barrel full of water handy to the pump for priming purposes. Once the pump is running, top up the barrel for next time.
"Anybody who knows anything about plumbing..."
That sentence is where your entire comment falls down. Plenty of sensible people know nothing useful about plumbing - myself included. Actually I find plumbing to be one of those topics that seems simple on the surface but has many ways to come back and bite you in the ass!!
I always tell anyone unlucky enough to have me train/tutor/mentor them, "The only really stupid question is the one you don't know the answer to but don't ask because someone said "everybody knows that!" - you don't know so there is a good possibility other people don't know either.
Asking questions is never stupid if you do not know or are not sure - not asking them can be very stupid indeed.
The only stupid question is one you ask twice.
unless you have short term memory loss and don't even remember asking the question in the first place... then it is only (possibly) stupid to those being asked the question if it is their 2nd or more time around the barrel...
yes, i do know where the ANY key is
... knows you don't plumb two water sources together. Ever.
Erm - AFAIAA most CH boilers in the UK have a means of temporarily coupling the CH circuit directly to the mains water supply in order to fill or top-up the CH system. In my recently installed boiler it is achieved by opening two ball-valves using a screwdriver. In my case I do so once or twice during Winter when the CH pressure falls below 0.8 bar (probably due to small leakages in the CH circuit). This is explicitly detailed in the instruction manual for the boiler - it is not an unapproved arrangent put in by a cowboy plumber.
The CH circuit has chemicals in it to prevent scaling and freezing, so we would definitely not want it to leak back into the mains water supply. AFAICS the only thing that prevents this from happening when both ball valves are open is the assumption that the mains water pressure will always be higher than the CH circuit pressure. While there might be a non-return valve somewhere inside the boiler to prevent flow from CH to mains (I don't know), a non-return valve (if it exists) would not negate the fact that the two water sources are plumbed together.
So I am not sure that your observation is correct.
Your mains water supply is a source. Your CH system isn't, it simply holds water that the source supplied. You're not coupling two sources together.
Once you store water, the water in that storage becomes a new "source". If I fill a barrel with water from the mains, then add a kg of potassium cyanide to that barrel, you are surely not saying that it is then OK to put that barrel on the top of a tall tower and plumb it into the mains water supply?
In the UK, in order to prevent the possibility of CH water (which is probably full of anti-corrosion chemicals) flowing back into the mains, the filling loop must be fitted with a double check-valve (non-return valve). In fact the regulations used to say that the loop must be completely disconnected when not in use. I think this has changed these days, as some boilers are permanently connected to the mains and can indeed auto-top-up.
In the case of a gravity-fed system, there must be an air gap between the filling valve outlet in the header tank and the high water level.
While you can't initially pump up the CH circuit to more pressure than is in the mains, mains pressure does fluctuate and of course the action of heating the CH water and indeed pumping it around will raise its pressure, so it's possible that the water in the CH circuit can be at higher pressure than the mains.
"... knows you don't plumb two water sources together. Ever. For lots of reasons."
I'd guess that your average person would not know that the cooling system was pressurised and so it wasn't a "source" in that respect, even if they did know that you should not mix two or more sources in that way.
I swear I've heard this story before with a flooded parking garage as a bonus.
There was talk around the office that one of our customers would be returning their Cray-2 instead of buying it, which meant we'd have to test it to resell for the most money.
Ron was against testing. He didn't trust that fancy chlorofluronated crap Cray used, and he didn't like 'water' cooling in general.
"Last time I worked on something like this I had to replace my Mustang!"
Ron claimed to have worked at a company with a Univac, and the thing kept getting hot. One of the 'engineers', who had worked on nuclear subs in the past, thought it was air bubbles in the cooling system. "Starvation caused by insufficient reserve volume", he said.
Mister Bull Nuke decided they would 'burp' the cooling system by attaching a couple of valves and hoses. One to vent the air and steam from the hot side into a city drain in the garage, another to add water on the cold side from a tap.
Ron continued on. "There isn't any real reserve volume in a Sperry-Univac, and it wasn't starvation, it was cavitation."
By the time the 'engineer' gave up, they had eight inches of water in the small underground garage and his Mustang was a total loss.
Probably required a little more effort than described, like the outflow hose ejecting itself in an utterly cartoonesque way from the city drain and spraying the chemically enriched water coming out of the Univac into some particularly sensitive areas of the Mustang.
It would also have to be a rather rundown Mustang already to be declared a total loss after a liquid encounter like this.
Not a "who me?" but it reminds me of the time (also back in the early 1980's) when the firm I was working for had their own IBM mainframe to control company stock, manufacturing and finance. It sat in a separate building, in its own room with its own water cooling. Back up, BTW, was a duplicate sitting across the pond, with a nightly satellite link. Two design flaws in the building:
1) One of the above ground pipes was plastic, and
2) The floor drain wasn't at the lowest point.
Needless to say, one day maintenance needed a bit of height to fix something (probably something as innocuous as changing a light-bulb) and a water pipe provided a convenient step up. Being plastic. it sheared off and water sprayed into the room. Fortunately, the spray didn't target any vents but the bunded floor started to resemble a paddling pool. Panic ensued as the computer was hastily shut down. I wasn't working in that department but was one of the folk who had to start placating the several hundred other people who now found they had no access to the data they needed to input or withdraw stock, run any of the four manufacturing shops or any of the myriad other back office jobs.
Fortunately, the computer was unharmed and, following some urgent baling and re-plumbing, normality resumed by the following day.
This post has been deleted by its author
Yeah, I recall one hot summer many years ago, someone had ordered a load of air conditioning units from a local hire place - and assured me they were the right sort. When the guy arrived to deliver them, I just told him to take them back. Stick a swamp cooler ina closed room (or at least, one with limited ventilation) and the only result is a temporary cooling before you quickly end up with the same temperature - just more humid and uncomfortable.
We did have a portable heat-pump type unit - and one day we went into a training course to find ... It was hot as hell, and the maintenance guy had wheeled it in and switched it on - oblivious to the need for somewhere for the heat to go (I think he thought it went back up the mains lead, like a fan heater in reverse !)
But, at my last place, when things got really hot in the server room, we did have a swamp cooler - and it was pretty effective. Though it made the metalwork at the front of the servers go rusty. First one in one hot days got to carry 10 gallons of water to fill it up.
We have a larger sized version of one of those to spot cool a piece of test equipment. During a recent hot spell, the technician asked for some help, as the tester was running out of spec. I discovered that someone had pointed the portable chiller at the tester (good) but had also routed the dryer vent style flexible tube from the back of the chiller into the side of the tester (very ungood).
If you're unfamiliar with this style of chiller, the flexible tube is used to discharge the hot air from the system. Normally this is discharged through a window, into a plenum, or just away from sensitive equipment.
We had one of those portable units semi-permanently installed in my old office (a small room containing 5 cubicles), taking up about half a cube. Two tubes coming out the front blowing cool air, a big tube going up into the ceiling for hot air, and a tub for condensation that had to be emptied occasionally. It was obnoxiously loud, and didn't seem to do much aside from dehumidify.
At some point, we took a look at the tube going up into the ceiling. It only went to the false ceiling, blowing the hot air into the space above - which, having nowhere else to go, then came back down into the room. Hmm...
Eventually, someone found a damper closed in the line going to our one small vent. Once that was opened, the office got pretty comfy, and they got rid of the portable air conditioner.
Not IT, but involving many tonnes of copper and heat; I have seen organisations in the USA calling in local fire response services to point their pumps at substation transformers to increase their cooling capabilities over and above what the traditional oil radiator-and-fan approach can do.
It is, from the standpoint of cooling highly effective. One has to question whether asset management or planning practises are fit for purpose to "survive" on one transformer rather than installing two rated to take the load in question.
For the record, in the UK, it is can be a criminal act to operate HV equipment beyond it's rated capability.
A/C because not hard to identify what organisation I must be in to be able to write this.
Similar thing can happen in the UK too. Some years ago during a hot summer, the area around Carnaby Street in London was blacked out due to overheating transformers in Carnaby Street main substation. The "cure" was to drape plastic hoses over the cooling fins with tiny holes punched into the hose along its length, and the hose connected to a cold water tap.
It wasn't designed that way.
I was working as a computer operator during my sixth form hols at a well known engineering company in the Black Country. They were a go-ahead organistion housing the kit and a resident ICT engineer in a state-of-the-art glass fronted office block facing south. Is the penny starting to drop?
Anyway it being the swinging 60s the company had purchased a spanking new ICT 1904. It magically had <drum roll> magnetic tapes. And magnetic tapes needed air conditioning whereas the 1301 drum just rolled along unpeturbed in any conditions.
So - the false ceiling was torn down to install the AC revealing an extensive network of water pipes. Cold water pipes as it turned out. On a blistering August summer's day the un-airconditioned air produced an absolute deluge as it condensed on the pipes right on to the top of the 1301.
The ICT engineers horror - not to mention the DPM as meltdown looked inevitable. But equally magically a dozen or so fire buckets were rapidly emptied and arranged along the top of the 1301. But buckets are round so complete coverage was impossible and the waterfalls were fickily moving along the exposed pipes necessitating constant re-configuration of the the computer's new bucket collection. But it worked the 1301 never missed a beat. But then it wasn't running GEORGE.
Perhaps that was the hardest day's work I have ever done - except, possibly the night before payroll day when one of the punch card machines supplying the data went unaligned and the 1301 reader rejected and halted after every fifth card. This had to be re-punched by one hand while holding up the errant card to spot the holes with the other before the payroll program would continue. The prospect of the Friday pay packet being empty was not an option for a company that didn't want its management lynched.
But buckets are round
I had a similar issue* much more recently that necessitated a quick trip to the nearest garden centre for the largest plant trough I could find!
*In my case an under-specified and badly installed aircon unit that iced up and was dripping onto the server rack.
George! I loved George (V3 I recall was the one I cut my teeth on). And those temporary files "!" whose name incremented each time you created a new one - ! became !1 then !2 if you created 2 more temp files, so you had to be on your toes to keep track of them. Were the command files called macros? We had the use of an ICL 1904S the other side of the city, a PDP 11/70 on-site and then, wonderfully, a VAX 750 which of course blew the socks of everyhting with it's unlimited virtual memory, all 500Mb of it.
But it MUST be the right stuff and properly installed. It has to be closed cell so water laden air can't constantly meander in and out, and it must be installed with all joints sealed for the same reason. Ignore this, and sooner or later you'll be ripping off a load of waterlogged insulation and having the right stuff installed by someone with a 'kin clue.
Even back then the building should have had a "break tank" with an air gap to prevent the buildings water getting back into the main district water supply. They may have contaminated the buildings own water, but not the surrounding district. A filling valve on a toilet cistern works on the same principle, the water sprays in from the top with an air gap, this prevents the possibility of water back syphoning into the mains water system.
We were 'endurance' testing a high-pressure pumpset with lots of control and monitoring gear attached, designed to give warning if 'anything went wrong'. Container-sized generator. Coolers to keep the water temperature down. Big set up, and because it had to run for a few days, access was restricted for safety and security.
I realised something was amiss when walking past the building, I noticed water coming out of the ventilation louvres 25 metres above ground level, then a shoe float out of the emergency exit. It's not easy stopping these 'temporary' lash-ups especially when there's an upside-down tropical-rainstorm going on in a building packed with electrical equipment. Incredibly, all the electrical and electronic gear survived. On subsequent tests, I made sure we had a stop button outside what was laughingly called the 'Control Area'.
We put a Prime mini into and office, and the business decided to partition off a corner of the room as "the computer room".
One day the outflow for the toilets on the floor above, which ran through the false ceiling in the main office, blew. Guess where the effluent went? The stuff literally hit the fan, which didn't cool very well after that!
There was a TV show in the later 70's that followed members of a certain fire station in the Los Angeles County Fire Department. In one episode, the station is called out for a house fire, which turns out to be a toilet on fire. The homeowner had flipped a lit cigarette into the water, only to have it essentially explodes. The crew hook up to a hydrant, and go to douse it, only to have it flare up. A neighboring yard has a sprinkler on, and a pedestrian flips his cigarette into the wet grass, only to start another fire. This time the crew uses only the water already in their engine to put it out, and they begin searching for the contaminant. They find a crew with the water main hooked up to the fuel oil line, which is supposed to be flushing the fuel oil line in preparation for welding the line. But the oil line was running at roughly triple the pressure of the water main, and the crew had failed to install a one-way valve, so the oil was flooding into the water system, not the other way around.
Fictional show, of course, but perhaps it took a page from genuine incidents of the time.
Only once in my 27 years of IT have I had that 'do I power off the ops room or not' decision to make.
One afternoon in sunny Surrey our cleaner spotted something shining through the air con vents of the raised flooring and asked me as the IBM Mainframe Operator on duty to take a look.
Looking down I realised there was a pond of water there instead of the usual bone dry concrete, with the air con still happily blowing away above it.
Rather than lift floor tiles to see how far up the sloped room the water had got - and I was thinking of those four-bars that were stashed under the AIX / Token Ring cabinets... - I did consider powering down the Ops room entirely via the big red button. However as I stood there I realised nothing had stopped working, nobody was dead, nothing was on fire etc., so I legged it up the corridor to pull my boss out of the monthly IT meeting. I'd never seen his eyebrows raised as high as when he looked through the same vent. When we did lift the tiles shortly afterward we had less than a foot to go before the four-bars would have been flooded. The IBM engineer said it was an impressive amount of water for a pin-hole sized water leak...
1st one was when I were a young and keen apprentice at all things CNCery... we had a spiffing 5 axis mill, the hallowed area of the gods(well the guy with 5 years experience of loading punch tapes anyway), one bit of it was water cooled for some reason, trickled through and then dumped. until the maintance guy decided "whats this pipe here? " and cut it off.
Luckily we never started the machine as no one could get near it due to the stench coming out of the drain pipe where the water was dumped into the street sewer.
The other was a few years ago, when the alledged mangler could'nt get the HP coolant circuit running after the coolant tank was cleaned, however he had seen me prime the pump by loosening a hose fitting on the HP circuit when the pump was replaced..... and we all know that this one ends up with the coolant tank being emptied in about 10 secs by a 300 PSI pump .... was impressive how far it went too ...
If only he'd listened... loosened.. and not while the pump was running....
Biting the hand that feeds IT © 1998–2021