back to article Hmmmmm, how to cool that overheating CPU, if only there was a solution...

Feeling the burn? Stress getting to you? Today's edition of Who, Me? concerns pressure of a different sort as a Reg reader experiences a most unexpected deluge. Regomised as "Peter", our hero was working in the IT department of what he described as a "medium-sized city" somewhere on the West Coast of the US. It was the early …

  1. ShadowSystems

    They wanted to play W:ET!

    They just ended up flooding their infrastructure when the servers retaliated with a buffer overflow.

    1. Korev Silver badge
      Coat

      Re: They wanted to play W:ET!

      Sounds like the Mainframe was hosed...

  2. Richard 12 Silver badge
    Boffin

    Should have used the sprinkler

    Spraying cold water over the outside of the radiator would have worked!

    1. Stoneshop
      Flame

      Re: Should have used the sprinkler

      I've referred to such a bodge before: a site run by the Ministry of Agriculture was down half its cooling capacity as it had an apparently unfixable pinhole leak in one of its two circuits, and they had already exhausted their Freon quotum refilling it. This was in a temporary barrack, one storey, low, tarpaper roof, and with even just moderate sunshine temperatures in the offices next to the computer room would already be approaching 30C and the computer room would hit 25C with the one chiller running flat out. So they had a tap installed, with two fairly serious sprinklers underneath the working half of the heat exchanger.

      So once the computer room temp would pass 25C, someone would open the tap, which then often had to stay running a fair bit into the evening until the temperature had sufficiently dropped again. "Tap duty overtime" must have been a nice little earner for some, hence the reluctance to fit a valve controlled by a thermostat and/or timer. Dropping a few buckets of white paint, or tacking aluminised bubble foil on that section of the roof was similarly declined.

      They also suffered from a well over average number of parts failures,

      1. cosmodrome

        Re: Should have used the sprinkler

        "White paint" was my first idea, too. Followed by the thought of just running water over the roof all day. That's what it's for, after all.

  3. jake Silver badge

    Anybody who knows anything about plumbing ...

    ... knows you don't plumb two water sources together. Ever. For lots of reasons.

    A cousin of mine decided it would be nice to build a manifold so he could close a valve or two, open a couple others, and thus easily prime either of the two jet-pumps on his irrigation systems ... with the city water supply as the priming water.

    Several thousand dollars in fines, and quite the tongue lashing from the raised-rural Judge later, and now he understands why most of us have a pickle/olive barrel full of water handy to the pump for priming purposes. Once the pump is running, top up the barrel for next time.

    1. John Doe 12

      Re: Anybody who knows anything about plumbing ...

      "Anybody who knows anything about plumbing..."

      That sentence is where your entire comment falls down. Plenty of sensible people know nothing useful about plumbing - myself included. Actually I find plumbing to be one of those topics that seems simple on the surface but has many ways to come back and bite you in the ass!!

      1. Pascal Monett Silver badge

        And those sensible people will sensibly ask a plumber how to go about doing something concerning plumbing, otherwise they're not so sensible.

        So I fail to see how Jake's comment falls down.

        1. John Doe 12

          Well to be honest there is a hint of arrogance in Jakes original comment which narked me. It has that same vibe as "everyone knows that..." which always backfires as the person saying it is usually doing so to someone who didn't know :-D

          1. not.known@this.address

            no such thing as a stupid question?

            I always tell anyone unlucky enough to have me train/tutor/mentor them, "The only really stupid question is the one you don't know the answer to but don't ask because someone said "everybody knows that!" - you don't know so there is a good possibility other people don't know either.

            Asking questions is never stupid if you do not know or are not sure - not asking them can be very stupid indeed.

            1. swm

              Re: no such thing as a stupid question?

              "The only really stupid question is the one you don't know the answer to but don't ask because someone said "everybody knows that!"

              The only stupid question is one you ask twice.

              1. waldo kitty
                Big Brother

                Re: no such thing as a stupid question?

                The only stupid question is one you ask twice.

                unless you have short term memory loss and don't even remember asking the question in the first place... then it is only (possibly) stupid to those being asked the question if it is their 2nd or more time around the barrel...

                signed,

                yes, i do know where the ANY key is

              2. cosmodrome

                Re: no such thing as a stupid question?

                > The only stupid question is one you ask twice.

                ...unless the person you were asking was unable to give you a reasonable answer. "Eeowah hullahoowida agaah ronn" for example does not qualify as "reasonable".

            2. G.Y.

              Re: no such thing as a stupid question?

              USMC: "The only stupid question is the question you don't ask!"

        2. A. Coatsworth Silver badge
          Unhappy

          >>I fail to see how Jake's comment falls down

          Two words: Dunning Krueger

        3. Anonymous Coward
          Anonymous Coward

          And those sensible people will sensibly ask a plumber how ...

          And if they are lucky, they'll have found a plumber who actually has a clue. One of the reasons I do my own plumbing is that I haven't found a plumber I'd trust to change a tap washer - let alone something more than that.

      2. cosmodrome

        Re: Anybody who knows anything about plumbing ...

        Well, there are civil engineers and architects who are supposed to know how everything that's in a building is done right.

        It's not without a reason that their golden rule says "Never do it yourself!".

    2. Stoneshop

      Re: Anybody who knows anything about plumbing ...

      The tap that can be used for topping up the central heating circuit is fitted with a non-return valve, as are, as required, the taps for the washing machine and the dishwasher

    3. Cynic_999

      Re: Anybody who knows anything about plumbing ...

      "

      ... knows you don't plumb two water sources together. Ever.

      "

      Erm - AFAIAA most CH boilers in the UK have a means of temporarily coupling the CH circuit directly to the mains water supply in order to fill or top-up the CH system. In my recently installed boiler it is achieved by opening two ball-valves using a screwdriver. In my case I do so once or twice during Winter when the CH pressure falls below 0.8 bar (probably due to small leakages in the CH circuit). This is explicitly detailed in the instruction manual for the boiler - it is not an unapproved arrangent put in by a cowboy plumber.

      The CH circuit has chemicals in it to prevent scaling and freezing, so we would definitely not want it to leak back into the mains water supply. AFAICS the only thing that prevents this from happening when both ball valves are open is the assumption that the mains water pressure will always be higher than the CH circuit pressure. While there might be a non-return valve somewhere inside the boiler to prevent flow from CH to mains (I don't know), a non-return valve (if it exists) would not negate the fact that the two water sources are plumbed together.

      So I am not sure that your observation is correct.

      1. Doctor Syntax Silver badge

        Re: Anybody who knows anything about plumbing ...

        Your mains water supply is a source. Your CH system isn't, it simply holds water that the source supplied. You're not coupling two sources together.

        1. Cynic_999

          Re: Anybody who knows anything about plumbing ...

          "

          Your mains water supply is a source. Your CH system isn't, it simply holds water that the source supplied. You're not coupling two sources together.

          "

          Once you store water, the water in that storage becomes a new "source". If I fill a barrel with water from the mains, then add a kg of potassium cyanide to that barrel, you are surely not saying that it is then OK to put that barrel on the top of a tall tower and plumb it into the mains water supply?

      2. Martin an gof Silver badge

        Re: Anybody who knows anything about plumbing ...

        In the UK, in order to prevent the possibility of CH water (which is probably full of anti-corrosion chemicals) flowing back into the mains, the filling loop must be fitted with a double check-valve (non-return valve). In fact the regulations used to say that the loop must be completely disconnected when not in use. I think this has changed these days, as some boilers are permanently connected to the mains and can indeed auto-top-up.

        In the case of a gravity-fed system, there must be an air gap between the filling valve outlet in the header tank and the high water level.

        While you can't initially pump up the CH circuit to more pressure than is in the mains, mains pressure does fluctuate and of course the action of heating the CH water and indeed pumping it around will raise its pressure, so it's possible that the water in the CH circuit can be at higher pressure than the mains.

        M.

    4. Antron Argaiv Silver badge

      Re: Anybody who knows anything about plumbing ...

      Check (anti-backflow) valve is usually a code requirement.

    5. John Brown (no body) Silver badge

      Re: Anybody who knows anything about plumbing ...

      "... knows you don't plumb two water sources together. Ever. For lots of reasons."

      I'd guess that your average person would not know that the cooling system was pressurised and so it wasn't a "source" in that respect, even if they did know that you should not mix two or more sources in that way.

  4. technos

    I swear I've heard this story before with a flooded parking garage as a bonus.

    There was talk around the office that one of our customers would be returning their Cray-2 instead of buying it, which meant we'd have to test it to resell for the most money.

    Ron was against testing. He didn't trust that fancy chlorofluronated crap Cray used, and he didn't like 'water' cooling in general.

    "Last time I worked on something like this I had to replace my Mustang!"

    Ron claimed to have worked at a company with a Univac, and the thing kept getting hot. One of the 'engineers', who had worked on nuclear subs in the past, thought it was air bubbles in the cooling system. "Starvation caused by insufficient reserve volume", he said.

    Mister Bull Nuke decided they would 'burp' the cooling system by attaching a couple of valves and hoses. One to vent the air and steam from the hot side into a city drain in the garage, another to add water on the cold side from a tap.

    Ron continued on. "There isn't any real reserve volume in a Sperry-Univac, and it wasn't starvation, it was cavitation."

    By the time the 'engineer' gave up, they had eight inches of water in the small underground garage and his Mustang was a total loss.

    1. jake Silver badge

      A total loss from sitting in a mere 8 inches of water?

      Story sounds all wet to me. I'm fairly certain I've driven Mustangs of most vintages through more than 8 inches of water before, with nary a problem.

      1. Pascal Monett Silver badge

        8 inches of water barely gets to cover half the height of the wheels. I don't see how a car of any kind can be totalled by that.

        You might need to replace the wheels, depending on the corrosiveness of the liquid, but worse than that I can't think of.

        1. Dave314159ggggdffsdds Silver badge

          So very nice work getting a not-working Mustang into position for a claim, then.

      2. Stoneshop
        Holmes

        Probably required a little more effort than described, like the outflow hose ejecting itself in an utterly cartoonesque way from the city drain and spraying the chemically enriched water coming out of the Univac into some particularly sensitive areas of the Mustang.

        It would also have to be a rather rundown Mustang already to be declared a total loss after a liquid encounter like this.

      3. Antron Argaiv Silver badge
        Thumb Up

        Probably an older Ford.

        The doors on those things would rust out at the bottom seam, unless you dumped a few tablespoons of old engine oil or motorcycle chain lube into them.

        All it took was a whiff of moisture for them to start rotting.

        1. Anonymous Coward
          Anonymous Coward

          Indeed.

          And if the drain holes blocked (usually with rust and dead flies), you could have up to a quart of stagnant water sloshing around in there - as I discovered one time when I took the inside panel off to fix the window mechanism.

        2. The Oncoming Scorn Silver badge
          Coat

          FORD

          Fluid Outside Rusted Destruction in this case.

          Makes a change from Found On Roadside Dead

          1. Anonymous Coward
            Anonymous Coward

            Re: FORD

            Fix or Repair Daily

          2. Anonymous Coward
            Anonymous Coward

            Re: FORD

            Fraked Over Rebuilt Dodge

          3. Stoneshop

            Re: FORD

            Spanish: Fabricacion Ordinario, Reparacion Diario

        3. Stoneshop
          Coat

          All it took was a whiff of moisture for them to start rotting.

          That, plus Lucas electrics on British-built Fords ...

          No, that's not a jacket, that's the tarp to cover my car.

  5. chivo243 Silver badge
    Facepalm

    Oh, oh, D'oh

    A little knowledge is a dangerous thing!

  6. Anonymous Coward
    Anonymous Coward

    Who thought water and computers mixed?

    Not a "who me?" but it reminds me of the time (also back in the early 1980's) when the firm I was working for had their own IBM mainframe to control company stock, manufacturing and finance. It sat in a separate building, in its own room with its own water cooling. Back up, BTW, was a duplicate sitting across the pond, with a nightly satellite link. Two design flaws in the building:

    1) One of the above ground pipes was plastic, and

    2) The floor drain wasn't at the lowest point.

    Needless to say, one day maintenance needed a bit of height to fix something (probably something as innocuous as changing a light-bulb) and a water pipe provided a convenient step up. Being plastic. it sheared off and water sprayed into the room. Fortunately, the spray didn't target any vents but the bunded floor started to resemble a paddling pool. Panic ensued as the computer was hastily shut down. I wasn't working in that department but was one of the folk who had to start placating the several hundred other people who now found they had no access to the data they needed to input or withdraw stock, run any of the four manufacturing shops or any of the myriad other back office jobs.

    Fortunately, the computer was unharmed and, following some urgent baling and re-plumbing, normality resumed by the following day.

    1. This post has been deleted by its author

      1. SImon Hobson Bronze badge
        Facepalm

        Re: Who thought water and computers mixed?

        Yeah, I recall one hot summer many years ago, someone had ordered a load of air conditioning units from a local hire place - and assured me they were the right sort. When the guy arrived to deliver them, I just told him to take them back. Stick a swamp cooler ina closed room (or at least, one with limited ventilation) and the only result is a temporary cooling before you quickly end up with the same temperature - just more humid and uncomfortable.

        We did have a portable heat-pump type unit - and one day we went into a training course to find ... It was hot as hell, and the maintenance guy had wheeled it in and switched it on - oblivious to the need for somewhere for the heat to go (I think he thought it went back up the mains lead, like a fan heater in reverse !)

        But, at my last place, when things got really hot in the server room, we did have a swamp cooler - and it was pretty effective. Though it made the metalwork at the front of the servers go rusty. First one in one hot days got to carry 10 gallons of water to fill it up.

      2. Anonymous Coward
        Anonymous Coward

        Re: Who thought water and computers mixed?

        We have a larger sized version of one of those to spot cool a piece of test equipment. During a recent hot spell, the technician asked for some help, as the tester was running out of spec. I discovered that someone had pointed the portable chiller at the tester (good) but had also routed the dryer vent style flexible tube from the back of the chiller into the side of the tester (very ungood).

        If you're unfamiliar with this style of chiller, the flexible tube is used to discharge the hot air from the system. Normally this is discharged through a window, into a plenum, or just away from sensitive equipment.

        1. Anonymous Coward
          Anonymous Coward

          Re: Who thought water and computers mixed?

          We had one of those portable units semi-permanently installed in my old office (a small room containing 5 cubicles), taking up about half a cube. Two tubes coming out the front blowing cool air, a big tube going up into the ceiling for hot air, and a tub for condensation that had to be emptied occasionally. It was obnoxiously loud, and didn't seem to do much aside from dehumidify.

          At some point, we took a look at the tube going up into the ceiling. It only went to the false ceiling, blowing the hot air into the space above - which, having nowhere else to go, then came back down into the room. Hmm...

          Eventually, someone found a damper closed in the line going to our one small vent. Once that was opened, the office got pretty comfy, and they got rid of the portable air conditioner.

  7. Anonymous Coward
    Anonymous Coward

    Not IT, but involving many tonnes of copper and heat; I have seen organisations in the USA calling in local fire response services to point their pumps at substation transformers to increase their cooling capabilities over and above what the traditional oil radiator-and-fan approach can do.

    It is, from the standpoint of cooling highly effective. One has to question whether asset management or planning practises are fit for purpose to "survive" on one transformer rather than installing two rated to take the load in question.

    For the record, in the UK, it is can be a criminal act to operate HV equipment beyond it's rated capability.

    A/C because not hard to identify what organisation I must be in to be able to write this.

    1. Cynic_999

      "

      ... calling in local fire response services to point their pumps at substation transformers ...

      "

      Doesn't sound safe. One slightly misdirected stream hitting a connection on the HV side of the transformer could result in permanent damage to the hose operator.

      1. Anonymous Coward
        Anonymous Coward

        Indeed. The USA has a rather different attitude than us Europeans do.

        "Hot" washing in the UK is occassionally practised; though you do so with a remotely aimed hose rather than putting a person in the line that can arc back down the water supply.

    2. KBeee

      Similar thing can happen in the UK too. Some years ago during a hot summer, the area around Carnaby Street in London was blacked out due to overheating transformers in Carnaby Street main substation. The "cure" was to drape plastic hoses over the cooling fins with tiny holes punched into the hose along its length, and the hose connected to a cold water tap.

  8. JeffB

    Water main street

    Should the headline not have been 'Water main street'?

  9. Lon24

    The first/only water cooled ICT 1301?

    It wasn't designed that way.

    I was working as a computer operator during my sixth form hols at a well known engineering company in the Black Country. They were a go-ahead organistion housing the kit and a resident ICT engineer in a state-of-the-art glass fronted office block facing south. Is the penny starting to drop?

    Anyway it being the swinging 60s the company had purchased a spanking new ICT 1904. It magically had <drum roll> magnetic tapes. And magnetic tapes needed air conditioning whereas the 1301 drum just rolled along unpeturbed in any conditions.

    So - the false ceiling was torn down to install the AC revealing an extensive network of water pipes. Cold water pipes as it turned out. On a blistering August summer's day the un-airconditioned air produced an absolute deluge as it condensed on the pipes right on to the top of the 1301.

    The ICT engineers horror - not to mention the DPM as meltdown looked inevitable. But equally magically a dozen or so fire buckets were rapidly emptied and arranged along the top of the 1301. But buckets are round so complete coverage was impossible and the waterfalls were fickily moving along the exposed pipes necessitating constant re-configuration of the the computer's new bucket collection. But it worked the 1301 never missed a beat. But then it wasn't running GEORGE.

    Perhaps that was the hardest day's work I have ever done - except, possibly the night before payroll day when one of the punch card machines supplying the data went unaligned and the 1301 reader rejected and halted after every fifth card. This had to be re-punched by one hand while holding up the errant card to spot the holes with the other before the payroll program would continue. The prospect of the Friday pay packet being empty was not an option for a company that didn't want its management lynched.

    1. GlenP Silver badge

      Re: The first/only water cooled ICT 1301?

      But buckets are round

      I had a similar issue* much more recently that necessitated a quick trip to the nearest garden centre for the largest plant trough I could find!

      *In my case an under-specified and badly installed aircon unit that iced up and was dripping onto the server rack.

    2. AndyMTB

      Re: The first/only water cooled ICT 1301?

      George! I loved George (V3 I recall was the one I cut my teeth on). And those temporary files "!" whose name incremented each time you created a new one - ! became !1 then !2 if you created 2 more temp files, so you had to be on your toes to keep track of them. Were the command files called macros? We had the use of an ICL 1904S the other side of the city, a PDP 11/70 on-site and then, wonderfully, a VAX 750 which of course blew the socks of everyhting with it's unlimited virtual memory, all 500Mb of it.

      1. Lon24

        Re: The first/only water cooled ICT 1301?

        When I later joined ICL it was in the early days of GEORGE 3 (or 4 if you wanted paging). MTBF was 15 mins!

        But perseverance, Elon's "crash to learn" style, eventually made it into quite a nice stable OS.

      2. Anonymous South African Coward Bronze badge

        Re: The first/only water cooled ICT 1301?

        It's crazy to think that 500Mb was once the bees' knees... if you look at today's systems with 128Gb (and more) of RAM and Terabytes of storage space...

        It just is mind blowing that we had functional computers with 4k or 8k of RAM...

    3. Antron Argaiv Silver badge

      Re: The first/only water cooled ICT 1301?

      Got those in my basement.

      Foam pipe insulation is your anti-condensation friend!

      1. SImon Hobson Bronze badge
        Go

        Re: The first/only water cooled ICT 1301?

        But it MUST be the right stuff and properly installed. It has to be closed cell so water laden air can't constantly meander in and out, and it must be installed with all joints sealed for the same reason. Ignore this, and sooner or later you'll be ripping off a load of waterlogged insulation and having the right stuff installed by someone with a 'kin clue.

  10. Phrontis

    Even back then the building should have had a "break tank" with an air gap to prevent the buildings water getting back into the main district water supply. They may have contaminated the buildings own water, but not the surrounding district. A filling valve on a toilet cistern works on the same principle, the water sprays in from the top with an air gap, this prevents the possibility of water back syphoning into the mains water system.

  11. Sam not the Viking Silver badge
    Facepalm

    Water and electricity don't mix well.

    We were 'endurance' testing a high-pressure pumpset with lots of control and monitoring gear attached, designed to give warning if 'anything went wrong'. Container-sized generator. Coolers to keep the water temperature down. Big set up, and because it had to run for a few days, access was restricted for safety and security.

    I realised something was amiss when walking past the building, I noticed water coming out of the ventilation louvres 25 metres above ground level, then a shoe float out of the emergency exit. It's not easy stopping these 'temporary' lash-ups especially when there's an upside-down tropical-rainstorm going on in a building packed with electrical equipment. Incredibly, all the electrical and electronic gear survived. On subsequent tests, I made sure we had a stop button outside what was laughingly called the 'Control Area'.

    1. SImon Hobson Bronze badge

      Re: Water and electricity don't mix well.

      We need to know, what was the fate of the previous owner of the shoe ?

  12. Sequin

    We put a Prime mini into and office, and the business decided to partition off a corner of the room as "the computer room".

    One day the outflow for the toilets on the floor above, which ran through the false ceiling in the main office, blew. Guess where the effluent went? The stuff literally hit the fan, which didn't cool very well after that!

  13. VanguardG

    As seen on TV

    There was a TV show in the later 70's that followed members of a certain fire station in the Los Angeles County Fire Department. In one episode, the station is called out for a house fire, which turns out to be a toilet on fire. The homeowner had flipped a lit cigarette into the water, only to have it essentially explodes. The crew hook up to a hydrant, and go to douse it, only to have it flare up. A neighboring yard has a sprinkler on, and a pedestrian flips his cigarette into the wet grass, only to start another fire. This time the crew uses only the water already in their engine to put it out, and they begin searching for the contaminant. They find a crew with the water main hooked up to the fuel oil line, which is supposed to be flushing the fuel oil line in preparation for welding the line. But the oil line was running at roughly triple the pressure of the water main, and the crew had failed to install a one-way valve, so the oil was flooding into the water system, not the other way around.

    Fictional show, of course, but perhaps it took a page from genuine incidents of the time.

  14. Custard Fridge

    Almost time to hit the big red button

    Only once in my 27 years of IT have I had that 'do I power off the ops room or not' decision to make.

    One afternoon in sunny Surrey our cleaner spotted something shining through the air con vents of the raised flooring and asked me as the IBM Mainframe Operator on duty to take a look.

    Looking down I realised there was a pond of water there instead of the usual bone dry concrete, with the air con still happily blowing away above it.

    Rather than lift floor tiles to see how far up the sloped room the water had got - and I was thinking of those four-bars that were stashed under the AIX / Token Ring cabinets... - I did consider powering down the Ops room entirely via the big red button. However as I stood there I realised nothing had stopped working, nobody was dead, nothing was on fire etc., so I legged it up the corridor to pull my boss out of the monthly IT meeting. I'd never seen his eyebrows raised as high as when he looked through the same vent. When we did lift the tiles shortly afterward we had less than a foot to go before the four-bars would have been flooded. The IBM engineer said it was an impressive amount of water for a pin-hole sized water leak...

  15. Boris the Cockroach Silver badge
    FAIL

    2 happening occur to me

    1st one was when I were a young and keen apprentice at all things CNCery... we had a spiffing 5 axis mill, the hallowed area of the gods(well the guy with 5 years experience of loading punch tapes anyway), one bit of it was water cooled for some reason, trickled through and then dumped. until the maintance guy decided "whats this pipe here? " and cut it off.

    Luckily we never started the machine as no one could get near it due to the stench coming out of the drain pipe where the water was dumped into the street sewer.

    The other was a few years ago, when the alledged mangler could'nt get the HP coolant circuit running after the coolant tank was cleaned, however he had seen me prime the pump by loosening a hose fitting on the HP circuit when the pump was replaced..... and we all know that this one ends up with the coolant tank being emptied in about 10 secs by a 300 PSI pump .... was impressive how far it went too ...

    If only he'd listened... loosened.. and not while the pump was running....

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like