back to article So the data centre's 'getting a little hot' – at 57°C, that's quite the understatement

Welcome to another edition of The Register's On Call in which incompetence saves a reader's bacon from a close encounter with the frying pan. Our story takes us back to the halcyon days of 2014 and a medical facility where our hero, Regomised as "Chris", worked. As has been the trend over recent years, the on-premises data …

  1. Anonymous Coward
    Anonymous Coward

    'Dave's syndrome. Poor bastard.'

  2. Anonymous Coward
    Anonymous Coward

    That reminds me...

    of something very similar that happened when I was a server engineer. There was an urgent call from the DC that the AC had fallen over (turns out the seawater feed pipe that drew water in from the nearby harbour had been blocked) and things were getting rather toasty. Our solution was a lot easier (and didn't require turning anything off): we opened the windows once Security had been notified that the alarm was about to go off. Only downside was that they faced a street so we had to post people by each open window to ensure nobody tried to get in.

    Temps came down rather quickly and an interim solution (big fan, open internal doors) was put in place until the pipe was unclogged.

    Have posted this story before, but always worth repeating :)

    1. Wally Dug
      Coat

      Re: That reminds me...

      "so we had to post people by each open window to ensure nobody tried to get in."

      Also, to ensure nobody tried to get out...

      1. Anonymous Coward
        Anonymous Coward

        Re: That reminds me...

        We had the same idea when there was an AC failure. Unfortunately, the systems were graded "UK Restricted", so security weren't that happy, even though this was on a campus site.

        They allowed us to man the open windows and doors for just long enough for us to do as clean a shutdown as we could, and then we had to close everything up again and do an asset check to make absolutely certain everything was still present in the server room.

        1. Anonymous Coward
          Anonymous Coward

          Re: That reminds me...

          The irony of an AC posting about an AC failure... pointed out by another AC!

    2. big_D Silver badge

      Re: That reminds me...

      We had something similar in the 80s, the AC for the data center collapsed (literally, the stilts it was mounted on were rammed out of the way by a delivery vehicle) and the 1st and 2nd floors were without cooling. The "cargo" door as either end of each floor were opened (double width door opening onto a sudden drop, to allow mini computers to be craned into the building) to allow cool air to circulate.

      Another time, the server closet became very loud. The temperature warnings in Nagios set off alarms. The AC was defective and the room had raised from 20°C to over 45°C in about 20 minutes, there was also water dripping from the AC into the top of the rack! AC was turned off and door and window opened and fans placed in the doorway to blow cool air through the room. It managed to hold the temperature at 35°C (middle of summer, so around ambient temperature outside mid-afternoon).

      Luckily a subsidiary was a home heating and sanitation company. They managed to get a new AC installed the next day - and it was properly placed, so that it couldn't drip water on the servers!

      A couple of years back, in the middle of winter, we had a heat alarm in another computer room. When we arrived, the AC was iced up, literally, there was a block of ice hanging out the ventilation flap! Being minus several degrees outside, we just opened the windows. Once the AC was turned off, the ice melted fairly quickly.

      1. SImon Hobson Silver badge

        Re: That reminds me...

        When we arrived, the AC was iced up, literally, there was a block of ice hanging out the ventilation flap!

        Common problem, and often caused by incorrect specification/selection of the cooling system - put another way, the air is too dry so it's iced up. I've had this conversation a few times, because it sounds crazy - how can the air being too dry cause ice formation ?

        With a system designed for comfort cooling in a typical office environment, there will be a fair bit of water vapour in the air. As the air is cooled, the water will condense, and this takes a lot of energy extraction to do. As a result, for a given rate of heat extraction, the evaporator will run warmer than if it wasn't having to extract all that latent heat of evaporation from the water. If everything is within specs, then the evaporator will stay above freezing point (of water), and so the water will dribble out and go down the drain.

        But put very dry air through that same system, and for the same (or less) energy extraction, the evaporator will now run a lot colder - and as a result, what water vapour does get condensed will freeze. As bits of ice will prevent airflow over/through localised bits of the evaporator, then those will get colder still as will the adjacent areas - thus the ice will get colder and harder. Left unchecked, this will continue until airflow is blocked completely by the ice, and the ice will be set "like concrete" in a now exceedingly cold evaporator. The best way to deal with this is by turning off the compressor but keeping the fan running - if that's possible. This will (unless things really are completely blocked) pull above-freezing air over the heat exchanger, warming it up and melting the ice. Better systems will detect such conditions and automatically defrost the heat exchanger - whether that's in the inside unit in cooling mode, or the outdoor unit in heating mode, many don't (I've seen a building with a whole wall full of "blocks of ice" !)

        1. Stevie

          Re: That reminds me...

          Also a possible indicator that the working fluid in the cooling circuit is running low.

          This is why old through-the-window a/c units freeze and newer ones don't.

        2. irrelevant

          Re: That reminds me...

          Thank you for that explanation!

          Early 1980s, I was an apprentice for Ferranti Computer Systems and spent a stint in the department that looked after the computers that work was done on, rather than the ones that we built. Big room, couple of VAX 11/780s, an 11/750 (or might have been the other way around) with the associated tape drives, and a smattering of MicroVaxen to round things off. All linked to terminals all over the various buildings...

          I remember coming in one morning to a panic as one of the three 8' high air conditioning units had failed, causing the other two to ice up, and the room temperature to skyrocket... It's nice to know the mechanics of what happened, after all these years!

      2. John Brown (no body) Silver badge

        Re: That reminds me...

        "A couple of years back, in the middle of winter, we had a heat alarm in another computer room."

        I've never experienced that. It's a very rare occasion for me to rock up at a data centre and actually go in, whether an aircraft hanger type place or a smaller on-prem site. But one I did go to a number of times in the early 00's was bloody freezing inside! I remember setting off one day, outside temp was 30C (that's feckin' HOT for NE England!) and my wife wondered why I was taking my coat :-)

        1. Anonymous Coward
          Anonymous Coward

          Re: That reminds me...

          The AC in our border comms room was utterly and totally US and had been for a long long while. Nobody had bothered telling us this of course. The room always seemed quite comfortable though. Turns out that the HPC that had been put in there had a chiller that was so efficient that it cooled the whole room. Until both circuits failed that is. One had accidentally been left shut off after a test leaving the A unit taking the whole load for months. When A failed the B unit should have been able to cope but the feed had been isolated so the room got hot. Very hot. Very quickly. My servers carked it at 60 C for over 1 minute and went into emergency shutdown. The comms gear flipped out minutes later. The HPC had already shut itself down. The environmental monitoring picked up a peak air temperature of over 55 C and as it was a basement with no windows the gear sat cooking fir a long time indeed! The faulty AC units were identified and replaced soon after.

    3. Anonymous South African Coward Silver badge

      Re: That reminds me...

      Baaaaah. Should've asked Simon for assistance with the windows...

      1. EarthDog

        Re: That reminds me...

        defenestration?

        1. Anonymous Coward
          Anonymous Coward

          Re: That reminds me...

          That's a Praguematic solution...

    4. Anonymous Coward
      Anonymous Coward

      Re: That reminds me...

      In my first job in tech support our DC was retrofitted to a space inside a Victorian town hall, it was completely internal, the AC filters were in our office (also Internal) so when there was a AC issue we were the first to know.

      We noticed that the office had gone very quiet and sure enough we had had a failure of both AC units, the first had failed and then because it was a very hot day and we had some duplicated kit in the DC (we were in the process of migrating mainframes) the remaining unit couldn't handle the full load and tripped out.

      We had some legacy doors from the DC into the 'rates hall' of the town hall and these were thrown open, we brought every fan we could find into the DC to blow hot air out but it was futile effort.

      The staff dealing with the public were caught behind beautiful mahogany counters which trapped the heat around them and the DC just got hotter and hotter. The ops manager fought against powering down the DC until the manufacturer's labels on the new mainframes disk cabinets fell off as the glue melted.

      I was only in that job for a year but needless to say we had many issues with disk errors having cooked everything so well.

  3. Sam not the Viking Silver badge

    Be careful with your sprinklers

    Setting the scene: Three standby generators in individual cells; Set one complete, set two semi-complete, set three in assembly with covers off and engine open.

    The customer wanted to check their sprinkler system before accepting the first plant into service (even thought the station need two to be operational). So the machinery in the 'completed' cell 1, engine, alternator, etc. was wrapped in polythene and tightly protected for the test. Naturally, the customer wanted to witness the event and they donned their deluge-protective clothing: think Hazmat suit although Captain Birdseye might be more appropriate. All other personnel had to clear the building.

    We guessed something was amiss when my shoe floated out of cell 3. In their co-ordination meetings between the various contracts, they had omitted to label the cell-machinery in a uniform manner. Sprinkler cell 1 was not Engine cell 1........

    As the dogged, suited Captain, still completely dry, stepped from cell 1 and into cell 3 he kindly offered his handkerchief to help dry-off a 1MW alternator spouting pretty waterfalls. Early pub lunch for us.

  4. Sgt_Oddball Silver badge
    Flame

    I once had to do something similar in a Skoda...

    Nose to tail traffic because there's nothing like having a busy A road go from dual carriageway to single and back again... And the engine overheating light comes on. Slow enough not to cool the engine, fast enough that I couldn't stop the engine.

    Cue turning the heating on full blow - high heat. Did the trick in saving the engine but cooked the occupants somewhat.

    1. OwenMc64

      Re: I once had to do something similar in a Skoda...

      In a similar situation with my 1st car, I had the windows & sunroof open (top spec when new - £100 to me :-))

    2. Dave314159ggggdffsdds Silver badge

      Re: I once had to do something similar in a Skoda...

      I've had to do the same - unfortunately in a tropical climate, which didn't improve the passenger experience.

      1. The Oncoming Scorn Silver badge
        Pint

        Re: I once had to do something similar in a Skoda...

        You are James May, the occupant was Clarkson & I claim my Five pounds (Icon)!

    3. Edwin

      Re: I once had to do something similar in a Skoda...

      I once had a similar experience in sunny New Mexico in the late 80s or early 90s - a somewhat older vehicle was standing by the side of the highway showing clear signs of overheating. We stopped and offered assistance and showed them the trick of running the heater to help cool the engine.

      They drove off happily, and we followed for a bit to make sure all was well. Sure enough, after a little while, they pulled over again. Turned out it was too hot in the car with the heater running (it was nearly 40 degrees outside), so they'd switched on the A/C, and a late 70s/early 80s A/C compressor represents a significant engine load...

      Advised them to roll down the windows instead, which got them to the next exit.

    4. Anonymous Coward
      Anonymous Coward

      Re: I once had to do something similar in a Skoda...

      Nothing ICT related, but that car story reminds me of the time, many years ago, my (now ex-) wife and I were taking a caravan from near Bristol to her parents who had moved into a former croft near Inverness. It was an old twin axle 16 beast and my 1200cc Mk1 Cortina would have stood no chance even shifting it. We borrowed my brother-in-law’s “tweaked” Triumph 2000 (triple carbed, high lift cams, etc). It managed to tow fine (even in overdrive on the motorway).

      All well until we turned onto the final bit of the journey, off the main roads. We came to a steep hill (1:4, I found out later, when I got the map from the missus). All still OK until we came up behind a coach. The car’s temperature gauge started to creep up, something my brother-in-law had mentioned as a warning. Nothing for it but pull out - straight section of road, nothing coming - and floor it in bottom gear. The look on the coach driver’s face as we passed was a picture…

      I got to respect that car!

    5. GlenP Silver badge

      Re: I once had to do something similar in a Skoda...

      Going back about 50 years that wasn't all that unusual. A relative had to drive everywhere for months with the heater on full blast to prevent the engine overheating. Very unpleasant on a hot day on the M1 with all the windows wide open.

      I can't remember now (if I even knew) whether he ever fixed it or scrapped the car.

      1. Giles C Silver badge

        Re: I once had to do something similar in a Skoda...

        It could be worse - I drive a tiger (lotus 7 type) so fibreglass bonnet and nose with a fairly large Ali fad inside. Doesn’t help when the fan stops working (thermal switch broke) and in a queue of traffic.

        At road speeds the airflow works fine at no speed it overheats and there isn’t a heater fitted to allow for cool downs. The only thing you can do is to pull over and wait for a cool down.

        I do have a manual switch that can be wired in and taped to the bonnet to get it to near me!

        1. Aitor 1 Silver badge

          Re: I once had to do something similar in a Skoda...

          Had that happen to me on a 2000s Kia Shuma II.. on London. Not nice.

        2. adam 40 Silver badge
          Flame

          Re: I once had to do something similar in a Skoda...

          My TVR used to do this but with less discomfort as top-down motoring is mandatory.

          You could direct the majority of the heater air out the side windows too.

          But now I've sorted out the head gasket leaks permanently, and no more trouble (in that area at least).

      2. Kildare
        Facepalm

        Re: I once had to do something similar in a Skoda...

        My experience was back in the 70's driving a Commer van around Taronga Park (think Longleat) in Australia. 5 - 10 MPH.with Syney at it hottest. Heater on or engine overheats - windows closed to keep the wildlife from coming onboard for a free ride (or lunch!). What a relief to get out of the exit!

    6. Tomato Krill

      Re: I once had to do something similar in a Skoda...

      Similar experience excepting the location was the wild dogs enclosure at West Midlands safari park and the warning was steam coming from the coolant tank…

      Fun times…

    7. Andy Taylor

      Re: I once had to do something similar in a Skoda...

      My dad's 1978 Skoda Estelle (in which I learned to drive) had a similar issue. Dad fitted an additional fan to the radiator with a manual control in order to improve the airflow and stop the engine overheating catastrophically.

    8. NXM Bronze badge

      Re: I once had to do something similar in a Skoda...

      Old mini's were pretty similar. No electric radiator fan, it was driven from the crank by a pulley.

      1. John Brown (no body) Silver badge

        Re: I once had to do something similar in a Skoda...

        "Old mini's were pretty similar. No electric radiator fan, it was driven from the crank by a pulley."

        That was normal for most if not all cars back in the day. Checking the fan belt tension was part of the normal servicing of cars. You could always hear a car coming up the road with a slipping fan belt because of the squeaking! It wasn't unusual for the fan belt to also drive the alternator too, so a slipping fan belt was quite a serious issue. Not only did you not get the expected cooling, the battery might not be charging either!

        1. Alan Brown Silver badge

          Re: I once had to do something similar in a Skoda...

          In such vehicles the fan was usually mounted on top of the water pump pulley, so losing the belt was doubly inconvenient

          You can go a fair way without an alternator in a pinch. Not so much when there's no water circulating (and then there's the issue that such overheating issues usually cause all the radiator fins to fall off, making a bad situation even worse over time)

          1. Potty Professor Bronze badge
            Boffin

            Re: I once had to do something similar in a Ford...

            I was once commuting along the M6 between Solihull and Rugby, when there was a loud bang from under the bonnet of my Mk5 Cortina, and the ignition warning light came on. I glanced in the rear view mirror and saw my fan belt writhing about in the middle lane, rapidly disappearing behind me. As I was just going past Corley services at the time, I knocked it into neutral and switched off the engine, making a dead stick landing on the hard shoulder, unfortunately about 1/4 of a mile past the exit ramp from the services. Had the belt failed a mile earlier, I could have made that dead stick landing up the off ramp into the services, but no, I was marooned on the hard shoulder. I had a spare fan belt in the boot, but didn't fancy fitiing it alongside rush hour motorway traffic, so I hopped over the Armco and waited until the radiator stopped boiling. I then started the car and cruised along to the next exit, about a mile, then stopped at the end of the shoulder to fit the new fan belt. Just a matter of negotiating Junction 3 and back onto the motorway, and arrived safely back home, just a little bit later that usual, to face an interrogation about where have you been from a very worried SWMBO. (No mobile phones in those days).

            1. Anonymous Coward
              Anonymous Coward

              Re: I once had to do something similar in a Ford...

              Reminds me of an old Car Talk story. One of the guys had been working on his own car ("That was the LAST time we let him work in the shop!" his brother then said) and headed home in it. He had forgotten to tighten the lug nuts on one wheel. His first warning that anything was wrong was the wheel passing him. He managed to get off the road and into a gas station, sparks showering from the rotor dragging on the ground, and finally skidded to a stop. He jumped out of the car and shouted to the attendant:

              "Fill it up and check the oil!"

    9. Charlie van Becelaere
      Flame

      Re: I once had to do something similar in a Skoda...

      Mine was a 1964 Rambler in about 1985, so the poor machine had a right to be a bit touchy. Driving in 35C+ temps with the heater blowing as much heat as possible made for a less than wonderful drive home from the office.

    10. Martin Silver badge
      Happy

      Re: I once had to do something similar in a Skoda...

      About thirty years ago, I had an Series 1 E-Type (1965) which had a tendency to overheat at the slightest provocation. The trick with the heater was not much use either. Replacing the original 4 bladed fan with a modern 16-blade fan solved the problem for good.

      Not a very interesting story, I admit - I just wanted to boast that I owned an E-Type....

      1. David Woodhead

        Re: I once had to do something similar in a Skoda...

        About thirty years ago, I had an Series 1 E-Type (1965) which had a tendency to overheat at the slightest provocation. The trick with the heater was not much use either. Replacing the original 4 bladed fan with a modern 16-blade fan solved the problem for good.

        Not a very interesting story, I admit - I just wanted to boast that I owned an E-Type....

        Blimey - this is my second Skoda related post in 30 minutes. Anyway: my boast is that my three year old Skoda Superb automatic estate is very slightly faster to 60mph than a 3.8L E-Type, which was always my ultimate benchmark for a whizzy car. (It's not nearly as beautiful as an E-Type though.)

    11. SImon Hobson Silver badge
      Mushroom

      Re: I once had to do something similar in a Skoda...

      Well if we're swapping overheating car stories ... the icon illustrates what an overheated car can look like.

      A while ago, a friend used to do off-road rallying, and at the time I drove a Discovery (an early 3 door V8 - the "pass anything but a petrol station" engine option, except this was converted to gas to make it vaguely affordable to feed). Normally I had no problem, but unknown to me the viscous coupling for the fan had failed. So off we went, having picked up the rally motor from his mates farm, and headed ... up Shap. Well actually, we overheated before we got to the bottom of Shap - the higher load and lower road speed towing another vehicle showed up the cooling problem.

      Pulled over, diagnosed the problem, let it cool a bit and topped up the water - now what to do about the fan as we faced the climb up "that hill". Rummaged around, and found a piece of baler twine in the hedge. Wrapped the loose ends round the hub of pulley and put the resulting loop over a fan blade - then started up. Rotation just took up any slack and it gripped really well, driving the fan at full speed - noticeably noisier, but kept us cool for a few hundred miles up to Scotland, around the various stages for a couple of days, and back home again. Always keep a bit of baler twine lying around, it has many uses :-)

    12. Tom 7 Silver badge

      Re: I once had to do something similar in a Skoda...

      I remember my Dad driving us through the Alps on the way to the Italian lakes and having to have the heating on full blast to keep the engine temp down - and we had to have the windows open to keep us cool. This was in a CND Cortina! We even stopped at the top to roll in the snow to cool down!

      1. KBeee Silver badge
        Joke

        Re: I once had to do something similar in a Skoda...

        Even worse when Dad won't let you roll the windows down because he's afraid other drivers will think he's a cheapskate and hasn't got air con in the car

    13. Tony Mudd

      Re: I once had to do something similar in a Skoda...

      We had the thermostat (that controls the electric fan) fail on a Golf when out in the middle of no-where.

      Managed to short the connection out (so the fan stayed on) with a coke pull-ring - that got us home.

    14. John PM Chappell
      Thumb Up

      Re: I once had to do something similar in a Skoda...

      Exact same thing, even down to the A road going dual-single-dual, on a warm day, in a Skoda.

      I was young at the time (hence a Skoda - was a gift from my parents who figured that if I were going to crash something, it may as well be something cheap) so no direct experience, but luckily I did (and do) understand some basic physical realities, and I had the "brainwave" of doing exactly what you did.

      Passenger and I were not exactly thrilled with hot air being blasted at our faces, on a hot day, but it beat breaking downing in nose-to-tail traffic and a few hours later we arrived at our destination, a little sweatier than we'd have liked but with a working vehicle. :)

    15. NITS

      Re: I once had to do something similar in a Skoda...

      I needed to do that in my first car. A Hillman Imp. In a suburb of New York City. They don't call the Long Island Expressway "the world's longest parking lot" without reason.

      1. Anonymous Coward
        Anonymous Coward

        Re: I once had to do something similar in a Skoda...

        Sorry, Mate, the world's longest parking lot is actually the M25. As it is a complete circle, it is therefor of infinite length.

        1. Soruk

          Re: I once had to do something similar in a Skoda...

          Almost complete circle, it is broken by a short length of the A282.

      2. AanotherAardvark

        Re: I once had to do something similar in a Skoda...

        The Devil's Elbow, now bypassed, was a very steep double bend at the end of a long steep hill on the main road north in Scotland.

        In the 1950's there was a layby at the top. It had a watering can, with the AA logo, filled from a stream. There was a very large sign warning drivers to allow the engine to cool down before removing the radiator cap.

        My grandfather's car had to stop there every time we travelled north.

    16. imanidiot Silver badge

      Re: I once had to do something similar in a Skoda...

      Had the same happen to me in my Fiat in the middle of one of the coldest winters in a long time in NL. Cue me trundling down the highway in the middle of winter with the drivers window opened most of the way, in a t-shirt with the heating cranked to 100. Gotta give the spaghetti-munchers credit, for an Italian car the Brava had surprisingly good heating through the heater matrix and it was keeping the car quite comfortably warm with -7 degrees C air blasting in through the window and into the heater from outside.

  5. MisterHappy
    Alert

    I can finally tell this one!

    Back in the 90's the company I worked at upgraded the mainframes from 2 ICL ME29's to a nice shiny ICL Series 39 Lv40 with the corresponding disk cabinet and tape drives for backup. The problem was that the replacement took longer than expected so the air-con in the room was struggling to cope.

    Initially this wasn't a problem, the 2 ME29's would run the production work while the S39 was being commissioned, things got a little warm but the S39 was shut down at the end of the day. However once production jobs started running on the new box we couldn't shut it down as easily. As everyone except project managers knows, projects always have snags, in this case it was that some of the batch jobs needed more work before they could be moved over which meant that for a few weeks we would be running both MEs & the S39 from 8am to 11pm.

    In the first day it got warm, then hot, then hotter! Then the over-temp alarm in the S39 started going off, a quick check of the manual showed that the alarm would be followed by a shutdown, this would be sub-optimal so a call to the friendly ICL field engineer followed. He told us that there was a reset button located just under the top vents in the box & that pressing it would silence the alarm for 30 mins and cancel the shutdown.

    For the next week there was a step-ladder beside the S39, after about midday whoever was on shift would leave the computer room because it was too hot to work in for more than a few mins at a time. When the temperature alarm in the S39 went off, someone would dash in, go up the step-ladder, reach in and hit the alarm reset. This would then repeat every 30 mins throughout the day until all the batch jobs had finished and everything could be shut down. Remote operator consoles were used to monitor things like tape requests etc & these would be done as fast as possible to avoid the heat in the room.

    After a week the management had a big chiller unit installed. Of course, after the old Me29s were decommissioned the room was usually too cold to work in because the new chiller was not linked to the air-con and constantly blasted chilled air into the room.

    1. Admiral Grace Hopper

      Re: I can finally tell this one!

      There's a reason they were painted in Hot Tango.

      1. PM from Hell
        FAIL

        Re: I can finally tell this one!

        Hot Tango was for the 2900 Series, S39 were painted in Argentinian Grey

  6. Doctor Syntax Silver badge

    "We can but hope that whatever the eventual repair was, the engineers took the opportunity to move on from just water in the DC."

    Halon? Could have been even worse for Chris & colleague if not for the servers.

  7. Anonymous Coward
    Anonymous Coward

    Comms gear went for a swim

    I was called to a certain university in Central Scotland back in the 90s where, during a particularly harsh winter, the main clean water riser for a large multi-story building had completely frozen and shattered at the junction where this entered the building from the street.

    Unfortunately for the university the buildings main comms cabinet, which also functioned as a main campus node point, was directly below the junction.

    When the thaw started, the pressure and volume of water coming out the shattered junction was sufficient to completely fill the cabinet with water until the riser was empty and the cabinet could drain.

    24hrs later and a lot of hard work getting loan gear, all was operational again.

  8. John Doe 12

    What A "Lovely" Guy

    "Hopefully," said Chris, "they never 'fixed' the sprinkler after I left."

    What a douchebag comment to make - if there had ever been a real fire and people hurt / killed those words would have come back to haunt him.

    1. adam 40 Silver badge
      Flame

      Speaking as a fire marshal

      A water sprinkler on top of electrical equipment - not a good idea.

      1. John Doe 12

        Speaking as a rational human being

        A water sprinkler in a fire situation is better than no fire suppression whatsoever!! The fire brigade still use water on things like Tesla cars when they catch alight.

        Also I cannot believe a so-called fire marshal would advocate a non-functional but looking-functional sprinkler system which 5 years down the road people will forget is connected up backwards.

        1. John PM Chappell
          Facepalm

          Re: Speaking as a rational human being

          Firemen use water or something more appropriate in their judgement (based on what may be present) on vehicle fires because it is a *vehicle*, I.E. Not wired up to mains electrical supply. They would not thank anyone for a real risk of electrocution that comes from inappropriately using water on electrical equipment wired to a mains supply.

      2. Nick Ryan Silver badge

        Re: Speaking as a fire marshal

        ...especially where UPSs are involved. They are highly unlikely to be waterproof and when waterlogged will likely happily discharge their electricity through the water.

        1. Richard 12 Silver badge

          Re: Speaking as a fire marshal

          Through the water, yes.

          It doesn't magically conduct up the stream or through the firefighter, it goes to the other battery terminal or the nearest piece of earthed metal which is of course inside the rack that's currently on fire- so who cares?

          The battery may dump its energy by venting-with-flame, but as the place was already burning...

          Firefighters do use water in building fires. They don't check whether the mains supply is off before they start, because they don't need to - the breakers are going to trip it off soon enough anyway.

      3. The Basis of everything is...
        Facepalm

        Re: Speaking as a fire marshal

        True story from a former employer:

        Buildings insurer insisted that a room full of servers and comms kit was a fire risk and a sprinkler head had to be be fitted in there.

        Equipment insurer then insisted that we fitted hoods and doors on the racks to protect the equipment in the event that the sprinkler activated.

  9. Ikoth

    Water Cooled Racks? What Could Possibly Go Wrong?

    A few jobs ago I worked for a UK company that had a site in Sweden, on the campus of a university. Being Swedes, they were keen on environmental stuff and the campus had a fancy, experimental, heating & ventilation infrastructure. The system was comprised of water filled pipes running everywhere to move heat from where it wasn't needed, to where it was. This included the server room in our leased building. The uni's facilities people connected all their plumbing and heat exchangers into special server racks they'd had designed for water cooling. Everything worked well for about 18 months or so, until the day there was a leak elsewhere on campus. We lost several Proliants and a storage array. The liability arguments were still raging when I left the company.

  10. IGotOut Silver badge

    Two questions

    Did no one think of turning the AC off?

    Why have you got sprinklers in a server room?

    1. Rich 11 Silver badge

      Re: Two questions

      Why have you got sprinklers in a server room?

      It was a ground-breaking design experiment. Who knows what exciting new physics might have been discovered in future years?

      The case for the defence rests, m'lud.

      1. Ken Moorhouse Silver badge

        Re: Why have you got sprinklers in a server room?

        We had a problem with the humidity control on the Air Cond in our server room, causing tracking problems with moisture on equipment boards.

    2. Richard 12 Silver badge

      Re: Two questions

      Most likely not designed as a server room, so got the generic fire suppression kit.

      It'll work just fine, the problem is any false trips due to broken aircon and similar will destroy all the very expensive kit in the room.

  11. Stuart Castle Silver badge

    A few years back, I was called by a friend to advise his bosses on providing live streaming. This was back in the early 2000s, so it wasn't as easy as it is now. I'd had some experience through researching it, and advising my own employer.

    As part of the project, I'd been asked to put together a demo.

    So, I met my friend at his office one hot Saturday morning. Made slightly worse by the fact that not only was I wearing a heavy suit (don't usually wear a suit for work), but I'd also been out for quite a heavy drinking session with the very same friend the night before, so we both had hangovers.

    I turned up at his office, and he let me in. We went to the server room so I could work. The Air Con had failed. Not sure what the temperature was because the maximum reading on the Thermometer on the wall was 40 Celsius. The mercury just reached the top of the tube, quite a bit above the 40 degrees level. It was hot enough that half the servers in the server room had overheated and turned off. We couldn't restart the Air Con, so we went around the building, opening every window we could find and getting every fan we could find. We wedged the door to the server room open, and just pointed all the fans at the door, turning every one on.

    As the room was far too hot for humans (even walking in there resulted in being sweaty), and too hot for the servers, we explained to the security guard what we had done, then went for a full irish breakfast at the local O' Neills, all on expenses. No alcohol though, we both needed clear heads.

    After about an hour, the temperature was still well above 40 degrees, but had decreased enough that we could start to bring the servers back online. My friend logged an emergency call with the Air Conditioning engineer, but they clearly have a different definition of Emergency, and had no engineers available for a couple of days.

    So, with the servers back online and running (if a little too hot for comfort), I was able to set up a demonstration system that merely took the output of a local TV station and streamed it on their local network.

    My friend demonstrated this to his bosses the next week. While they were happy with what they saw, they had concerns with the delay between the broadcast and the stream. They ran a small Auction TV channel, so as far as they were concerned, any millisecond of delay was a potential lost sale. I tried to explain that those milliseconds were unavoidable latency in streaming (after all, it takes a finite amount of time to receive the signal, then encode and stream it), but ultimately they decided not to go ahead due to the latency.

    Actually, ultimately, they went bankrupt, but that's by the by. I still got paid for a day's consultancy, and the resulting pay bought me a very nice monitor that I'd otherwise have been unable to afford, so I was happy, if extremely sweaty when I finished.

    1. tfewster
      Flame

      > It was hot enough that half the servers in the server room had overheated and turned off...

      One night I got a call that some of my HP servers were reporting over temperatures and would automatically shut down at some point. From my knowledge of the DC layout, I could tell the Ops over the phone exactly which AC unit had failed and which server would report overheating next.

      Of course, they didn't believe me and had to send someone to check. In the meantime, I was shutting down non-Production servers.

      IIRC, the Sun boxes hit their limit and shut down without warning. The IBM boxes just kept running...

    2. Ken Moorhouse Silver badge

      re: I turned up at his office, and he let me in....

      The way that para reads reminded me of HBO's Chernobyl epic... spine chilling stuff... the mother of all On-Calls.

  12. chivo243 Silver badge
    Alert

    A week without Aircon!

    Back in May we had one of our ACs take a powder, and our service company couldn't source parts for it, nada, so they had to order a new one took 5 days for it to arrive, and 2 days to get it installed correctly. I put fans up on stands at the top of the doorways (3) to the server room, blew the hot air out, with windows open in the adjacent rooms letting cool air in along the floor. Worked OK until we had a warm day... The howling was deafening, from both me and the gear. I was screaming at facilities, they were screaming at the AC company about the SLA not being worth the paper it was printed on! The new unit went down 3 times in the next two weeks. Finally somebody who knew what they were doing was sent in, and solved the problem in about 5 minutes! Been solid ever since, touch wood!

    1. SImon Hobson Silver badge

      Re: A week without Aircon!

      Seen something similar with a client a few years ago.

      Small server room, one rack for servers, one for switches and cabling. One small AC system screwed on the wall. I knew the AC was suitable as before it had been installed, I had the client forward me the details so I could check it was suitable for the continuous use and very dry air - the manual was very detailed with it's performance tables for the various combinations of humidity and temperatures :-) All worked well for a couple of years - then we got a call from the client to say there was a lot of beeping from the server room, it was getting a tad warm in there.

      So they propped the door open, and directed a fan to circulate air which kept it cool enough until the AC engineer could get there. The symptoms were a bit like the system had lost it's gas - there was hint of cooling, and then the unit tripped.

      To be fair, the engineer did turn up in a reasonable time. But then declared that it was the wrong sort of system. When challenged he then declared that it was "the wrong sort of room" and too much heat was coming in through the ceiling - so the clients went and got a roll of insulation to put over the ceiling which, not surprisingly, made no difference.

      After a few rounds of this sort fo thing, they phoned and asked for me as one of those "knows a bit about everything" people. I had a discussion with the AC engineer who was adamant that the system could never cool that room, and he didn't care that it had been working for a couple of years just fine because it can't have been.

      So I rang the AC company, spoke to the service manager and (politely) expressed my opinion that the service tech was "not making sense", the service manager agreed with my opinion, and within a few hours the fault was found (stuck reversing valve) and the unit was fixed. But until then the client had been in a standoff with the AC tech who refused to accept there was anything wrong with the system. So just getting the tech on-site isn't necessarily the end of your troubles.

    2. Tom 7 Silver badge

      Re: A week without Aircon!

      One place I worked at had the server room on the 2nd floor and the aircon worked fine. Until the admin on the 1st floor got some too. On a hot calm day the hot exhaust from their aircon rose straight up into the intake of the server room and basically cancelled out any effort it made to cool it down.

      Turns out its quite a problem in a lot of places though my favourite still remains the portable aircon where the exhaust pipe is hung our of the window to rise back in through said window.

    3. MadManInABox

      Re: A week without Aircon!

      Having spent many years in various data centres and dealing with facilities management people and their usually outsourced service minions, I can definitely atest to the fact that getting the AC Engineer onsite usually means the problem is going to get worse before it gets solved.

  13. Kev99 Silver badge

    Where I used work, the server room in an unused office. The designers had the HVAC exhaust out a repurposed window. The cooler was mounted on a wall about a meter from the server rack. Ever stand behind a jet while it was spooling up? That was the blower on LOW. On high, you weren't allowed to enter the room. All to cool about 8 2U Dell pizza boxes. Oh, and guess where the techs' work area was.

  14. tweell
    Coat

    Power outages, UPS's and overheating

    My team had just finished upgrading a remote (as in >200 miles away) site. We'd pulled fiber and cat5e, changed out their analog phone system for VoIP and installed two shiny new server racks, complete with a very nice UPS. A week later, there were screams of rage from them and demands for my still beating heart as a sacrifice.

    The site had had a long term power outage. After an hour on battery, the phone system along with the rest of the servers had gone down. This was unacceptable! Heads will roll! As the head of the team, I was called on the carpet.

    Brought into the Star Chamber, I mustered my feeble defenses. I had designed the UPS setup to last an hour, and it had done so. I had not been given the budget to do more, and (heresy!) did not believe that it made sense to provide more battery backup. You see, there was a large emergency diesel generator that was supposed to kick on automatically when grid power went down, and it had not done so. Also, my servers had shut down from overtemperature, not power, after running an hour with no cooling in the tiny closet deemed appropriate to install them.

    Well, that wasn't good enough for these great men. They demanded that I reprovision the system for 4 hours - nay, 8! When I pointed out that the system would still shut down in an hour, I was told to put that air conditioning system on the UPS as well. My laughter at this point was not appreciated. I was told to go and get a quote for this, to provide to the board tomorrow.

    They weren't happy with the quote. I helpfully stated that there would need to be some building modifications in order to host this new battery room, and that they'd need to talk to someone qualified to provide that information. As an addendum, I noted the price of a replacement diesel generator, and that it was 25% of the cost of the battery extravaganza.

    The incident was not spoken of again. I understand that the generator was overhauled and tested regularly afterwards. My rise in that organization came to a halt. Thou shalt not laugh at thine superiors, nor shalt thee make them look like fools, and I had violated those commandments. It was just that I've never had anyone demand a battery backup for a 5 ton A/C unit before (or since).

    1. SImon Hobson Silver badge

      Re: Power outages, UPS's and overheating

      Oh, that sounds a bit familiar - but not on that scale.

      But you raise a good point - it's no good providing a long runtime off a UPS if the systems are going to overheat from lack of cooling before the batteries give up.

      At my last gig we used natural air cooling - a fan to draw air in, and another to blow the hot air out. As it took very little power, that was wired into the UPS. Running a big chiller plant would be a different prospect and, as you point out, just fixing the diesel genny is likely to be a better option.

      But going back another job hat, I worked for a smallish giftware manufacturer. At the time we had a fair number of power cuts due to being on the end of a long rural electricity network. Each time we had a power cut, manglement would ask about the price of a diesel genny - what's the point of having backup for the server and phones if the lights and PCs are off in the offices ?

      Each time the facilities manager would dust off the old quote from his drawer - choice of big genny, runs whole site; or little genny, runs specific services but needs investment in switching - and each time they'd decide (with the lights now back on and hence no obvious problem) it was too expensive.

      1. Anonymous Coward
        Anonymous Coward

        Re: Power outages, UPS's and overheating

        (with the lights now back on and hence no obvious problem) it was too expensive.

        Oh God! If I'd had a couple of quid for every time a proactive measure was forgotten/blocked once the original crisis that prompted it was over I'd probably have managed that world cruise.

    2. John Brown (no body) Silver badge

      Re: Power outages, UPS's and overheating

      "I understand that the generator was overhauled and tested regularly afterwards."

      Hopefully, part of that testing including checking the amount of fuel in the tank. There's nothing worse than doing frequent generator tests and then finding you used all the fuel up on the day there's an actual power cut :-)

      1. Anonymous Coward
        Anonymous Coward

        Re: Power outages, UPS's and overheating

        Everybody thought that somebody would check the tank so nobody did.

        1. KittenHuffer Silver badge

          Re: Power outages, UPS's and overheating

          And anybody could have!

  15. petef

    Reverse situation

    I used to provide field service for systems in the deserts of the Middle East. There was one fault that was annoyingly intermittent. My approach was to turn off the AC and bake until the fault persisted. I was then able to diagnose it and so effect a repair.

  16. Anonymous Coward
    Anonymous Coward

    I was working in the lab, late one night ....

    When my eyes beheld an eerie sight - a dripping liquid nitrogen line. Shut everything down, loosened the connector but it still wouldn't disconnect. With the application of a gentle hammer - I discovered I hadn't depressurised the system.

    When you dump the entire pressurised storage tank of liquid nitrogen into the lab it gets interesting very quickly. Queue very scifi alarms and flashing lights - it lacked only Sigourney Weaver and a cat.

    Since we were getting regular fines from false alarms I called the fire brigade to tell them it was all OK. Which is how we discovered that our fancy fire/chemical spill/gas monitoring system wasn't connected to anything. But since I had "informed them" they were on their way - more flashing lights, big men in breathing apparatus and excitement

    It only got scary an hour later when the Police arrived to cone off the "chemical spill" that the fire service had logged.

    "Now then 'sir', Where's this spill then?"

    "Oh it was just air, it's blown away by now!"

    Some very suspicious looks by the local constabulary about what these boffins get up to and have they doomed the local town to some sort of zombie plague

    1. Anonymous Custard Silver badge
      Boffin

      Re: I was working in the lab, late one night ....

      Although to be fair, L-N2 (and indeed any cryo-gas like L-He too) should be treated with respect.

      Not just for the temperature, but for the sheer amount of gas produced when it vapourises at room temperature, and that whilst air may be ~80% of the stuff, if you get too much more of it around you quickly get into a nasty asphyxia situation. Something akin to being in the server room when the halon systems go off, but a little more subtle.

      The big men in BA would be OK to pull you out, but lesser mortals who may try to come to your aid in an almost pure N2 environment will just end up dropping beside you through lack of oxygen. A pure N2 environment doesn't look any different to a pure air one after all.

      One I know of both from past academic life (having L-He and L-N2 magnets quench on me) and also current day job, where we work with tools containing pure N2 mini-environments. Worldwide across the last couple of decades that I've worked on them, there have been at least 3-4 cases either of near misses or unfortunately people being killed through taking short-cuts with interlocked doors and not bothering to let things fully come back to an air environment before opening/entering.

      Whenever I teach newbies about those tools and environments I always ask what is the most dangerous chemical used on the tools (and they use some nasties like Silane, Fluorine and HF). Simple answer is it's the N2, as everyone is so familiar with it and thinks it's safe as it's most of the air. No-one "fears" it like they do the other nasties, and that complacency can be lethal.

      1. Anonymous Coward
        Anonymous Coward

        Re: I was working in the lab, late one night ....

        Yes I opened all the windows and checked the rest of the building was empty before calling 999.

        We weren't allowed to be in the lift with any amount of He (4mL -> 24L !) but LN2 was thrown around. Main danger with LN2 was burns from touching the wrong bit of metal - or pouring it on your trousers

      2. imanidiot Silver badge

        Re: I was working in the lab, late one night ....

        have a coworker that had a bit of an incident with an N2 leak. He was even wearing a canary (O2 monitor) and had done some checks. All seemed clear. While checking for loose connections he ducked into the back of a cabinet. And almost immediately blacked out and fell back out where he quickly came around again. Turns out the back of the cabinet was badly ventilated and there was an N2 leak back there. He had inhaled right as he had bent down and stuck his head in. And if you're breathing absolutely pure N2 with NO oxygen there's basically no hypoxia effects apparently. It's just straight to lights out. The O2 monitor clipped to his chest never even went off.

        Had a lucky escape that he fell straight back out without injuring himself any further and had coworkers on hand with knowledge of the effects of N2 to catch him and make sure he was alright. Came off as far as anyone can tell no worse for wear.

  17. DS999 Silver badge

    Underpowered AC

    When I worked at a university long ago, there was a lab for grad students that doubled as a "server room" of sorts, with a dedicated Liebert AC. It had worked well for years, but as acquisitions were made that included such hot items as a maximally configured SGI Onyx and a stack of two dozen PA-RISC workstations used as a compute cluster it became stifling in there - because the system had been set up to maintain a fairly high humidity I guess to avoid static electricity since it was carpeted (it was also a lab, remember) in the summer it would be as hot and humid in there as it was outside on the worst days!

    I had facilities look at it and they basically said it was working fine, just woefully undersized for the amount of heat that was being generated. Short of replacing the entire system for a budget breaking amount, there was nothing to be done. So we just ran that way my last couple years, and probably long after I left. Never had any hardware in there fail, not even a single hard drive, so other than the grad students who didn't want to use the lab any later than 10am or so in the summer it didn't really cause any problems. I always wondered what would happen if that AC failed, but it never did during my tenure.

  18. Steven Guenther

    similar and worse

    We had a "automated" server facility, no need for humans to be in the room.

    We had a power drop on a Saturday morning. It turned off the A/C but the servers got past it with UPS. The servers hummed along merrily. The A/C did not come on. Did I mention it was Florida in the Summer? By Sunday afternoon the servers were shutting down due to heat. Once they cooled, they would try to boot back up. Then they would crash again. It took a while to piece the data all back together.

  19. Old Used Programmer Silver badge

    More than one way for A/C to fail...

    In the early 1970s, the company I worked for had the offices in a converted wharehouse in San Francisco, CA. When I started there, the computer and the programmers offices were in the basement. Eventually, the company remodeled space on the top floor for a new computer room (and a newer, larger, computer) as well as our desks.

    Everything was on the same A/C system. Said A/C worked by putting out REALLY cold air and then using hot water heat exchangers at the vents to bring it to the desired temperature. During the winter, the hot water system failed. We were told that the A/C couldn't be shut down, lest the mainframe overheat, so we were allowed to wear sweaters and gloves at work.

    Even though the office area was against an outside wall over the former (railroad) loading dock, we couldn't just open the windows, as the remodel had put in ones that couldn't be opened. This despite the fact that, even in winter, it was warmer outside in SF than the A/C was making it inside.

    1. Anonymous Coward
      Anonymous Coward

      Re: More than one way for A/C to fail...

      On a side note, that's a pretty normal commercial-scale A/C system - supply cold air to the general area, then each thermostat controls how much hot water (or steam) goes through the coil, allowing for much more precise temperature control. After all, the east side of a building will have a higher heat load in the morning and lower in the afternoon than the west side.

      Which comes with the odd side effect that keeping the temperature cold is always cheaper, even in summer, as it uses less hot water or steam. (Chilling the air also dehumidifies it, so increasing the cold air temperature isn't necessarily a good idea.)

  20. Peter Gathercole Silver badge

    When 'hot' rooms started becoming fashionnable

    In the 2010 decade, thinking switched from keeping the whole server room cold, and beyond hot and cold aisles, to putting containment curtains around the systems that needed most cooling, and trying to channel the cold air to the hottest systems.

    At the time I was looking after some IBM Power6 575 supercomputers, and the server room managers put containment curtains around the whole clusters.

    Very quickly, we started getting over-temperature alarms coming up on the hardware management consoles for some of the air-cooled components (IBM large systems just have so many sensors of different types, and only the compute nodes were water cooled), normally at quiet times when there was nobody in the server room. Just to check, I put a max/min thermometer on the rack closest to the system that was complaining, and indeed the input air being pulled to the racks was too hot.

    There then ensued a bit of an argument, with the room managers pointing out that the air leaving the air ducts was well in spec, and us pointing to the thermometer and the readings from the temperature sensors (I was even shown by the 3rd level support how to read the individual sensors in the systems, and there were actually dozens for these systems).

    It took a while for the penny to drop. Although the air was cold enough, the rate at which it was being delivered to the contained area in front of the systems was too low for the fans pushing the air through the racks, and the systems were making up the shortfall by pulling hot air over the top of the racks, leading to a feedback loop allowing heat to build up. Before the curtains, the air would just come from the whole aisle in the front of the rack, but once the curtains were in place, that could not happen.

    The initial solution was to remove the curtains, but eventually they managed to get the right amount of air delivered.

    We didn't have the problem with the replacement Power7 775's, as they were almost completely water cooled, and the small amount of air put through the racks came out colder than it went in!

  21. Anonymous Coward
    Anonymous Coward

    I have to be careful with details

    But I was on a project that was emulating Sun's container datacentre design for portable systems in case of disasters etc.

    There were two containers, 1 for systems and 1 for all the cooling.

    We were still in project and hadn't handed over the environment. Our system monitoring had been put in, but the group implementing the environmental monitoring hadn't got round to it yet.

    On a Thursday afternoon around 3pm, the coolant started to leak. We started to get alerts on systems around 10pm that they were shutting down due to maxing their highest heat running tolerance, but not all systems do this. The switches for example just kept going and were found to have reached 90 degrees Celsius.

    The person that went to open the door of the container nearly got blasted out of the way by the positive pressure it caused in the container. The only reason there wasn't a fire was probably because the container acted as a huge heatsink and it was quite a cool summer night.

    Amazingly, not one system failed or had to be replaced but we strongly advised for the company to submit an insurance claim as their warranty was shot on a lot of the systems. They didn't.

  22. Ashto5

    Jeans company

    Had a halon system installed

    On the test day it fired off and dumped the halon into the server room

    All good, until we tried to leave and we found the positive pressure held the door in the shut position as it opened inward

    That bit was a bit scary

    1. Anonymous Coward
      Anonymous Coward

      Re: Jeans company

      One particular company I know did their first halon test in a while.

      A couple of slightly blocked valves meant that the higher pressure knocked over half the server racks.

      They were counting themselves lucky they found the fault outside an emergency.

    2. Anonymous Coward
      Anonymous Coward

      Re: Jeans company

      Folks think about 0.1 psig being a really low pressure, and it is - until it's on a 7ft x 3ft door. Then it's applying 300 pounds of force to that door. Make it 1 psig and you'll knock the door right out of the wall.

  23. dl1

    had to go in on a sunday evening for the same kind of thing, but I forgot the door alarm pin, so had to do the shutdown with added stress of insanely loud alarm and imminent arrival of the soho cops (they never came)

  24. Unicornpiss
    Flame

    Similar story..

    One of our facilities was housed in a few units of an old strip mall while a new facility was being built. This was in Texas. The server room was just an average room with a few racks in there. There was a big A/C vent in the room, and normally it was quite cool in there. One thing that hadn't occurred to the planners was that they took occupancy in May, just when things were starting to warm up. Come September, the heat was turned on for the first time, and with the 'server room' lacking separate temp controls, that nice big vent efficiently heated the room nice and toasty. The thermostat for the system was located in an area that never got properly warm, so it just ran and ran. We did have a monitoring module added onto the UPS, and it started spewing emails (on a Sunday of course) that the temp was going up and up. I think it got to about 110 before someone was able to get to the facility and turn off the heat, open doors, run fans, etc. None of the equipment suffered any ill effects. The door open/fan solution remained in place until summer came again, and we dialed down the threshold for temp reporting on the UPS. This would happen a couple more times as staff with short term memory loss would shut the door until we just took it off the hinges for the season.

  25. Anonymous Coward
    Anonymous Coward

    Ever had your bacon saved from a frying pan encounter thanks to negligence of others?

    Long time ago (late 90s?) working for the NHS in a hospital in Scotland, we had a server room which was protected by a fire suppression system (it wasn't Halon, but everyone called it Halon). There was a stupid problem with the manual/automatic key switch where it didn't have a battery backup, and every time the Estates department did a generator test, it lost power, and switched to manual. They then had to come round and insert the magic key and switch it back to auto, however, they were pretty slow in getting round to it, no matter how many times we badgered them about it.

    Cue the inevitable, someone notices smoke coming out from under the door to the server room, and the smoke alarm has triggered inside the room. Fire brigade are on an automatic attendance, because its a hospital, and when i come back from my lunch, there are 6 firemen, 3 estates officers and 2 of my IT colleagues outside arguing that you can't go in because you don't know if the "Halon" has been discharged.

    I confidently looked at the control panel, saw it was still on manual, 4 days after the generator test, and said "of course it hasn't gone off, the panel is still at manual because estates haven't changed it back to auto after the generator test, even though we've been asking them since Monday" and opened the door.

    Fire brigade duly go in and discover one smoky APC UPS, which gets unplugged, sprayed with a CO2 extinguisher and removed.

    Estates got a bollocking from the fire department, and rather than do the right thing and fix the control box to have a battery backup, they gave us 3 spare keys to the control panel so we could turn it back to auto ourselves.

    A/C as i still work for the NHS

  26. Anonymous Coward
    Anonymous Coward

    server room

    At work there is a server room in the basement (no windows) it has cooling units to keep it cool but in the weekend it get warm. It gets the air from the corridor and the ventilation stops out of office hours.

    On another thought most server rooms are treated if they are in the Nevada dessert. A computer manages to get it's own warm air out, just vent that outside and no need for huge coolers in our climate. But no it's just recirculates it own hot air through the room.

  27. Moonrunner

    Another car story

    I don't have much to contribute about on the job cooling system issues. I did work for a company that was winding down and wouldn't invest in new kit as the soon to be new owners had their own DC and only needed to move some specialised encoders over to their own DC. Did have a midnight call to drive out to a truck stop and get some diesel oil when the backup genny shut down due to low oil pressure. Fortunately, I did have some lying around from my motorcycling days (at that time, diesel oils didn't have moly in them, so they made a great, cheap alternative to 'motorcycle-specific' oils for wet clutch uses). That got us through the night, until we got someone in to fix the genny.

    On the topic of cars, while visiting the old country, my stepmom loaned me her old '95 Escort diesel. At some point, the cooling system crapped out. Equipped with a couple of lengths of wire, it took me about 5min to determine that the thermoswitch needed replacement. Having spent most of my life in the colonies, I didn't know where to get one down there, so I contacted a childhood friend, who referred me to his mechanic. I called the owner and arranged to bring the Escort in to replace the thermoswitch. Once I got there, he left me in the capable hands of his apprentice, who proceeded to tear the car apart and poke around with a multimeter for 3 hours, before replacing the thermoswitch. Then, he took me to the owner, so we could figure out how much I owe for the repair:

    Owner: What did you do?

    Apprentice: Thermoswitch replacement and 3 hours of diagnostics

    Moonrunner: [If looks could kill, mine would have outperformed Hitler/Mao/Stalin/Hirohito in terms of body count]

    Owner: No charge. Have a nice day, sir!

  28. Bruce Ordway

    Reminds of a site in northen U.S.

    HVAC related but... much lower potential hazard at a site in northern U.S.

    They had a small server room, original install/setup occurred during the summer.

    On an especially cold winter day I had to perform some maintenance and noticed it was too cold where I was working and very hot in the opposite corner.

    It din't take long to identify the reason, central heating duct was blasting hot air and the AC was trying to compensate... 24 hours a day.

    As a temporary workaround I plugged the heat vent with rags.

    However, I heard that my temporary fix remained in place for several years before operations finally got around to closing off that heating duct.

  29. Hazmoid

    Server room in multistory tenancy

    We have a server room with dedicated AC and had arranged with building maintenance to have it hooked into their chilled water loop (everyone knows that you look after building maintenance and they will look after you). anyway we started getting alerts that the room was heating up, so I attended ( being the closest to the office) and found that the AC was spilling water all over the desks outside the Server room (fortunately it was not mounted above the servers.)

    I hunted around for desk fans to circulate the cooled building air through the open SR door and let my boss know.

    We ended up with a massive temporary portable unit that had a 10 Litre catch can for the water that had to be emptied every 6 hours so guess who the muggins was that had to visit the office every 6 hours? for about 2 weeks until we could get a replacement for the ceiling unit.

  30. Bob Scrantzen

    I was a PCB designer for Dealing Rooms. You know, the Mission Control, Apollo 13, rooms for the Bankers in the City of London. Buy, Buy, Buy, Sell, Sell, Sell

    Our biggest was the £15 million Broadgate, Union Bank of Switzerland dealing room on Liverpool Street.

    3000 miles of coax in completely full false floors and false ceilings (no room for airflow)

    About 30 x 19" cabinets served the 600 dealers. All with 4 x RGB CRT monitors on their desks.

    I specified the power supplies in the Machine Room cabinets. 4 x 500W, 2kW each.

    We needed it and I hadn't designed all the boards. Not my fault, honest!

    A Dealing Room Cannot Fail. But ours was reaching 50C.

    I had my only Site visit (I was a Designer, remember. I wasn't involved with the Installation)

    At Reception they directed me to the first floor from the lovely, posh granite foyer.

    I went to the stairs. The door was locked! Reception said I had to take the lift.

    So I went up one floor and was trapped in the Elevator foyer while my host came to escort me.

    3 revolving doors we had to go through. Tiny, one person at a time.

    Very Secure.

    On arrival at their team's Den, I asked where the toilets were. He rolled his eyes, handed me his magnetic security key and pointed me back out the way we had come.

    Anyway, I measured temperatures and recommended what the Broadgate Building Manager and our Installation Manager should do

  31. neilo

    Worked for an Air Conditioning manufacturer once

    The server room was small; three racks with mostly switches, a handful of actual servers and a big tape library system.

    The cooling for the room was a wall mounted unit. It was a nightmare; every hot day it would fall over, allowing the server room to quietly cook. "But this was an A/C manufacturer" you say. It certainly was, and too cheap to actually put a decent unit into the server room. Heck; we had to book service calls for that unit! And we were staff!

  32. Dave Null

    I was once responsible for a couple of racks of mainly Compaq Proliant G380s in a Clinical Trials ward of a large pharma company. Everything had to work in isolation from the rest of their infra, so the server hardware was placed in what was effectively a specialised large safe with built in aircon in the back of the nurses' changing area. It was assembled on site around the racks by the specialist supplier, Lampertz, and in general worked well. It was deafening to work on as you had to open the front access door and from the rear the built in air con blasted out which meant that doing maintenance on the servers left you unable to hear anything else going on, including people coming in to get changed, which led to some embarrassing scenarios...

    That aircon unit failed once, and I can tell you that at pretty much precisely 57 degrees, that era of Proliant will just shut itself off. It didn't appear to have any long lasting damage...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022