back to article I heard somebody say: Burn baby, burn – server inferno!

Welcome to On Call, The Register's weekly dip into the big bag of woe unleashed by users on those responsible for picking up the telephone. Today's remembrance takes us back to 2003, the year of Concorde's final flight and a time when Reg reader "Chris" was working as a CAD engineer at a semiconductor corporation. It wasn't …

  1. Anonymous Coward
    Anonymous Coward

    Oh so special's

    Yes, I've worked with the "Oh so special's" in the past.

    Brilliant at what they did, but speech and common-sense were not amongst those things. Some were even banned from learning to drive.

    Handled with care, they were valuable members of the workplace, if not society.

    But they should not be "handled with care" to the detriment of the whole company.

    1. Anonymous Custard Silver badge
      Headmaster

      Re: Oh so special's

      Reminds me of my post-grad days in academia (too many decades ago).

      A lot of professors who were the absolute authority in their (often highly specialised, deeply theoretical/mathematical and abstract) fields of research, but who seemed to have difficulties with high tech things like making a cup of tea or tying their shoelaces. Not to mention actually being able to talk to people on any other topic.

      1. MiguelC Silver badge

        Re: Oh so special's

        One of my wife's PhD colleagues once had to take the tube to go to a meeting. She asked around for tube schedules... not maps for directions, mind, she really wanted to know at what time did the tube depart from the U's closest station.

        1. MJB7 Silver badge

          Re: Oh so special's

          Should come to Zürich. Technically there is no such thing as a tube here (the locals vetoed it in a referendum some years ago), however there are a lot of suburban railways that run through tunnels - and all the services are timetabled. The busses and trams also have timetables ... and keep to them.

          1. Kevin Johnston Silver badge

            Re: Oh so special's

            By co-incidence on a vist there at the moment but in the past I worked in Zurich for almost two years. It took me less than three months to reach the point of tutting whenever the tram was a minute late

            1. A.P. Veening Silver badge

              Re: Oh so special's

              I see you've reached the apex of Swiss humor.

          2. Stevie Silver badge

            Re: Oh so special's

            Well where's the fun in that?

        2. Hans Neeson-Bumpsadese Silver badge

          Re: Oh so special's

          For someone unfamiliar with the Tube, that's probably not an unreasonable question. Where I live there is an electrically-powered mass-transit urban rail system, but at some periods during the day the trains only run one every 15 minutes. If I'm making a journey at that time I take a quick look at the timetables to make sure I'm not going to be hanging around on a platform when I could be doing something more useful with my time.

          1. ICPurvis47 Bronze badge
            Angel

            Re: Oh so special's

            We spent several consecutive summer holidays camping in the Bernese Oberland in Switzerland. We bought a Swiss Rail Card each and travelled all over the area by bus, train, cablecar, and funicular, and the longest wait we ever had between two connecting services was seven minutes. Except on one occasion when we turned up at Lauterbrunnen station at about 9PM and decided to walk up into town and get a takeaway. As a result of this, we missed the connection and had to sit on the station platform, eating our chicken, for a whole 15 minutes! The whole of the Bernese Oberland is like a huge model railway, and everything just works.

            1. H in The Hague Silver badge

              Re: Oh so special's

              "Bernese Oberland in Switzerland"

              Yes nice area, used to go there for family holidays when punched cards were a thing. A few decades later Pater went back with his mother and two sisters and a railway employee mentioned to them that despite their somewhat advanced ages they still qualified for a cheap parent-and-children ticket.

            2. Paul Cooper

              Re: Oh so special's

              Agree entirely about your comments - a train a minute late is a cause for severely raised eyebrows! However, I happened to find out how they manage it when I was staying at a hotel in Interlaken while attending a conference in Thun (the hotel in Interlaken was run by people we knew well). I was commuting daily by train from Thun to Interlaken along lake Thun, and one day we were held up for about 10-15 minutes at one point along the line - long enough that there was an announcement on the train apologizing. But we STILL made it to Interlaken West on time. It turns out that the amazing time-keeping is because there's enough slack in the timetabling to allow them to make up time for almost any conceivable delay. ISTR that some years (probably decades!) ago, the Swiss railways seriously considered taking on British Rail - but backed off when they realized that they couldn't run timetables with that sort of slack in them.

        3. Tom 7 Silver badge

          Re: Oh so special's

          As we rattled around the emptying Uni after degrees had been handed out I bumped into one girl from the Maths dept (1st probably with 150% as was allowed in Maths!) standing outside her hall of residence with a hefty suitcase and looking lost. She wanted to know how to get to the station - her parents having decided she should be able to cope on her own not dropping her off and picking her up every term terminus. Turns out she hadn't even been off campus in three years except in her parents car.

          However my dad was a uni head of dept and I'd spent most of my life rubbing shoulders with people with astronomical academic achievements and some of them could creep you out from across a lecture hall. A technique developed when doing PhDs and having to earn money helping out in lectures and practical classes and realising they hate human interactions!

      2. ColinPa

        Re: Oh so special's

        40 years ago we had a lecturer whose specialty was 10 dimensional space. He was considered "odd". He was always muttering to himself, and he rode a bicycle because he didn't drive. In his view it was easier to be in 10 dimensional space than the real world.

        He is a different lecturer from the one who put his pipe in is jacket pocket, and the jacket caught fire.

        1. Stevie Silver badge

          Re: Oh so special's

          I had a friend a the University of Climategate who did that! He was at a debutantes ball and was approached by a pretty young thing whole he was sitting on a walled fountain smoking his pipe. Realizing he was about to Make Contact he placed his pipe in the side pocket of his green velvet jacket (we all had them in '76 - mine was midnight blue/slate grey depending on the direction of the light). He was, in his own words, "doing well" when he realized his jacket was on fire, and he leapt into the fountain to effect fire suppression.

          All this was revealed when I ran into him at term start and noticed the gaping burned hole in the jacket, and it's somewhat bedraggled nap. I'd have disbelieved it from anyone else, but this chap was utrterly believable.

          1. jake Silver badge

            Re: Oh so special's

            "we all had them in '76"

            Who is "we", Kemosabe?

            On the other hand, do you think it would have gone with my Rastaman Vibration Tour tshirt?

            1. Stevie Silver badge

              Re: Oh so special's

              The "in" crowd, Tonto-sorry-Jake. I can't help it if others were such unwhacky vacuous glooks they neglected to obtain a nice velvet jacket. I pulled like a quasar in that jacket (and my 9 foot crocheted white scarf thank you gran).

              But they only worked with collared shirts I'm afraid. All the coolness ran out if improperly accessorized.

              As for T-Shirts, my "Trust Me" T featuring a lecherous goblin was a firm fave until some git swiped it from the D block drying room in Suffolk Terrace. I recently found a pic of myself as a handsome youth wearing a Caravan shirt in Sheffield (I was there for a halloween party) that must've gotten swiped in '75 too. Happy days. I was thin, the music was great and all the girls were gorgeous.

              1. jake Silver badge
                Pint

                Re: Oh so special's

                Ah, yes ... the Canterbury scene. I saw most (all?) of that bunch in the mid-late '70s, in one guise or another ... to the point where Steve Hillage recognized me[0] at a gig in LA in 1976 during his L tour. Seems that me being the only Yank in a couple of the smaller British venues was rather memorable to him. To say I scored points with my date is an understatement ...

                Somewhat strangely in light of this conversation, Malcolm Cecil's T.O.N.T.O. (The Original New Timbral Orchestra) came out of that crowd, and was rather heavily used by Hillage on his next album, Motivation Radio. (TONTO was played by Cecil and Miquette Giraudy on the album. Giraudy went on to play a subset of the kludge live on the MR tour. Fun to watch.) The link above is well worth eyeballing when you've got ten minutes to kill.

                Happy days indeed. And the girls are still gorgeous. This round's on me :-)

                [0] No, not by name, it was my butt-length pre-punk hair that he recognized. Probably didn't hurt that I wearing a 1975 Gong concert tshirt ...

                1. mdubash

                  Re: Oh so special's

                  Shame on you for not wearing an afghan coat as well. Preferably inside out...

                  1. Tom 7 Silver badge

                    Re: Oh so special's

                    Ah the old olfactory fashion statement, You could tell someone was wearing an afghan before you opened the door for them to enter your flat.

                2. Stevie Silver badge
                  Pint

                  Re: Oh so special's

                  Ta. Slurp.

                  'Course, if you'd had a velvet jacket you'd have been recognized by Didier Malherbe *and* Daevid Allen too. 8o)

                  Flying Teapot. Angels Egg. Fish Rising. Blimey, takes me back. Saw Hillage & Co. in Norwich once, somewhere in town. Can't remember the venue.

                  Also: One of the ladies on my 3rd year Suffolk Terrace corridor* knitted me a pot-head pixie hat. Still have it somewhere.

                  * - yes, it was a mixed corridor. Experiment started that year. Lads 'n' ladies mixed in one big happy family. Mostly. Better days.

              2. DiViDeD Silver badge

                Re: Oh so special's

                ... my "Trust Me" T featuring a lecherous goblin ...

                Oh bugger! I had one of those too, except it was a yellow scoopneck with those trumpetlike flared sleeves.

                Wonder what ever happened to it? And my matching pale blue with the "Patience my arse - I'm going to kill something" vultures.

                Perhaps it's best not to know. I mean I know what happened to my floor length white afghan coat, and it wasn't pretty!

                1. Stevie Silver badge

                  Re: "Patience my arse - I'm going to kill something" vultures.

                  I remember that one too. I never had an afghan coat. *I* had the altogether more fabulous Civil Defense Greatcoat. Buttoned up top to bottom and I looked like either a blue milk churn or "Clunk" from that Dick Dastardly cartoon show. Weighed a ton. Warm though, and it provided good body armour if things got bellicose.

                  Fun times.

                  1. jake Silver badge

                    Re: "Patience my arse - I'm going to kill something" vultures.

                    I got thrown out of SantaPod in '75 for wearing a Ed Roth airbrushed "Rat Fink" tshirt ... Was of the eponymous character driving a blown hemi powered dunebuggy with the caption "Injection is nice, but I'd rather be blown" on the front, and the schedule for the '74 Northern Nationals at Fremont Raceway[0] on the back ...

                    Everybody who talked to me about it wanted to know where to get one, including the track officials who escorted me out ... but it seems that one fuddy-duddy threatened to have the place shut down for public indecency if I wasn't removed. The same Roth design (sans schedule) was available at racing events (and flea markets, etc.) all over the West Coast of the US that very weekend ... and they say us Yanks are prudes!

                    [0] When the air was right we could hear the high horsepower cars making practice runs from my elementary school in Palo Alto in the late '50s and early '60s ... The track was about 10 miles away to the North East across the bay, and the sound would somehow become focused between the inversion layer and the bay/salt flats. Fortunately I had a gear-head uncle who took me to events on a regular basis, sparking a life-long love of working on large lumps of noisy, hot, oily metal.

          2. Stevie Silver badge

            Re: Oh so special's

            whole==while

            it's == its

            dammit.

          3. Tom 7 Silver badge

            Re: Oh so special's

            My dad had the ability to make a biro leak profusely simply by putting it in his jacket front pocket. He could pop one in there and and 10 minutes into the lecture he would have a large blue stain on his white shirt. Some of his PhD students (who often came for dinner and drinks) said they put pens in while he was distracted and there was never any evidence of any activity that could cause the pen to leak and yet...

            Not the best superpower.

            1. Robert Carnegie Silver badge

              Re: Oh so special's

              Isn't it a basic design flaw of minimal-design Bic pens and similar... pocket clip is on the cap, so you put the pen in your pocket with the clip at the top... which means that the tube full of ink has one end open - at the bottom. Add body heat and there goes another shirt. Or trouser, but since I mostly wear basically black trousers, it only showed on my leg, when I undressed.

              Bic four-colour biro has a clip along with the complicated apparatus at the top, and your ballpoint comes out the bottom. That just leaves the problem of not losing the beggar. I have mine on a plastic "phone cord" clip (with elastic band to the pen actually), so that if it falls out of my back pocket, it just hangs down and tickles my knees. My elastic band broke today, but I've got loads at home.

              1. Anonymous Coward
                Anonymous Coward

                Re: Oh so special's

                The pens I've had leak have always done so from the tip, rather than the top. (Sometimes due to, er, dropping the ball. Literally. Ball of ball-point disappears, pen leaks.) Clip on cap, so pen hangs upside-down, seems reasonable. But I'd be open to seeing evidence to the contrary.

        2. disgruntled yank Silver badge

          Re: Oh so special's

          The American author Guy Davenport claimed to have once seen Jean-Paul Sartre when the latter's jacket pocket was on fire. After trying and failing to call Sartre's attention to this ("Monsieur vous brulez!" or thereabouts), he or his friend poured a glass of water into the pocket.

      3. ClockworkOwl
        Boffin

        Re: Oh so special's

        I'm not sure Richard Feynman would fit into any of those stereotypes, however Paul Dirac was only vaguely human at times...

        I think the 'Doddery Professor' is a learnt routine that just overtakes some people when they have all day to day concerns taken care of, for them. No common interaction, no common sense.

        1. Daedalus Silver badge

          Re: Oh so special's

          The all time prize must go to math genius Paul Erdős, "Have satchel and sandals, will travel" who would go from conference to conference, show up on doorsteps expecting (and getting) hospitality, and basically live the peripatetic academic monk lifestyle across Europe and the world in general.

          Honourable mention though, to a particle physicist I once helped move to the great cathedral of the prairies. His most precious possession was a very expensive classical guitar, respect for that, but the rest of the contents of his, for want of a better word, hovel, seemed to consist of rusty and delapidated items barely worth the expense of transport. I was also informed that he was an identical twin. Two of them? The mind boggles.

          1. Rich 11 Silver badge

            Re: Oh so special's

            Two of them? The mind boggles.

            That's the secret as to how he managed to peripatate™ between venues so readily.

            1. Daedalus Silver badge

              Re: Oh so special's

              Ah but in quantum theory you can be in two places at once, if nobody is looking.

      4. swm Silver badge

        Re: Oh so special's

        I once watched a full professor unsuccessfully try to dial (rotary dial) a telephone call when they replaced real people operators.

        1. jake Silver badge

          Re: Oh so special's

          Just a couple days ago I offered a teenager $50 if he could place a call on my '50s Western Electric Model 500 desk phone. He refused to touch it!

          1. heyrick Silver badge

            Re: Oh so special's

            I have a rotary phone with a proper bell inside. It's a proper phone. Get angry and slam the handset down, it's your hand that breaks.

            Unfortunately the Livebox doesn't understand pulses, so there's this cool little gadget inside (RotaTone) that converts the pulses to the required beeps and meeps.

            1. keith_w Bronze badge

              Re: Oh so special's

              Bell Canada used (and may still) to charge you for tone rather than pulse even though none of their equipment supported pulse.

              1. DiViDeD Silver badge

                Re: pulse dialling

                The old rotary phones with pulse dialling were very handy at Uni when a certain landlord of my unfortunate acquaintance had a coin operated phone installed for the use of scruffy students (you know - the one where you set your own charge rate so you can keep students safe from the evils of drink by letting them pour all their beer money into phone calls - usually calling home to ask for more beer money) and put one of those nifty phone locks on the main house phone in the hall, not realising how adept most students were at dialling long IDD numbers by tapping the popup buttons in the cradle.

    2. Muscleguy Silver badge

      Re: Oh so special's

      We had a bin fire in the lunch room in one lab I worked in. One of the bright ones had been working late and had brought in some microwave popcorn for a snack. Except they failed to read the instructions thinking they knew better and nuked it, causing a fire in the microwave which was dealt with by, putting the bag in the metal sink? No, the bin. It ignited the conents, causing smoke and the deployment of a fire extinguisher and the smoke alarm telling all what had happened.

      We had to stage an intervention on one colleague who got sick from malnourishment having left home and existed without the ability to cook anything. Dinner was crisps and whisky. The intervention worked, happily.

      I have known any number of very bright but incompetent in life people. The head of the place was forced to drive a Ka because they had had so many accidents it was the only car the insurance would insure them on.

      Being a Boffin who exercises and is not at all bad at life has involved me helping all manner of colleagues with issues. I'm not trying to seem all powerful, just competent in general. It seems many cannot achieve that in all areas. Some manage it only one or a few.

      1. Sam not the Viking

        Re: Oh so special's

        Reminds me of a very clever guy who came to work with us. A genuine polymath on all things mathematical, physical and chemical.

        He was so obviously brilliant he was taken on straight away.

        After a couple of days at work, his company car arrived.

        "I can't drive," he replied.

        The car was put in reserve until he passed his test, but he became/was a terrible driver. I feel sure he was driving in a different set of dimensions.

  2. Pascal Monett Silver badge
    Facepalm

    But of course

    Oh dear, I got some water on my laptop. Should I bring it home and use a hairdryer to try and fix the problem ? Nah, I'll just use the company server room equipment, which I will reconfigure to my needs. Oh, there's a reason why the aircon is set to cold ? Doesn't matter, I need warm. There, my problem is solved. I shall now leave everything the way it is without a care in the world. If anything bad happens, it's not my budget.

    Man, I would have so fired that guy.

    1. Olivier2553 Silver badge

      Re: But of course

      And that is why I have locked the thermostat for the server room... Air cond setting cannot be changed saved of turning off the power in the building (our setting is like described in the article, air cond shared between lab and server room but I am nice, machines nowadays don't need to be freezing cold).

      1. phuzz Silver badge

        Re: But of course

        Hide the thermostat, and leave a fake one in plain view that's not connected to anything.

        That way they'll spend their time playing with the fake one, not trying to break into the real one.

        1. Nunyabiznes Silver badge

          Re: But of course

          I guessed the password to Facilities aircon control application and locked out the thermostat in our office. The boss wears an undershirt, long-sleeved dress shirt, tie and sweater so he is constantly putting the thermostat at absolute zero. Now he can do it all he likes.

          1. Robert Carnegie Silver badge
            Joke

            Re: But of course

            Or you could dress for the job you want. Supposing that you do.

        2. ICPurvis47 Bronze badge
          Devil

          Re: But of course

          When I was in charge of the Energy Management System at the electrical engineering company I worked for, I once noticed that the temperature in one office was significantly lower than the setpoint, and the local heating zone was going full blast. I took a stroll through that office, and noticed that someone had suspended a plastic bag full of ice cubes in front of the temperature sensor. I strolled nonchalantly by, pretending not to have noticed. When I returned to my office, I changed the setpoint for their area to -5°C and waited for some reaction. About an hour later, the phone rang, and someone was complaining that the heating had gone off, and that they were freezing. I said "Hang on, I'll check. Oh, yes, your ice cube trick has sent the thermostat off scale and it has rolled over. Apparently you are currently basking in a 35°C heatwave. If you remove the ice cubes, it will reset itself and your heating will come on again". A few minutes later the temperature reading started to increase, so I reset the set point to 20°C, and the heating came on again. They never tried that prank again Mwahahaha!

          1. Anonymous Coward
            Anonymous Coward

            Re: But of course

            Hmmm, I need to setup this trick at my office.

          2. Unicornpiss Silver badge

            Re: But of course

            We used to do that trick in high school, in one room that never was warm enough. A bit of snow from the windowsill and the heat would keep running.

      2. CountCadaver Bronze badge

        Re: But of course

        server rooms are pleasantly cool for me vs generally inferno temperature offices where its often sweltering to accomodate the office "cold potato"

        1. Rich 11 Silver badge

          Re: But of course

          Liftshafts can be used to resolve that issue.

    2. Shooter

      Re: But of course

      "Man, I would have so fired that guy."

      Out of a cannon!

  3. Olivier2553 Silver badge

    SPARC burn?

    I still have a CPU module with burn scars on the PCB. It was not the CPU but some of the voltage regulation on the 4 CPUs board.

    But I cant remember what flavor of SPARC that was for.

    1. Korev Silver badge
      Flame

      Re: SPARC burn?

      In case people haven't seen the "thermal event" on a Sun box that used to run part of Cambridge's email setup. The system was so well architected that end users like myself didn't notice a thing.

      1. J. Cook Silver badge

        Re: SPARC burn?

        Wow. Now That's what I call a 'thermal reluctance anomaly"...

      2. CrazyOldCatMan Silver badge

        Re: SPARC burn?

        Many years ago (2001? Maybe) we stood up a new server room in our offices in France (which acted as a European hub). Lots of Sun boxen and a few PC-type servers.

        All was plugged in, configured and working. What we didn't know was the the server room aircon was tied into the building aircon management and shut off during out-of-hours..

        We arrived the next morning to find that the server room was doing a good imitation of an oven (50C) and almost everything was shut down. Once the room was cool enough, we managed to get the switches up and then sequentially bought up the Sun boxes. Most came up fairly cleanly - the shutdown had been caused by a thermal warning and, in most cases had happened cleanly.

        The PC server boxes and the Sparc storage arrays however were basically toasted. The SSA chassis was OK but the majority of the SCSI drives had so many errors as to be unusable - fortunately, since they had just been provisioned, there was very little data on them and we could just get new drives.

        Most of the PC servers had to be replaced.

        There was an amount of French-type shouting going on at a senior level afterwards.

    2. Stevie Silver badge

      Re: SPARC burn?

      Pah! Last night I removed a socket from my house's wall that had been arcing internally and partially melted due to pass-through current draw from a socket upstream (kettle).

      How it didn't burn us all to death in our beds I don't know. Stank like old cat food though. House reeked.

      1. A.P. Veening Silver badge

        Re: SPARC burn?

        That is the problem with the UK standard ring circuitry, it wouldn't happen with continental star circuitry.

        1. jake Silver badge

          Re: SPARC burn?

          "That is one of the problems with the UK standard ring circuitry"

          FTFY ... there are other glaring problems.

        2. Stevie Silver badge

          Re: SPARC burn?

          Nah. New York wires were cooking off.

        3. Paul Crawford Silver badge

          Re: SPARC burn?

          If there was a bad connection on a star-wired socket you get exactly the same problem of fire risk, only difference here is the fault was not local to the one drawing the high current.

          AFDD breakers are supposed to stop this, mostly, but are a bit pricey so far. At least with a ring circuit you only need 1 expensive AFDD breaker to protect a typical floor's worth of sockets!

  4. Anonymous South African Coward Silver badge

    Had my share of overheating server rooms, but every time it was caught in the nick of time.

    Next time we won't be lucky. More equipment means more heat... so...

    *shrugs*

    death by Coronachan is more preferable than dealing with the fallout from an overheated server room.

    1. Alan Brown Silver badge

      "Had my share of overheating server rooms, but every time it was caught in the nick of time."

      It's a good idea to have a thermal crowbar on the power so that if the room gets towards overheat, the power goes off before things start burning

      And it gives a second thermostat for the users not to fiddle with - or face the possibility of the room suddenly getting _very_ quiet.

    2. Baldrickk Silver badge
      Alert

      Coronachan

      I think you win the prize for most cursed comment.

  5. Admiral Grace Hopper Silver badge
    Mushroom

    "Yeah, but it's a dry heat"

    There's a uniquely horrifying feeling to the heat in a data centre when the AHUs have gone down, accented by the screaming of the fans as the server struggle to cope with an environment for which they weren't designed. That's when you discover which of your machines are truly mission critical as you frantically switch off everything that isn't.

    Anyone pulling this sort of stroke needs some firm corrective action. Don't fuck with my servers.

    1. MAF

      Re: "Yeah, but it's a dry heat"

      No they're special. Treat them well.

      Take them out for a trip in the woods....and leave them there.

      1. Anonymous Coward
        Anonymous Coward

        Re: Trip to the woods

        ...and leave a decent, untraceable firearm (shotgun) with them.

        If they manage to use it, they'll either shoot at whoever is coming to investigate the sound and go to jail, or shoot some food and still starve not wanting to get bloody.

        If they can't figure it out, they might look up the barrel and blow their head off.

        It's a win either way as long as the firearm itself can't be traced back.

  6. Manolo
    Flame

    Not just computer equipment

    I once had a pharmacy tech who, in my temporary absence, decided that protecting the medications in store from the tropical heat was less important than her urgent need to wear a tight sleeveless top at work and therefore decided to turn of the airconditioning.

    1. J. Cook Silver badge

      Re: Not just computer equipment

      Ouch. that sounds about as expensive as a dead server room...

      1. Manolo

        Re: Not just computer equipment

        I caught it within a few hours, so no harm done (except for the bollocking I had to give her).

        1. A.P. Veening Silver badge

          Re: Not just computer equipment

          That bollocking should have been only three words: "You are fired!"

          1. Anonymous Coward
            Anonymous Coward

            Re: Not just computer equipment

            That would depend what she looked like in the tight sleeveless top with the a/c at the correct setting.

  7. Giovani Tapini Silver badge

    I've been there

    DC1 aircon failure - larger room-v hot - circa 45C went round site collecting desk fans, propping open fire doors, shutting stuff down..

    DC2 aircon failure - small server room - room so hot you couldn't stand in it (probably 60C or more - we couldn't tell) reached in with hand wrapped in jumper to flick off switch - prop doors open. Probably lucky nothing caught fire in this one.... boot floppies melted into drives.. far more tricky to recover...

    Probably should have just ordered beer and changed into shorts...

    1. The First Dave
      Flame

      Re: I've been there

      What's going to catch fire at a mere 60°C ?

      1. TSM

        Re: I've been there

        If the general air temp of the whole room is 60°C, the temperatures in the parts of the equipment that are genrating all that heat are going to be a lot higher.

  8. Philip Storry

    How about a nice long hot summer?

    We thought we were fine, because we had decent aircon in the server room and nobody else could control it. But then we had a hot summer.

    Very hot. Specifically, the 2006 heatwave hot. https://en.wikipedia.org/wiki/2006_European_heat_wave#United_Kingdom

    And it turns out our aircon had a serious flaw. The exhaust port was in direct sunlight, and at the wrong angle. The warm wind and the warm air rising from the (black, naturally) roof isn't blowing heat away, it's blowing it back down the exhaust pipe...

    Yep, our server room aircon kept cutting out due to overheating, despite the room being freezing. It just kept tripping the internal sensors on the units. We were very puzzled. We were also somewhat frazzled as we kept having to come in early, spend long days nursing minimal systems along in a room cooled with one inadequate portable air conditioner. Clusters ran on only one node. Some servers were powered up only on demand.

    I've just realised that in some ways we spent that week as a crappy physical version of AWS or Azure. Damn, we should have patented the concepts! ;-)

    Each night everything except email and network/AD servers were powered down, to try to cool the room as much as possible for the next day. Staff were advised that systems were strictly 08:00 - 18:00, due to the emergency. I suspect most were grateful for an excuse to leave early and get some sun!

    After a little over a week the exhaust port was temporarily fixed, and later in the year a more permanent fix was put in place. But I now have much more respect for HVAC engineers and the work that they do, because it seems that there definitely some cowboys out there!

    1. GlenP Silver badge

      Re: How about a nice long hot summer?

      there definitely some cowboys out there

      Like the ones who installed two inadequate heat pumps in our server cupboard at previous premises (before my time I hasten to add) putting the "outside" part on the other side of the internal wall, in the factory part of the building a few inches below a metal roof.

    2. Killfalcon

      Re: How about a nice long hot summer?

      Rural Surrey wasn't the worst place for an office, fantastic views of the wooded vales, the sounds of wind through the trees, the distance whisper of the M25, but it had it's drawbacks at times.

      We had a lot of sim work to do, and for a long while people worked at their desktops, and if they needed more firepower, they'd borrow other people's (sometimes without asking). Eventually it was decided (around 2008) we needed some machines dedicated to Production work. Not quite enough will to get actual servers, mind, but still.

      We had a tiny meeting room converted into a "server" room, and dedicated air-con stuck on the roof. It was just Pentium 4s, but fifty of them in a 5'*5' room needs cooling!

      All was well until the first autumn, when the intakes jammed up with wind-blown leaves and the room got up to 45-ish, hot enough that the door handle was painful to touch from the outside!

      1. Anonymous Coward
        Anonymous Coward

        Re: How about a nice long hot summer?

        I've been inside stability testing chambers at [$employer] set at 50 °C and 95 %RH. It's distinctly harder to breathe, and you might as well leave glasses outside (they fog instantly and won't clear). Of course, the 50 °C / 10 %RH chambers aren't that much better - ever feel the moisture evaporating out of your mouth?

        1. J. Cook Silver badge
          Flame

          Re: How about a nice long hot summer?

          Yes; Arizona in the middle of the hottest day on record.

        2. Alan Brown Silver badge

          Re: How about a nice long hot summer?

          "set at 50 °C and 95 %RH"

          If the environmental wet-bulb equivalent temperature is over body temperature, then you can't radiate heat and will eventually go into your OWN version of thermal overload - where a temperature rise of only 1-2C in the brain can be fatal

          I'd rather take the dry heat. It takes far longer to dehydrate than to broil

        3. Andrew Norton

          Re: How about a nice long hot summer?

          Am I just weird then?

          Because when I worked on a certain robot tv show, 50C was 'normal' for in the arena.

          One season we filmed in San Fran, in an old WW2 warehouse converted into an event space... kinda in the bay. There's no AC, and it's June. And then we bring in the lighting. This was 2000 (in fact, I'm lucky in that the day after we'd finished filming and packing up - also my one sightseeing day - it hit 111F (44C) and a new record for San Fran, and 6 people passed out in the then-new PacBell park) and there was no LED lighting, it was all incandescent. We had full on TV lighting, and to ensure the lighting was constant, the lighting guys had taken the double-line of windows at the crown of the buildings room, and blacked them out with black curtains (so they were nice and hot). And by 'full on TV lighting', we're talking a 100,000W storm-strobe, and dozens of 1500W lights (maybe 60+) pointing into the arena. Oh, and the arena was lexan, between 1 and 1/4 inch thick throughout, the only air escape was the two 8x8 doors when they were open, or the two 3ft ducts for ventilation for IC engines to be used.

          And me and the other 5 arena crew had to go in there, guard the arena hazards, and help the teams load upto 350lb bots on a cart, then clean up debris, then help the next team in there, and help them unload, Maybe 3 minutes inside every 10 minutes. Oh, and since we were safety, full protection gear - so kevlar pants, steel toe/shank boots,and body armour (one guy, living up to his ex-marine status, also then put some highly chromed plate armour on top).

          We went through a case of water every 30 minutes or so. And we did it for 15 hours a day for 4 days filming. Inside the arena was easily 55C, hell the building was 45C minimum. And again, this building was literally ON the water, humidity was a bitch!

          The building has since has AC put in.

          The next filming was in Vegas in November. The pits were in tents in the parking lot, and most of the time it was below freezing. From one extreme to another, and I ended up running the pits/safety/testing area as being the only non-californian crew member (and, in fact, a brit) the cold didn't bother me a damned bit.

      2. Morat

        Re: How about a nice long hot summer?

        Pentium 4s were known as "Space Heaters" for a reason!

        1. Ogi

          Re: How about a nice long hot summer?

          Aaah yes, we used to call them "Presshot", as a corruption of their core codename ( https://en.wikipedia.org/wiki/Pentium_4#Prescott ).

          The P4 was excellent at turning electricity into heat, and was the CPU that first made me consider trying AMDs offering (which was not as performant, but more efficient per Watt). Since then I have stuck with AMD on my machines.

          Aaah... memories....

    3. short a sandwich

      Re: How about a nice long hot summer?

      I have met one or two of the cowboy HVAC chaps. A flour mill control room that needed to be a 20C to measure dough water absorption accurately. Too much water or too little in the bakery dough meant either too much space in the plastic bread wrapper or a mess of crumbs and plastic as the machines try to stick big loaves in too small bags!

      Control room had 2 or 3 500w meat based heat radiators, a 1 megawatt facitilty outside and sundry control PC's or PLC's installed. Said cowpoke couldn't work out why the temperature couldn't get below 25C with a 2.5KW unit. The 7.6KW unit that we replaced it with coped much better but the 'engineer' was unhappy that a 'civilian' had worked it out for him.

      1. Anonymous Coward
        Anonymous Coward

        Re: How about a nice long hot summer?

        "2 or 3 500w meat based heat radiators"

        Got a lol out of that one. But isn't the usual estimate about 100W of heat per person?

        1. WonkoTheSane
          Headmaster

          Re: How about a nice long hot summer?

          "But isn't the usual estimate about 100W of heat per person?"

          Depends how much of the company's product they'd had.

        2. CrazyOldCatMan Silver badge

          Re: How about a nice long hot summer?

          estimate about 100W of heat per person?

          Depends on age, gender and BMI..

          (Women, in general, run slightly hotter than men. I, in specific, run cooler than the average male..)

          1. Anonymous Coward
            Anonymous Coward

            Re: How about a nice long hot summer?

            I'm sure it does, and probably a number of other factors as well. Would the amount of time since a meal have a difference? Would higher BMI tend to decrease heat output, due to extra insulation? LOTS of possibility for variability.

            My point was that the 500W originally quoted is closer to a professional athlete while exercising, as opposed to the average for an individual sitting at a desk.

      2. SImon Hobson Silver badge

        Re: How about a nice long hot summer?

        I think we've probably all met at least one of the A?C cowboys - or at least, one of their installations.

        A favourite of mine was for a client who arranged a dedicated little server room int heir new building, and had A/C installed in it. To be fair, the client asked me (me by name) to look at the A/C proposal they'd had since I knew a little about it. Having checked that the capacity was about right (with a bit of spare), and especially that it could handle dry air[1] I said it should be OK. And to be fair, it was for a year or two - even though they put in in the wrong place in the room.

        After said year or two, the room got hot, and the A/C wouldn't run - so it was down to propping open the doors and directing a fan into it. A (so called) engineer was dispatched to look at it and ... declared that it was "the wrong sort of room", and there was no way that unit could have kept it cool (despite the empirical evidence to the contrary for the last year or two). He insisted that it was gaining too much heat through the ceiling, but needless to say, adding an extra layer of insulation did nothing.

        After a few days of this finger pointing, the client asked for my opinion, I spoke with the (so called) engineer and he was not be being persuaded that there was anything wrong with the system - in spite of it tripping out a minute or two after it was reset.

        So I rang the engineers employer, spoke to the engineering manager there and explained the situation - to get a "that doesn't sound right" response. An hour or two later it was magically discovered that there was a fault (stuck reversing valve in the outdoor unit) and it got fixed.

        [1] That being a favourite problem when systems designed for comfort cooling of homes and offices are put into a server room without all that sweat and breath moisture us meat bags put into the environment. Without the extra energy removal needed to overcome the latent heat of fusion in condensing that moisture, the evaporator coil runs too cold, what moisture there is condenses and freezes on it, and after a short time the system stops working with the evaporator coil embedded in a solid lump of air-flow blocking ice.

        Some better systems can detect the onset of this and go through a defrost cycle - most don't.

  9. jake Silver badge

    Tricks of the trade.

    Pretty much everywhere I've ever worked there has been a long standing feud between two halves of the population that I can't mention without being accused of being sexist ... One side always says it's too hot, the other side always says it's too cold. Facilities says "set it all to 72F, that's what the HVAC is optimized for" ... and so we listen to pretty much everyone bitch about the temperature.

    Until one place I worked at upgraded the AC, and all the controls that go along with it (had to do with a couple of new clean rooms). Naturally, the folks installing all the new gear left all the old thermostats in place. They were no longer connected, so why worry about them. A friend of mine noticed one of the secretaries would inevitably turn one of these controls up, and then keep an eye on it from her desk. Within an hour, one of the engineers would stroll by & turn it down again. Then she'd turn it back up, and so on ... This dance went on all day.

    So we hatched up a Plan ... with the Boss's permission, we installed unconnected thermostats quite near both the secretary and the engineer ... and removed the one they were "fighting" over. Now both could happily set "their" temperature to whatever they wanted. It worked. Both were happy, and both commented how comfy the office was with the "new, improved" controls. People in their circle of friends made similar comments. The complaining about the temperature stopped, virtually overnight.

    That would have been the end of it, except ever since then I've installed faux thermostats for 'special" people. It has never failed to shut them up about the office temperature. However, be warned ... that type can always find something else to bitch about. Don't say I didn't warn you.

    1. J.G.Harston Silver badge

      Re: Tricks of the trade.

      You'd think that with half the population wearing nearly nothing and half the population being properly dressed, somebody would notice there would be a disconnect between different group's ambient temperature desire.

      Put a damn jumper on!

      1. My other car WAS an IAV Stryker Bronze badge
        FAIL

        Re: Unequal clothing options

        I would like to be able to wear "nearly nothing", but office dress codes demand I have a minimum of a short sleeve (polo for the collar), long slacks, and shoes with dress socks.

        There are many times with my last employer I would have loved to downgrade to T-shirts, shorts, and sandals (or at least sneakers with more comfy socks due to the size of walking across the campus). Ladies are allowed more options that are still too taboo for us blokes; the customer (hint: who buys armored vehicles?) especially wouldn't appreciate it.

        I'm an a smaller office now, where I can get away with popping the shoes off, and the ambient is generally cooler. Since I still prefer polos over long sleeves, I sometimes have to put on my jacket/coat when I get chilly, and I'm fine with that.

        1. A.P. Veening Silver badge

          Re: Unequal clothing options

          There are many times with my last employer I would have loved to downgrade to T-shirts, shorts, and sandals (or at least sneakers with more comfy socks due to the size of walking across the campus). Ladies are allowed more options that are still too taboo for us blokes; the customer (hint: who buys armored vehicles?) especially wouldn't appreciate it.

          You should have opted for a (short) skirt and scream discrimination the moment your employer said anything negative about it.

          1. jake Silver badge

            Re: Unequal clothing options

            " would have loved to downgrade to T-shirts, shorts, and sandals"

            When I was at HP, I fairly regularly wore tshirts, shorts and no shoes to the office.

          2. Paul Crawford Silver badge

            Re: Unequal clothing options

            Just get a kilt and boast about your Scottish roots, invented if necessary.

            1. Tom 7 Silver badge

              Re: Unequal clothing options

              Kilts are great in winter - warmer than trousers but you have 5 yards of heavy cloth in them so they have sufficient insulation power to be pretty shit in a warm office unless wafted ala Carry on up the Kyber every 2 seconds.

      2. SImon Hobson Silver badge
        WTF?

        Re: Tricks of the trade.

        Put a damn jumper on!

        Good god man, don't you realise how dangerous it is to say something like that ? When I tried it, I think it was only the fact that there were witnesses that saved my skin from the (daring to play the sexist card) ladies in question. I'd simply observed that certain of the ladies tended to come to the office on freezing days, wearing light summer outfits - and hence were freezing before they even set foot in the office. A polite suggestion that they might consider dressing for the season did not go down well.

        These same people would also complain that their stylish but impractical shoes weren't very good at navigating the walkway through the factory to the canteen - it was always slippery because we made candles and so everything was always covered in wax (if only a fine film from it condensing out from the atmosphere). Oh no, there was no happy medium to be found - my suggestion that they consider shoes suitable for where they know they are going to be walking was met with "I'm not wearing safety boots in the office".

        And sorry ladies, this is true even if you want to deny it or call it sexist. When some of you get to what's often referred to as "a certain age", then I know it's a waste of time trying to get the temperature right - since "right" changes dramatically from minute to minute.

    2. Alan Brown Silver badge

      Re: Tricks of the trade.

      "a long standing feud between two halves of the population that I can't mention without being accused of being sexist"

      "Comfort cooling" (ie: aircon for PEOPLE is 90% about HUMIDITY control, not about actual temperature) - it's worth watching the RH when people are griping as a mismatched system can be swinging over wide ranges and this gets perceived as temperature changes by the users

    3. Daedalus Silver badge

      Re: Tricks of the trade.

      Problems arise when you put more than six people in the conference room that is on the same thermostat as the general office area. Conf room quickly becomes overheated while the general area remains at "acceptable" temps. I have been known to step above my station for the sake of personal comfort, and even to keep potential customers from thinking we're idiots (we mostly are, but that's another story).

    4. Andy Miller

      Re: Tricks of the trade.

      I had one colleague who always complained about the heat. One morning he came in early and super-glued the thermostatic valve on the office radiator shut, but still he complained.

      At some point he decided to stop drinking coca-cola by the 2-litre bottle full. Soon he stopped complaining about the heat, and actually closed the window....

      1. J.G.Harston Silver badge

        Re: Tricks of the trade.

        Oh good god. I've just spent a week working in an office where the radiator is on full blast and they control the temperature by opening the window.

  10. Ross Luker

    My favourite tale of aircon woe is actually adjacent to the server room - in the mailroom there was an aircon unit which, as it turned out,didn't drain to the outside, but to the adjacent room. If you cut the power to it, instead of siphoning the water away, the water just overflowed in the other room... through the false floor...to the floor under the server room....where some wag was running 2 racks off some Argos-standard 4-ways.... The company employed a new cleaner, who dutifully turned off the aircon in the mailroom on her first night - cue hi-jinks :-(

    1. Alan Brown Silver badge

      "The company employed a new cleaner, who dutifully turned off the aircon in the mailroom on her first night"

      We have had security guards decide that the AC in machine rooms was wasting power and shut it off at night - triggering the overheat crowbar an hour or so later.

      The number of times this gets reported I'm surprised that more people don't start tossing lorawan kit everywhere to monitor stuff being turned off when it shouldn't be and alarm before things go wrong.

      1. A.P. Veening Silver badge

        The number of times this gets reported I'm surprised that more people don't start tossing lorawan kit everywhere to monitor stuff being turned off when it shouldn't be and alarm before things go wrong.

        Why should they? This way they have a very decent chance of getting new kit.

  11. Sequin

    Many years ago I was working on a project for the department that installed and maintained comms equipment for the emergency services, from the Police handheld radios to full hilltop transmitter sites with 100ft aerial arrays etc.

    At the time the stock control at their warehouse and central stores was all paper based and we were sourcing and installing a computerised system that ran on a PRIME minicomputer. We told them that they would need a computer room building and they said that they would do this themselves to save money, as they were used to doing things like this when setting up comms rooms in Police and Fire stations.

    Sure enought, They boxed off a corner of the office with partition walls, put in the power supply, wired in the supplied UPS etc. What they did not do was check what was running through the cavity in the dropped ceiling.

    A month or two later we got an emergency call saying that everything had gone down - what had happened was that the pipework from the toilets on the floor above had become clogged and a build up of pressure had caused a joint in the pipework to pop - right above the computer room. The sh*t literally hit the fans!

    1. SImon Hobson Silver badge
      Facepalm

      Did a business continuity course many years ago, and a comment was made that water (of all colours) flows downhill, and many businesses put their computers in the basement.

  12. OssianScotland Silver badge

    Boss Fail

    Worked under a FD who penny pinched on everything (except himself) and decided the server room AC should be off at the weekend ("because no-one is in"). After the second occasion - fortunately with no equipment casualties - I got the lock changed.

    Unfortunately everything was on the ground floor, so I could not emulate the current BOFH episode.

    1. Alan Brown Silver badge

      Re: Boss Fail

      "After the second occasion - fortunately with no equipment casualties - I got the lock changed."

      Too bad you couldn't "manufacture" some casualties - because manglement with attitudes like that frequently treat a changed lock as a challenge, whereas a threat to be directly invoiced for the damage if it happens again is a good dissuader.

      "We've had to change out all this kit due to overheating damage having seriously shortened its life and making it liable to fail at any moment"

    2. Andrew Moore

      Re: Boss Fail

      I would have backed up the servers and run them soooooo hot. One thing a FD doe not like is unexpected bills for replacement equipment, and breakdowns of losses due to critical systems being offline.

  13. Joeyjoejojrshabado

    Disturbing

    "One Sunday afternoon one of our brightest and best arrived in for some extracurricular activity," explained Chris. The boffin had managed to dampen his personal laptop...

    This is what gym socks are for.

  14. Ima Ballsy
    Mushroom

    Snark .....

    One of the State agencies I used to maintain the DB for, moved to a Custome renovated floor in a high rise building.

    The created a centrally located 'server' room out of a 12 by 10 closet located in the middle of the floor.

    To keep things secure they kept the door locked to THAT server room which housed 4 new rack servers, 4 old rack servers, high end routing equipment and the complete PBX system as well as UPS's - Your could barley walk around the racks.

    Guess what the renovator forgot to do and the on contract network / system administrator / hardware manager forgot ?

    Yeah, no AC to the room. -

    Second DAY after they moved in, everything shut down

    1. A.P. Veening Silver badge

      Re: Snark .....

      I am completely flabbergasted it didn't happen on the FIRST day.

      1. SImon Hobson Silver badge

        Re: Snark .....

        You'd be surprised.

        While it may not be much, there is a mass fo air in the room, plus all the metalwork, and of course the mass of the walls (and floors and ceilings). If the heat input isn't too much above what can be lost through conduction through all these surfaces, then the thermal mass can be enough to delay heat rise by some amount.

        The downside is, when you have got the racks, cabinets, server hardware, walls, and thick concrete floor slab up to some silly temperature - it also takes time for it to cool off.

        This is noticeable in houses etc. I've know people who live in old stone houses of the sort where doorways are more like short corridors. They tend to be cool in summer, and cooler but not cold in winter - because of the thermal mass in the walls. That does depend on having a typical British summer where there are a few very hot days, and winters where there are a few very cold days - if it's really hot or really cold for long enough then the thermal mass is eventually used up. Unfortunately, we've yet to persuade one of our church wardens that leaving the building cold all week, then heating it "only just enough" to be survivable for the length of a service, does not do much for the fabric of the building - take cold walls, heat the air (but not long enough to warm the walls at all), add moisture generators (humans), and you can see the paint change colour with the condensation, revealing the internal structure of the wall in a "mottled" effect. When he's not around, we set the timer to come on a couple of hours earlier to try and give the plastered surface a fighting chance of survival.

        1. No Yb

          Re: Snark .....

          Perhaps finding a local museum curator/archivist to talk about the issues would help. Someone who can be a "neutral third party" might get more response.

  15. Trbonja

    A couple of decades ago I came in as a contractor to support two server rooms in two diferent buildings. AC was in place in each location but backup power generator was at capacity so if street power goes out no AC...what could go wrong?!

    Many hot summer days, power outages and intel servers later (even one AS400), I got approval....for two portable AC units for each location. At least these units are on backup power and temps stay below 27C...most of the time.

  16. earl grey Silver badge
    Devil

    lovely AC

    Many decades ago i was hired into an unmentionable large corporation to be a computer room supervisor. Our floor was all raised and there were tiles with holes to supply extra air for the needed mainframe components. I always liked having one of those tiles right under my desk.

    1. jake Silver badge

      Re: lovely AC

      "I always liked having one of those tiles right under my desk."

      In the days before the common availability of cordless tools, I had one of these in my tool bag. Took a few minutes, but I was able to tailor the airflow to suit myself ... Last time I was at SLAC, my unapproved improvement was still in operation, much to the delight of the current occupier of that office. Judging by the leftover adhesive residue, one of the prior occupiers did NOT approve, and had taped the holes over ...

      Before anyone says anything, we had several tons and a couple thousand CFM of extra cooling capacity. How do I know? It was part of my job to know :-)

  17. Anonymous Coward
    Anonymous Coward

    HCF

    We were a preview site for one of the Sun Enterprise series. E6000 or E10k I think it was.

    We discovered that the on prem data centre didn't have smoke detectors when the shiny new box paged a SysAdmin in the wee hours about an IO issue. Bleary SysAdmin drove in, switched on the lights and the glass wall was an featureless grey expanse of smoke where Blinken lights normally blinked. Fsck!

    Fortunately more smouldering than flame and a new server was air freighted in (and defective server removed) faster than I have seen before then or since.

  18. Anonymous Coward
    Anonymous Coward

    Don't forget your environmentals

    Anonymous to save the guilty.

    Worked on a project where the company had followed Sun/Googles idea of container based datacentres.

    This was two containers 1) Refrigerant and air con source and 2) a two row DC with a lot of Cisco UCS and NetApp kit. The BTU must have been quite high, but the container was kept cool, until it wasn't.

    System was almost in production, but the team doing the environmental alerting wasn't directly related to our project. One Thursday after 3, the Refrigerant leaked but wasn't picked up. By 11, systems had gone offline, but we didn't know why.

    Technician went to the DC and nearly got blown off his feet when he opened the door. The container was too hot to touch, but was acting like a massive heatsink. NetApp had shut down gracefully, but the core switches had stopped at about 95 degrees.

    We got round to getting the air con sorted and testing what kit was toast. Not one failure. We told the customer to file an insurance claim and replace everything, but they refused to do so. Last I heard, it was all running fine, but I'm pretty sure they voided all their warranty.

  19. chivo243 Silver badge
    Windows

    3Com racks had cooked and were as dead as a dodo

    Walked many, many miles those shoes...

  20. Stuart Castle Silver badge

    Never had to deal with a server having issues caused by someone turning down the Aircon. I have dealt with a failed Aircon system at a friend's company though. Thanks to a project I was involved in at work, I have a fair amount of knowledge of setting up a streaming video system. His company wanted to stream Live TV (legitimately, they were a TV channel), and had asked me to go in to advise them on it. We arranged for me to go down their one Saturday. The hottest Saturday of the year, as it turns out. So, on that Saturday, I got up (hungover - I'd been out drinking with my friend until the small hours), got to his office suited and booted, and met him. We went to the server room, and the Air con had failed.

    When we went in the room , it was well over 40 degrees C. I don't know how far over, as the thermometer on the wall maxed out at 40, but suffice to say doing anything in the room involved a *lot* of sweat. Obviously, every piece of equipment in the room overheated.

    So, we opened all the doors and windows we could, and got every fan and extension cable we should. We arranged them to blow air on the servers, and turned the servers off (shutting them down safely if possible), and the fans on.

    Nothing was cool enough to power up, so we went to the local Weatherspoons, and had a Full Irish breakfeast on expenses.. An hour later, we went back. The temperature was still well above 40 degrees, but had cooled down enough that we could bring the servers back online. We did get everything back online, and also set up a live stream of the channel, which my friend was able to demonstrate to his boss. Who, after all that, decided not to proceed. I still got paid though. Profitable day. I got paid £320 for my day's work, and got a free breakfast to boot.

  21. Paul Hovnanian Silver badge

    Not the server room, but ...

    ... I used to work in an office that was maintained at 82 to 84 °F (28 °C). According to management, this was set "to save money on air conditioning costs". During the summer it was pure misery. During the winter it was sort of odd. But we assumed that the heat load in the building (people, lighting, computers, etc.) was so high that it just kept the building hot.

    Then, we had the Nisqually Earthquake in the middle of winter. Operations were shut down for a week. When we finally did return to work, the building started out at 62 °F and warmed up to 68 °F by the end of the day. Pretty comfy. I asked one of the facilities guys what was up. It seems that they could not restart the gas fired boilers until a thorough inspection was conducted. So, no heat. Yeah, but what about the typical 82 °F? "We are not to touch the thermostat setting. Management edict due to air conditioning costs." But during the winter? You could put the AC setpoint at 82 °F and the heating setpoint at 66 °F. Turns out, the thermostat was too old to have two setpoints. And during the summer, it wasn't just delaying the AC from coming on, it was heating the building. But you couldn't tell the boss that. He was the kind of guy who would get red-faced and fly into a rage if someone challenged his edicts. And that included not going to the hardware store and picking up a new thermostat just to work around his orders.

    1. A.P. Veening Silver badge

      Re: Not the server room, but ...

      Penny wise and pound foolish. Somebody should have shown him how much extra he overpaid on heating (in summer) to save a bit on air conditioning. With a bit of luck he would have flown into such a rage he would have dropped that. And even if not, a nice movie created out of it attached to that calculation sent to his manager would have seen him out of the door.

  22. Nick Ryan Silver badge

    Not AC heat related...

    ...but in a previous job where the only thing that mattered in the organisation was sales, and trying to get a total monopoly in the market (which when neglecting service and installation got harder and harder as customers did tend to speak to each other), the sales team decided that they were the only ones in the office that were too hot to work therefore had AC purchased for them. The rest of us had to sweat it out somehow. To save time and money the AC extractor and dehumidifier was put into the open warehouse space rather than being external and were, of course, located immediately above the server racks which due to nothing other than sales being important, were located in a secure(ish) cage in the warehouse. I raised the positioning of the AC as an issue and was told that I was just jealous about the sales team getting AC and not us. Needless to say, AC unit leaked water straight into the server rack a week or so later. Pure genius.

  23. Anonymous Coward
    Anonymous Coward

    The server room is HOW HOT?

    While working for a small bureau service in Kingston-upon-Thames, right on the river with a great pub downstairs, one summer in the 80's the HVAC gave out. We had to wait 3 days for repairs but...we had to keep our 2 IBM 4381's and all manner of disks spinning as Clients were doing their monthly pay cheques etc.

    During the day temperatures would get over 40 degrees C. We had to break all seals on the windows and open them wide just to try and let air through. Not a pretty few days.

    All equipment survived and kept humming away (the operators were humming but for different reasons).

    Needless to say when the HVAC was fixed it was a very careful operation in trying to bring temperatures down slowly...

    Good thing was company had a flat for people to stay over if necessary - it got well used after every shift and visits to the bar.

  24. Unicornpiss Silver badge
    Flame

    Overheat

    A couple of decades ago I inherited a small house in a terrible part of town that was my grandfather's. I did not want to live there and it was more of a tax liability than a boon. (I ended up selling it for a few thousand, eventually) Anyway, while trying to decide what to ultimately do with it, I kept the utilities on, and the heat in the winter at a minimal level so the pipes wouldn't freeze.

    One winter day, I decided to check on the house, as I hadn't been there in 3 days. I drove over, and used my key to come in the back door. Two things I immediately noticed: I could hear the burner from the ancient 'gravity' furnace running, and while it was well below freezing outside, the unheated back room was well over 80 degrees F. With some dawning horror I opened the door to the main house and was shocked and startled to find it over 130F. (actually I had no way of knowing how hot it really was, as one thermometer was pegged at 130 and the other, in the thermostat, had exploded and sent glass and its red contents everywhere} The thermostat was inoperative. I hurriedly dashed to the basement and shut off the furnace there. Coming back upstairs, I opened all doors and windows. The wall where the chimney was located was too hot to lay a hand on. All the other walls were uncomfortably hot, and anything metal in the house was almost too hot to touch. A legal pad that I'd left lying on the floor a foot from the heat register had curled itself into a spiral. A can of shaving cream I kept there hadn't burst, but was bulging ominously, and I kept well away from it.

    The house and the furnace suffered no apparent ill effects from what I figured was at least a 2-day heat soak. It rather frazzled my nerves, and I had to wonder what the neighbors thought with the front door wide open for a couple of hours in freezing weather. You could see the heat waves billowing from the door if you looked at it right. If I'd waited another day to check on the place, I'm sure it would not have been standing.

    1. A.P. Veening Silver badge

      Re: Overheat

      If I'd waited another day to check on the place, I'm sure it would not have been standing.

      And you could have collected a nice insurance pay-out (assuming you did have it insured), might have been the better option in hind sight.

    2. A.P. Veening Silver badge

      Re: Overheat

      actually I had no way of knowing how hot it really was, as one thermometer was pegged at 130

      Actually, you also know it was below 451F, see:

      A legal pad that I'd left lying on the floor a foot from the heat register had curled itself into a spiral.

      If it had reached 451F, that legal pad would have started burning by "spontaneous" combustion. With special thanks to Mr. R. Bradbury.

      1. jake Silver badge

        Re: Overheat

        That wouldn't be spontaneous combustion, that would be auto ignition. And it's not necessarily exactly 451F ... There are many different kinds of paper, made from many different fibers and formulas, igniting at many different temperatures. I've seen as low as around 350F and higher than 700F.

        1. Anonymous Coward
          Anonymous Coward

          Re: Overheat

          Whooosh!

      2. Unicornpiss Silver badge

        Re: Overheat

        Well, I didn't think it was likely even 160. But it was so hot I had some difficulty breathing and was instantly drenched with sweat.

  25. Anonymous Coward
    Anonymous Coward

    I worked on a Nuclear Power Plant, and...

    ...the cooling of the Process Computers room was hardwired to the reactor cooling power supply, as in, you can't kill the cooling of that room by 'normal' means, without proper authorization. (Nothing short of a RPG to the padlocked breakers, mind you.) These were behind thick fire doors and card swipes, 4 redundancies of them.

    The IT servers - not the process computers - were just next to the, er... "Bridge" of the NPP and was just a regular server room with raised tiles, but these could be shutdown at the IT discretion.

    You could tell which screens were NPP process and which were 'support', because the Process didn't run Windows.

    I think it was a "keep your enemies closer" kinda of deal, where the M$ shit was right there in the Command Center where the reset and ctrl-alt-del buttons were accessible, and the Process was 2 floors down, behind thick sheet rock firewalls, with an ambient temperature between 'nippy' and 'downright freezing'.

    I'd use the coat icon, because you'd need those at all times...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020