The developer who came in from the cold and melted a mainframe

The world is rapidly becoming a more uncertain place, but The Register tries to offer readers one small point of certainty by always delivering a fresh Monday morning instalment of "Who, Me?" – the reader-contributed column in which you admit to your errors and elucidate your escapes. This week, meet a reader we'll Regomize as …

[Korev]

They robbed the hot air

[The Oncoming Scorn]

(Hot) Oxygen thieves.

[Anonymous Coward]

We'd installed some comms equipment in long relocatable cabin. It had kit at one end that got warm and a couple of workstations at the other. It was fitted with air con whose main aim was to keep the internal temp liveable for the operators across a wide range of external temperatures and, obviously, keep the equipment in it's comfort zone. The kit didn't get specially hot but the whole thing had to work across a broad external temp range so the HVAC was a bit more special than a couple wall-mounted units. When we got it on site for integrate and test the operators complained that the temperature oscillated from too hot to too cold. They were notoriously fussy people so we ignored them and got on with important stuff. Then the kit logs started showing temperature warnings and those, too, seemed to be regular. In fact, you could almost set your watch by the cycle so the initial suspiscion that the operators were fiddling with the temp setttings were quickly set aside. When we put temp monitors in we found the temp at each end of the cabin went from hot to cold over a very regular ~1.5 hour cycle and the temps at each end of the cabin were exactly out of phase. We spent a lot of time diagnosing and in the end the HVAC contractor stripped down the whole system and it turned out that they'd installed one section of ducting incorrectly that was enough to set up the oscillation.

[blu3b3rry]

Always rather it too cold than too hot

However it always seems to be the vocal ones who whinge about it being "too cold" and insist on making everyone else uncomfortable by getting the HVAC temps turned up. I'm someone who naturally runs "warm". Over 20°c is bearable but only if I'm not doing any physical work, and it takes me very little to start struggling in warmer temperatures.

If it's too cold you can warm up by putting on extra layers. Wandering around shirtless or in underwear due to the heat would probably be considered unwelcome at most workplaces (unless your employer is paying you to do so, anyway).

Perhaps the winner I encountered was one of the periods I had purgatory employment in ISO Class 5 cleanrooms. In one the HVAC unit was on the ground floor of the building in a storeroom, and pulled in air from the room itself rather than outside. After two new starts complained endlessly at the boss about how the room was too cold, he nudged the temperature up to 22°C. Perfectly fine, but not if you're wearing thick tyvek suits, overshoes, hairnets and two pairs of gloves (cotton with vinyl gloves over the top). Unpleasant isn't the word - on a busy day some of us were coming out effectively drenched in sweat under the cleanroom gear.

Not long after this we noted the HVAC system struggling to maintain the correct humidity and temperature in the room, making intolerable conditions even worse.

We had a service engineer out, he gave the HVAC unit a once-over and found it was fine. The issue was eventually rooted down to the product packers who used the storeroom having a habit of stacking flatpacked cardboard boxes against the air inlet...

[BJC]

Re: Always rather it too cold than too hot

I firmly agree that it's better for the temperature setpoint to be on the low side, rather than the high side, allowing some to add layers, if required. However, there's another issue that crops up often - the location of the vents.

Most HVAC systems seem to maintain the temperature by pumping in air that is either very warm or very cool, depending of the required correction. Seldom is the incoming air just above or below the setpoint. That creates localised issues for those under the vents. In one place I was at - with a wall of south facing windows, the solution adopted was to block the vent above the biggest complainant. [All fudged with cardboard.] The remaining vents then had to work harder, and so became more uncomfortable. The solution was to block those too. Obviously, that only gets worse as the process repeats!

Perhaps unsurprisingly, this tended to start with those that only wore a t-shirt.

[sebacoustic]

Re: Always rather it too cold than too hot

Worked in a clean room in UK and our temperature was set to 20°C bit our Taiwanese fab (and office, which is where I work these days when I'm there) work at 23°C to save considerable amounts of energy. Operators wear a thin cotton suit under the clean room garment. And office attire is a t-shirt etc. I find it more comfortable than my UK office which struggles to get above 15°C in winter

[Pascal Monett]

Re: Always rather it too cold than too hot

I agree totally.

You can always add more clothes if required.

The other way around, there comes a point where, even if you ignore basic social etiquette, there comes a point where you can't take any more off.

The only solution at that point is the wet T-shirt - again, not something that most workplaces will be tolerating.

Those who are lucky enough to be able to work remotely will enjoy.

[Vincent Ballard]

Re: Always rather it too cold than too hot

Yep, I wore underwear only when working from home in July and August.

[Philo T Farnsworth]

Re: Always rather it too cold than too hot

While you can certainly "layer up," there are parts of your body that can't be effectively layered, such as hands and fingers.

I once had an office which was perpetually freezing and I'd have to wear a hoodie and sometimes a coat over it just to avoid shivering all day long.

Layers were fine for most of my body, but mittens were more or less out of the question, seeing as how I am an overpaid typist. . . er. . . trained software professional who needs to type a lot. My fingers and hands would get cold to the point of pain.¹

I tried multiple solutions, including pasting an electrical terrarium heater to the underside of the desk to create a warm spot² and wearing disposable gloves to keep some of the body heat in.³

Dinking with the thermostat didn't really help⁴ since there was a draft coming from somewhere that I could never identify.

It only was cured when I just upped and moved to a recently vacated office on the other side of the building and dared management to say "boo" about it.

________________

¹ Perhaps due to incipient arthritis, being of the "older persuasion," as I am. . . and no wisecracks, kiddo, it'll happen to you. . . if you're lucky.

² An idea quickly discarded due to the obvious fire hazard.

³ Wearing them all day long made my hands begin to itch. I'm not sure whether it was from the powder used to make the gloves easier to put on but whatevr it was, that experiment only lasted a couple of days.

⁴ I'm not sure the thermostat actually controlled anything or was user-adjustable. I'd heard rumors that they were there just to give the illusion of control and the temperatures were actually controlled from the central physical plant office but I was never able to confirm that local legend.

[Dave314159ggggdffsdds]

Re: Always rather it too cold than too hot

Yes. Sitting still at a desk at 20c, there is no practical amount of clothing which will stop my digits losing feeling. Whereas the selfish bastards who complain we should layer up can just turn a fan on if they're too warm.

[Philo T Farnsworth]

Re: Always rather it too cold than too hot

A major cause of the "battle of the thermostat" is that people's tolerance to heat and cold varies by gender and age and in a mixed office there are always going to be differential tolerance.

Younger males tend to like lower temperatures than females at the same age largely due to differences in body mass.

Curiously, as one ages, both my partner and I have observed that tolerance tends to switch by gender, with older males liking warmer temperatures while females seem to like it cooler (at least for my sample set of 1).

It's biology.¹

_________________

¹ Why Women Often Feel Colder Than Men

[J.G.Harston]

Re: Always rather it too cold than too hot

If it's biology, it's sex not gender.

[Robert Carnegie]

Re: Always rather it too cold than too hot

It's probably hormones, which have a complex relationship to gender and sex these days.

I could ask someone who may know more, but I'm inclined not to.

But it's commonplace that menopause makes your personal thermostat misbehave - then again, I think I remember a Victoria Wood character played by Julie Walters who found that her dog had somehow changed the HVAC setting, it wasn't hormones at all.

[C R Mudgeon]

Re: Always rather it too cold than too hot

The dog had caught a late showing of "Gaslight" on TV.

[J.G.Harston]

Re: Always rather it too cold than too hot

How would circulating warm air do anything for people finding warm air uncomfortable?

[Mimsey Borogove]

Re: Always rather it too cold than too hot

How would circulating warm air do anything for people finding warm air uncomfortable?

That's what my mom always said, but I've found that a fan, even if I have to hold and move it myself, at least keeps my face from getting sweaty.

[joed]

Re: Always rather it too cold than too hot

I'd consider upper respiratory airways to be the biggest issue. I can go out and spend considerable amout of time in subzero conditions (basically until I cant feel my digits any more) but staying at refrigerated office with near zero relative humidity is a punishment of its own. Literally everyone without an office facing sun gets sick int a weekly cycle. And all this at significant expense (monetary and environmental). Efficiency typical for an American office or preferred vehicle choice out here.

[JT_3K]

Re: Always rather it too cold than too hot

I fought this at a previous employer through use of USB-heated fingerless gloves. I looked ridiculous, particularly in a senior position, but it worked.

[Mimsey Borogove]

Re: Always rather it too cold than too hot

I was a database searcher for many years, most of it in a building that most people considered too cold. People brought in heaters, wore coats while working, etc., and many of them got what I think of as Bob Cratchitt gloves, with just the palm and back of the hand covered and the fingers free to type. Don't know if it helped or not - I loved the temperature!

[DS999]

Re: Always rather it too cold than too hot

People shouldn't have to bring sweaters to work in July just because there are some people who "run hot" and insist on keeping the AC at arctic levels. How about they bring a desk fan to work instead?

[J.G.Harston]

Re: Always rather it too cold than too hot

"How about they bring a desk fan to work instead?"

And circulate the hot air across everybody? The basterds!

[I could be a dog really]

Re: Always rather it too cold than too hot

If it's too cold you can warm up by putting on extra layers.

I agree, but have you ever tried telling an office full of women of differing ages and with differing ideas on fashion ?

At a place we worked at, I tried suggesting that if people know they are going to be a bit on the cool side, they might consider wearing appropriate clothing - it "wasn't well received". I merely pointed out that certain of the young ladies would come to work in the depths of winter wearing light summer clothes - and then complain that they are cold. The fact that they were chilled to the bone before they even got into the office was irrelevant - it was "my fault" that the office was uncomfortable.

At the time, one large space had a large A/C unit in the ceiling - IIRC around 14kW capacity. I'd repeatedly set it to sensible settings and tell people to "turn it up or down a degree or two at most". But sure as ursines defecate in the woods, people would turn up first thing in the morning, be cold (note above about being cold before they are even in the office) - and so whack it up, not by a degree or two, but to 30˚C. Needless to say, after a while it's then too hot - so it gets turned off, not down. Then by early afternoon the place is still too hot, so it gets turned back on and whacked down to 18˚C. Then after a while, the place is freezing so it gets turned off. The next morning, the office is cold because there's been no heating on - and it's all my fault.

We also had a few ladies of a certain age where they would change between too hot and too cold several times a day - but they were of an age where they understood that the world didn't revolve around themselves.

[JPeasmould]

Re: Always rather it too cold than too hot

I didn't realise how cold the air coming out of the floor vents in the control room in Wessex Studio 1 was until I had to line up a half-inch tape machine wearing a kilt (a story for another time). The vents were directly in front of the machine. That was the quickest alignment and tones-run I ever did.

[Anonymous Coward]

Re: Always rather it too cold than too hot

Why was the tape machine wearing a kilt? Inquiring minds want to know,

[jake]

Re: Always rather it too cold than too hot

Error in translation. The mag tape itself was Scotch brand.

Easy mistake to make.

_{My round, I think ...}

[JT_3K]

Re: Always rather it too cold than too hot

Re-record, not fade away. Re-record, not fade away.

- I'll get my coat

[Anonymous Coward]

Re: Always rather it too cold than too hot

If your desk is too cold, move away from the HVAC vent, instead of negatively impacting everyone else.

[David Hicklin]

Re: Always rather it too cold than too hot

Ah the temperature wars, the bane of the open plan office - we rarely had this when we had individual offices. I don't think there will ever be a solution for it as people are just to varied.

As for why men are usually warm and women colder? Try removing your body hair neck down (No Don't ask - that is another story!) and you will notice a heck of a difference. Women (usually) don't have much whilst some guys look like their mother mated with a bear.

As for me? I was always to warm but had reached the point where I could not remove anything else to stay in an acceptable state... WFH for the last few years fixed that however.

[Robert Carnegie]

Re: Always rather it too cold than too hot

Tricky to investigate, but your clean room new hires (if not in the storeroom instead) may have misunderstood what clothes can and can't be worn inside the clean room worker suit.

There's a slightly tall tale of a traditionally minded Church of England priest, "High Church" with full church robing, who invited a much more liberal pastor to co-host a church service. Then the host asked his guest about the experience, and was told that the guest appreciated being able to put his trousers back on afterwards... which, in fact, he wasn't really meant to take off.

[Mimsey Borogove]

Re: Always rather it too cold than too hot

In about 1984 I was working in a hospital library. The building was very old, and, as the real hospital part where patients were seen had been moved to a newer building, didn't have central air. There was a window air conditioner in the librarian's office, but she was one of these tiny <100-lb. women who was always cold, so there was a lot of back-and-forth about it. We finally settled on a temperature of 80F, because she couldn't function under 80 and I couldn't function under it. I managed a couple of years there, largely by turning up the air when she left the office (I covered evenings).

A better situation was later when I worked in a room with 5 high-capacity printers, each of which could hold 8 reams of paper. That room had to be kept cold for the printers, so I was allowed to complain if it got above 65F. Aaaaaaahhhhhhhh!

[C R Mudgeon]

Similar problem, opposite cause

It was a regular office building with fully functional HVAC. Problem #1: as is often the case, the thermostat was set too low for my liking. I think I'm part cat -- I like things warm, dammit.

Problem #2: one of my co-workers found the temperature too high for his liking. And he had access to the machine room. So he tended to block the door open, so the cold machine-room air would grant him relief.

Problem #3: The machine-room door was just across from my desk, which meant that I -- who was already chilly -- got most of the "benefit" of his efforts, while he, on the far side of the office, got fairly little.

And as a lowly contractor, I wasn't in a position to ask to swap desks with him.

As for what his workaround meant for the servers' security (and temperature)? Well, I grumbled to myself, but as low man on the totem pole, it wasn't my place to raise a fuss...

[Giles C]

I thinkk air con

Is the biggest cause of problems in offices, especially where the HVAC was designed around one layout and the internal partitions were rearranged afterwords.

At a previous employer it got to the point that you could walk out the building at the end of the week, and when you came back in you would find a wall in the middle of the route to your desk.

This had the problem of light switches that didn’t work the lights in the room, AC controls that did nothing (well not in the room where your were sitting etc.

The lighting I think they finally sorted with motion sensors, but the AC nope…..

[breakfast]

Re: I thinkk air con

Motion sensor office lights are genius. Sitting thinking through a problem and suddenly you're thinking in the dark.

[ChoHag]

Re: I thinkk air con

Even better if you do your best thinking on the toilet...

[Anonymous Coward]

I seem to have had more than my fair share of aircon related issues!

Not once, but twice had machine room underfloor space flooded when aircon pipes burst. 2 different companies too.

Then there was the time the aircon was first switched on after a move to a new building/machine room only for it to spit little bits of rust/debris all over the place - not good for the mainframe (ICL2966) and the disk packs we had to use. Turns out no-one thought to even test the aircon before starting up the mainframe post-move, until it got a little 'toasty'.

My favourite was the time, during hot periods, one of the machine rooms would get so hot systems would start powering themselves off. The only warning we'd get was when machines started going offline, which gave us about 10 minutes before everything else went. Bear in mind getting in the machine room was a security nightmare - and the 'fix' was to have all doors open to the outside and hire some very big fans to get air moving through the room.

Nearly forgot this gem. We used to have racks that had a foot bolted to the front to prevent it toppling over. One rack's foot partially blocked an air vent (poor planning), and someone unbolted it to move the vent. Unfortunately they forgot to bolt the foot back on again, and when an engineer came to fix a problem on a machine near the top of that rack, he pulled it out and the rack started to fall over. Luckily I was there and managed to stop it from falling on him. He hadn't pulled out the machine far enough to lock the rails, so we were able to push it back in quickly enough to prevent a more serious problem. We were both a bit shook up after that!

[trevorde]

Freezing in an Aussie summer

One place I worked at in Perth, Australia, had a fire exit which opened onto the car park. Every lunch time, everyone used this door to go to their cars and thence onto a lunch bar. Only problem was the air conditioning sensor was right next to this door. Every time the door was opened, a blast of hot, 42C air wafted over the sensor. As a result, the air conditioning was working overtime to cool down the building. After lunch, the office temperature plummeted from a pleasant 20C to an arctic 12C!

[Tim99]

Re: Freezing in an Aussie summer

Another Perth aircon story. Over 30 years ago I was working in a building that had evaporative air condition - also known as "damp fug" - It worked fairly well when the weather was hot and dry, but seemed to make the air worse when warm and humid. I needed a reverse-cycle system to control the room temperature when we were installing a very expensive instrument that required stable temperatures. The MD was not impressed when he found out, and I had to explain that it was for the instrument, and not the staff. Purely by coincidence, the summer when it was installed and working found most of the staff sitting in that room.

[I could be a dog really]

Re: Freezing in an Aussie summer

Also known as swamp coolers.

They do in fact work reasonably well in dry environments - simple physics, it takes energy to change water from liquid to vapour, the result is that the air gets cooled but also more moist. Fine in very dry conditions as long as you have plenty of ventilation to keep replacing the moist air with more dry air.

If the air is already moist, you don't get much cooling, but you also end up with moister air - so the room gets "muggy". Add in people who won't accept that the laws of physics apply to them and that they need to keep the room ventilated, and the result is very quickly turning a warm room into something resembling a sauna. More than once I'd sent evaporative coolers back when people had hired them (without asking first) for the simple reason that they stood no chance of working in the rooms people wanted them for.

[herman]

Re: Freezing in an Aussie summer

In a Calgary meeting room directly underneath the A/C equipment it could reach a balmy minus thirty Celsius in Winter - a whole ten degrees warmer than outside. We went in there with parkas on.

[DS999]

Re: Freezing in an Aussie summer

If its a fire exit you could have surreptitiously added one of those alarm bars to it. That would stop people taking that shortcut to the parking lot lol

[alain williams]

Remember to switch it back on

Many years ago I did some contract Unix sysadmin at a large insurance company in South London. They ran a pair of Sequent machines.

The previous sysadmin had gone in to do some work at the weekend that had to be done in the machine room. He felt cold so switched the HVAC off, did what he had to do and went home.

By Monday the machines had cooked themselves. One of them never quite worked right after that, occasional strange problems.

[David Hicklin]

Re: Remember to switch it back on

Not quite the same but at an early job in the 1990's the little cupboard that passed as our server room had an aircon and the condensate pump was a strange affair.

It had to pump up quite a way to get out to the drain pipe for all the aircons (and it was on the ground floor), and consisted of a piece of thin plastic tubing held in a half loop with a rotating contraption in the middle with 3 rollers that went round and round, basically squeezing the water round the tube.

Of course this eventually work hardened causing the water to back up and freeze in the Aircon. At which point the Aircon shut down to preserve itself and as the temperature skyrocketed the ice melted leaving the floor of the room looking like a shallow paddling pool - a very hot, humid one at that.

And of course it waiting until a weekend to get in that state ! How nothing failed or shut down I will never know.

[MiguelC]

I once did a stint at an insurance company during a hot western European summer. At the time, wearing business attire was mandatory, even for IT. So I wore the jacket on my arm, got into my "office" (the server room), put my jacket on, buttoned it, and toiled away. And the end of the day I would take my jacket off and go home... Fortunately it was just for a pair of weeks, but it felt really awkward.

I was wise enough not to mess with the aircon though, I'm sure the client wouldn't be pleased otherwise.

[BJC]

That reminds me of the times when I used to fairly frequently cross the pond from the right side to the wrong left ;-) side - specifically Houston. Travelling in summer, I didn't anticipate much need for warm clothing. That was true outdoors. However, indoors I found the HVAC so severe that I quickly learned to pack a jumper for inside the office.

[Chloe Cresswell]

Friend of mine, a US coder, once remarked it to me as "America. Where we want to wear sweaters in summer and tshirts in winter"

[Giles C]

With comms rooms I have worked in the jacket would be needed as it is so cold in there.

One job I went to the site in Sunderland discovered I had not bought a jumper with me and had to go to the nearest supermarket (only thing still open) to buy one so I could work without shivering.

In contrast my current employer has a site in a brand new DC complex and you can walk around that comfortably, it is kept at a reasonable temp (19C ish) and the aircon doesn’t attempt to blow you out of doors.

[I could be a dog really]

Yes, people are starting to realise that machine rooms don't actually need to be all that cold for systems to be happy. It saves a lot of cash.

[GlenP]

Way back when computer rooms were kept at 15-16C I always kept a jumper in the office, actually it was even more needed in summer as moving from a warm office into the cold wasn't pleasant.

[TeeCee]

One day, outdoors having a cigarette with a Data Centre manager in SE Asia who'd just given us "the tour" and whom I knew from working together elsewhere, I asked about the hot and cold aisles:

"So, how much space is running warmer?"

"XYZ square metres".

"How much money does that save?"

"More than you'd think, x thousand dollars a month! TPTB are very impressed."

"What's the total size of the building?"

"XXXXYYYYYYZZZZZZ square metres."

"Right. Given how you've just had to get a lad with a screwdriver to tweak the thermostat to stop us all freezing our balls off in the meeting room, how much would, say, two degrees on the entire building save?"

He left at Warp 10 to do some maths and get more lads with screwdrivers on the case.

[Mugs]

More hot air stories

James Capel stock brokers used to have Tandems in the basement of their city office. When the IRA bombed Baltic Exchange they lost power to the office but the Tandems stayed up (internal and external UPS). The air-conditioning didn't, they ended up propping the basement doors open. The Tandems overheated but still stayed up.

Later, I was working at NatWest. We needed some extra Tandem CPUs and bought some second hand. The Tandem rep who installed them looked up the serial numbers and took great delight in telling us that they were the ones that had been run over temperature for days at James Capel. Still worked fine.

Also at NatWest, we had a small Tandem CLX in a basement. To prevent overheating the basement windows were left open. It started to report errors, the engineer found corrosion where the rain had blown in. We got a bollocking for that one.

[billdehaan]

Management (nearly) caused a similar problem

When I was contracting at $BIGFIRM, they had a lot of legacy big iron. Other than the reception desk, it took up the entire ground floor office of the company's high rise. Although $VENDOR offered on-site support, and in fact had provided it for several years, if not decades, the new management had cancelled it two years earlier, because "we can maintain our own computers, just as well, for a lot cheaper".

Narrator: They could not, in fact, maintain the computers just as well.

$VENDOR had addressed HVAC by providing what were essentially refrigerators at strategic locations in the ground floor's system room. And by providing, I mean renting, at a high cost, which the new management considered an unnecessary expense, and so they were cancelled. The IT crowd objected, saying that a melting computer center would be a bad idea, but management persisted.

Their idea was cheaper, and just as good. Even better, in fact. They would water cool the ground floor. They rerouted plumbing so that the system room would be water cooled. It seemed to work in general, but the room temperature was uneven, fluctuating based on water consumption, being noticabely warmer on weekends and holidays.

One day, I was charged with installing a new network cable. I entered the system room, along with a new hire I was training, and when we entered the room, it was stifling. It should be between 18C-22C, but it was just under 29C. I said we'd better report this to Facilities, and he casually mentioned that they already knew. In fact they'd told him that there was a "known temperature issue in the machine room" 20 minutes earlier, he'd forgotten to mention it.

I grumbled, but one must always be polite to wives, politicians, and new hires, so I just continued on.

I was to show him how to hook up a cable a cable, and I explained the raised floor setup we had. I got the suction cup grip, and lifted the tile on the raised floor, preparing to show him the 18" elevation below, with the "eel farm" (as it was called) of network, electrical, and god knows what other cables that were underneath the floor.

I was not prepared to show him the 12" of running water that a broken cooling pipe had flooded. The new hire, unaware that this was not normal, asked me if it was a good idea to have stick my hand in running water that had power cables like that. No, I agreed, that it would in fact be a bad idea, a very bad idea.

He asked me what we should do, then. Sadly, this predated the Pinky and the Brain cartoons' (in)famous "We shall flee in terror. Yes. That would be the wisest choice" quote, because it would have been tremendously appropriate.

I restored the floor covering, we left the room at speed, and casually mentioned to Facilities that there was a foot of running water in the system room. The Facilities person was fairly senior, and responded appropriately. There was an announcement on the PA system about five minutes later informing everyone in the building that they were shutting off all computer systems down, immediately, so save your work. People calling Facilities to ask them to delay were greeted with ringing phones, as the entire Facilities department was in the system room, powering everything off as quickly and safely as possible.

My network cable installation task was deferred for two months. When I did it, the system room was 18C again, I noticed those portable refrigerator units were back, and there was not only a lack of running water under the floor, I saw some moisture sensors there that had running wires, presumably to a system that would alert Facilities if there had been another broken pipe.

[gosand]

I joined a small company that had been acquired by a bigger one. We were on the 6th floor of an office building, and had our server room on our floor. Just before Christmas I was one of only a couple of people in the office, and we heard the server alarms going off. Turned out the AC unit had died and temps were rising. Nobody in the office that day had access to the server room, but we were able to contact someone at corporate and convinced them to unlock it remotely.

We were able to open up some of the drop ceiling tiles to let the hot air out, and ran some fans to pull in cool air from the office (through the propped-open door). The AC was fixed the next day, and no servers died - which was good, because we didn't have access to those either.

[goblinski]

As for most things life-related, College Humor had that one nailed as well.

[Xalran]

How about wet from sweat ?

I'll probably send it to On Call at some point since it was more a on call than anything else...

Lets picture a container filled with 3G telecom equipment [parse that as a few racks of SUN Servers], an home grade air cond unit, in summer, under the sun... And a $TELCO that lost contact with everything in that container. I literally melted when I opened the door of the container (instant massive amount of sweat caused by the wall of heat that hit me despite the 35°C+ temperature outside), it was impossible to enter it, and the reason why all the servers in it were down was obvious : thermal protection.

I went to the nearest bar (we don't have pubs here) and got a pint while waiting for the temperature to lower to a semi livable level, then after a cursory check, I went back to the office to tell my boss that they had to call the aircond maintenance.

[Lazlo Woodbine]

Back in the mid-to late 90's, I enrolled on an IT college course.

For the first few months, the course was all theory, as the college hadn't yet set up an IT lab.

This all changed after Christmas, when we were ushered into a room full of shiny beige desktop computers.

Each computer, assembled by hand at a nearby technology college, consisted of a desktop chassis equipped with floppy disc and a CD Rom drive, the case had an air inlet on the left and a hot air outlet on the right.

There were 12 computers on a long bench on the left of the room, and another 12 on the right.

It wasn't terribly long into our first lesson that the computer at the far end of each row started showing signs of distress, and gradually the computers started to crash as it ingested ever hotter air from the preceding computer.

We were back to theory lessons for a couple of months before they could wangle another 24 computers, and this time they were arrayed in 4 well-spaced rows of 6 computers...

[Throatwarbler Mangrove]

A mobile solution

Many years ago, my team had just moved into a new office, where Facilities had not quite worked out all the issues with the HVAC system. As a result of the recent construction, many of the ducts would apparently get clogged with latent dust, causing the HVAC system to shut down or work inefficiently, at which point a technician would need to clean them out and we'd then wait for the temperature to normalize. As the move happened in July, this situation was not ideal. Fortunately, we had a set of Movin' Cools (basically, giant mobile AC units) that could be positioned as needed, but the trick was figuring out where to vent the hot air; eventually, I figured out that a set of paper clips was strong enough to anchor the air vent to the HVAC intake vent, thus both ridding ourselves of the hot air and fooling the HVAC system into thinking the ambient was much warmer and thus, hopefully encouraging it to work harder.

Pretty much of the opposite of a cool story, I know.

[Zed 2]

I do telecom for a living.

At one office, the only place they could place the phone system was in the room that was roof access.

In Detroit, it could go from -30C to +30C (in a sauna in the summer), so there were many trips down there that would need many different layers.

Rather not do that again.

The demark was in the boiler room with cloth-wrapped wires in a WWII vintage site. Not fun to deal with T-1s in the '90s

[NXM]

Get Carter

You're a big man, but you're out of shape. I do telecom for a living! (Then pushes big man off the car park)

[Anonymous Coward]

IT Solution

When we moved in to our new office, evil open plan, our area was much too warm. Facilities would/could not do anything about it so we hacked the control PC. We modeled the air temperatures on our floor with a spreadsheet and planned and made subtle adjustments over a week or so. Nobody seemed to notice.

[MattieD]

The shivering server

A decade or so back we had just virtualised an old (and I mean OLD) legacy system that had sat on it's own server in the datacentre rack. As is always the case we'd been instructed to keep the physical server around 'just in case'.

Every now and then we'd get a temperature warning email and a few minutes later it'd clear. As we didn't care too much about the server and the alert had cleared we didn't bother to investigate. Eventually these alerts were getting almost constant, so we decided to take a wander down. Nothing was using the server so we were intrigued by what was causing the warning.

We logged in through the console and checked the stats. The temperature warning wasn't because it was too hot, it was complaining that it was too cold. The CPU temp was down around 12°C and that was just below the range where the temperature warning would trigger.

First time I've ever seen a computer complain about being a bit chilly.

[lordminty]

Since when was

A Tandem box a "mainframe"?

Of the same era an IBM 3081 was a mainframe, with masses of disks, comms controllers, tape drives etc. And if the air-conditioning and chilled water supply (a 3081 was water cooled) went off, the machine room would get very hot, very, very, very, quickly. Like 40+°C.

Don't ask me how I know.

And said 3081 would be bleeping its alarm very loudly, just before it turned itself, and everything else, off.

[Richard Pennington 1]

HVAC settings

UK, mid-late 1980s.

My first job out of University was at a software company which still had a typing pool and a mainframe (the latter housed in a dedicated room with environmental controls for humidity and temperature). The local rumour was that by proper (i.e. highly improper) settings on the environmental switches (probably something along the lines of "humidity right up, temperature right down") it was possible to cause it to snow in in the machine room.

I don't know whether this theory was ever tested.

Penguins because they like snow.

[Robert Carnegie]

Re: HVAC settings

Do penguins like snow, though? You never seem them flying in it. :-)

[JT_3K]

Re: HVAC settings

Finishing reading this thread and intending to add my own, yours seems entertainingly to have crossovers.

Gaining one of my first ever fiefdoms in ~2007 I was lucky to arrive to find a ~4m x 4m dedicated computer room, which was a 3m high breeze-block single room in the corner of a UK business-park type warehouse with ceiling tiles on it. Less fortunate was I to find the motley crew of "servers" that were pushed together under a desk and that the door was permanently open to allow a combo of the warehouse staff to use one of them as a personal computer, and for """cooling""".

Over time I put in a fairly early and rudimentary VMware stack on a few pieces of HP tin. Real bang for the buck at the time. To protect the new investment I asked if we could put aircon in, but the cost was deemed too much so I got a household thermostat connected to a bathroom extractor fan, with the fan being the type with rotating flaps that opened when it turned on. I was polite but made it clear that I didn't find it suitable.

Late one Friday evening, my on-prem Exchange stopped routing and my VPN stopped taking connections. As the company was an 8-6 M-F affair with a roving sales team that it wouldn't impact and 1hr from home, I assumed a power cut with little impact, and decided to stop by as I happened to be passing the next day. On arrival I found a 4m high "scorch mark" from a large hole on the outside of the building where the back of the fan had once been. Transpired that the rotating flaps had *not* opened when the fan had started, the fan had superheated and caught fire, and then turned itself in to plastic flakes and soot. The servers had gleefully sucked this in and diligently, as they'd become more packed with plasti-flakes, ramped up the fans in a desperate attempt to cool further, then having packed themselves (beautifully) had thermal shutdown at ~85c. All this before the fan had gone "enough" for the power to drop out of the plastic housing and short itself to blow the power to the room.

I cleaned the room for two days, all the servers too. Removed the fire blackened roof and swapped cables. Lo and behold, the HP kit booted without any losses - somehow, and the business was online as normal on Sunday night.

Suddenly and inexplicably, there was budget for a pair of humidity controlling aircon, a raised floor and the room to be re-fit with a solid roof with insulation. There was a great moment where, as a younger and more gung-ho (idiotic) guy, I agreed to crane/forklift/sling the live server rack from the concrete floor on to the new raised floor. In my (limited) defense, the forklift operator was one of those guys that could pick up a coin, and understood the assignment, and we had no issues.

====

The bit that aligns with your story however is shortly after. My new CLIMate monitoring device, hooked to my shiny new PRTG monitor emailed overnight just after the aircon was installed, just after I went to sleep. It told me my beautiful new TIA-942 compliant server room was running at 14c and 90% humidity. Seems the HVAC firm on the business park had limited experience with humidity or server-room aircon units and had misconfigured. Some oscillations we'd had for a week or so and been complaining to them about had got out of hand.

By the time I got to site (at quite the clip) early morning, it was colder and more moist. I've never experienced anything like it in my life. Was like walking in to a tropical rainforest just after a downpour, but without the heat.

I shut down all the kit and opened the door, then left it for ~24hrs whilst the air-con was finally set up properly and the kit re-acclimatised, and it somehow dried out and came back on with no complaints. And that, is the story of why I now have a very strong affliction for HP enterprise server hardware...

[[e-mail]]

During the dot com boom, our office got very crowded, and also very hot with all the people and equipment.

But, the management listened and had some air conditioners installed. The hot exhaust air was dumped in the double ceiling.

A few days later, I heard on the local news that a street was closed because an office was on fire. When I arrived the next morning, it turned out to be my office and investigations showed that the fire had started in one of the AC units.

Interestingly, the fire was discovered by the branch in Silicon Valley. Their system manager phoned our system manager why he could not reach the servers. When our system manager checked it out, he found the office on fire.