Servers hated Mondays until techie quit quaffing coffee in their company Mornings are hard, and Friday mornings doubly so. Which is why The Register gives readers a little kick along on the last day of the working week in the form of a new installment of On Call, the reader-contributed column that tells your tales of tech support treachery and triumph. This week, meet a reader we'll Regomize as " …
I used to design boards to environmental level 5.
Part of the testing involved putting the boards in an environmental chamber to temperatures of -40C (-50C for some special boards) and +80C (+85C for some special boards). Would they start up after an hour at either extreme was a basic test. (Emulating a night in a Finish tank, or a day in a Saudi helicopter.)
One particularly nasty test involved cycling from one extreme to the other and back again (within single digit minutes).
Condensation was a real killer until we insisted on conformal coating before testing (something that made rework much harder and so previously resisted - pun intended).
One of our customers does a lot of food transport. During summertime the warehouse got quite warm, and their walk-in freezer remained rather cold. They went through a lot of portable bar code scanners before switching to a different model!
(We also had to place a wifi access point within their walk-in freezer, because insulated metal boxes aren't particularly good at letting rf signals through. That also provided some challenges)
I spent 6 months working in a cheese wholesaler while they went through the process of installing Red Prairie warehouse management system.
Their warehouse was a 70,000 sq ft fridge, and we needed to install 16 APs for the WiFi to enable the portable barcode terminals to stay connected down the long aisles.
One July day, the hottest day in memory in that part of the UK, the warehouse pickers started to complain about their terminals not connecting in one corner of the chiller. After checking the network, we found 4 of the APs were off, so me and the network guy donned hard hats and hi-viz bibs and ventured into the roof space of the chiller to check on the cables to the APs.
Matt walked ahead of me checking the CAT6 cable run with a torch. As we approached the corner with the dead APs I heard a yelp and a string of profanities.
It seems the chiller units were working so hard to keep the big chiller below 3°C, the power cables were very hot, and they'd melted the CAT6 cables to those APs, and burnt Matt's hand as he tried to move it out of the way.
The solution was running more cables and putting a thermal blanket between the network cables and the chiller power cables...
Ouch! Yet another reason why network and power should ideally be in separate conduits (magnetic interference being the main reason).
Reminds me of one very large building, used to be industrial, but was converted to a large store with a bunch of different stands. They have these wonderful super wide conduits running along the ceiling everywhere, with separators for different kinds of cabling, but unfortunately everyone who came before us decided to disregard the separators and put the electrical wiring wherever :')
Yikes!
I would think the solution would be "call an HVAC tech and an electrician" to address those 'very hot' power cables, whether the cause was under-spec cabling or a failing compressor that's over amping!
Personally I would not "solve" the problem of the cat6 cables with a thermal blanket - they are what alerted you to the danger in the first place. Losing part of the network is better than a fire!
They make WAPs that use an external antenna. Seen 'em used in big-box stores with the antenna inside the cooler and the WAP outside. Also. with the antenna outside the building and the WAP inside, to provide coverage in the parking lot, truck dock, outside lawn-and-garden, etc. One thing I've never figured out is why in bl**dy heck they made these so that if you install them with the writing right-side-up, the LAN connectors face up, and collect any condensation from cold roof trusses, water from roof leakage, etc.... I have seen several destroyed when moisture had made the PoE sizzle inside the connector and fry the contacts. I would remount the replacement WAP "sideways" if possible, and/or improvise a tent out of a piece of cardboard and a plastic bag, to reduce the risk of liquid contamination recurrence. It wouldn't do to mount it upside down, because gravity + vibration would encourage the WAP to slide off its mounting bracket. I've always wondered if this design was the result of someone not paying attention to how their product would be used, or was a subtle form of planned obsolescence (a.k.a. a triumph of marketing over engineering).
On the subject of RF and metal boxes... I've been to several locations of a chain restaurant (think "neighborhood bar and grill" or something like that) where the intrusion alarm had been updated, and all the contacts and sensors were replaced with wireless ones. The installer had concealed the transmitters for the door contacts inside metal Handyboxes* with metal covers. Not a perfect Faraday cage, but it did confine the Hertzian waves to the extent that the control panel reported "loss of supervision". My solution was to install plastic junction-box covers from the DIY store.
I later learned that this installer had done the same thing at over 20 sites, and was not pleased to have to revisit them all to rework them.
* a Handybox is a North American single-gang electrical junction box. It is intended to be surface-mounted, not concealed in a wall. It is drawn, so that it has rounded corners, versus being made of folded-and-welded flat sheets of metal and having sharp corners.
A previous employer of mine had a thermal test chamber for very similar purposes.
One day a colleague spilled coffee on his keyboard. It being a bombproof Model M or equivalent, he rinsed it clean under the tap and put it in the TTC to dry it out.
Which was a brilliant plan except that he forgot about it in there.
He was reminded, when the smoke alarms went off and the entire site got evacuated to the car-park.
Apparently, the keyboard, once retrieved, looked somewhat Dali-esque.
A few jobs ago I used to be an end of line QC technician. The medical products the company was producing for a client had to have their electronic performance tested by being immersed in a very strong saline solution, inside a lab incubator at a wide range of temperatures - from memory about 50c to -5 when they put in the fridge afterwards.
The rigs designed for this were supplied by the client, made of acrylic and superglue, with the PCB handling all the RFID stuff underneath the liquid reservoir. Even with conformal coating we used to lose one or two a week out of the sixteen or so that would be in use thanks to corrosion from leaking saline solution or the constant temperature cycling.
Not much fun when the custom PCB's came in at about £1000 and the client refused to change the design to make them more robust. Still, it was their money when the rigs broke.
Such temperature swings with high and low humidity is how CSR (now Qualcomm) used to test Bluetooth chips, running test software continuously. This does accelerated ageing of the package, leads, interconnects &c. Apparently condensation can get inside a solid chip package given enough time! But of course if the failure is after 10+ simulated years that's OK.
It wasn't enough to catch one nasty bug whereby booting the chip, it calculates a table for setting the radio on the 80 channels. Since things can warm, or cool, a load of extra entries are calculated, and the base pointer steps up or down the table as temperature changes so the radio still works. But not enough if it booted at -40° and it actually goes all the way to 80°C, when the table wasn't long enough...
It was a car company that found this one, in their testing of Bluetooth chips deep in the dashboard - as you said, Finnish winter to an hour's driving gets the whole car interior toasty....
Chips generally come with a "shelf life" after which they have to be put in a (lowish) temperature oven for some time to drive out the moisture that gets in.
If they are used after that time without the bake, then they can explode when soldering when the trapped moisture turns into steam.
Working for a company that develops equipment for installation into public areas of buildings, some types of product require us to consider not only obvious environmental threats such as condensation, but also other types of fluid ingress. Some, like rain, fire suppression systems etc, are obvious. Others, such as can be encountered in, say, a poorly monitored car park stairwell/lift, are rather less obvious to the uninitiated product designer, and are best left to the fertile imagination of the dear reader to consider exactly what types of fluid we may be dealing with here - suffice it to say, whatever you might think up here is probably correct for at least some install somewhere in the world...
ESTEC (the European Space Technology & Research Centre - one of the ESA establishments) used to have environmental chamber for test satellites. During one test sequence the satellite would be heated to about +200C on one side, while being cooled to -150C on the other side - AT THE SAME TIME.
Apparently the chamber cost a bomb to construct, but earned its cost back many time over in robust satellite design/builds.
(Four Yorkshiremen voice)
You had it easy!
I once had a desk in a place like that. The heating 'system'* for a 3 story building was a pipe that pushed hot water around each floor starting at the top. By the time it reached the ground floor, it was somewhat tepid. Added to the long, south facing glass clad frontage (yup, hot in summer, cold in winter) and that the air conditioning was on a 'one size fits all' per floor basis (management had the south facing offices and control of the thermostats, of course), it was far from ideal. The 'killer' for me was that my desk on the ground floor was directly under the 'fresh air' vent. Every couple of hours, unheated outside air was dumped on my head to make sure we were not breathing too much recycled air. In winter I was wearing my serious mountaineering** down jacket with hood up and thermals.***
* Run from an irreplaceable and unmaintainable Windows 3.11 'box' in the 2000's because no one knew how it worked or dared to see what happened if it got turned off.
** Kilimanjaro summit, Iceland, Bhutan, Everest base camps in Nepal (Khumbu glacier) and Tibet (Rongbuk glacier), Antarctica and, of course, Scotland. (Yeah I've been around a bit, but then I am a tad over 60 and we didn't have the InterPleb in those days, y'know. You young people don't know you're born. insert lengthy tedious old fart complaint of your choice here...)
*** Surprisingly this is actually true. It was b**^&y freezing.
Hot summers with rows of servers sat in a server room with barely enough airconditioning on a good day, let alone a bad one. It go so bad one year that the company had to open up all the doors to the machine room and hire massive fans just to try and get air circulating. The only warning we'd get of impending doom and hours of recovery time were one or two servers crashing with overtemp alarms about 15 minutes before the storage array would shut itself off (nowhere near enough time to shut down the remaining servers cleanly before that happened).
Didn't help we were a good hour away from the building the machine room was in, as were any DC engineers who could have warned us in advance. Yes, we comlpained about it all the time, and no, the company couldn't be bothered to spend any money to fix the problem.
A long time ago I worked on the top floor of a listed building that had been converted to offices.
Of course it turned into a sauna every summer, but staff requests for aircon were always declined.
And then we got new servers ... which turned themselves off on the first hot day.
Portable aircon arrived the following morning.
After that neat demonstration of where staff ranked, turnover increased considerably.
Sounds like the experience of my friend who was a manager in a suite of hospital labs and the staff sweltered every summer without relief despite his requests for air-con until they got new analysers which started falling over in the heat, and suddenly the funds for infrastructure were found.
I once was in a very small (5-cube) office. It only had one small air vent, and was often warm in summer and cool in winter. There was a big portable air-con unit in the room, but while it faithfully blew cool air, it didn't do much. For some reason (why now eludes me), we had to peek above the drop ceiling. Turns out the air-con was dumping the hot air above the drop ceiling... pushing it right back into the space. So it was a very large, very noisy dehumidifier.
In a long ago job I worked at a university and my division had a former classroom turned into a combination graduate computer lab and server room on the third floor of an academic building. The building AC was sufficient for rooms filled with people but not rooms filled with a dozen Sun workstations, a stack of a couple dozen headless HP-UX workstations acting as a computing cluster, an AIX box and a full height SGI Onyx. Fortunately that room had been equipped with its own dedicated Lieber AC unit. Every once in a while that would break down and we might have to shut down some stuff in there until Facilities came to fix it, but for the most part it was a good setup, or at least as good as we could do without a proper server room.
Since facilities would turn off the building AC on summer weekends (because there few students/faculty wanted to come in on weekends) if they had to work they'd come to that lab instead of their offices because it was kept cool. Once we had a particularly hot weekend with heat indices touching 110F and nights where the low didn't break 80F. When we came in Monday morning our office (which was next door) was fine since the building AC turned on a few hours earlier, but walking into the server room it felt like a sauna. At first we thought the dedicated unit had gone out, but it was pushing out cold air. It simply wasn't able to cool that level of heat, especially without the building AC to help. Probably didn't help that there were 1" + gaps under the doors so a lot of that cool air was likely sinking to floor level then going into our office or out the other door into the hallway.
We later heard from a couple grad students who said they came in over the weekend but it was hotter in there than it was outside! As far as we know nothing broke down as a result - at least not in the immediate aftermath that we could conclusively link to it. We had Facilities install sweeps on the doors so the cool air couldn't escape so easily in the future.
I image that the HDDs couldn't spin up because the spindle bearing lubricant had got sticky with the cold.
There was most probably a minimum operating temperature specified for the drives by their manufacturer that was not complied with in this case.
I remember when, in about 1984, the design engineers got Compaq i386 PCs that had Western Digital 40MB IDE drives. After a year or so (i.e. just out of warranty), one of them stopped booting up. I was asked to investigate & found that the HDD wasn't spinning up because the spindle bearing had seized, the reason for this being that Compaq had ignored WD's fitting instructions and mounted the drive upside down, thereby putting the load on the bearing in the wrong direction.
With a gentle bit of hand rotary motion of the whole drive with it the right way up, I managed to free it and it spun up normally but there was no way without serious modification that it could fitted into the PC the right way up. Over time all of the other Compaqs failed in the same way.
I seriously doubt that the un*x boxen in the story were PCs with WD harddrives. The drives would have probably been CDC/MPI or the like.
Compaq's first 386 computer (the Deskpro 386) was released in late 1986. The hard drives were ESDI, not IDE.
WD's first 40 meg IDE HDDs were after they bought Tandon's hard drive manufacturing assets in 1988 (1989?).
Not so in this case, definitely a WD 40 MB IDE. I know because I used one of the more reliable ones that had been replaced to make a 286 PC from parts that were evaluated & rejected when we were trying to make our own PC from parts to put into our engine analyser products. I used the PC for the first two years of my Open University degree.
Of course it's possible that it was later than I thought when they got the the Deskpros, it was rather a long time ago now....
Thank you for the beer, though I suspect it'll be a contributing factor to my failing memory :)
I tend to remember things by association to the products that were being developed at the time rather than the actual years so, thinking about it, it was probably nearer 1988/1989 as the Deskpros were used with Intel I2ICEs to develop & debug 80196 based products, whereas COBEST was 80188 & 8032 developed on Intel Series 4 & AUTOMATE 450 was 8085 developed on Genrad.
I remember buing some ruggedised 2.5" 80GB drives from Toshiba around two decades ago. Not only had they increased operating altitude limit up to 5 km (it's no fun if your drive crashes when you need to work at an observatory or such), but also an operating temperature from -30 to 80℃ or so.
The spec that stroke me as quite funny was the non‑operating altitude of 12 km – that's well over the *user's* operating altitude...
That upper altitude limit is nice, but what bit a couple of manufacturers seriously was the lower altitude limit. They specified altitudes between zero and a nice, impressive positive number. The rather important and security conscious client*) requested a negative number for the lower altitude number as they were located below sea level. Upon being told about the to them unexpected circumstances, the manufacturers faced the choice between retesting to the requested specifications or lose the sale. Most chose to retest and respecify.
*) Schiphol airport, the largest airport world wide (and one of very few at all) below sea level.
In a past life I did user support and in the winter months I can't tell you how many people would complain their laptops wouldn't boot...always the ones who left their machines in the car overnight !.15 mins at room temp sorted them out. I guess these days with laptops using SSD's instead of spinning rust that sort of problem has gone away.
"IMO consumer and enterprise grade is a legacy from the days of spinning rust (where it sometimes did make a difference). Vendors will of course apply that business model to SSDs because gullible customers expect to see it."
AFAIK enterprise SSDs have a larger percentage of "spare" capacity (10-15% ?) to automatically replace dead blocks compared to consumer SSDs, also typically higher performance, higher supported lifetime writes, and have capacitors to ensure data writes complete on power failure (*some* consumer SSDs have this).
"In a past life I did user support and in the winter months I can't tell you how many people would complain their laptops wouldn't boot...always the ones who left their machines in the car overnight"
Likewise, our field support manager would send an email out at the start of the "cold season" reminding us all to make sure any spares in the boot or collected from the couriers needed for that day were moved into the passenger cabin before setting off to the job. Mostly a reminder for the newbies or younger members of the team. Mostly because something stored just above freezing was likely to attract condensation when placed in a warm environment such as the inside of a PC.
I worked for a company some years ago whose "server room" like a lot, was basically a broom cupboard. Well maybe slightly bigger, say a stationary cupboard.
It had a useless, tiny, aircon unit and the servers ambient temperature averaged out at 85-90C over the winter I was there.
I tried to convince them to get proper cooling but it always fell on deaf ears.
From what I head, come the following summer things crashed on a regular basis and they *still* didn't want to spend on cooling the room.
When I was at Bigger Blue there was a rather foreboding elderly secretary who ran my Boss's department with an iron fist. One of her trophys was the only key[0] to the rather prominently labeled "Stationary Cabinet". We moved the thing across the room at least once a month when nobody was looking, and added "Not Very" above the original. The woman Was Not Amused ... but the Boss was.
Who needs keys when you have a couple lockpicks in your wallet?
> "say a stationary cupboard."
Yeah, I hate it when cupboards move about regardless of their size.
------------------------------------------------------------------------------------------
When you say mobile phone, why do you point at that blue box?
The Doctor: Because it's a surprisingly accurate description
That's quite normal for small businesses - stuff the noisy servers anywhere that gets them out the way. More than once I've had to explain that the nice cool space under the stairs (or other random cupboard) is only cool because it doesn't have a fan heater (a.k.a.server) sat in it with no ventilation - once you stick a server in it, it will be toasty warm until the server shuts down or just dies.
We had a nice office with a great view, but the only problem was this was over an outside corridor. When it was cold outside we got cold feet, so we brought in insulation, and put a carpet on top. ( We did wonder why lowly minions were given such a nice office - every one else knew about the office). We complained to site services who said - there was not a problem - and we were wusses.
A director took over our office (and the ones next to it). The first cold day she complained the floor was cold - and whoosh site services were round "oh yes there is a problem". They found that water from the rain pipe from the roof was leaking into the space under the floor. When it was cold - it froze - and the cold flowed up to the floor.
It took them a few days to fix the leak - and a few months to replace all the insulation along the corridor - having to work from the outside.
If they weren't needed on the weekend (i.e. not a 24/7 manufactory) why waste the electricity?
While as you rightly say, UN*Xy systems are capable of long-term uptime (had in the past a file-server running on FreeBSD with uptime in years), it's not _necessary_ that they are kept permanently up.
There's nothing in the story that said that was the case. In fact the opposite is implied, hence they were switched off at the weekend when no humans were working. Which is why poor "Mark" had to turn them on every Monday morning. You did read the article didn't you?
Also, I'm quite aware of typical CPU clockspeeds of that era, thank you very much.
The first office I worked in had a server room in which we had a VAX 780 with a pair of CDC washing-machine sized drives. Since our office was empty over the weekend and there was a youth centre next door, our fire certificate required us to switch off everything on Friday evening, and restart them on Monday.
It wasn't unusual for the systems to fall to boot on Monday until they'd been powered up and warm for an hour or so. Eventually DEC field service found a compromise alignment setting for the drives that could survive all the week's temperature range.
Back in the very late 20th century, I worked in an IT office on the sixth floor which was near the top of the main building. We would set the computers up with the software they needed and then cart them to their new location. In this case, the employee had a shiny new Viglen 286 to replace their old Amstrad PC - we set it up, moved it to the second floor and not long after we got a phone call to say that it had crashed. We went back down, reset the machine and poked at in to hope the problem went away. It didn't. After several crashes, we re-installed the old computer and took the new one back upstairs for testing.
It worked perfectly, of course. But we knew the user wasn't making this stuff up, so we left it on soak test for an entire weekend and it was fine. So, we swapped the computers back around again and of course the new one crashed again not long after, and we were swapping them back again.
The faulty computer sat in the sixth floor office and worked perfectly. The old computer on the second floor was working perfectly too, so we didn't think that there was something wrong with the second floor. This seemed frustratingly illogical.
Eventually the reason was found - because of the clapped-out nature of the building's heating system the sixth floor was slightly cooler than the second floor. Although it seemed unlikely that this temperature difference would cause a problem, we found that introducing just a faint waft of warm air from a fan heater was enough to make the PC crash. Cause identified, the machine was repaired and the employee got to enjoy their super-speedy 286 once more.
Now there's a name I haven't heard in years.
Back in the early 1990's we had one in my engineering department, and for reasons I've long forgotten, it was unable to recognise a replacement (probably larger) hard drive it needed. I recall contacting their support team (possibly by fax) asking for assistance, and to my amazement they said I needed a BIOS upgrade, and they would put the necessary chips in the post, free of charge! Yes kids, the BIOS was in a couple of EPROMs and couldn't be updated by mere software. Once installed, all was well and the new drive was installed with no further issues.
Because of this experience, I regularly recommended Viglen to anyone and everyone who needed a new PC.
I miss the old days; Real support by real humans. Progress, eh?
>the BIOS was in a couple of EPROMs and couldn't be updated by mere software
It made it so much harder for persistent virus authors:
1, Download naked_tennis_star.exe from dodgy link
2. Follow instructions to remove BIOS chip and peel off sticker covering window
3, Leave on window ledge in sun for 2-3 days to UV erase
4, Buy an EPROM programmer
5, Burn updated BIOS and repalce chip in motherboard
Oddly enough we had a UV eraser and EPROM programmer ( I still do ... ) but the BIOS chips in that PC didn't have a quartz window, but were simply plastic and therefore much cheaper (Technically they were PROMs, but they kept the EPROM part number which made life easier)
Of course back then it would have involved Zmodem / Kermit and a 14400 baud modem to connect to Viglen's own BBS to get the hex files, but that wasn't offered as an option at the time. People these days have no idea just how difficult stuff could be last century!
"Now there's a name I haven't heard in years."
I was going to say they still exist as a brand under the new owners, but looking at https://www.viglen.co.uk/ it seems the company which bought Viglen have abandoned the brand entirely now, since at least 2022 according to Nominet domain registration history. The Viglen logo looks suspiciously close to the old trademark too. Looks like it's owned by a German "domain operations" business, which looks like some sort or re-seller/parking operation. The content of the "Viglen" site is 4 years old and pretty much dead, so I guess it's just parked there by a the owners as squatters.
At my first job, the central heating was set to be significantly lower over the weekend so that when I turned on my Sirius micro (this was 1984) of a Monday morning in the winter months it would often whirr away before displaying some v weird characters and then not doing anything. Normally had to wait 15 minutes and hard reboot it when it would then work fine.
The radiators also had "tamper proof" controls to prevent us peasants from warming our bones excessively but we rigged up a complex multi elastic band system to bypass the lock without giving the game away (which we returned to normal on Friday afternoons). Even with this warming of the room, I still had to wait a quarter of an hour before the Sirius would get going but that meant I could go and make a nice brew instead.
Icon to warm me up after thinking of the above.
In the 1970s we had a PDP 11/40 that was turned on every morning, and it too would often not start in cold weather. We were told that this was because the 14" platters of the RK05F disk drives would contract until they were no longer within the tolerance for track positioning.
(The RK05F was a fixed version of the removable RK05 and was able to have twice the track density because it removed the manufacturing variability between disks.)
I used to work with a Sequent which had a LOT* of huge (4Gb I think) disks. Folk lore was that if they were shut down they wouldn't start up again. That was left running for a years. Eventually it had to be moved & they started up without problems.
* Part of the reason for the number was that they were mirrored at disk controller level and pairs of mirorred pairs were mirrored at database level. Whoever set it up wasn't taking chances.
Indeed... I worked at Sequent at that time and the issue was IBM disk drives which had a striction issue which meant the heads would freeze in place when cooled down.
The short term solution was the judicious use of a hammer in the right place for stubborn drives... the long term issue was to replace every version of these IBM drive ever installed by Sequent.
I had customers that had EMC frames with 4000 drives (striped and mirrored) supporting a single DSS database...talk about a nightmare.
Bluck
Not quite that old, but I had an Apricot 386 SX16 worked fine in the summer, but come the winter it often wouldn't boot in the morning but give a drive error, but if you left it running for a few minutes then rebooted once the drive had been spinning for a while it would then work ok.
We eventually worked out that the drive was too cold in the far corner of the living room unless the heating had been on for a while, so we moved it to a slightly warmer spot and it booted fine every time.
I later found out the drives that were fitted were IIRC RLL (or was it MFM?) and they did something along the lines of wrote some data on the platters during formatting and if the drive got significantly hotter or colder than the day it was formatted it would not be able to read the data, hence it was fine in the summer, fine when the heating was on abut not fine if it was in a cold room turned off over night.
It's been about 30 years since I used that machine, but still remember the "Oh" moment when I found out what the specific problem was.
The computer store I worked at back then had an ordinary building in front, attached to a mostly-empty, unheated warehouse we shared with a car-stereo sales/installation company.
Our build/test procedure included running the drives -- mostly Seagate ST-225s and ST-238Rs -- for 45 minutes in the heated building, before doing the low-level formatting, testing, and high-level (OS-level) formatting.
MFM or RLL drives?
I had a similar problem with an Apricot 386, but it wouldn't boot if the drive was cold (the drive spun up fine, just didn't read).
The fix was to either reboot after about 10 minutes of the drive running, or move the computer from the coldest corner of the living room to be nearer a heat source..
Many years ago, I worked in an ICL mainframe environment which was housed in a small bungalow (apropos of nothing but it once used to be a morgue). There were several air con units spread about the room and generally they managed to keep the temperature stable. However, about a dozen times a year, they would throw a shit fit overnight and when we came in the next day, opening the door to mainframe room was like opening an airplane door when landing in the Caribbean; we were hit with a blast of hot air that felt like it could remove eyebrows and exfoliate skin. A quick examination of the air con units revealed that one or more of them had frozen solid and now had large icicles (sometimes a foot long) blocking the air vents and zero air movement. The operators would quickly check to see what was working and what wasn't, throw open every door and window (it was an old building remember, and had "ye olde worlde" lead-paned windows) and then start the laborious job of chipping away at the ice with what ever came to hand - stanley knives, scissors, screwdrivers - until the mechanism could be freed up again and restore the air con units to some sort of working order. We also had to place buckets (always on standby) underneath to catch the drips from the ice-melt that we couldn't get to in the internals of the air con units.
This went on for a few years until someone at a senior enough level got bored of not having the IT systems available on Monday morning and decided it was time to upgrade the air con systems, so out went the ineffective peculiarly-brown units and in came some larger, more effective units. We missed those air con units ... for a brief moment once a month we imagined ourselves as Hillary and Tenzing, hacking away at ice on the Khumbu Icefall on their ascent of Everest. :-)
It's an interesting fact that many aircon systems can't cope with air that's too dry - such as in server rooms. This is is issue when people just buy an off the shelf system designed for comfort cooling in occupied spaces and use them for the server room, or re-purpose a small office as a server room.
The reason is down to the energy that needs to be extracted to condense water vapour. If the air is wet, much of the cooling energy goes into condensing water and the evaporator coil stays relatively warm. With dry air, there's a similar level of cooling ability, but now it's just cooling the air - so both the air coming off the coil, and the coil itself, are much colder. The result is often that the water that does condense will freeze on the coil before it can run off - with the result that the coil blocks up.
And worse, as the ice interrupts the airflow, that bit of the coil will get colder still and the ice will spread to adjacent fins. Eventually, all airflow stops, and the coil is encased in a block of very cold ice. It takes a lot fo energy to unfreeze it - just getting it up to freezing temp takes a lot of heat (which is hard to get into the ice), but then you need to supply the latent heat of fusion to melt the ice.
A decent control system will detect this problem and switch off the compressor while leaving the fans running - or even reverse the cycle to heat the coil that would normally be cooling. In smaller systems (like a freezer), there may be an electrical heater. I recall Land Rover Defenders of 80s/90s vintage with the option air con would freeze up in hot desert conditions - the fix being to turn them off for a few minutes every so often and let the airflow melt the ice out, but you have to do that before the ice completely blocks the airflow.
All this should be a "past problem" - these days all systems should have sensible control systems that will detect and fix any icing without the user noticing.
Worked at a place that had a very old server room with massive air-con units around the edge providing underfloor directed cooling for min-computers that were no longer there.
These were hellishly noisy and since we had to work in the room a lot more than the old mini-computer guys it was a PITA.
They finally started failing and were replaced with ceiling units which were a lot quieter but up to the job.
Anyway, came in on a Monday morning to find that half the servers had powered down due to heat, room was like an oven.
Turns out the power had blipped on the Sunday afternoon or evening, and while the UPS had kept the servers up the electrician's hadn't wired the aircon units so that they started up again automatically when power came back. Oops.
It would actually have been better if the power had stayed off for longer since everything would then have shut down gracefully and somebody on the server team would have been called out.
Funnily enough a few days later a temperature sensor was added feeding back to the security office.
Obviously the aircon setup was adjusted as well.
Before I entered recovery from IT, I would frequently get very early morning emergency calls, causing me to quietly scuttle from my bedroom to my living room, laptop in hand. My place is rather poorly insulated, so it was not uncommon for me to be able to see my breath, and I would huddle under a blanket while fixing whatever had most recently broken. I was latterly assigned a Dell Precision laptop with a Xeon processor and nVidia graphics card, and one of its chief virtues was to emit enough heat to substantially keep me warm during my troubleshooting sessions.
Back when I was in the power company biz, we had a customer who had his own hydroelectric plant, selling back to the local utility (us). One day, when I noticed that he was in, I stopped by just to say hello. And get a peak at the plant's inner workings. He was muttering darkly that yet another monitor and control PC (home/office grade) had its had drive fail yet again.
I observed that the interior of the building was reasonably but not uncomfortably warm with a couple of hundred kilowatts of generation running. The owner commented that with the system off on a winter day, it could get quite chilly inside. I also noted that the concrete floor had its fair share of puddles, probably resulting in a relative humidity of near 100%. I mentioned to him that the combination of temperature swings and high humidity were probably causing the hard drives to "breath" damp air through their pressure equalization ports (which he was unaware of), causing premature failures. I suggested that he either invest in an industrial grade PC, rated for that environment (expensive) or build an enclosure and condition its interior to isolate the electronics from the room full of leaky pipes.
I don't know what he did because politics and the local fisheries people eventually shut down his plant.
Steel will expand 1/1000*K. That one millimeter per K and meter. For a backplane, maybe 20cm at the longest side, room temperature to assumed -10°C for a not unusually cold winter morning we've got about 30K difference. 30*0.2m makes an interesting difference of 6mm. Should be enough to disconnect something that should remain connected.
I do recall the other extreme, FAT testing DCS systems in the factory which used to have an evaporative roof for cooling, but due to leaks it was reverted to a dry roof, felted and with sticky-up window lines, too. Recipe for Very Hot Factory in warm weather. We regularly had 50°C with the cabinets hotter than that. Surprised - we didn't get too many failures at all. Well soak tested once shipped!
In the early 2000s I was working in an oil and gas field in Kazakhstan. In winter, in the dark, the temperature can easily hit -40C (Kazakhstan has the largest summer/winter temperature swing of any country in the world. -40c in winter, +40 in summer).
Anyway, I worked on the RTUs and PLCs out on the well sites, these units were obviously in enclosures, but the cold got to them. The fibre modems would regularly stop working at night, and come back on in the mornings when a little sunlight hit the steel enclosures and warmed the air inside. The Kazakh techs I worked with solved the problem. They fitted 40w domestic incandescant light bulbs to the bottom of the enclosure. The bulbs generated enough heat when the enclosure was closed to keep things just warm enough.
However, if we had to power-down an RTU things got dicey. The RTUs were rack-based, with the CPU and IO being in the form of plug-in cards onto a backplane. I would have to remove the cards and either put them in the car, or down my jacket to keep them warm, otherwise they wouldn't restart at power-up. Once restarted, they generated just enough residual heat to keep themselves running.
Fun (but cold) times. I miss the place. Jacket ---> because it's cold out there!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
