"all self-inflicted therefore warranty on our beautiful $15m system was void"
What did you expect?
icon => Lady in Schweppes ad
Here is the latest festive instalment of On-Call, the place to be for techies with tales to tell. It might be cold outside but this may just make the blood boil as it goes to show what happens when bean-counters take over the asylum. Meet P, who told El Reg about troubles at a brand spanking new data centre that he managed in …
This post has been deleted by its author
> 336 calories to melt 1 gram of ice at 0deg C
(actually 334 joules/79.5 calories)
79.5 calories later, you have water at 0C
> 1 calorie to raise 1 gram of water by 1 degree C
(which is more or less right)
what happens when you put in 334 joules/79.5 calories to 1g of water at 0C ?
As for BTU, the only time you see that used in the UK is in adverts derived from US equipment. All our suppliers use kW (latent and sensible)
Be careful when talking about calories.
There are "physics" calories, defined as 1 calorie to raise 1 gram of water by 1 degC.
There are "diet" calories, defined as 1000 "physics" calories. Usually on food packaging they are annotated as kcal, but they can also be annotated as Calories. kCal, however is just wrong.
Stick with kW. That way nobody knows what you are talking about, but everyone knows that they don't know. If you talk about calories, everyone knows that they *do* know, and *all* of them are wrong.
"Be careful when talking about calories."
A friend of a friend, a physicist, looked at the number of Calories stated on a beer bottle label. Did a quick calory -> joule conversion, decided the amount of energy you spend repeatedly lifting the glass up to your mouth offsets the energy contained in the beer. Enjoyed a few pints. Then it was pointed out to him that the Calories on the label were actually kilocalories, so his calculations were off by three orders of magnitude :)
Stick with kW. That way nobody knows what you are talking about, but everyone knows that they don't know.Actually, it's dead easy. A kilowatt of cooling will exactly undo a kilowatt of heating, by definition. And a person emits heat energy at about 100W (= about 2000 kCal per day, = about 8.4 MJ per 86400 sec, which can be safely presumed to end up as heat.)
The thing about energy in physics, and why physicists get so passionate about it, is that it's the same stuff. And SI, by avoiding proliferation of measuring units, makes this about as clear as it can be made. You might not see an obvious connection between, say, pounds-feet of gravitational potential energy and BTUs of heat; but if you do the dimensional analysis which is baked right into SI, kg.m2.s-2 are unmistakably Joules, even if that doesn't get you very much closer to designing an apparatus to convert gravitational potential energy to heat. (Maybe something like a torque converter with the output shaft held in a stall, to exploit friction between fluid particles flowing past one another?)
And I still maintain that the continued pretence that there are different units in which different forms of energy should be measured is an obfuscatory tactic.
"As for BTU, the only time you see that used in the UK is in adverts derived from US equipment. All our suppliers use kW (latent and sensible)"
Nope, UK radiators are still specced in BTU as well as those funny new metric units for those who like to drive on the wrong side of the road.
Bought my imperial Mears calculator yonks ago - why waste money - just measure up the rooms in feet. Made sure my laser measure worked in both systems. Horses for courses, which of course are in furlongs, but my job does not involve the gee gees.
>> We need a horse sh*t icon.
In the real (UK version) world we mix units like anything. We buy petrol in litres, but still measure speed and fuel consumption in miles/ per gallon/hour. We weigh recipes in g and Kg units, but weigh ourselves in Lbs and Stones. We measure temperature in Celsius even if we call it Centigrade. But we at least think of beer or milk in pints. It's partly I'd guess because metric units, while technically precise and decimal, are some how not on a human scale.A litre is either too big or too small. A centimetre is not big enough to be useful, but a metre is too big, so we end up with annoying, fiddly amounts. A Kilometre is Ok for longer distances, but somehow a mile fits the significance better. (And a metre is just a yard with with some avec on it.)
Well this ex-pat was recently put in the position of explaining why the Slade original of Merry Christmas Everybody has the colonially confusing and dubious lyric "Does a ton upon his sleigh" after I noted with disgust the inaccuracy of an American cover version on the wireless that opined that Santa "Has a ton upon his sleigh".
And I should add that before any revisionist foetuses wade in, the term is "Do a ton" and *NOT* "Do a tonne". The term comes from pre-frenchified-UK biker slang in parlance when I were a lad and the Vincent Black Shadow were King o' road and people weren't afraid of things that came in fractions other than tenths and tenths of tenths and would tell the French, Belgians and Germans to bugger off if they told you to buy petrol in litres.
Few vehicles of the day were capable of "tonning up" and all were completely incapable of stopping in a safe distance when they hit the magic 100 mph* should danger and excitement rear their heads, which was kind of the point.
"Sensible countries". The recent round of voting should have put paid to that conceit on all fronts. Pfft!
*That's miles per hour an' orl, not namby-pamby frenchy-metres.
Unfortunately, in the UK they only teach the metric system. So when the poor kids get slung out into the real world and hopefully get JOBS in that real world they find themselves confronted with lots of old equipment built to Imperial standards (not to mention the bastardised Imperial units used on the other side of the pond) and they haven't got a clue.
Not to mention crusty old sods like me who were dragged up in the good old days and still THINK in imperial units. I was taught in the cgs / mks era when SI was just coming in. Made physics FUN.
I had an assistant at one time, and we were measuring up an area. She was holding the other end of the tape and I was singing out the cutting lengths. She was quite confused at the seemingly random figures until I explained that some of the figures were in inches and others were in metres. It was obvious which were which as I was in plain view.
Yes, I am a bastard. I should have been a rocket scientist.
We should at at least make 'em aware of the past, and warm 'em that these units are still in common use, with enough knowledge to cope.
"Not to mention crusty old sods like me who were dragged up in the good old days and still THINK in imperial units. I was taught in the cgs / mks era when SI was just coming in. Made physics FUN."
Same here. Started with imperial measures at infant school and started the switch to metric in junior school and more or less fully transitioned to SI in secondary school. I can visualise in either system when given the numbers but still have problems estimating lengths in either! I'm more likely to revert to feet and yards when estimating and then converting to metres.
After World War II, NASA commandeered the scientists and engineers who had been working on ballistic missiles for the German army to work on their space programme. Being German, they had done all their original calculations in SI units, before their hosts converted them back to US units.
The last step turns out to have been a mistake. Since all the old European scientists are long dead and gone, NASA's own graduates have been trying to do the calculations in US units throughout. With a level of success that you can witness for yourself .....
To get something weighing one kilogram to speed up by one metre per second every second, you need to exert a force of one Newton on it. Which part of that is so complicated?
"Surely when they teach science in school they teach using SI units in this day and age? Or do they still do everything in feet, pounds etc?"
In the US - yes, both. Gravity is 9.8 m/s^2 or 32.2 ft/s^2. Road distances are in miles. Doctor's office weighs in pounds and ounces, and measures height in feet and inches. Yet most physics classes are taught (primarily) in SI units. The common joke here is that only pharmaceutical companies and drug dealers use grams - the rest of the country uses pounds and ounces.
"...an anachronism from the colonies."
That probably is from here, but we haven't been colonies for over 200 years. "The States" would be an acceptable replacement.
Here in Blighty there is also the BTU (British Thermal Unit) which, in typically Imperially-measured style, equates to the energy needed to heat/cool 1 pound of water by 1 degree Fahrenheit at sea level.
So when you see heating and air-con units advertised here, it's not unusual to see a rating in BTU/hr
Since I had to enter a server room where temp got well above 40°C after an AC failure, I set my servers and other devices to shut down automatically in such situations - but I was never able to get the budget to install room temp sensors and the required system to act automatically on everything in the room.
Early in my working life I worked in a purpose built school ( I'm guessing 70s) that was all beautiful and bright with walls constructed of big glass windows. It was also orientated East-West so that one side was facing South. And there was no air flow across the building. From mid-March until October all the rooms on the side facing South were furnaces if there was a hint of sunshine. Some days the thermometer wasn't able to show the temperature on the tables near the window. (Think car parked facing into the sun).
I've had a deep mistrust of architects, even deeper than my loathing for beancounters, ever since.
One or two minutes? Lucky guy!
I once had to troubleshoot and fix an industrial PC located in a ***working*** production line in a fish canning factory. The thermometer marked 38ºC and the humidity was >95%. To make things worse, the device was placed into a massive stainless steel box that held lots of other electronics stuff.
Cue an hour or so of contortions in hellish conditions including also very high noise levels, stopping every few minutes to wipe the sweat from my eyes or going out to cool myself down.
In those conditions you feel as if your IQ has been lowered 30 points!
Afterwards, I told the management that I'd never work in those conditions again. They pointed out that the workers in the line worked in those conditions in 4 hours turns and I pointed out that said workers were walking over a metal grid that provided them with a steady flow of dry, cool air; otherwise, they'd be suffering half a dozen cardiac arrests per day!.
They didn't call me again, and good riddance! 8^)
Done that. Prior to my IT and engineering days I spent a year working moulding presses. In their wisdom they placed some presses in the side room that had been attached to the side of the existing building. To cut costs they built it with a tin roof. Temperatures in summer regularly hit 40 degrees due to the combined effects of ambient outside temperatures, solar radiation and the fact that the presses were kept heated at 200 degrees C. And one single air-con unit at the far end that had the only use of blowing straight onto the moulding plates of the press at that end causing most of the devices made on that press to fail QA for moulding issues (with the plates being several degrees cooler than they should be the plastic would stick to the plate, causing it to break within the mould and remove large chunks from the devices.)
Indeed. Nothing will match the year or two I and some colleagues spent in a windowless, flat-roof, single layer of block office block room (it was a single room) in India. In the summer, pre-monsoon, it was nearly 50 degrees outside, and more in the room. We took turns drying ourselves in front of the one working air-con unit, spraying our humidity onto each other.
It was freezing in the winter too!
"I'd hate to have had to work in that kind of heat...."
A long time ago as an apprentice, I fairly regularly ended up in house ceiling spaces running cables (the "boy" always does this stuff, mainly because he can fit through the manhole that the average electrician's gut precludes).
Steel roofing, summer days and insulation below usually meant the ambient temperature was in excess of 50C + a blasting space heater above you. Being fibreglass wool insulation for the most part, stripping down to shorts and tshirt was not advisable, it was bad enough getting it between skin and coveralls. In low pitched roofs there was often only 2 feet of space to work in and touching the roof on the inside, even through the waterproofing, meant blisters.
fibreglass wool insulation
I was a Part-P certified self-employed electrician for a while, and one of the reasons I got out of the business (other than the lure of a regular paycheck from a regular employer) was attic work. Freezing in winter, sauna in summer and particularly the blasted glass wool insulation.
If we need a wool material to insulate our new house, we are definitely using actual sheep wool.
What a delicate flower you are - there are large parts of the world where it is consistently that temp or substantially higher for major parts of the year.
It's called adaptation it takes a while. You don't just step into somewhere that's twenty odd degrees hotter than what your body thinks is a really sunny day and instantly adapt, the body registers only a couple of degrees as a temp change, It's a process that takes months. I currently think 18C is a bit nippy, last year in the UK I would have been sleeping with the window slightly open all the way down to near 0C with just a light duvet.
Also those people in hot climates. If it's humid especially they suffer as well, there's a reason people in hot countries often take siestas. Real feel this year was hitting 48C in the cities there were plenty of locals who suffered, not as badly as me, but enough that there was articles in the paper about how deadly the heat could be.
"I've been in our conservatory when it's been over 40 degrees. I was lucky enough to not have to spend more than a minute or two there, getting some doors open, I'd hate to have had to work in that kind of heat..."
I've had to go into a hotroom at 50 degrees dry heat to do calibration measurements, never had to be in it for more that several minutes, though. The effect is interesting. A wall of hot air hits you as you enter. After a few tens of seconds all your pores suddenly explode open, and you are bathed in sweat. You're fine then.
50 deg C wet heat on the other hand ... Marconi Comms Chelmsford had an environmental chamber you could get a small house in. ISTR it could go from at least -30 to +50 deg C and up to approaching 100 per cent humidity. Bloody great big red button inside with klaxon and a flashing beacon on the top. I was told that the procedure for that one was to change into swimming trunks before entering, colleague outside watching you continuously through the viewing window in case you passed out. Nasty.
Did the same, but I managed to sneak network cards into the UPSs. Meant I could monitor the power to see if it went down (and if it stayed off for 3 mins I could suspend all VMs and power down), and also temperature (same thing if the temperature spiked).
This was after the AC failed in that teeny room, and the Proliants ended up rebooting constantly overnight one night. They were so hot that you couldn't hold onto them, but bless them they were trying.
Only lost one hard disc. The rest recovered with no apparent ill-effects (although I suspect that lifespans were shortened).
That sounds familiar. Data centre in Victoria, Aus?
Nope. Teeny one-rack cupboard-with-a-window in Edinburgh. Come to think of it, it might have been over the weekend. I had MRTG / Cacti monitoring the servers, and I think the serge started on Saturday night.
Ended up running with the door closed and window open during the day, and door open, window closed at night until I could convince management to replace the (stretched) portable aircon with a decent fixed unit.
I was called out one Monday morning to reports of PCs losing LAN connections at random. Sounded like a failing switch, commonly sorted for a while by rebooting it.
Collected the server room key from security and headed to the 5th floor.
As I approached the server room, I could feel the heat. Both aircon units had failed over the weekend. The temperature was over 50C.
I eventually got the fancy key needed to open the windows. The concrete building pillar was HOT.
Restarted the switch and all was well. The ProLiant servers never missed a beat, though their monitoring logs showed the temperature spike in detail. Aircon 1 died about 8pm on Friday, with aircon 2 deciding to follow it 7 hours later.
In any system I saw with this capability embedded, I've never seen one with it enabled by default. It may be in some situations you could prefer it to stay on until it melts or disks are erased - but I guess in many situations when it gets really hot you prefer on orderly shutdown, or even an immediate turn off.
Now I have a warning when temp gets above a threshold, and a shutdown if it gets hellish enough. Some less critical machine may shutdown earlier, the real critical one a little later - but all before they could get damaged.
I've seen very few admins turn on these protections - it looks only overclockers monitor their CPUs and GPUs...
"I've seen very few admins turn on these protections"
Which is a shame, because mrtg monitoring of system temperature can frequently show problems starting to happen before they knock things over (eg: fan failures, where the fans aren't monitorable).
In a pinch I've resorted to using HDD SMART temperature reporting.
One night, I got a call from the Operators to say the HP PA-RISC servers I looked after were reporting > 30C temperatures. From the pattern of reporting and my intimate knowledge of the layout of the machine hall, I could tell exactly which AC unit had failed, overloading the others - I despatched one operator to the spot, another started phoning the other sysadmins, and I started shutting down my systems* cleanly before they hit 35C and shut down hard.
The Suns and IBM systems weren't complaining. If I recall correctly, the Suns fell over at about 40C, the IBM systems just kept running.
* Yeah, I got a bollocking for stopping Production systems. But they restarted OK, which is more than could be said for the Suns.
"It's not that expensive."
No, but it might be politically impossible.
A/C failure, equipment room 35+ degrees - detected because the noise of the CPU fans desperately trying to cool the gear down was audible from outside. Afterwards I checked, each rack had an APC environmental monitoring unit, wired to ... nothing.
"WTF?" sez I to the Operations manager.
"Simples", he replies, "HVAC & environment belongs to the Facilities Management guys, and they refuse to accept any inputs / connections from the ICT group. Seems like they don't actually monitor their own gear though. Pity."
"So why not connect the rack sensors into your all-singing, all-dancing SNMP monitor system?"
"Oh no, I'm not allowed to connect HVAC / environment sensors into my network, they belong to FM"
AC to protect the guilty.
[actually, that ops guy has been moved sideways, so maybe I should have a quiet word with his replacement ...]
>I was never able to get the budget to install room temp sensors
Same here, but I was able to find the OID for the Air Inlet Sensors on my servers and I monitor those.
Years ago we had a "comfort cooling" system in a server room go wrong and piss water all over the floor. The temperature then rocketed and the server room turned into a sauna. It may surprise you to learn that servers don't like saunas...
( I never did get the building manager to explain the difference between aircon and comfort cooling... )
The server "cupboard" I inherited at the current employers had two heat pump units instead of proper air con. Instead of placing the condensers outside they were mounted just the other side of the wall in the factory/warehouse right underneath the metal roof. Now what could possibly go wrong?
In summer they struggled to cool things sufficiently, bearing in mind they were being fed with hot air to start with, in winter they shut down as apparently they weren't rated to run in sub zero outside temperatures. They constantly dripped water (fortunately they weren't right over anything too critical) and eventually iced up completely and destroyed themselves.
I specced the replacement, 7.5kVA of proper cooling power is a bit excessive but I know it will be reliable.
Reminds me of the old story of the prat who stacked up a, er, stack of Marshall amps. The bottom one drew in cool air and vented out warm air. The second drew in warm air and vented hot, the third drew in hot air and vented scorching air, and the last one, well, it kind of burst into flames.
This post has been deleted by its author
In a previous company, one day I was shown the new server room. The racks were aligned in neat lines, just all in the same front-rear -> front-rear -> front-rear configuration.
I asked them if they never heard at least of cold/hot aisles configurations for server rooms - and their setup would have just fed hot air to many machines...
But they didn't listen to me until the sysadmin installed some machines.... and he had to work in the hot airflow...
After the last 10 years, there are many USAians who would not object very strongly if Texas acted upon its frequent threats to secede.
// it would mess up the nice star field in the flag, though
// perhaps we could get Puerto Rico to fill in the gap?
// ...not likely after the way we've treated them since the hurricane
This post has been deleted by its author
"Used to be a part of Mexico till the US nicked it in a war (that started in a dispute about who owned Texas)
Net result was Mexico lost nearly half its land area to the US, - hence New Mexico as a US state name."
It's a bit more complicated than that. Mostly, though, what happened was that Mexico had the biggest, best equipped, and best trained army on the North American continent. They also had Lopez de Santa Anna, eleven times president of Mexico, a military genius in his own mind.
Firstly, Santa Anna was responsible for there being a Texas to be in dispute. He let gringos (and their slaves) into what was then considered useless land, fit only for scorpions and Comanches (and he didn't think there was much difference between the two). This was despite the fact that officially slavery was illegal in Mexico. When he woke up to the fact that Texas wasn't as useless as he'd thought it was, he made a big deal out of the slaves and went to show the gringos who was boss. He showed them. Hint: not him. After Sam Houston kicked his ass, Texas was nominally independent, but Houston wanted to make it part of the US. Santa Anna objected. Santa Anna also had a difference of opinion on exactly where the border between Texas and Mexico was. Mostly, Santa Anna opposed whatever Houston wanted. Houston wanted Texas to be part of the US, so Santa Anna was against that. The Mexican-American War was basically fought because Santa Anna had a grudge against Houston.
As Santa Anna was, he thought, a military genius (despite having been beaten by gringo militia) he put a large chunk of the Mexican army up on the border. The US sent troops who danced around Texas, California, Arizona, and New Mexico. They also sent an amphibious force south. Santa Anna discounted them, thinking that the mosquitoes would get them. Unfortunately for him, Winfield Scott, who _was_ a military genius, had heard what had happened to the French in Haiti (10,000 troops, including the commanding general, killed by yellow fever) and landed during the cold season when there weren't many mosquitoes and got the hell out of the lowlands as fast as he could. Most of the Mexican army was in the wrong place to stop Scott, so he advanced up the mountains to Cuidad Mexico. The Marines went 'to the halls of Montezuma' when they took Chapultepec Castle, above the city; Chapultepec's last defenders were Los Ninos Heroes, five military cadets and one of their instructors, Chapultepec being then the Mexican Army's military academy. Five were killed in action, the last wrapped the Mexican flag around his body and jumped off the top of the castle rather than surrender and let the gringos get the flag. That was pretty much it for the war. Some of the gringos wanted to just annex the whole of Mexico, but that was a non-starter. Too many brown, Catholic, non-slave-holders. Instead the gringos took the northern chunk. The Mexicans weren't that sad to see it go; this way the gringos got to deal with the Comanches and the Apaches and the Utes and the rest. And the scorpions. And good riddance to the lot of them. The southern gringos wanted a nice strip of land they could turn into slave states. The northern gringos wanted a nice strip of land they could turn into free states. Neither group considered that what they got was a lot of sand, a lot of hostile Indians (Indians didn't come any more hostile than Comanches and Apaches) and a lot of scorpions.
You'll notice that no-one asked the Comanches etc who they'd rather have running their land. The US Army spent most of the time between the official end of the Mexican war and the start of the US Civil War chasing Comanches and Apaches. Bobby Lee was in Texas, hunting Comanches, until just prior to the Civil War. After the Civil War, the US Army went back to hunting Comanches etc. until the Spanish American War, and then back to Apache-hunting and to the Philippines and moro-hunting. The last of the Indian Wars in the territory liberated from Santa Anna wasn't over until 1901, Apaches being very, very, VERY hostile. Cheyenne and Sioux might or might not have considered that it was always a good day to die; they got stepped on by the end of the 1880s. Apache thought that it was always a good day to kill gringos. Still do. They ain't particularly fond of Mexicans, either.
> Apache thought that it was always a good day to kill gringos
Patton said “No dumb bastard ever won a war by going out and dying for his country. He won it by making some other dumb bastard die for his country.”
Geronimo agreed and did a damn good job of it. He just couldn't kill 'em all.
BTW - If you need to quickly reduce the heat of a scorching machine room armed with nothing more than natural ventilation and all the desk fans you can lay your hands on, you get much quicker results if you set everything up to blow all the hot air out, rather than by trying to force lots of lovely cold air in.
I learned this the hard way, so you don't have to.
Had an issue with humidifiers many years ago myself. We rocked up at work one day (this was at a local council) to find the server room covered in puddles of water. It was dripping through ceiling tiles, all the vents, everything. Obviously the first thing we did was to immediately shut everything down and kill the power to the room to protect the servers and reduce the risk of electric shock.
After the maintenance guys had taken a look, we found out the problem. Turned out that the previous day, a new humidity control system had been installed for the building's central ventilation system. The job however had only been half done and the control unit for the humidity regulator wasn't set up properly. They'd then left the system pumping out maximum humidity overnight. Unfortunately, the main plant room was adjacent to our server room, and the first place this nicely humid air went was through the ducts/vents above the server room.
Of course, the server room is a cool room due to the air-con, so the moisture in the air immediately condensed and dripped out all over the place. My colleague's LCD screen had over a litre of water in it when we tipped it upside-down.
The solution was to install the proper control unit for the humidity system, and re-direct the vents so that the "central air" system no longer pumped through the vents above the server room. Amazingly we lost no servers due to this. The only casualty was a desktop PC that ended up with blue/green rust on the motherboard. Even the previously mentioned LCD screen worked fine once we took it apart, dried it out thoroughly and then re-assembled it.
Should add, I've also lost air-con to a server room at a later job due to the weather. This time I was working at a University and was in charge of a small server room for one of the departments. During winter, I came in one morning to find the room like a sauna and the air-con system pumping out warm air.
The reason was that the external air-con units had been mounted on the ground outside (this was a first-floor room) as it was cheaper than running chiller pipes up to the roof. The building had a sloped roof, and we'd had some snow a few days before. As the snow began to melt, a huge pile slid off the roof and buried the external units which promptly tripped out due to the sudden lack of airflow. And just like that, every air-con unit stopped and up rocketed the temps!
I worked on military operations computers for a certain air force. Specifying an (at the time) unusual system of dual-resilient mini-computers, running master and tracker, with all clients and the high speed link between the two minis running on fibre optics meant that if one of the computers (always located in a hardened bunker) was lost, the other would instantly takeover, and the only interruption the users saw was a forcible log out. Installing a fairly thermally inefficient minicomputer was fine as these were going into hardened operations centres that were manned for war operations, and therefore human habitable (ie pressure ventilated, air conditioned). All was well until installing at one air base, where the alternate operations centre was a grotty little hole in the ground, without even an emergency escape, and didn't have room for a minicomputer, the comms racks and the multiple dustbin lid Winchester drives, so somebody elected to put all that in a spare hardened air shelter. That didn't have air conditioning, but was a large space able to take a fast jet, and somebody assumed that the shelter would just dissipate the heat Obviously, they didn't stop to consider that the accesses to this building were all highly sealed against NBC threats as well as conventional blast damage. You know where this is heading. After a few days of unmanned operations, the tracker system went down, and when the sysadmin went to find out why, he entered the building to find ambient temperatures of around 45C, and the minicomputer looking rather like it had been designed by Salvador Dali. Back some three decades, the half million quid of this little episode was serious money, and you can imagine the blame game between the multiple contractors and the operator. In the end, that taxpayer forked out, because that's what taxpayers are for.
A repeat, but the beanies cheaped out even further...
I was in a new "satellite" branch of a company, and someone had the bright idea of sticking our VAXcluster in a cleaning cupboard behind Reception. A sparky was duly dispatched to install "air conditioning", which turned out to be a toilet extractor fan jerry-rigged with a home-brew temperature sensor.
Unfortunately, it was all set up to keep the power ON until a certain temperature was exceeded, rather than kicking in when things got too warm.
A hot Easter break later, we discovered this the hard way, The enitre cluster was roasted, but recovered once allowed to power off and cool down.
Did I mention this was an electronics company that sold to a number of safety-critical industries ?
Some electronics engineers shouldn't be allowed anywhere near electrics. When we moved into our house the kitchen had been re-wired by the previous owner, an electronics engineer. It was a death trap waiting for an accident. 5A under cupboard lighting was wired to an open connection box situated directly under the taps of the sink. From there to the light switch. From the light switch back to another box behind the cooker. From there wired direct to the cooker main. Yes, 5A lights connected to a 30A fuse. Needless to say those lights didn't work for very long after I discovered what he'd done.
Yeah no. One of those 46' days in Melbourne was when the glue holding my shoes soles on melted and they dropped off. Needless to say this was the day before I was due to fly back to cooler climes and I was already deep in the middle of a project and had no time to source new shoes. I am sure the customs/immigration people widdled themselves laughing at my shoes held together with duct tape.
The joys of ad-hoc server rooms. The male toilet block was cut in half, and one side repurposed as the machine room. Air-con was a wall mounted effort more commonly found in domestic siutations. It was positioned above shelving that contained all the mag tape backups for an imaging system and the tapes we received from various organisations such as NATO and the EU. One weekend the air-con went berserk and poured water all over the tapes. We didn't manage to dry them out and it took months to get a full set of backups again as dumping data from the imaging system was extremely sssslllooowwww.
At a well known managed data center 2 or 3 years ago, some genius decided that because it was a hot aisle it did not matter that 14+ blade servers on each side were going to produce a lot of heat.
It was incredibly noisy due to what i suspected were failing fan bearings all round the place, every surface was hot, all the 16amp c19 connector power cables that normally are stiff, were like wet noodles and the kit I was asked to have a look at because it was sending home over-temp alerts had a temp of 44c at the back, but because the hot air was spilling out over the top of the cabinets, it was 28c at the front, unfortunately for the customer our specs say max operating temp of 35 c.
Cue me telling the dcm to sort that out, any parts replacements will be on T&M until it gets fixed.
Note that we used that very hot aisle to pre-warm a bunch of cables to make them easier to install
Worked at one place that had a steam trap above a cabinet full of industrial controls. (Trap was in the ceiling, where it was nearly impossible to get to.) Something was wrong with the trap that caused it to dump water about every 6 months - triggering replacement of archaic controllers and computers below. We're talking Pentium II for the interface, and a PLC that was configured using DOS-only software - in 2006. In checking for damage after one particular flood, I pulled out a functioning controller just to look. I swear it had a little tuft of moss growing on the circuit board, from having gotten wet so many times over the years.
So I think what the story tries to say is that the steam produced by the humidification equipment(I guess they were using kettles but maybe some ultrasonic system) somehow produced so much steam / water that it froze and sealed up the pipe.
This makes zero sense. Firstly you do your humidification after you control your temperature.
That makes sense when you consider that what you are trying to control is rH (ie humidity relative to temperature)!
If for some bizarre reason you did your humidity control before your temp control then it still wouldn't cause a flood because you don't need a ton of steam to get +-50rH at 0C (the air inlet temperature).
In my humble opinion.. Facilities engineers blame bad design when proper maintenance / operations (eg finding the leak into the humidity control / responding to an increase in pressure alarm in the HVAC) is actually the solution.
I remember being called to a data centre in Scotland, for a large bank based there (hint), to investigate why the AAA system wasn't working reliably. Walked into their DC and it was so hot I had to take my jacket and tie off just to walk around... kind of obvious what the problem was. This state of affairs had also been created by the "cost savings".
Years ago while working in Network Ops we had a member of the support team who would turn off the air con in the server room because he was "too cold". The fact he didn't need to be in there because we had remote access didn't seem to occur, the third time I got called out at 5am to a steaming pile of servers and a knackered 2 grand tape drive I confiscated the remotes. Didn't work, they got steps and used the power buttons. So I had those disabled. So they flipped the breakers.
It was at this point I ascertained they were doing it deliberately, and when nothing was done about that I left them to it...
Years ago i worked for a major video organisation at one of their mpeg concentration and fibre distribution centres, originally designed as a remote site and manned after some high budget companies had come onboard with nasty potential service outage "fines". Most of the telemetry was still fed back to their HQ and it was they that raised alarms that we had to stay on the phone to check cleared.
At the start of a night you would still do a tour and check Ups status, fault change overs temperatures from the front panels even if nothing flagged in the log, or usually verbal handover from responsible colleagues and call in anyway to see what they saw.
There was a room where all the encoders and kit for the fibres was, with just a double door to the main area, I walked in and it was toasting, I cracked the doors, went for the thermometer probe and The BIG fan and as others have said got the hot air out, hung between bays by the door using patch cables, ( not elegant) and connected to the unreg mains. I had 35C in some places until it started working.
The cowboy who had already gone when I arrived 2 mins early had thought it felt warm and "turned the aircon up" every one of the units was set at it`s minimum 16 despite a big dymo tape by the control panel saying "Do not set below18C minimum". They had carefully sourced multiple units from the same manufacturer, so no single point of failure, but if the humidity and outside temp got into certain regions they would Ice up and go into a defrost cycle. At 20 or 18 it was random so 1 or 2 out of the units in the arrays would do this, at 16 they all frosted up more or less together, and 15-30 mins no aircon. Nothing died although alarms aplenty, so the favoured idiot didn`t get sacked. I voted with my feet after this.
There was a similar incident here in sunny Adelaide, Australia not long back. A popular co-lo data centre here in the city had an air conditioning failure that resulted in machine hall temperatures soaring up to 50+ degrees (or so I was told). The result was massive disruption to government and private enterprise all over the state. Many millions of dollars of warranties popped. The root cause? Rumour has it that the electrical safeties tripped, and the people supposed to be on call didn’t respond due to a dispute.
This is all hearsay, though. Thankfully I was not there.
Business schools indoctrinate their students to believe that when they get their degree, they can operate any business better than any mere technician. What they need to be taught is that there is never a perfect decision and if they are getting feedback from an IT department or the plant manager they need to understand that those people have also spent a considerable amount of time learning their craft and are in a better position to make valid recommendations.
In an age of "yes" men, there is a big lack in cynical bastards like me that are always trying to poke holes in a system or product. "What happen if this goes down?" What do we do if we run out of bolts on the production line? Whose authorized to book a charter flight if we need to get a flight crew to another city right now?
In this case it starts looking like some putz saved £30k-£40K in HVAC and might have cost the company several hundred thousand plus in productivity and £15m in computers. The overtime in the IT department to get everything back up will only be an incidental expense. Anything mission critical needs a back up. I've seen a data center go down when a lorry didn't see a compressor on the side of the building when backing up and it took several days to replace the bollards, the compressor and recharge the whole system with Freon.
I do DC design, audit & troubleshooting for Dell across EMEA. We regularly get asked to convert Watts or KiloWatts to BTu and not just UK customers either. Tons of cooling is still used occasionally in the US but not by us.
Generally for cooling it is Kilowatts, although Temps are quoted in Deg F & Deg C. Airflow is interesting I see Cubic Feet per Minute, Cubic Metres per Hour, Litres per second & other variations on a them.
Luckily I'm old enough to have been taught Imperial & the variations on Metric so can normally sort it out enough to get an answer for the customer.
I always quote Murphy's Law. Units will always be expressed in the least usable form. For example "speed will be expressed as Furlongs per fortnight!"
Biting the hand that feeds IT © 1998–2021