Reply to post: Desperate times demand desperate measures

Heatwave shmeatwave: Brit IT departments cool their racks – explicit pics

Anonymous Coward
Anonymous Coward

Desperate times demand desperate measures

Where I work earlier this year, at the beginning of the heatwave, the machine room just below where I sit suffered a total air conditioning failure.

By the time someone followed up the alerts, the ambient temperature was heading towards 40oC. What it was inside the racks we don't know, but it would have been much higher.

As we had just recovered from a major storage failure a few months earlier, and did not want to have to do the same again, the managers put everyone on an emergency footing, but decided not to do an EPO, but wanted a rapid ordered power down.

Well, that made a very busy afternoon, shutting down 100+ servers and the associated comms. and storage gear in the correct sequence, and in order to try to slow the rise in temperature, every single desk fan was commandeered, as was the portable air-con unit that was keeping the coffee shop cool. All the doors and windows in the corridors around the machine room were wedged open, and security were told to ignore all of the warnings they were getting. The managers themselves manned the doors and open windows to make sure that nobody tried to get in, while we did the technical work.

Fortunately, most of the systems in the room were for test and development, so the external facing systems were not affected.

Never seen the managers, project managers and PMAs do so much physical work!

It must have worked, because we did not suffer any more than the normal number of failed disks following a power down, and in general, it all came back cleanly once the air-con. was fixed. Took a lot longer to bring up than it did to shut down, though.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon