Police ignored the laws of datacenter climate control With Friday upon us once again, The Register finds itself glancing at weather forecasts as we prepare another instalment of On Call, the weekly tale of tech support in which we share readers' stories of being asked to fease the infeasible. This week, meet a reader we'll Regomize as "Sylvester" who, in the early 2000s, found …
A company I worked at had an old terminal (green text type) connected to a digitising table. The terminal needed degaussing quite often (with a button on the side) to clear the text which was semi persistent as did the digitising table with a large magnetic sweeper.
Guess where they stored the back-up tapes from the MicroVAX that ran the office network?
This post has been deleted by its author
This post has been deleted by its author
Back in the day I definitely worked with monitors with degauss functions (not sure if it was a button though). A quick google brings up lots of examples of degauss functions on monitors.
Something related that I was thinking about recently, it used to be that speakers used for the center speaker of a surround system were magnetically shielded, to stop them affecting the CRT televisions, presumably that isn't needed anymore with LED/plasma sets.
This post has been deleted by its author
I'm am old fart, not a single monochrome CRT I've ever seen in anything (I've seen metric fucktons of terminals, TVs and monitors because I repaired them for a living)
Yes, wonderful story Grandpa. The only problem is that the only person who has mentioned monochrome monitors is you. The OP said that it had green text. Have you never come across a monitor with green text that was not a monochrome monitor, because I'm a spring chicken of only 44 years, and I've seen several. One old CGA (perhaps EGA) monitor I used even had a slider so you could switch it between the full 16 colours, or green text, white text and amber text, as you desired. The wonders of modern technology!
I liked green text on the terminal, but for Sopwith I much preferred amber.
> Yes, wonderful story Grandpa.
> Have you never come across a monitor with green text that was not a monochrome monitor?
Unfortunately for you, rude little whippersnapper, the device has been identified as a Tektronix storage tube, which is very much a monochrome display.
And a very, very expensive one, so it is quite reasonable that someone working for decades on TVs and monitors never saw one - which is why the self-confessed old fart quite reasonably asked for the make and model (i.e. still willing to learn).
Looked it up on Google, that looks just like it! Even had that pocket on the side with a manual in that some of the photos show.
Cheers, nice to know I wasn't imagining it as some thought.
Yep the degauss was quite impressive, used to scare the crap out of me when I first started using it.
Reading through the comments, and the reference to the degaussing being used to clear the persistent letters on the display, I was starting to imaging some sort of Etch-a-Sketch display and the "DUNGGggg!" noise was to shake the display and spread the dust properly again :-)
Yeah, those built-in degaussers were scary the first time you came across them. I never saw a display like that, but did come across many 19" or larger colour CRTs used in CAD and process control rooms[*] over the years.
* control room screens were the ones most likely to have screen burn, since they invariably showed, 24/7, a mostly static schematic of the factory/refinery/chemical plant systems with things like kettle temperatures and valve positions.
This post has been deleted by its author
This post has been deleted by its author
Are you sure it wasn't a storage screen?
Conventional mono monitors use only a single electron beam, so don't need a shadow mask (which is what requires degaussing in a colour CRT).
The Tektronix 401x series graphics terminals used a bigger version of an oscilloscope storage screen, where a dot of phosphor struck by the beam with sufficient intensity carried on glowing afterwards until the whole screen was cleared. On the oscilloscopes, this was done electrically, by removing the voltage from an electrode near the screen. It's possible the screen clearing on the bigger CRTs in graphics terminals could have been done magnetically, which could be mistaken for degaussing.
Well, positioned correctly it wouldn't matter.
Many years ago, a certain air force put one of its air base operations management computers in a portakbin within a hardened air shelter, locked the door and left it to run. That was obviously fairly bomb-proof, from the outside or within. This sort of HAS was a very well sealed against NBC warfare, but because HAS are normally rather cold, nobody gave too much thought to the matter. After abour ten days of continuous running this computer stopped and a technician was sent to reboot the machione or whatever was required. When the doors were opened it became clear that the machine had go do hot that the whole thing had overheated and now featured a lot of melted and resolidified plastic and solder. Nobody was happy about that, but as always the taxpayer picked up the bill.
Oh that reminds me of an incident about five years ago. I went to see a client as their server kept restarting. I found they'd removed the side, front and back panels of the 2m rack to hang their wet cycling gear. On a wet morning the server kept restarting.
Can't think why......
sounds like a certain "hotels and resorts" chain I supported when working for a big global services provider. A pair of Slowlartus boxes, a pair of disk arrays, a UPS in a closet with a slop sink, mops, cleaning supplies etc.
They had a problem with one of the pair on a regular basis, the one closer to the floor. One day, we had the Sun guy in to fix it yet again, and we asked him to re-rack it to the top of the rack. Never had a problem with it again. He sent us pictures. It was a full rack, about 16 U consumed, all at the bottom of the rack. The room was moist, but regularly ventilated by people opening and closing the doors. No matter, it stopped complaining, so we stopped complaining.
I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away.
But I once found - after a network outage - a hub underneath a listed-building's parquet flooring. Literally just sitting under floorboards.
One office didn't come back up and we couldn't work out why because the cable to it was absolutely fine and had been working minutes earlier. Only after much investigation did we find a hidden 10Mbps powered hub was being used to "boost" the length of the cable run, and that hub was literally underneath a floor you couldn't rip up, which nobody had touched in decades, and was plugged into a hidden electrical socket inside the 3-inch gap between the actual floor and the parquet floor. Not documented. Nobody knew it was there. Not visible. Even the electrician had to come out because the plug socket was unknown to them and the building was so old it was a potential hazard (it still had round-pin plugs, toggle switches and those twisted-fabric electrical cables in places).
Found it by sheer chance by tugging on random nearby cables and then hearing something move. Even the LEDs on it weren't bright enough to see any more, probably decades caked in dust. From a networking viewpoint, it was invisible - while it was powered the cable just worked end to end. But if you tried to tone it or test it or trace it with the power off, it just stopped under the floor somewhere.
Long time back now but we moved into an old house which still had flash-cap wiring. For those that have never heard of it a carpenter/joiner(chippie) would run 1"x2" timbers with two channels cut into the length. Sparky would then come along and lay in two bare wires and the chippie would come back put a thin cap over it to protect the residents.
Apparently that was determined to not be a safe method to deploy mains round a house
Same house, most of the wiring was wooden conduit like that except those wires were gutta-percha with fabric covering. Metal conduit chased in down the walls.Only two wires of course. What to do when 3-pins were installed? Easy. Thread an extra wire down the metal conduit then twist it round the top of the metal to earth it. Or round a 6" nail hammered into the joint between two courses of brick. No brickwork handy? Hammer the nail into a wooden joist. I found all three options in that house.
My house is a bit over a hundred years old. The wiring for the overhead lights is knob and tube, fabric covered wire that runs through ceramic tubes placed in the ceiling joists and turns corners around ceramic knobs.
The gas pipes for the dual-fuel lights have long since been disconnected, but they were of course abandoned in place.
I've inspected most of the wiring, it's in good shape, and so I've deemed it safe enough for its current duty powering lights, given that they're all LEDs now.
All the rest of the wire was replaced with romex decades ago, I've added a few circuits here and there, most recently a 50A car charging circuit and a circuit for the bidet.
Our first house, (1893 rowhouse, Chicago USA) had the same knob and tube, as well as gas lines for dual fuel fixtures, when we moved in in 1977. To our horror, the gas lines were still filled and connected to the mains! Contractor called and gas lines disconnected, uncapped and emptied safely ASAP. Wiring got replaced in kitchen and bath, but all these years later, I wouldn't swear to there NOT being more antique wiring still in use in other parts of the house.
That describes the telephone cables between my village and the one on the next hill that has the exchange, except that the conduit had filled with water over the last 50 years. I got a friendly OpenReach Engineer to put a little hole in the pipe near the bottom of the valley. The 'water' (a disgusting sludge) ran out for weeks. The broadband was little better though.
I could never quite place why my old broadband was (more) shonky sometimes. Turns out that it still had paper-insulated cables, terminating in the chamber underground beside the poll. A little water would obviously seep in when the weather was inclement, and the insulation would become a little less insulating. None of that became apparent until an openreach van came to fix some neighbour's more blatant phone problem. Said engineer did not properly place the foam or cap properly on the chamber, a day or so before there was a proper downpour, and the paper insulation on the cables in there went from theoretical insulator to just not. That followup job was a more involved job than the initial fault/fix.
We moved into a house where the proud former owner had fully rewired it in the 1950s - using salvaged ex-GPO lead covered cable and the various-sized round-pin sockets. Surprise jolts from touching parts of it. Even better, he'd nailed a long run of it to the garden fence to aviaries at the end of the garden. Using uninsulated staples.
We had the only electrified fence in the neighbourhood.
"I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away."
I had a car wash as a client with several security cameras inside, all IP68 protected with a long, non-detachable coax+power cable - sold for underwater use in aquariums or pools. All auto-focus (and reasonably good analog image) so they were not meant to be opened, and I didn't.
They all stopped working before two years in the hot and humid (~100%) conditions, probably due to the chemicals involved. Since the cable and camera were single unit, replacing them required recabling in the filthies and slimiest possible cable ducts you can only imagine - in humid conditions and on top of A ladders over a super slippery floor.
Also, seeing really close the filthy and oily rotating brushes that clean cars - I have ever since only opted for touchless wash programs - though I prefer hand wash services.
On the positive side, the cameras produced loads of laugh-out-loud footage of people not heeding the instruction sign posts before entering and thus losing their antennas or rear wings, or not letting the conveyor belt move the car and instead driving and bumbing through like an idiot they were, opening of doors and windows during wash cycles and so on. Very few came back to collect the rear wings for some reason. Some tried to have the car wash owner pay for them with little success.
"losing their antennas or rear wings"
The guy in my local petrol station car-wash was adamant that I could go through with a roof rack and roof box attached. I assume he a) wants someone else to pay for a new car wash, and b) has a sideline running a youtube channel featuring videos of idiots.
>But I once found - after a network outage - a hub underneath a listed-building's parquet flooring. Literally just sitting under floorboards.
So in summary, there was an installed solution which had worked fine with no intervention for possibly decades.
And you broke it
= tell me you're a network admin without telling me you're a network admin !
"So in summary, there was an installed solution which had worked fine with no intervention for possibly decades."
I think if pressure were applied plenty of us would own up to dodgy installs like that. The boss "doesn't want to hear any excuses", make it work and make it work by 5pm! or you're going to be looking for a new job. The last guy would have had it done already... etc.
It sounds like it was a good job for lasting as long as it did. Running a fiber link at the time would have been expensive and overkill, but would have been the better solution. My aunt's VW bug, the one with the fuel injection that would often lose a hose and go up in flames got a stainless braided hose upgrade from me and my step dad after a minor fire. It wasn't cheap, but less than having to replace the whole car and the school were she was a teacher wasn't in a good neighborhood so having an incident anywhere near there could have other repercussions. There was never a problem with the fuel system after that. The VW repair shop where we took our Vdubs copied what we did for all of his fuel injected customers and got known for the mod.
A field engineer once told me he got sent to a server at a school. As this was a school, the staff didn't want the pupils playing with the server so they built a little box around it. The engineer opened the box and found that the plastic case for the server had heated enough to soften and had sagged onto the motherboard.
Personnally, I'm suspicious of this: Surely if the box had gotten warm enough to soften the plastic, wouldn't it have gotten hot enough to fry the chips?
You'd have thought so, but it's surprising how well many ICs will operate well above their rated temps - perhaps unsurprising given the often very high process temps that silicon dies are processed at, or indeed the temps they'll be exposed to by soldering. Doesn't mean the silicon will always function as intendedoutside it's rating, but it's easily possible for an enclosed processor to push the temps up until something melts - see my little tale above about the computer in the hardened air shelter.
I had a 1U rack mountable case I built a server into in '96, the heat sink touched the lid. The lid got so hot, the paint came off the lid in a square around the footprint of the heat sink. And it only took a few minutes to do that, and the server was perfectly happy running during those few minutes. I wound up drilling holes into the lid in a pretty pattern over the heat sink, which improved airflow.
"10Mbps powered hub" ...
Luxury!
We installed "modern" 10base2 with stretches of 10base5 to get extra length ... which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ...
We even ditched the Apple2 server and its RS232 "network" ... eventually.
Did the job for small offices (and more than a few large ones) for about two decades.
Was really cheap once NE2000 clones started showing up, and thus became nearly ubiquitous. A couple of techs could crank out a couple hundred seat network in a long weekend, given enough coffee. I must have run several hundred miles of the stuff.
I don't miss it a bit ... but I still run it at home when it makes sense. Why not, I still have all the tools and several 2,500 foot rolls of the stuff. And I maintain a couple of legacy machine shops that use it, along with a few SCADA odds & ends.
Quote
"Luxury!
We installed "modern" 10base2 with stretches of 10base5 to get extra length ... which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ...
We even ditched the Apple2 server and its RS232 "network" ... eventually."
You 'ad RS232? all we got was paper punch tape.... god that was fun when someone put "program end/rewind" on the end of the punch tape instead just "program end" wheeeeeeeeeeeeeeeeee rip rip rip rip shred rip......
Meanwhile back on topic....
PCs found in disgusting places..... bottom of our metal plating tank anyone? and before you ask saying I've posted that one before, its happened again... this week. oh well another trip to pc werld for a cheapo laptop.
we need a vomit icon.
"Did they not flag it up as an unsuitable installation?"
Probably, yes.
But the clueless Boss said "do it anyway, it'll be fine and I've got a quota to fill".
Or perhaps the owner of the shop in question insisted, and was willing to sign a "get out of jail free" card for the installer.
Etc.
Waaay back in my past, we had a customer with a PC that controlled a glass sheet cutting machine. The glass panes were huge and thrown around on a bed of air.
The jobs were sent to the machine via a 3 1/2 floppy drive. The glass dust ate floppies and drives on a regular basis.
“How many replacements before you just put a simple shield or flap over the front of the floppy drive?”
Ah, yes a question for the bean counters, possibly fitting a simple shield (or maybe glass dust being, well, glass dust, it might not be as simple as that), comes out of ‘x’ budget who are unprepared to pay for it, because it’s not their problem, whereas just replacing the drive every month comes out of ‘y’ budget. As long as everything balances, then fair enough.
The simply fact that both ‘x’ and ‘y’ are ultimately paid for by the same company does seem to escape some organisations.
"How many replacements before you just put a simple shield or flap over the front of the floppy drive?"
My "solution" at a pottery where the optical sensor on the tape drive kept getting clogged with dust and ripping the tapes off the end of the spool, was to open the PCs PSU, turn the extractor fan into an intake fan by physically reversing its orientation, and put a filter on the the new "inlet", so clean air blew out the tape/floppy drive/other open holes at the front. The filter needed changing every week but that was no longer my problem.
I once got sent out to a site I'd never been to before to deal with some server issue. It wasn't dead, must have been something else. Despite being in a small 'where everything gets dumped' cupboard it was on something where it wouldn't get knocked off or have stuff piled on it. It was also very well cooled, by a small portable air conditioner. On a chair. One of those metal frame ones with thick foam seat and back, such that the air con unit didn't quite sit up straight. This might have been fine it it had automatic evaporation via the hot air hose (which did go out of the room) but instead the tilt meant the level sensor in the water tank didn't notice the tank getting full, overflowing so it soaked the seat foam it sat on, then dibbling down the chair leg and soaking the carpet tiles. The squelch when you walked in the room was a giveaway.
It's an on attitude isn't it? These machines will house your entire business. If they go down so does your business. Collectively they represent your entire business, whatever you value that at. Where do you put them? Somewhere where they'll be treated with the respect that that value deserves or whatever small space that you'll begrudge least, however unsuitable?
> whatever small space that you'll begrudge least, however unsuitable
>> but the cubicles were still there as the partitions were brick built.
Cubicles, plural, with decent brick walls? At least two (possibly more, to have plural partitions)? That could be a fair amount of volume, with solid structure you can safely hang the kit on and plenty of spare wall space to neatly hang your bits and bobs.
That sounds like a good deal more suitable than the cubicles in the rest of an average Business Link premises, the ones the human resources are supposed to live in.
Not sure why they still cling to chalk when whiteboard pens and smart displays have been around for decades now (albeit my school replaced whiteboards with chalkboards/blackboards (to use UK parlance because.well they are black in colour) for well "reasons" likely either someone was a traditionalist, pens were seen as a luxury and a new blackboard came out of capital budget but consumables came out of another and well we buy lots of chalk so why should we.buy pens?)
We have a site right beside the sea, and the work carried out from there involves going on the beach regularly. Not got any servers there fortunately, but any desktops which need repair have to be de-sanded first.
At another site they decided to re-purpose the comms room, where the main patch cabinet is, as a cleaner's store. I had to kick up quite a fuss about that one to get them to shift the mops and buckets elsewhere!
I worked for a fruity trillion dollar phone company. Third party had "issues" with rain leaking into their datacentre.
So they closed some of the toilets and put the servers in there. Hilariously they had to balance the servers on top of the toilets, as they couldn't put them on the sinks due to their weird shape, and couldn't put them on the floors as the server blocks weren't in their HUGE racks anymore.
Door propped open,cables leading down the hall, sellotaped (not even proper electrical tape) to stop people tripping over them.
It worked for about 4 weeks until they FINALLY agreed they needed to get the roof fixed, as each department wanted the money to come out of the others funding.
"until they FINALLY agreed they needed to get the roof fixed"
I once worked somewhere they were on the top floor of a mid-rise block, where it was owned on a share-of-freehold basis. The roof leaked, and had for years, because all the freeholders needed to agree to the work being done, but only my clients on the top floor actually suffered from the leaks. I've never seen an office with internal roofing and rain gutters leading to the drains anywhere else.
A New Zealand university had this in the Science block almost from new. It was startling to see this rig arching over a multimillion dollar xray crystallography machine
The block was built by the Ministry of Je^H^HWorks in the early 1970s and had 2 years of snagging before it was even safe to enter (Amongst other issues the staircases weren't fixed to the building)
This building, the Veterinary studies tower and virtually the _entire_ Teachers' College campus (later a polytechnic) had to be abandoned and were knocked down as inspections in the wake of the Christchurch and Kaikoura earthquakes revealed they were structurally unsound
On a computing related note there were servers stuffed into a number of closets, but that was to keep them out of the sight and knowledge of central IT, due to the god complex of the director of IT (he would actively kill anything that he didn't control)
I can understand the director of IT on this one.
Unauthorised servers on an internal (and privileged) network are a security nightmare. They are rarely kept in line with security standards, probably never backed up, or regularly patched, and cause unanticipated load on network infrastructure.
Plus, at some point, they go wrong or are crypotolockered, and then IT are called in to help fix the shitshow caused by the office cowboys.
In a university, which likely does not have much (if any) internal firewalling on the staff network segments, a vulnerable machine can cause chaos if it is compromised, and you can't recover what was not backed up.
I had an apartment like this for several years in the Colorado mountains..... it was very cheap and a nice location. Indoor guttering! All the fancy. The water would pool on the insufficiently sloped flat roof, especially in the spring melt there would be 8-12" of standing water, and it would seep through the roof and only into my apartment. For weeks at a time.
I knew someone with a flat like that in London in a large residential block. Flat roof. Years of investigations and patching trying to work out where the water was coming from - it tends to seep long distances. The person's bedroom had a funnel under the point in the ceiling where the water came in, with a drainpipe out of the window. He was a builder, so that bit worked. He and his wife were remarkably sanguine about it.
It was only when the residential block's roof was completely replaced as part of a major refurbishment project did the problem go away. Flat roofs are the work of the Devil.
Comms cabinets under toilet blocks - yes with a leak in the large pipe….
Misc kit under floors to extend circuits - yes
But the best was a non IT one builders knocking down some heavy duty partition walls to find someone had mounted a 40 gallon water tank on top of the false ceiling. The walls came down , the ceiling started moving and they found the tank help up by the copper pipes connecting it to the rest of the plumbing and nothing else….
In the mid 1990's, I got sent to a pizza restaurant to work on their late 1980's DOS-based point-of-sale system. Repair if possible, replace if necessary. Slimline style 386 boxes. The insides looked like they were covered in a soft gray felt, with small bumps where the various chips were. Forget about operational fans. Outside of the cases were sticky with that fryer grease which permeates a restaurant and settles on all surfaces.
They ended up being replaced with 486DX2 boxes running at 40 Mhz. They were the mainstay workhorse back in the day, complete with VESA Local Bus cards to handle serial ports, graphics, networking, and (get this) light pens.
Some of the best pizza I ever had though. Made their sausage in-house daily and cooked it on the pie - no pre-cooked toppings. Pina Pizza in Downey, CA. Cheers!
At an Air National Guard base, one building was given an air handler room to install the network switches for the building. One day their network went out. The air conditioner had died and the room filled with hot humid air. So humid that condensation started raining down the walls and into the wall-mounted switches.
Back in my youth I had a part-time job working for a department at the Uni I was attending. IIRC, I was being sent around to every workstation in the department to make sure the backup program was set up correctly after a poor grad student had their entire thesis and research lost due to a HDD crash when it turned out for reasons lost to the mists of time, the backup program wasn't actually backing anything up for them.
I was chatting with the department secretary and beancounter, who were in the same office, while checking their systems. They warned me about a particular professor's office and how I should always make sure to let someone know I'm going in there or I could end up trapped under a pile of papers or something and might not be found for days. I figured they were just exaggerating, but then my travels finally took me to that office. It looks like they converted an old janitor's closet into a professor's office, and literally every free surface was stacked with papers 2-3ft high, or higher. There was just a small pathway to the desk where the computer was. Think like what you see in those hoarder shows, only if the hoarder was specific to only saving various papers. If anything, the nice office ladies were probably underselling things. Oh, and the keyboard and mouse were sitting atop... no prizes if you guessed "more papers." Fortunately that was the one and only time I ever had to go into that office.
Back in the 1970s....
An early $JOB involved manufacturing special purpose data-collection terminals with a keypad and a manually fed 80-column punch card reader. They sold it to factories so machine-tool operators could scan a punch card every time they started a new job. A bit of computing prestidigitation and you got accurate to-the-minute labour and machine costs for each job.
In the theme of "dog fooding", that is, eating the same food you make for your dog, the business owners had an electro-plating shop next door. They put these terminals next to large vats of various acids used to clean the metal items before being chromium-plated.
The terminals were typical for the day, 2-layer printed circuit boards with no particular protection layers added.
On average they lasted about 12 months before corrosion killed them.
I maintained a local M.O.T. garages computers for a couple of years, but generally only had to deal with the office equipment which was what I would call run-of-the-mill support. Though one day the manager asked me to look at the box installed in the garage, you know, the one that held all the M.O.T. data etc. It was overheating and to cut a long story short it mirrors the main story above. Opened up the box, couldn't even see the motherboard/CPU block or PSU through the compounded brake dusr and gunge. BUT I got it working after a good steam cleam (joking).
It's not only companies... I couldn't count the number of times I've arrived at residences to find the computer I'm to work on is crammed in a little cubby in a credenza, with the door kept closed while the computer is running. Not only that, but it takes the hands of a 10 year old to hook/unhook the connections.
Take computer out, open case, clean dust, admonish people... lockups and blue screens stop :-)
I degaussed a truly monochrome CRT. 1939 Allen B. DuMont oscillograph (today called oscilloscope, but he was a pioneer in several ways that didn't catch on). Found on curb, rebuilt, but could not find the dot. Finally found it by experimenting with magnets. The idle position seemed to be about 5 inches outside the edge of the screen. Found my old reel-tape demagger, Hurrrnnn!, dot was on-screen and could be centered with deflection bias.
Strange Places:
A 286 PC (Panasonic!) in a piano technician's workshop. Goofiness finally cured by inverting the PC (we use to run them pizza-box orientation) and thumping the metal chips out.
Main department server (Novell Netware 3.11!) in the closet of a busy tinkle-toilet room. Gave little trouble but shocked visiting geeks and geekesses.
Two of my server rooms had buckets under the drips in the skylights.
But L---- took the cake. Despite repeated gentle and also loud warnings, she kept an IBM PC/XT directly under a hanging plant and watered it every day. Sometimes she over-watered. Somehow she never electrocuted herself or filled the office with cathode-ray smoke.
The NHS, too cheap to move the small server room at one site that, in heavy rain, would get slightly flooded as water would come in to hallway from backdoor then finds its way under the door into the server room. So all they did was raise the servers off the floor slightly.
The other server room had a leaking aircon. As always in my career I pick the shit places that don't listen to temps, so the 3 years I was there it was never fixed.
Then at last place building a new swimming pool. We warned them "You'll want to put it on the roof, the server room, on the top floor cause of the sea water and humidity in the basements because of all the pool water kit. We were ignored, the architects know best. Except they fucking didn't. The room has never been sealed well enough and shortly after all the kit started to rust, I still have the photos. Its been several years later, the expense at having to replace that kit regularly must be costing a fortune.
How about "the shop floor" - the factory space where a major control systems company (now defunct) used to build, configure, test and FAT test systems. It was fine in winter, spring and autumn. However, when summer arrived, its flat roof, designed to hold a cooling pond of water across its whole width but which had been drained of water some years ago (presumably cos it leaked), duly obeyed the laws of physics, notably those of radiation, and caused the building below to warm up. By warm, I don't mean 25 or 30°C. Or even 35°C. No, a cool 40°C was a fairly usual temperature. However, it would make 45 or even 48°C for short periods. Unsurprisingly, for quite a lot of the kit this was a heat soak test too far, and some bits of kit would start falling over. I was surprised that more didn't break than actually did.
A department store, several branches of which I have had occasion to visit, does not provide dedicated aircon to their server rooms. Rather, they see fit to install an outlet from the building's main HVAC system. Maybe, in theory, it worked OK when first installed; I'll give them the benefit of the doubt. But over time, Stuff Happens -- more equipment added to the MDF room, duct gets disconnected, system becomes unbalanced for some other reason. More than once, I've walked into one of their rooms and found it rip-off-all-your-clothes hot. I suggest to manglement that they *really* should get their Facilities folks to address the situation, and they say "Oh, it's always like that!"
I worked for a local authority who had converted an old school into a site for IT and finance. They were looking for a secure room to site a couple of finance servers related to the collection of cash from all the parking ticket machines across the authority and decided that the best option was an old boys toilet. There were no windows and the walls were all tiled. It was also quite humid as there was a drain in the floor and intially there was no air conditioning. After several overheating issues, the decision was taken to install air con and things were OK for a while. Nobody really needed to go into the server room/boys toilet, backups were automated to a robot in the rack and the tapes were cycled one a week.
There was a sudden outage one day and I was sent to investigate. When I opened up the room, it was all nice and cool, the servers were both dead and there was a large pool of water in the room. both severs were wet inside the cases and had cut out when they short circuited. The root cause was the air con and the humidity. As the air con ran, the humid air in the room started to condense and then freeze on the aircon unit. Eventually this had built up to the point where the fans were affected and stopped working. The heat then increased in the room, the ice melted and once the fans were able to spin again they did, spraying the water from the melting ice across the room and onto both servers.
There was a rapid re-evaluation of the site and the servers were moved somehwere less humid. Once they had been suitably dried out, we were able to boot them both and were amazed to see there had been no lasting damage.
Went to a place of learning once that we used to run. It was tiny and the chaps that had set what passed as the IT up had left and it looked like it had just been setup with the contents of someone's junk cupboard.
Anyhow I digress. Got there was shown where the servers were (partitioned off section of a room with full window at the front, during what passed as a british summer.
"Mind the locusts" I was told. Yeah they kept some locusts in glass tanks in there as it was nice and warm for them.
I then somehow became responsible for the site, so not long after portable Aircon thing and window blind installed.
I was requested to design a setup for a buffet style restaurant. Cameras, wifi, doors, internal data. Easy enough, POE cameras, door actuators and WiFi hotspots, cabling above the false ceiling, VPN programmed switch and UPS in the manager's office. They picked a reputable firm to do the installation and I went on to other things.
A few months later, I was yanked off another job to the scream of an impending lawsuit to find out why this setup was no longer working. A few minutes after getting there, it was obvious why the system no longer worked. The manager had decided that the wall rack should not go in his office, and instead had them put it in the back room. That room had their deep sink, cleaning gear, chemical storage and the very large hot water heater, and it had no ventilation. Every bit of exposed copper in the patch panel was green, the UPS had shorted out when it was doused with fluid of some kind, and I still don't know how the switch survived.
I took pictures and told the business to pound sand, they would get laughed out of court if they sued. After I was reprimanded for harsh language and folks had some time to cool down, I was asked how what they had could be made to work for the least amount of additional money. Since the cabling as built wouldn't go to the manager's office, I stuck the rack above the back room, and rigged up some ventilation for it. We cut a few feet off each wire, patched it into a new patch panel, replaced the UPS and it was good to go. The only problem was that you needed an eight foot ladder to get to it.
For my first job out of uni I worked for a pipeline company. They had two control rooms and two server rooms, one pair downtown, and one pair at their tank farm. Each was provisioned with custom control consoles and a PDP-10 mainframe to provide hot backup for controlling the entire pipeline system. Most of the time, the downtown site was live, with operators in the server room 24/7/365. The tank farm had operators during the day shift and on call at other times.
One day the tank farm operators came on shift to find an odd smell in the server room. After a bit of investigation, they lifted a tile from the raised floor. There they found a few inches of natural-gas condensate had flooded the room. (See e.g. https://en.wikipedia.org/wiki/Natural-gas_condensate) Needless to say, condensate is very volatile and flammable. It is not a good idea to submerge your server cabling in it.
Fortunately, the leak was fixed and the server room was drained safely.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
