back to article Police ignored the laws of datacenter climate control

With Friday upon us once again, The Register finds itself glancing at weather forecasts as we prepare another instalment of On Call, the weekly tale of tech support in which we share readers' stories of being asked to fease the infeasible. This week, meet a reader we'll Regomize as "Sylvester" who, in the early 2000s, found …

  1. Felonmarmer

    Fun with magnets.

    A company I worked at had an old terminal (green text type) connected to a digitising table. The terminal needed degaussing quite often (with a button on the side) to clear the text which was semi persistent as did the digitising table with a large magnetic sweeper.

    Guess where they stored the back-up tapes from the MicroVAX that ran the office network?

    1. Anonymous Coward
      Anonymous Coward

      Re: Fun with magnets.

      Except monochrome CRTs don't have a shadow mask or aperture grille because they only have one electron gun, so there's no need for a degaussing coil.

      Plus degaussing a CRT won't fix burn in, so either it was a colour terminal or someone was telling fairy stories.

      1. Felonmarmer

        Re: Fun with magnets.

        This one did. I know because I used it. It used to make a loud cracking sound when you degaussed. And the button had degauss written next to it. It was old in 1989 when I first used it.

        The magnetic fields in that room twice made my bank cards unreadable.

        1. This post has been deleted by its author

          1. Martin Summers Silver badge

            Re: Fun with magnets.

            I had quite a large CRT for home use with a degaussing button on it in 2000. Made a heck of a noise and clunk when it was pressed.

            1. This post has been deleted by its author

          2. Anonymous Coward
            Anonymous Coward

            Re: Fun with magnets.

            Back in the day I definitely worked with monitors with degauss functions (not sure if it was a button though). A quick google brings up lots of examples of degauss functions on monitors.

            Something related that I was thinking about recently, it used to be that speakers used for the center speaker of a surround system were magnetically shielded, to stop them affecting the CRT televisions, presumably that isn't needed anymore with LED/plasma sets.

            1. This post has been deleted by its author

          3. Tom 38

            Re: Fun with magnets.

            I'm am old fart, not a single monochrome CRT I've ever seen in anything (I've seen metric fucktons of terminals, TVs and monitors because I repaired them for a living)

            Yes, wonderful story Grandpa. The only problem is that the only person who has mentioned monochrome monitors is you. The OP said that it had green text. Have you never come across a monitor with green text that was not a monochrome monitor, because I'm a spring chicken of only 44 years, and I've seen several. One old CGA (perhaps EGA) monitor I used even had a slider so you could switch it between the full 16 colours, or green text, white text and amber text, as you desired. The wonders of modern technology!

            I liked green text on the terminal, but for Sopwith I much preferred amber.

            1. Anonymous Coward
              Anonymous Coward

              Re: Fun with magnets.

              > Yes, wonderful story Grandpa.

              > Have you never come across a monitor with green text that was not a monochrome monitor?

              Unfortunately for you, rude little whippersnapper, the device has been identified as a Tektronix storage tube, which is very much a monochrome display.

              And a very, very expensive one, so it is quite reasonable that someone working for decades on TVs and monitors never saw one - which is why the self-confessed old fart quite reasonably asked for the make and model (i.e. still willing to learn).

              1. An_Old_Dog Silver badge

                Re: Fun with magnets.

                Our uni had some Tektronix 4010 storage tube terminals running at 4800 baud. Tek had a FORTRAN package called PLOT10 that let you feed it numbers which it used would display pretty charts on the screen.

            2. garwhale Bronze badge

              Re: Fun with magnets.

              OT. My ex asked me to look at her monitor, it was only showing B&W, whilst her co-workers had colour. I fiddled with it a little and then said, "it's a monochrome monitor!?". Guess she just wanted me to come to her office when nobody else was there...

          4. CJ_C
            Go

            Re: Fun with magnets.

            Sounds like a Tektronix 14"storage tube graphics didplay to me. Probably cost over £20k in the 1970s. They were not common. As I recall the whole unit shook and made DUNG noise when the degauss was pressed.

            1. Felonmarmer

              Re: Fun with magnets.

              Looked it up on Google, that looks just like it! Even had that pocket on the side with a manual in that some of the photos show.

              Cheers, nice to know I wasn't imagining it as some thought.

              Yep the degauss was quite impressive, used to scare the crap out of me when I first started using it.

              1. John Brown (no body) Silver badge

                Re: Fun with magnets.

                Reading through the comments, and the reference to the degaussing being used to clear the persistent letters on the display, I was starting to imaging some sort of Etch-a-Sketch display and the "DUNGGggg!" noise was to shake the display and spread the dust properly again :-)

                Yeah, those built-in degaussers were scary the first time you came across them. I never saw a display like that, but did come across many 19" or larger colour CRTs used in CAD and process control rooms[*] over the years.

                * control room screens were the ones most likely to have screen burn, since they invariably showed, 24/7, a mostly static schematic of the factory/refinery/chemical plant systems with things like kettle temperatures and valve positions.

            2. Anonymous Coward
              Anonymous Coward

              Re: Fun with magnets.

              Yeah I vaguely remember the security logging machine at my first employer was like that, it didn't need a colour screen so I was told they had used the first screen that had come to hand.

            3. teebie

              Re: Fun with magnets.

              I thought the big DUNG noise was the point of degaussing?

          5. This post has been deleted by its author

          6. I Am Spartacus
            Flame

            Re: Fun with magnets.

            It was almost certainly a Techtronix vector graphics definitely did have a degausing button on them. Given that the terminal was next to, and probably associated with, the drafting table, it almost certainly was one of the then goto terminals for graphics.

          7. This post has been deleted by its author

      2. Alan Brown Silver badge

        Re: Fun with magnets.

        I've seen monochrome monitors with magnetised frames which needed degaussing

      3. Dabooka

        Re: Fun with magnets.

        I do love it when heroes come along and hide by the AC mask

    2. JulieM Silver badge
      Coat

      Re: Fun with magnets.

      Are you sure it wasn't a storage screen?

      Conventional mono monitors use only a single electron beam, so don't need a shadow mask (which is what requires degaussing in a colour CRT).

      The Tektronix 401x series graphics terminals used a bigger version of an oscilloscope storage screen, where a dot of phosphor struck by the beam with sufficient intensity carried on glowing afterwards until the whole screen was cleared. On the oscilloscopes, this was done electrically, by removing the voltage from an electrode near the screen. It's possible the screen clearing on the bigger CRTs in graphics terminals could have been done magnetically, which could be mistaken for degaussing.

  2. Korev Silver badge
    Coat

    It sounds like he wasn't the biggest fan of the server's location...

    1. b0llchit Silver badge
      Coat

      The flow had dried up.

    2. Inventor of the Marmite Laser Silver badge

      Well someone had to be drafted in

  3. IanRS

    Boom

    At least it did not blow up. Or in, or out, or any other direction.

    1. Anonymous Coward
      Anonymous Coward

      Re: Boom

      Well, positioned correctly it wouldn't matter.

      Many years ago, a certain air force put one of its air base operations management computers in a portakbin within a hardened air shelter, locked the door and left it to run. That was obviously fairly bomb-proof, from the outside or within. This sort of HAS was a very well sealed against NBC warfare, but because HAS are normally rather cold, nobody gave too much thought to the matter. After abour ten days of continuous running this computer stopped and a technician was sent to reboot the machione or whatever was required. When the doors were opened it became clear that the machine had go do hot that the whole thing had overheated and now featured a lot of melted and resolidified plastic and solder. Nobody was happy about that, but as always the taxpayer picked up the bill.

      1. Yet Another Anonymous coward Silver badge

        Re: Boom

        But we put the computer and the chiller inside a sealed room and left it to run.

        Why is it hot? shouldn't it be cold ?

  4. SVD_NL Bronze badge

    Convenience

    To be fair, if you've got a room right next to the vehicle bay that comes with a heating fan included, why wouldn't you use it as a drying room?

    1. Pascal Monett Silver badge
      Facepalm

      Re: Convenience

      To be fair, what the hell were they thinking putting electronic equipment in a room full of humidity ?

      That server would have been better off sitting outside the building. Maybe next to a door, for quick access.

    2. David Nash Silver badge

      Re: Convenience

      It's fine using it as a drying room. It's less fine using it as a server room!

      1. VicMortimer Silver badge

        Re: Convenience

        But without the server in there, what's going to heat the room?

    3. Yet Another Anonymous coward Silver badge

      Re: Convenience

      Before we went all "cloudy", the server rack was a lovely place to dry/warm my cycling gear

      1. Snapper
        Facepalm

        Re: Convenience

        Oh that reminds me of an incident about five years ago. I went to see a client as their server kept restarting. I found they'd removed the side, front and back panels of the 2m rack to hang their wet cycling gear. On a wet morning the server kept restarting.

        Can't think why......

  5. jake Silver badge

    One of my favorites.

    Punchdown blocks mounted on the wall of a janitor's closet.

    Directly over a mop bucket full of ammonia water.

    1. The Oncoming Scorn Silver badge
      Windows

      Re: One of my favorites.

      Posted one of these recently.....

      Finding network comms kit nine foot directly above a lavatory.

      Investigating a schools dead PC finding "rust" on the motherboard - Cause PC was directly under a leaking glass roof.

    2. Joe 59

      Re: One of my favorites.

      sounds like a certain "hotels and resorts" chain I supported when working for a big global services provider. A pair of Slowlartus boxes, a pair of disk arrays, a UPS in a closet with a slop sink, mops, cleaning supplies etc.

      They had a problem with one of the pair on a regular basis, the one closer to the floor. One day, we had the Sun guy in to fix it yet again, and we asked him to re-rack it to the top of the rack. Never had a problem with it again. He sent us pictures. It was a full rack, about 16 U consumed, all at the bottom of the rack. The room was moist, but regularly ventilated by people opening and closing the doors. No matter, it stopped complaining, so we stopped complaining.

    3. Yet Another Anonymous coward Silver badge

      Re: One of my favorites.

      >Directly over a mop bucket full of ammonia water.

      At least it was above the bucket

    4. chivo243 Silver badge
      Facepalm

      Re: One of my favorites.

      Hands down the best ever is a rack filled with UPS and POE switches, power injectors for cameras in a basement, under the roof drain pipe, in The Netherlands. I have the pics!

      1. Yet Another Anonymous coward Silver badge

        Re: One of my favorites.

        To be fair most of the country is built underneath the drain pipes

  6. Lee D Silver badge

    I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away.

    But I once found - after a network outage - a hub underneath a listed-building's parquet flooring. Literally just sitting under floorboards.

    One office didn't come back up and we couldn't work out why because the cable to it was absolutely fine and had been working minutes earlier. Only after much investigation did we find a hidden 10Mbps powered hub was being used to "boost" the length of the cable run, and that hub was literally underneath a floor you couldn't rip up, which nobody had touched in decades, and was plugged into a hidden electrical socket inside the 3-inch gap between the actual floor and the parquet floor. Not documented. Nobody knew it was there. Not visible. Even the electrician had to come out because the plug socket was unknown to them and the building was so old it was a potential hazard (it still had round-pin plugs, toggle switches and those twisted-fabric electrical cables in places).

    Found it by sheer chance by tugging on random nearby cables and then hearing something move. Even the LEDs on it weren't bright enough to see any more, probably decades caked in dust. From a networking viewpoint, it was invisible - while it was powered the cable just worked end to end. But if you tried to tone it or test it or trace it with the power off, it just stopped under the floor somewhere.

    1. jake Silver badge

      Now ask me why I've had a top of the line TDR in my toolkit since time immemorial.

      Yes, they were very expensive in the early days, but worth every penny. Recommended.

    2. Doctor Syntax Silver badge

      "twisted-fabric electrical cables"

      You had fabric? Luxury! We moved into a house where some of the cables were lead-sheathed with paper insulation in the last stages of rotting away.

      1. Kevin Johnston

        Long time back now but we moved into an old house which still had flash-cap wiring. For those that have never heard of it a carpenter/joiner(chippie) would run 1"x2" timbers with two channels cut into the length. Sparky would then come along and lay in two bare wires and the chippie would come back put a thin cap over it to protect the residents.

        Apparently that was determined to not be a safe method to deploy mains round a house

        1. Doctor Syntax Silver badge

          Same house, most of the wiring was wooden conduit like that except those wires were gutta-percha with fabric covering. Metal conduit chased in down the walls.Only two wires of course. What to do when 3-pins were installed? Easy. Thread an extra wire down the metal conduit then twist it round the top of the metal to earth it. Or round a 6" nail hammered into the joint between two courses of brick. No brickwork handy? Hammer the nail into a wooden joist. I found all three options in that house.

          1. Handlebars

            I'm picturing the original installer as a guy in a stovepipe hat who has been working there since the place was built

        2. VicMortimer Silver badge

          My house is a bit over a hundred years old. The wiring for the overhead lights is knob and tube, fabric covered wire that runs through ceramic tubes placed in the ceiling joists and turns corners around ceramic knobs.

          The gas pipes for the dual-fuel lights have long since been disconnected, but they were of course abandoned in place.

          I've inspected most of the wiring, it's in good shape, and so I've deemed it safe enough for its current duty powering lights, given that they're all LEDs now.

          All the rest of the wire was replaced with romex decades ago, I've added a few circuits here and there, most recently a 50A car charging circuit and a circuit for the bidet.

          1. CountCadaver Silver badge

            Knob and tube is NOT safe

            It's LONG overdue to be replaced

          2. Sherrie Ludwig

            Our first house, (1893 rowhouse, Chicago USA) had the same knob and tube, as well as gas lines for dual fuel fixtures, when we moved in in 1977. To our horror, the gas lines were still filled and connected to the mains! Contractor called and gas lines disconnected, uncapped and emptied safely ASAP. Wiring got replaced in kitchen and bath, but all these years later, I wouldn't swear to there NOT being more antique wiring still in use in other parts of the house.

        3. CountCadaver Silver badge

          Sounds similar to the American knob and tube wiring (that some STILL claim is "perfectly safe")

      2. Caver_Dave Silver badge

        That describes the telephone cables between my village and the one on the next hill that has the exchange, except that the conduit had filled with water over the last 50 years. I got a friendly OpenReach Engineer to put a little hole in the pipe near the bottom of the valley. The 'water' (a disgusting sludge) ran out for weeks. The broadband was little better though.

        1. the hatter

          I could never quite place why my old broadband was (more) shonky sometimes. Turns out that it still had paper-insulated cables, terminating in the chamber underground beside the poll. A little water would obviously seep in when the weather was inclement, and the insulation would become a little less insulating. None of that became apparent until an openreach van came to fix some neighbour's more blatant phone problem. Said engineer did not properly place the foam or cap properly on the chamber, a day or so before there was a proper downpour, and the paper insulation on the cables in there went from theoretical insulator to just not. That followup job was a more involved job than the initial fault/fix.

      3. LessWileyCoyote

        We moved into a house where the proud former owner had fully rewired it in the 1950s - using salvaged ex-GPO lead covered cable and the various-sized round-pin sockets. Surprise jolts from touching parts of it. Even better, he'd nailed a long run of it to the garden fence to aviaries at the end of the garden. Using uninsulated staples.

        We had the only electrified fence in the neighbourhood.

    3. Anonymous Coward
      Anonymous Coward

      "I am pitching to replacing a switch that's in a swimming pool pump room - hot, humid, full of corrosive chemicals, 24/7, and slowing rusting away."

      I had a car wash as a client with several security cameras inside, all IP68 protected with a long, non-detachable coax+power cable - sold for underwater use in aquariums or pools. All auto-focus (and reasonably good analog image) so they were not meant to be opened, and I didn't.

      They all stopped working before two years in the hot and humid (~100%) conditions, probably due to the chemicals involved. Since the cable and camera were single unit, replacing them required recabling in the filthies and slimiest possible cable ducts you can only imagine - in humid conditions and on top of A ladders over a super slippery floor.

      Also, seeing really close the filthy and oily rotating brushes that clean cars - I have ever since only opted for touchless wash programs - though I prefer hand wash services.

      On the positive side, the cameras produced loads of laugh-out-loud footage of people not heeding the instruction sign posts before entering and thus losing their antennas or rear wings, or not letting the conveyor belt move the car and instead driving and bumbing through like an idiot they were, opening of doors and windows during wash cycles and so on. Very few came back to collect the rear wings for some reason. Some tried to have the car wash owner pay for them with little success.

      1. Dave314159ggggdffsdds Silver badge

        "losing their antennas or rear wings"

        The guy in my local petrol station car-wash was adamant that I could go through with a roof rack and roof box attached. I assume he a) wants someone else to pay for a new car wash, and b) has a sideline running a youtube channel featuring videos of idiots.

        1. Ken Shabby

          In my teens, I worked at a petrol station that had a car wash. Seen a nice sports car (well think it was an MG), get a new go faster stripe, perfectly front to back, over the top, from his Ariel or a previous user’s.

          1. J.G.Harston Silver badge

            A mermaid?

    4. Yet Another Anonymous coward Silver badge

      >But I once found - after a network outage - a hub underneath a listed-building's parquet flooring. Literally just sitting under floorboards.

      So in summary, there was an installed solution which had worked fine with no intervention for possibly decades.

      And you broke it

      = tell me you're a network admin without telling me you're a network admin !

      1. MachDiamond Silver badge

        "So in summary, there was an installed solution which had worked fine with no intervention for possibly decades."

        I think if pressure were applied plenty of us would own up to dodgy installs like that. The boss "doesn't want to hear any excuses", make it work and make it work by 5pm! or you're going to be looking for a new job. The last guy would have had it done already... etc.

        It sounds like it was a good job for lasting as long as it did. Running a fiber link at the time would have been expensive and overkill, but would have been the better solution. My aunt's VW bug, the one with the fuel injection that would often lose a hose and go up in flames got a stainless braided hose upgrade from me and my step dad after a minor fire. It wasn't cheap, but less than having to replace the whole car and the school were she was a teacher wasn't in a good neighborhood so having an incident anywhere near there could have other repercussions. There was never a problem with the fuel system after that. The VW repair shop where we took our Vdubs copied what we did for all of his fuel injected customers and got known for the mod.

    5. CountCadaver Silver badge

      Urgh fabric covered Vulcanised India Rubber insulated cabling....where the rubber has a charming habit of falling off the wiring due to perishing and then either giving someone a nasty (or fatal) shock or starting a fire (hence why it's mandated to remove it in favour of pvc)

  7. A Non e-mouse Silver badge

    A field engineer once told me he got sent to a server at a school. As this was a school, the staff didn't want the pupils playing with the server so they built a little box around it. The engineer opened the box and found that the plastic case for the server had heated enough to soften and had sagged onto the motherboard.

    Personnally, I'm suspicious of this: Surely if the box had gotten warm enough to soften the plastic, wouldn't it have gotten hot enough to fry the chips?

    1. Anonymous Coward
      Anonymous Coward

      You'd have thought so, but it's surprising how well many ICs will operate well above their rated temps - perhaps unsurprising given the often very high process temps that silicon dies are processed at, or indeed the temps they'll be exposed to by soldering. Doesn't mean the silicon will always function as intendedoutside it's rating, but it's easily possible for an enclosed processor to push the temps up until something melts - see my little tale above about the computer in the hardened air shelter.

    2. Caver_Dave Silver badge

      I used to test milaero conduction cooled boards in an oven between -50 and +85 degrees.

      I once found the remains of a biro that had accidentally been left in the oven by the night shift. It had softened into an oval instead of round, and that was in less than 8 hours.

    3. Dave314159ggggdffsdds Silver badge

      You'd be surprised how low the softening temp of some plastics can be - well under 100c for some commonly used types.

      1. Alan Brown Silver badge

        It's inadvisable to exceed 65C in most cases (fumes) although softening is a problem for both PVC and Polyethelene at 80C, ABS is rated to 105C but I've encountered warped boxes which supposedly didn't get hotter than 80, so hotspots must factor into it

    4. Joe 59

      I had a 1U rack mountable case I built a server into in '96, the heat sink touched the lid. The lid got so hot, the paint came off the lid in a square around the footprint of the heat sink. And it only took a few minutes to do that, and the server was perfectly happy running during those few minutes. I wound up drilling holes into the lid in a pretty pattern over the heat sink, which improved airflow.

  8. Andy The Hat Silver badge

    "10Mbps powered hub" ...

    Luxury!

    We installed "modern" 10base2 with stretches of 10base5 to get extra length ... which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ...

    We even ditched the Apple2 server and its RS232 "network" ... eventually.

    1. jake Silver badge

      Thinnet.

      Did the job for small offices (and more than a few large ones) for about two decades.

      Was really cheap once NE2000 clones started showing up, and thus became nearly ubiquitous. A couple of techs could crank out a couple hundred seat network in a long weekend, given enough coffee. I must have run several hundred miles of the stuff.

      I don't miss it a bit ... but I still run it at home when it makes sense. Why not, I still have all the tools and several 2,500 foot rolls of the stuff. And I maintain a couple of legacy machine shops that use it, along with a few SCADA odds & ends.

    2. Boris the Cockroach Silver badge

      Quote

      "Luxury!

      We installed "modern" 10base2 with stretches of 10base5 to get extra length ... which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ...

      We even ditched the Apple2 server and its RS232 "network" ... eventually."

      You 'ad RS232? all we got was paper punch tape.... god that was fun when someone put "program end/rewind" on the end of the punch tape instead just "program end" wheeeeeeeeeeeeeeeeee rip rip rip rip shred rip......

      Meanwhile back on topic....

      PCs found in disgusting places..... bottom of our metal plating tank anyone? and before you ask saying I've posted that one before, its happened again... this week. oh well another trip to pc werld for a cheapo laptop.

      we need a vomit icon.

    3. Montreal Sean

      @Andy the Hat

      Wow! You had it so.much harder than us!

      At least we were allowed to keep our tongues!

      "which obviously had to be installed two hours before we got up and before we licked the road clean with out tongues ..."

  9. Doctor Syntax Silver badge

    When you hear of some of these installations you wonder about the installers. Did they not flag it up as an unsuitable installation? Maybe they'd strayed into the wrong genre - cops and robbers when they should have been in cowboys and indians.

    1. jake Silver badge

      "Did they not flag it up as an unsuitable installation?"

      Probably, yes.

      But the clueless Boss said "do it anyway, it'll be fine and I've got a quota to fill".

      Or perhaps the owner of the shop in question insisted, and was willing to sign a "get out of jail free" card for the installer.

      Etc.

      1. Doctor Syntax Silver badge

        Remember the shop was a cop shop so they ran the jail.

        The boss, of course, would be the cowboy.

  10. Dwarf

    Glass cutting

    Waaay back in my past, we had a customer with a PC that controlled a glass sheet cutting machine. The glass panes were huge and thrown around on a bed of air.

    The jobs were sent to the machine via a 3 1/2 floppy drive. The glass dust ate floppies and drives on a regular basis.

    1. Lee D Silver badge

      Re: Glass cutting

      How many replacements before you just put a simple shield or flap over the front of the floppy drive?

      1. Excused Boots

        Re: Glass cutting

        “How many replacements before you just put a simple shield or flap over the front of the floppy drive?”

        Ah, yes a question for the bean counters, possibly fitting a simple shield (or maybe glass dust being, well, glass dust, it might not be as simple as that), comes out of ‘x’ budget who are unprepared to pay for it, because it’s not their problem, whereas just replacing the drive every month comes out of ‘y’ budget. As long as everything balances, then fair enough.

        The simply fact that both ‘x’ and ‘y’ are ultimately paid for by the same company does seem to escape some organisations.

      2. Richard 12 Silver badge

        Re: Glass cutting

        Won't work, trust me.

        That stuff gets everywhere.

        Switching to something solid state would be the only real fix, and probably never an option.

      3. John Brown (no body) Silver badge
        Boffin

        Re: Glass cutting

        "How many replacements before you just put a simple shield or flap over the front of the floppy drive?"

        My "solution" at a pottery where the optical sensor on the tape drive kept getting clogged with dust and ripping the tapes off the end of the spool, was to open the PCs PSU, turn the extractor fan into an intake fan by physically reversing its orientation, and put a filter on the the new "inlet", so clean air blew out the tape/floppy drive/other open holes at the front. The filter needed changing every week but that was no longer my problem.

  11. GDM
    Facepalm

    Soggy Bottom

    I once got sent out to a site I'd never been to before to deal with some server issue. It wasn't dead, must have been something else. Despite being in a small 'where everything gets dumped' cupboard it was on something where it wouldn't get knocked off or have stuff piled on it. It was also very well cooled, by a small portable air conditioner. On a chair. One of those metal frame ones with thick foam seat and back, such that the air con unit didn't quite sit up straight. This might have been fine it it had automatic evaporation via the hot air hose (which did go out of the room) but instead the tilt meant the level sensor in the water tank didn't notice the tank getting full, overflowing so it soaked the seat foam it sat on, then dibbling down the chair leg and soaking the carpet tiles. The squelch when you walked in the room was a giveaway.

  12. DJV Silver badge

    Location?

    At a Business Link office I once had to work in, the server room was the 'Gents'.

    Admittedly, it was no longer still in use for its original purpose, and they had at least removed the, um, seats, but the cubicles were still there as the partitions were brick built.

    1. Doctor Syntax Silver badge

      Re: Location?

      It's an on attitude isn't it? These machines will house your entire business. If they go down so does your business. Collectively they represent your entire business, whatever you value that at. Where do you put them? Somewhere where they'll be treated with the respect that that value deserves or whatever small space that you'll begrudge least, however unsuitable?

      1. that one in the corner Silver badge

        Re: Location?

        > whatever small space that you'll begrudge least, however unsuitable

        >> but the cubicles were still there as the partitions were brick built.

        Cubicles, plural, with decent brick walls? At least two (possibly more, to have plural partitions)? That could be a fair amount of volume, with solid structure you can safely hang the kit on and plenty of spare wall space to neatly hang your bits and bobs.

        That sounds like a good deal more suitable than the cubicles in the rest of an average Business Link premises, the ones the human resources are supposed to live in.

        1. NITS

          Re: Location?

          Department store I've called on has their IDF located in either the shoe stockroom, or the ladies' lingerie fitting room, depending on the branch.

  13. eswan

    Had an IBM ps/2 model 30 that was in use at a gravel quarry. When I took the cover off, I couldn't see any of the chips due to the quarter inch (literally!) layer of rock dust over everything. Just kinda mounds here and there.

    1. Fido

      Around here it is not rock dust but chalk dust which clogs the computers in the classrooms used to teach mathematics and other sciences. Things have improved but the mathematicians still make a lot of dust with all their equations.

      1. CountCadaver Silver badge

        Not sure why they still cling to chalk when whiteboard pens and smart displays have been around for decades now (albeit my school replaced whiteboards with chalkboards/blackboards (to use UK parlance because.well they are black in colour) for well "reasons" likely either someone was a traditionalist, pens were seen as a luxury and a new blackboard came out of capital budget but consumables came out of another and well we buy lots of chalk so why should we.buy pens?)

  14. 43300 Silver badge

    We have a site right beside the sea, and the work carried out from there involves going on the beach regularly. Not got any servers there fortunately, but any desktops which need repair have to be de-sanded first.

    At another site they decided to re-purpose the comms room, where the main patch cabinet is, as a cleaner's store. I had to kick up quite a fuss about that one to get them to shift the mops and buckets elsewhere!

  15. xyz123 Silver badge

    I worked for a fruity trillion dollar phone company. Third party had "issues" with rain leaking into their datacentre.

    So they closed some of the toilets and put the servers in there. Hilariously they had to balance the servers on top of the toilets, as they couldn't put them on the sinks due to their weird shape, and couldn't put them on the floors as the server blocks weren't in their HUGE racks anymore.

    Door propped open,cables leading down the hall, sellotaped (not even proper electrical tape) to stop people tripping over them.

    It worked for about 4 weeks until they FINALLY agreed they needed to get the roof fixed, as each department wanted the money to come out of the others funding.

    1. Dave314159ggggdffsdds Silver badge

      "until they FINALLY agreed they needed to get the roof fixed"

      I once worked somewhere they were on the top floor of a mid-rise block, where it was owned on a share-of-freehold basis. The roof leaked, and had for years, because all the freeholders needed to agree to the work being done, but only my clients on the top floor actually suffered from the leaks. I've never seen an office with internal roofing and rain gutters leading to the drains anywhere else.

      1. Alan Brown Silver badge

        A New Zealand university had this in the Science block almost from new. It was startling to see this rig arching over a multimillion dollar xray crystallography machine

        The block was built by the Ministry of Je^H^HWorks in the early 1970s and had 2 years of snagging before it was even safe to enter (Amongst other issues the staircases weren't fixed to the building)

        This building, the Veterinary studies tower and virtually the _entire_ Teachers' College campus (later a polytechnic) had to be abandoned and were knocked down as inspections in the wake of the Christchurch and Kaikoura earthquakes revealed they were structurally unsound

        On a computing related note there were servers stuffed into a number of closets, but that was to keep them out of the sight and knowledge of central IT, due to the god complex of the director of IT (he would actively kill anything that he didn't control)

        1. Cheshire Cat
          Stop

          Kill the unauthorised servers!

          I can understand the director of IT on this one.

          Unauthorised servers on an internal (and privileged) network are a security nightmare. They are rarely kept in line with security standards, probably never backed up, or regularly patched, and cause unanticipated load on network infrastructure.

          Plus, at some point, they go wrong or are crypotolockered, and then IT are called in to help fix the shitshow caused by the office cowboys.

          In a university, which likely does not have much (if any) internal firewalling on the staff network segments, a vulnerable machine can cause chaos if it is compromised, and you can't recover what was not backed up.

      2. Brian 3

        I had an apartment like this for several years in the Colorado mountains..... it was very cheap and a nice location. Indoor guttering! All the fancy. The water would pool on the insufficiently sloped flat roof, especially in the spring melt there would be 8-12" of standing water, and it would seep through the roof and only into my apartment. For weeks at a time.

        1. Dave314159ggggdffsdds Silver badge

          Yeah, to be clear it's much more common with apartments to have shared freehold. I've seen quite a few residential blocks with leaking roofs because the freeholders on all the other floors don't want to pay to fix the roof.

        2. Norman Nescio Silver badge

          I knew someone with a flat like that in London in a large residential block. Flat roof. Years of investigations and patching trying to work out where the water was coming from - it tends to seep long distances. The person's bedroom had a funnel under the point in the ceiling where the water came in, with a drainpipe out of the window. He was a builder, so that bit worked. He and his wife were remarkably sanguine about it.

          It was only when the residential block's roof was completely replaced as part of a major refurbishment project did the problem go away. Flat roofs are the work of the Devil.

          1. John Brown (no body) Silver badge

            I winder if it would have been fixed sooner if the leak had been directed to affect the people below rather than out the window? :-)

            1. Excused Boots

              Absolutely it would have been, although you do need a BOFH level person to arrange things to work that way.

  16. Giles C Silver badge

    Seen too many of these

    Comms cabinets under toilet blocks - yes with a leak in the large pipe….

    Misc kit under floors to extend circuits - yes

    But the best was a non IT one builders knocking down some heavy duty partition walls to find someone had mounted a 40 gallon water tank on top of the false ceiling. The walls came down , the ceiling started moving and they found the tank help up by the copper pipes connecting it to the rest of the plumbing and nothing else….

  17. Marty McFly Silver badge
    Pint

    Restaurant computers...

    In the mid 1990's, I got sent to a pizza restaurant to work on their late 1980's DOS-based point-of-sale system. Repair if possible, replace if necessary. Slimline style 386 boxes. The insides looked like they were covered in a soft gray felt, with small bumps where the various chips were. Forget about operational fans. Outside of the cases were sticky with that fryer grease which permeates a restaurant and settles on all surfaces.

    They ended up being replaced with 486DX2 boxes running at 40 Mhz. They were the mainstay workhorse back in the day, complete with VESA Local Bus cards to handle serial ports, graphics, networking, and (get this) light pens.

    Some of the best pizza I ever had though. Made their sausage in-house daily and cooked it on the pie - no pre-cooked toppings. Pina Pizza in Downey, CA. Cheers!

    1. Gene Cash Silver badge

      Re: Restaurant computers...

      Hm! Google Maps says there's still a Pina Pizza House in Downey.... Rooooooaaadd TRIIIIPPPP!

    2. Tim_the_Unenchanter

      Re: Restaurant computers...

      Pina pizza is awesome! We would stay in Downey when visiting a customer in Lynwood

  18. LoPath
    Mushroom

    Make it rain!

    At an Air National Guard base, one building was given an air handler room to install the network switches for the building. One day their network went out. The air conditioner had died and the room filled with hot humid air. So humid that condensation started raining down the walls and into the wall-mounted switches.

  19. aerogems Silver badge

    Back in my youth I had a part-time job working for a department at the Uni I was attending. IIRC, I was being sent around to every workstation in the department to make sure the backup program was set up correctly after a poor grad student had their entire thesis and research lost due to a HDD crash when it turned out for reasons lost to the mists of time, the backup program wasn't actually backing anything up for them.

    I was chatting with the department secretary and beancounter, who were in the same office, while checking their systems. They warned me about a particular professor's office and how I should always make sure to let someone know I'm going in there or I could end up trapped under a pile of papers or something and might not be found for days. I figured they were just exaggerating, but then my travels finally took me to that office. It looks like they converted an old janitor's closet into a professor's office, and literally every free surface was stacked with papers 2-3ft high, or higher. There was just a small pathway to the desk where the computer was. Think like what you see in those hoarder shows, only if the hoarder was specific to only saving various papers. If anything, the nice office ladies were probably underselling things. Oh, and the keyboard and mouse were sitting atop... no prizes if you guessed "more papers." Fortunately that was the one and only time I ever had to go into that office.

  20. Bitsminer Silver badge

    Plating shop

    Back in the 1970s....

    An early $JOB involved manufacturing special purpose data-collection terminals with a keypad and a manually fed 80-column punch card reader. They sold it to factories so machine-tool operators could scan a punch card every time they started a new job. A bit of computing prestidigitation and you got accurate to-the-minute labour and machine costs for each job.

    In the theme of "dog fooding", that is, eating the same food you make for your dog, the business owners had an electro-plating shop next door. They put these terminals next to large vats of various acids used to clean the metal items before being chromium-plated.

    The terminals were typical for the day, 2-layer printed circuit boards with no particular protection layers added.

    On average they lasted about 12 months before corrosion killed them.

    1. NITS

      Pool supplies shop

      Copper networks corrode away. Been asked more than once to replace an entire cable run, including jacks and patch cords.

      I don't think that they get 3 years out of a PC, either.

  21. cornishman33

    Garages, they have a lot to answer for.

    I maintained a local M.O.T. garages computers for a couple of years, but generally only had to deal with the office equipment which was what I would call run-of-the-mill support. Though one day the manager asked me to look at the box installed in the garage, you know, the one that held all the M.O.T. data etc. It was overheating and to cut a long story short it mirrors the main story above. Opened up the box, couldn't even see the motherboard/CPU block or PSU through the compounded brake dusr and gunge. BUT I got it working after a good steam cleam (joking).

  22. 82412

    I was once called to a garage premises to look at a PC which would no longer start.

    Inside it was absolutely spotless.

    A very nice metallic bronze colour throughout.

    It had been sitting in an office next to the spray booth for years.

    1. Dave314159ggggdffsdds Silver badge

      No Fordite, then?

      https://en.wikipedia.org/wiki/Fordite

  23. Grogan Silver badge

    It's not only companies... I couldn't count the number of times I've arrived at residences to find the computer I'm to work on is crammed in a little cubby in a credenza, with the door kept closed while the computer is running. Not only that, but it takes the hands of a 10 year old to hook/unhook the connections.

    Take computer out, open case, clean dust, admonish people... lockups and blue screens stop :-)

  24. Nifty Silver badge

    ...When Sylvester opened the rack and looked at the fans, he found "a solid, unrecognizable black lump."...

    So that's where they were hiding the cannabis.

  25. PRR Bronze badge

    I degaussed a truly monochrome CRT. 1939 Allen B. DuMont oscillograph (today called oscilloscope, but he was a pioneer in several ways that didn't catch on). Found on curb, rebuilt, but could not find the dot. Finally found it by experimenting with magnets. The idle position seemed to be about 5 inches outside the edge of the screen. Found my old reel-tape demagger, Hurrrnnn!, dot was on-screen and could be centered with deflection bias.

    Strange Places:

    A 286 PC (Panasonic!) in a piano technician's workshop. Goofiness finally cured by inverting the PC (we use to run them pizza-box orientation) and thumping the metal chips out.

    Main department server (Novell Netware 3.11!) in the closet of a busy tinkle-toilet room. Gave little trouble but shocked visiting geeks and geekesses.

    Two of my server rooms had buckets under the drips in the skylights.

    But L---- took the cake. Despite repeated gentle and also loud warnings, she kept an IBM PC/XT directly under a hanging plant and watered it every day. Sometimes she over-watered. Somehow she never electrocuted herself or filled the office with cathode-ray smoke.

  26. steviebuk Silver badge

    Places

    The NHS, too cheap to move the small server room at one site that, in heavy rain, would get slightly flooded as water would come in to hallway from backdoor then finds its way under the door into the server room. So all they did was raise the servers off the floor slightly.

    The other server room had a leaking aircon. As always in my career I pick the shit places that don't listen to temps, so the 3 years I was there it was never fixed.

    Then at last place building a new swimming pool. We warned them "You'll want to put it on the roof, the server room, on the top floor cause of the sea water and humidity in the basements because of all the pool water kit. We were ignored, the architects know best. Except they fucking didn't. The room has never been sealed well enough and shortly after all the kit started to rust, I still have the photos. Its been several years later, the expense at having to replace that kit regularly must be costing a fortune.

  27. Anonymous Coward
    Anonymous Coward

    Is the Sahara Desert still there?

    I spent a few years working in a town in the Sahara Desert.

    One trip back to UK, I took my laptop to a repair shop.

    A few hours later I had a phone call: "Is the Sahara Desert still there? I think I've just typed out the entire desert from your laptop"

    1. NITS

      Re: Is the Sahara Desert still there?

      I seem to recall reading that in the days of Desert Storm there was high demand for circumcisions among GIs that hadn't had the procedure early in life, because that fine sand gets *everywhere*.

  28. Nematode

    How about "the shop floor" - the factory space where a major control systems company (now defunct) used to build, configure, test and FAT test systems. It was fine in winter, spring and autumn. However, when summer arrived, its flat roof, designed to hold a cooling pond of water across its whole width but which had been drained of water some years ago (presumably cos it leaked), duly obeyed the laws of physics, notably those of radiation, and caused the building below to warm up. By warm, I don't mean 25 or 30°C. Or even 35°C. No, a cool 40°C was a fairly usual temperature. However, it would make 45 or even 48°C for short periods. Unsurprisingly, for quite a lot of the kit this was a heat soak test too far, and some bits of kit would start falling over. I was surprised that more didn't break than actually did.

  29. NITS

    Hot server rooms

    A department store, several branches of which I have had occasion to visit, does not provide dedicated aircon to their server rooms. Rather, they see fit to install an outlet from the building's main HVAC system. Maybe, in theory, it worked OK when first installed; I'll give them the benefit of the doubt. But over time, Stuff Happens -- more equipment added to the MDF room, duct gets disconnected, system becomes unbalanced for some other reason. More than once, I've walked into one of their rooms and found it rip-off-all-your-clothes hot. I suggest to manglement that they *really* should get their Facilities folks to address the situation, and they say "Oh, it's always like that!"

  30. Agincourt and Crecy!
    Facepalm

    A p*** poor choice of site

    I worked for a local authority who had converted an old school into a site for IT and finance. They were looking for a secure room to site a couple of finance servers related to the collection of cash from all the parking ticket machines across the authority and decided that the best option was an old boys toilet. There were no windows and the walls were all tiled. It was also quite humid as there was a drain in the floor and intially there was no air conditioning. After several overheating issues, the decision was taken to install air con and things were OK for a while. Nobody really needed to go into the server room/boys toilet, backups were automated to a robot in the rack and the tapes were cycled one a week.

    There was a sudden outage one day and I was sent to investigate. When I opened up the room, it was all nice and cool, the servers were both dead and there was a large pool of water in the room. both severs were wet inside the cases and had cut out when they short circuited. The root cause was the air con and the humidity. As the air con ran, the humid air in the room started to condense and then freeze on the aircon unit. Eventually this had built up to the point where the fans were affected and stopped working. The heat then increased in the room, the ice melted and once the fans were able to spin again they did, spraying the water from the melting ice across the room and onto both servers.

    There was a rapid re-evaluation of the site and the servers were moved somehwere less humid. Once they had been suitably dried out, we were able to boot them both and were amazed to see there had been no lasting damage.

  31. gh1978

    Critters and chips

    Went to a place of learning once that we used to run. It was tiny and the chaps that had set what passed as the IT up had left and it looked like it had just been setup with the contents of someone's junk cupboard.

    Anyhow I digress. Got there was shown where the servers were (partitioned off section of a room with full window at the front, during what passed as a british summer.

    "Mind the locusts" I was told. Yeah they kept some locusts in glass tanks in there as it was nice and warm for them.

    I then somehow became responsible for the site, so not long after portable Aircon thing and window blind installed.

  32. tweell
    FAIL

    Cafeteria IT

    I was requested to design a setup for a buffet style restaurant. Cameras, wifi, doors, internal data. Easy enough, POE cameras, door actuators and WiFi hotspots, cabling above the false ceiling, VPN programmed switch and UPS in the manager's office. They picked a reputable firm to do the installation and I went on to other things.

    A few months later, I was yanked off another job to the scream of an impending lawsuit to find out why this setup was no longer working. A few minutes after getting there, it was obvious why the system no longer worked. The manager had decided that the wall rack should not go in his office, and instead had them put it in the back room. That room had their deep sink, cleaning gear, chemical storage and the very large hot water heater, and it had no ventilation. Every bit of exposed copper in the patch panel was green, the UPS had shorted out when it was doused with fluid of some kind, and I still don't know how the switch survived.

    I took pictures and told the business to pound sand, they would get laughed out of court if they sued. After I was reprimanded for harsh language and folks had some time to cool down, I was asked how what they had could be made to work for the least amount of additional money. Since the cabling as built wouldn't go to the manager's office, I stuck the rack above the back room, and rigged up some ventilation for it. We cut a few feet off each wire, patched it into a new patch panel, replaced the UPS and it was good to go. The only problem was that you needed an eight foot ladder to get to it.

  33. Anonymous Coward
    Anonymous Coward

    Flooded server room

    For my first job out of uni I worked for a pipeline company. They had two control rooms and two server rooms, one pair downtown, and one pair at their tank farm. Each was provisioned with custom control consoles and a PDP-10 mainframe to provide hot backup for controlling the entire pipeline system. Most of the time, the downtown site was live, with operators in the server room 24/7/365. The tank farm had operators during the day shift and on call at other times.

    One day the tank farm operators came on shift to find an odd smell in the server room. After a bit of investigation, they lifted a tile from the raised floor. There they found a few inches of natural-gas condensate had flooded the room. (See e.g. https://en.wikipedia.org/wiki/Natural-gas_condensate) Needless to say, condensate is very volatile and flammable. It is not a good idea to submerge your server cabling in it.

    Fortunately, the leak was fixed and the server room was drained safely.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like