back to article It's alive! Farmer hides neglected, dust-clogged server between walls

Welcome again to On-Call, our regular look at the messes readers find themselves confronting when asked to go to help out their clients. This week, reader Matt brings us a tale from the time “I was working in a tiny system-integrator firm that has some big clients.” One of those clients was a substantial farm and food …

  1. chivo243 Silver badge

    it's early but

    it's a farm server! I get it. Thought I was reading it wrong.

  2. P. Lee

    "The farm decided to go with a more modern, off-the-shelf software solution."

    I hope they don't expect the same longevity and robustness from the "new and improved" system.

    1. Ken Hagan Gold badge

      Re: "The farm decided to go with a more modern, off-the-shelf software solution."

      In terms of the hardware, it is surely now possible to provide the same processing grunt in a box that requires *no* air cooling (except for exposing the case to room temperature).

      I find it "surprising" that we still build, and buy, machines that require the circulation of clean air to keep them running, knowing full well that they will spend their entire working lives in a dusty environment, or worse. If you are a gamer -- fine, it's your choice and you know how to take care of your machine. If you are operating a server farm -- ditto. For just about everyone else, whether it be a laptop or a traditional desktop, just ditch the bloatware and run on a fanless, sealed, system that still has more power than money could actually buy just a few years ago.

      1. MacroRodent

        Re: "The farm decided to go with a more modern, off-the-shelf software solution."

        In terms of the hardware, it is surely now possible to provide the same processing grunt in a box that requires *no* air cooling (except for exposing the case to room temperature).

        True, any modern smartphone has more processing power than that Pentium server. However, implementing the same functionality using modern languages and software engineering methodology would result in a much slower system, unless you go for a bigger system ... that requires cooling.

        It is like booting Windows: for the past 30 years it has seemed to take about the same time, regardless of the increase in the power of the computer I run it on! What Intel giveth, Microsoft taketh away.

        1. Phil Endecott

          Re: "The farm decided to go with a more modern, off-the-shelf software solution."

          > It is like booting Windows: for the past 30 years it has seemed to take about

          > the same time, regardless of the increase in the power of the computer I run

          > it on!

          In other news this week, all animals take about 21 seconds to piss irrespective of their size.

          Must be related somehow.

          1. Anthony Hegedus Silver badge

            Re: "The farm decided to go with a more modern, off-the-shelf software solution."

            "In other news this week, all animals take about 21 seconds to piss irrespective of their size." - not everyone, apparently <starting at penis telling it to hurry up he has work to do>

            1. TonyJ

              Re: "The farm decided to go with a more modern, off-the-shelf software solution."

              "In other news this week, all animals take about 21 seconds to piss irrespective of their size." - not everyone, apparently <starting at penis telling it to hurry up he has work to do>

              Your penis has work to do???

          2. Dan 10

            Re: "The farm decided to go with a more modern, off-the-shelf software solution."

            Really? I need the loo - I'm going to count.

        2. Gene Cash Silver badge

          Re: "The farm decided to go with a more modern, off-the-shelf software solution."

          Not really. There's water cooling now. I have an i7-3770 with 32GB and an SSD with water cooling on the CPU & chipset. I have a large radiator and it works fine without any fans for normal use.

          Only when I fire up KSP do I switch on the fans. The nvidia card (960) also has fans which automatically stay off except for gaming.

        3. Charles Manning

          Re: "The farm decided to go with a more modern, off-the-shelf software solution."

          I have had some experience working in agricultural computing, both in-cab and in-shed.

          Dust is a real killer, but these days it is trivial to set up a box with some sort of ARM CPU that does not need any ventilation.

          Storage is the only real bastard. An RPi or such depends on SD Card storage and I would never recommend that. There are, however, plenty of boards that will run more robust storage - even SSDs and not need any cooling.

          These will often cost no more than a PC - often less - and will provide the sort of robustness a PC can't.

          Of course it means you're running Linux and not WIndows.

          1. sisk

            Re: "The farm decided to go with a more modern, off-the-shelf software solution."

            An RPi or such depends on SD Card storage and I would never recommend that.

            You could run the OS itself off an SD card and mount a USB SSD for the /home, /var, and /tmp directories. That would solve the problem of write fatigue on the SD card nicely and still give you a ridiculously small system that requires no real effort to cool. But of course, as you said, that means you're on Linux. You CAN run Windows 10 on a RPi2, but it's not a particularly functional build of Windows 10.

            1. This post has been deleted by its author

        4. Anonymous Coward
          Anonymous Coward

          Re: "The farm decided to go with a more modern, off-the-shelf software solution."

          "It is like booting Windows: for the past 30 years it has seemed to take about the same time, regardless of the increase in the power of the computer I run it on! "

          That's not my experience. A current laptop / desktop with Windows 10 boots in about 5 seconds.

          1. Rich 11

            Re: "The farm decided to go with a more modern, off-the-shelf software solution."

            That's not my experience. A current laptop / desktop with Windows 10 boots in about 5 seconds.

            It'd be faster still if it didn't waste three seconds sending your browsing history to Microsoft.

            1. Pookietoo

              Re: Windows 10 boots in about 5 seconds

              It's not booting, it's resuming from suspend to disk.

              1. Anonymous Coward
                Anonymous Coward

                Re: Windows 10 boots in about 5 seconds

                Here? It's booting. Suspend and Windows, while it seems ideal, Can have real-world consequences. It still only requires five seconds.

          2. John Brown (no body) Silver badge

            Re: "The farm decided to go with a more modern, off-the-shelf software solution."

            "That's not my experience. A current laptop / desktop with Windows 10 boots in about 5 seconds."

            You sure? The POST usually take longer than that, before the BIOS/UEFI even goes looking for a boot block, never mind the OS actually booting up. Then there's the windows log-in screen appearing before the boot has completed because MS know you will take time to notice it's there and type in your details so they can claim a fast boot time. And even then, it's probably not fully usable for another minute or so while all the auto-update programs phone home and "helper" programs load up.

            TL;DR. I call bullshit.

      2. Havin_it

        Re: "The farm decided to go with a more modern, off-the-shelf software solution."

        Mm. It's almost as if the PC industry *wanted* your box to die of heatstroke long before it would otherwise have needed replacing ...

        Also consider the Dell et al will sell you a "shitty-environment-proof ruggedised enterprise workstation" jobbie for a hefty premium if you're so inclined.

        1. Anonymous Coward

          Re: "The farm decided to go with a more modern, off-the-shelf software solution."

          It's actually not that hard to buy (or build) fan-less systems, then you can run a stripped back OS and your good to go.

          After all, haven't seen many smart phones with 5" fans on the back.

  3. Anonymous Coward
    Anonymous Coward

    Simply denying servers are necessary is an interesting approach. I still remember visiting a customer on-site who wanted solutions for his data analysis.

    Visiting the employee involved I noticed that his network-card was half hanging out of his PC (which was still working) and when I asked the employee if it had fallen and pulled out the card he indicated that that was not what had happened. He was annoyed about warnings and messages that he did not need and had lifted the desktop and given it a good jerk, but it had replaced the old messages with newer messages he had not seen before.

    After consultation with the boss we did not offer to provide solutions.

    1. elDog

      Funny - giving the old desktop a random drop has frequently solved problems

      You know, redistributed dusty bunnies or reseated a board or CPU.

      Then you hear about frustrated users pulling out their concealed weapon and plugging the beast. Perhaps just waving it around would be enough to make it start to behave...

      1. Richard Altmann

        Re: Funny - giving the old desktop a random drop has frequently solved problems

        Step 1: Push it

        Step 2: Kick it

        Step 3: Drop it

        Still not operating? There might be a failure then.

        1. Anonymous Custard

          Re: Funny - giving the old desktop a random drop has frequently solved problems

          Ah the joys of percussive maintenance.

          The intermediate step between swearing by the machine and swearing at it...

        2. Anonymous Coward
          Anonymous Coward

          Daft Punk's Guide to Computer Repair

          "Step 1: Push it

          Step 2: Kick it

          Step 3: Drop it"

          Step 4: Smash it.

          Technologic. Technologic.

          1. Richard Altmann

            Re: Daft Punk's Guide to Computer Repair

            You don´t know anything about technology, don´t you?

  4. Sokolik

    I am Evil and Must be Destroyed!

    Sounds like something my late father would have done.

    I spent (and, I do mean, *spent*) my high-school years on his small farm.

  5. Andy Tunnah


    I was expecting it to be covered in poop

    1. king of foo

      Re: Disappointed

      ...or hatching ickle chickens

  6. gregthecanuck

    On failure...

    All in all it was just a brick in the wall....

    1. choleric

      Re: On failure...

      "Wonderwall" by Oasis

      Anything by "Living in a Box"

      "Sweet dreams are made of these" by The Eurythmics, but with "Green screens" instead of "Sweet dreams"

      "West Country Boy" by The Shirehorses (that's a bit of a stretch, but California is in the west...)

      1. sisk

        Re: On failure...

        "Sweet dreams are made of these" by The Eurythmics, but with "Green screens" instead of "Sweet dreams"

        And that'll be stuck in my head for the rest of the day...I'm sure I'll get some funny looks later when someone overhears me singing "Green screens are made of these..."

    2. Steve Knox

      Re: On failure...

      We don't need no ventilation

      We don't need no temp control

      No circulation in the wall space

      Farmers leave that box alone

      Hey farmer leave that box alone!

      1. Steve Davies 3 Silver badge

        Re: On failure...

        you forgot two essential lines

        Welcomes to the Machine


        Careful with that Axe Farmer Eugene

      2. Fibbles

        Re: On failure...

        Why does this have a down-vote?

        Cheer up you miserable bugger.

  7. Lee D Silver badge

    Work in a school that had a network switch buried in a radiator cabinet inside some antique wood panelling, with power coming from underneath the floor. The switch ran all the offices. When it went off, it was an... interesting... exercise in cable-chasing to find the problem.

    Replaced it with one a bit more visible, a bit faster (10/100 doesn't cut it nowadays), but still need to re-run all the various cables that come off it.

    1. Voland's right hand Silver badge

      This is going to happen more often now

      Well, in those days you had to provide cooling for the server. Nowdays, a RasPi, something running OpenWRT or MIPS Creator can happily live without any active cooling. So frankly, I would expect more of that in let's say 5 years time.

      Buying a house from a geek? What controls the lights and central heating? Ooops, you may need to get that laminate flooring on the fist floor all up and replace it tracing cables because the fecking bastard has not left a cable map. Not to mention that there is no source either so you have no clue how it worked once you find that RasPi with a duff memory card somewhere in the ceiling void between the ground and the first floor which for reasons unknown to you happens to be under the floorboards in the toilet (Disclaimer - this is imagining what will happen if I sell my house :)

      1. Roland6 Silver badge

        Re: This is going to happen more often now

        >Buying a house from a geek? What controls the lights and central heating?

        Well either the lava lamp your partner doesn't like and doesn't understand why the geek is leaving it behind, or the phone in the geek's pocket (the joy of 'cloud' computing)...

        >which for reasons unknown to you happens to be under the floorboards in the toilet

        Not visited many new builds? it seems to be surprisingly common for the main fuse box to be located directly below an upstairs toilet - I've got two spare breaker slots in my fusebox but without doing a bathroom refit (or cutting a hole in the downstairs ceiling) they ain't going to be used...

    2. Alan Brown Silver badge

      "Replaced it with one a bit more visible, a bit faster (10/100 doesn't cut it nowadays), "

      A lot of the schools I was dealing with would use 10Mb hubs everywhere and then wonder why the network would go titsup when a classrom would simultaneously start their networked MS office for the lesson.

      "But it worked when we tested it"

  8. elDog

    And I was guilty of leaving a laptop stuck in an attic used as a router

    Only problem is that when I went to sell the townhouse, the couple who bought it worked for an agency in Langley, VA. On a security sweep they were not too happy to find that the laptop was still forwarding packets...

    1. Anonymous Coward
      Anonymous Coward

      Re: elDog: And I was guilty of leaving a laptop stuck in an attic used as a router

      Since you're still here, I'm guessing you're not of Chinese or Russian descent then... ;)

  9. Anonymous Coward
    Anonymous Coward

    Seen it too

    At a school. The server was old enough to take slot processors. When it went in the school had no server room, and nowhere safe to store the server - so they removed a panel from a false wall, put the server in, and screwed the panel back in. Then forgot all about it until about fifteen years later when it was discovered, long-disused and powered-off but still hooked up, during a network audit.

    1. Destroy All Monsters Silver badge

      Re: Seen it too

      > old enough

      > slot processors.

      > 15 years later

      Hold on, do you you mean those recently released Intel Pentium 3 cards? But that was just last .... FUUUUUUUU.

      1. Sgt_Oddball

        Re: Seen it too

        Lets not be hasty, it could have been a mythical AMD 'Slot A' Athlon.

        1. sisk

          Re: Seen it too

          I actually had a Slot A Athlon at one point. I can confirm they existed, though not where I got it or why I had it since to my knowledge I never had a matching motherboard.

          1. Steven Raith

            Re: Seen it too

            My mate had a Slot A Athlon of some ilk, we thought it was wicked cool.

            He ran Red Hat on it (back when you bought it off the shelf in a box? Remember that?).

            I think he eventually put Windows on it to play GTA though.

            That is all

            Steven R

        2. Maelstorm Bronze badge

          Re: Seen it too

          Watch it. We still have a AMD Athlon 700 MHz Slot A computer that is still running. Rock solid hardware even though it's severely dated. Nowadays, it's running Linux and is being used as a test bed for software builds before deployment onto the network. It's had a couple of new HDDs over the past 15 years.

  10. Mark Allen

    Paving Slab Construction

    Visited a client working in a big old farmer's barn. Making paving slabs, kerb stones and similar. At the other end of this barn was the office. No door. When opening up the PC case there was an inch deep heap of fine concrete dust on top of every surface. Hate to think how much of that had got inside the hard disk breath hole as this stuff was really fine grained.

    1. Linker3000

      Re: Paving Slab Construction

      I had a similar experience with some of the computers at a cement factory in Wellingborough. They were pretty wrecked.

      One of my engineers serviced a machine that came back from a local farm's milking shed. The machine needed a complete clean out and the floppy disk drive was replaced. The engineer wrote up the repair description as "Half a field removed from computer".

      1. harmjschoonhoven

        Re: Paving Slab Construction

        No cement dust here. I can only mention my neighbours PC full of perfumed cat hair.

        1. Trixr

          Re: Paving Slab Construction

          The best one I encountered along those lines was the guy who smoked heavily and had a Persian cat. Yes, mate, your computer was overheating within 5 minutes of boot because of all the FELTED cat fur bonded with the cigarette tar. I literally peeled it out in layers (it came away nice and clean).

          1. Danny 14

            Re: Paving Slab Construction

            Ive seen similar cigarette/cat/dog fur combinations. The sheets look like loft insulation, if it wasn't so icky and gooey it would be quite neat.

          2. SImon Hobson Bronze badge

            Re: Paving Slab Construction

            I see your tar and fur (we've had one like that in at work, my advice was to refuse to touch it), and raise you ...


            ... wait for it ...

            ... hydrogen sulphide

            Highly corrosive and highly toxic, but present in the atmosphere at the (now long gone) local cellophane factory - if the wind was in the right direction, you could smell it from 10 miles away !. They had some Apple II systems in there doing some mundane monitoring of something - they'd run for years and then stop. No point trying to repair them as the legs of the chips corroded away - by the time one was gone completely through (hence the machine stopping), the rest of the legs had naff all left either (not enough to pull them out of what was left of the sockets).

    2. Richard Altmann

      Re: Paving Slab Construction

      Don´t talk to me about grain mill infrastructure

    3. Nolveys

      Re: Paving Slab Construction

      I encountered a few similar situations in the mid '90s while doing the field circus thing for a little independent out in the boonies.

      This guy had a wood shop in a make-shift shack on the side of a hill in the middle of the bush. No attempt had even been made to flatten out the dirt floor. His computer was located under a saw. When we opened it up we found it was full of sawdust along with a family of dead, desiccated mice that were huddled around the CPU.

      The other one was the computer from the chain smoking office. *Everyone* in there was smoking and in the time we were there we watched three people light cigarettes using the last cigarette they had been smoking. We the dead computer they had called us out for back to the shop. When we opened it everything was covered with about 1mm of a greyish-yellow paste-like residue.

  11. Anonymous Coward
    Anonymous Coward

    H.P. BOFHcraft: "The HP Servers in the Wall"

    On July 22 occurred the first incident which, though lightly dismissed at the time, takes on a preternatural significance in relation to later events. It was so simple as to be almost negligible, and could not possibly have been noticed under the circumstances; for it must be recalled that since I was in an office redone and newly partitioned except for the old drywalls, and surrounded by a well-balanced staff of yuppies, apprehension would have been absurd despite the locality. What I afterward remembered is merely this—that my most trusted PFY, whose moods I know so well, was undoubtedly alert and anxious to an extent wholly out of keeping with his natural character. He roved from room to room, restless and disturbed, and sniffed constantly about the walls which formed part of the old office structure. I realise how trite this sounds—like the inevitable dog in the ghost story, which always growls before his master sees the sheeted figure—yet I cannot consistently suppress it.

    1. hplasm
      Thumb Up

      Re: H.P. BOFHcraft: "The HP Servers in the Wall"

      This does, of course, demand a book...

      1. Anonymous Coward
        Anonymous Coward

        Re: H.P. BOFHcraft: "The HP Servers in the Wall"

        This does, of course, demand a book...

        In a story collection together with "Herp-Derp Jest, Project Reanimator", "The Call of Gantt-ulu" and "At the Meetings of Baldness".

    2. Anonymous Coward
      Anonymous Coward

      Re: H.P. BOFHcraft: "The HP Servers in the Wall"

      Your name is Charlie Stross and I claim my £10.

  12. Frumious Bandersnatch

    something similar

    Was called in to a factory to look at a PC doing CNC duties (probably not directly) for cutting up various wooden sheets to size. The PC was in the same area as all the cutting equipment and it was apparently being controlled by software stored on a floppy disk. The whole thing apparently had no IPS rating. Needless to say, I had to tell them I couldn't fix it.

  13. DropBear

    For extra LOLs you make sure that a cable just like the one going into the walled section comes out on the other side, only to end chopped off and frayed in the air. BOFHs would also tie the two cables together inside the wall with something, so if you tug on one of them, the other moves too...

    1. Lee D Silver badge

      How about finding a whole bunch of cables connected in two different locations. And then eventually tracing to find that - in the middle of the building, deep in the rat's nest of trunking and piping and tubing and other things running through the corridors, someone had removed a 6-foot-section of the bundle, without explanation, but left everything else of the 100+ metre run there, with both ends connected and patched in.

      That was a fun one.

      The other one was finding that in various lofts, wall-panels and other assorted places, loads of Cat5e couplers were in place, unlabelled, joining various cables to various other cables. Cable in, cable out, connectivity, but you can't pull the cables out without finding all the locations. Probably explains the first, actually.

      1. Destroy All Monsters Silver badge

        Are you SURE this wasn't your stint on a dark & gothic generation ship?

        Maybe you are in a VR now, thinking you are posting to El Reg?

      2. Anonymous Coward
        Anonymous Coward

        Could have been worse.

        Could have been chocboxes rather than couplers. just don't ask.

        1. Steve Cooper

          Ahh I've seen that too!! It's standard Ethernet's fault for being so tolerant...

  14. Diodelogic

    Not quite a farm

    Back in the late 70s, I got a call one morning from a local computer store asking me if I would mind stopping-by that day. Seriously intrigued, I said yes and proceeded to the store, wondering the whole time what on earth they could possibly want of me. As it turned out, they had a customer running an S100-bus computer who was having an odd problem, and asked me to take a look at it (apparently I was the only person in the entire area with any experience of S100 computers), for a fee, of course. I merrily proceeded on my way and found the company--a feed-and-supply warehouse business. I was shown to the "computer room" and the manager rather forlornly explained that their 8 inch floppy disks were constantly losing all of their data. At first all I could think of was that the drive itself might be defective, but I did some quick write/reads and nothing bad happened. Then there was a huge BANG from behind the wall where the computer was situated. After pulling my skin back on, I asked the manager what had happened. He showed me: Behind the wall was a 1000 horsepower electric motor driving a grain elevator screw. The BANG was from the motor starting up and engaging the screw. Back inside the computer room, I mentioned that the magnetic field from the motor must be something fierce. The manager laughed, picked up a two-foot wrench, and stuck it to the wall while the motor was still running. Moving the computer and floppy drive to the other side of the room solved the problem of the disappearing data.

    1. Nick Ryan Silver badge

      Re: Not quite a farm

      Similar-ish: We had in a PC on a site where every hard disk started to fail randomly losing or corrupting data until they wouldn't boot. We replaced the system a couple of times, the second time we had a two week long burn in PC that suffered no problems at all until it went onto site...

      Turned out that the PC was leaning against a pillar that contained the conduit in which ran the multi-phase power for the entire complex. We moved the PC 2m away and never had the same problem again.

  15. Linker3000

    Sssh...Top Secret

    In the early 1990s, I turned up at a Ministry of Defence (MoD) site in the UK to load some software updates onto a Netware server. Having duly passed through all the security checks, I was led to a busy office that looked like it was last refurbished in the 1950s.. "It's in the corner..' I was told. After about half a minute of trying to locate aforementioned server, I asked for some guidance..."Oh, filing cabinet - bottom drawer.. I'll remove the padlock." The padlock came off and a solid steel strap that ran from top to bottom through all the handles was withdrawn like Excalibur from the stone. There it was - a Toshiba T3200 with orange plasma screen running Netware and hosting who-knows-what secrets on its hard disk. A hole had been cut out of the back of the cabinet to poke through the power and data cables. "Most secure server in the building, I was advised".

    So I got to work, crouched down in front of the filing cabinet. About 10 minutes later, a phone started to ring in the next drawer up - a muted bell, it's ring deadened by something..a war surplus sock maybe!? "There's a phone ringing in here", I volunteered. "Oh,we don't answer that one", came the reply.

    Funny day.

    1. elDog

      Re: Sssh...Top Secret

      I've seen similar things in the bowels of the pentagon. Equipment plugged in with cables running through the lead and concrete walls to somewhere. No one could take the chance of yanking or cycling power since some type of Armageddon might ensue.

  16. Richard Altmann


    The management of the IT- system in the company i mainly work for has been outsourced to a new "IT" company. The system was based on the now no longer supported "windoof server whatever version". So, against my strong vocal advice to make the switch to a Linux based system they decided to go for yet another Windows system which will be no longer supported in 4 or 5 years. So it was new servers, migrating the rental and sales database, xchange ... I went on leave for two month during this period. Last week i came back to find that on every keyboard, nice and tidy, had been placed a P-Touch sticker with Username and Password. The Password being: 1234 for every account. I´m not shitting you. Getting my coat.

  17. Anonymous Coward
    Anonymous Coward


    I know a chap that has a nas box buried in with a septic tank in a field with the only clue being the power consumption from a power cable that runs to a barn. that has been split underground.

    Not sure about the data stored but he said that if the government wanted to see his private information then they can come and dig through $-hit to get it.

    The genius part was the NAS was managed over powerline ethernet.

    Unfortunately I was called out as it had a faulty disk and no sensible way to service it.

    He bought a replacement disk and I left him instructions on how to swap it.

    Don't think that has happened yet.

  18. Steven Raith

    Not quite as bad....

    ...was a site I used to look after, which does farm machinery repair. That's only tangentially related, as the building was quite old itself, stretching back to at least the 19th century, with the rest of the site built around it. Very cute.

    Server lived upstairs. In the attic. Which was floorboarded, in a simple manner, probably in the 40s - proper oak floors, but no real insulation, so it got warm in summer, cold in winter, and wasn't hugely secure, but certainly less obvious than some installs I've seen. Nice little 24u wheelie rack, with rackmount servers in it to run the Unix based stock control system. It had a very good security system.

    Spiders. Hundreds of them, all around the attic. All seemingly giant house spiders. Some bloody monsters in there, could span a CD.

    Apparently only one member of staff dared change the backup tapes. I converted them to USB disks so they could do it faster and for other reasons that I won't bore you with.

    Not surprised, though. The spiders seemed to keep out of the racking, so no real risk, but it was quite surreal having this set of rolly racks living behind it's very own arachnoid army.

    I suggested a NAS in the office to take the backups, which could be backed up offsite (IE backup the backup NAS to USB, take it home) but the arachnophobia wasn't enough to justify the cost - which is fair enough.

    Notes added to documentation: "Not a bad setup, but if you're arachnaphobic, send someone else; don't ask, because you really don't want to know"

    Steven R

    1. TRT Silver badge

      Re: Not quite as bad....

      Was it a web server?

      1. Steven Raith

        Re: Not quite as bad....

        You know, in the three or four years since finding that place, that pun never struck me (not sarcasm - I literally facepalmed when I read that).

        Probably because it isn't a webserver, or I'd have come across it myself.

        Ho-hum, award yourself a biscuit for spotting something I should have noted within seconds, given my normal 'dad-humour' punnage levels :-)

        Steven R

  19. Anonymous Custard

    Pi in the sky

    Fun reading all the comments (and the article itself of course) but it does make me think of the future, and the current trend for the ickle computers like the Pi's and Odroids. Given how much fun people seem to have had with these big ol' servers behind walls etc, combined with how small and simple Pi's are, and quite where they could get hidden (especially when run headless)...

    Personally I've already got 2 of them, although they are only mildly tucked away behind the TV (the media player Kodi Pi) and in the video cabinet (the headless server Pi).

    By the way - am I the only one who thought some of this sounded like one of Dabbsie's columns?

  20. wardster

    Dust bunnies? Boring.

    When I took over a IT management job for a furniture production company many years ago there was a PC running some custom software in a spray booth - this ran a zippy little motor system for spraying enamel paint onto steel frames for chairs. It was just a little old 386 happily singing away on its own, but as I had been informed that there was no backup, I thought, ok, lets get one sorted and clone the HDD.

    I couldn't actually remove the case due to the amount of paint that had covered it over the years - and when I eventually managed to dremel the bloody thing apart, the entire mainboard was, well, covered in enamel paint about 3 mm thick. It was impossible to see any markings on ANYTHING whatsoever, and even the IDE cable was rigid due to the covering of minuscule paint particles over the years.

    So, challenge accepted, I eventually managed to get the IDE off the HDD by liberal amounts of trichlorethylene. WIN!

    And yes, I did get a backup, and yes, it carried on in service for another 4 years before the painting was outsourced.

  21. Mike Flex

    Oooh, I haven't seen a walled-up server story for yonks and now we have a whole comments section full of them. Of course, back in the day, all these hidden servers used to run Netware rather than Unix.

  22. Gordon 11

    >20 years ago I set up 6 Sun IPC boxes (headless) as DNS/NIS servers. We had 3 sites, and one went to the Server room and the other to the Comms room on each site.

    6 years and 3 re-organizations later I was no longer in charge of these, but got a 'phone call.

    We want to decommission dns2.


    Where is it?

    In the Comms room.

    That's what we thought - we can't find it. Could you come and have a look?

    The Comms room consisted of lots of things on deep shelving accessible from both sides. We eventually discovered that other systems had been placed in front of, behind and to each side of it. It was hidden in the middle and had been for several years (they had been looking for it before...)

    When one of the others was decommissioned I requested a reprieve of 2 days - just so I could type "uptime" and see it show 1000 days.

  23. Allan George Dyer

    Missed the important question...

    What was the uptime?

    And can someone compile a table of uptime, system, software, location and additional hazards?

  24. Dale 3


    Just be thankful they hadn't invented WiFi when it was installed (presumably), or you'd likely still be looking for it.

  25. Dan McIntyre

    Loving reading these. There's surely inspiration for several BOFH episodes in this comment thread alone...

    1. Anonymous Custard

      BoFH, Dabbs, Worstall, Bong...

      Hell there's enough inspiration here for a good few Friday afternoons and beyond...

  26. Anonymous Coward
    Anonymous Coward

    Worked for a specialist vehicle manufacturer

    And some machines had to be on the factory floor.

    Fine metallic particles formed the dust there from cutting, welding and shaping metal. Also makes a great screen visor as even on full brightness some of the screens were nearly invisible.

    This has a much more interesting effect than run of the mill dirt.

    I also noted that large vacuum packaging machines would disrupt local power supplies or emit RF interference just enough to crash PC's running nearby or disrupt printing in progress. That was both hard to diagnose and impossible to fix with the usual reinstall software or screwdriver solutions I was able to make at the time.

  27. 404

    About 8 years ago

    Shop I was (temporarily - place went out of business) working at received a small footprint Dell running XP that had 'simply quit'. I'm looking at this thing, wondering what/where all this brown stuff was oozing out from, popped open the case that was seemingly glued together with the brown stuff. The entire inside was coated in the same brown stuff and here I am wondering WTF? Didn't realize what it was until my hands started shaking, then my arms, which brought out another WTF is going on here?

    Turned out the customer was a heavy chain smoker and the brown stuff?


    Got cleaned up, put some gloves on, put it back together and placed the damn thing in a hazardous materials bag. Nope, Nope, NOPE!

    1. Anonymous Coward
      Anonymous Coward

      Re: About 8 years ago

      More likely tar, but yeah, seen that in iMacs and Macbooks. Those thin, sleek airways block up with tar and fur much quicker.

      Once got a tower PC in that was held together with parcel tape and was truly manking - tar, dusty bunnies, dead spiders inside. As the only smoker in the office, I was assigned that job. Fine.

      I got special permission to take it outside, and clean it out straight into a bin, hoover up the dead insects, replace the PSU (fuck leaving it in there, the tar alone could probably bridge the mains/DC insulators) PAT test it, get some IPA on the mobo to clean it up the various heatsinks and main contact points, and finally give Windows a tickle to make it happier. All while smoking a few rollies, as the smell of ciggies was *never* coming out of that machine, not unless you stripped it and jetwashed the fucker.

      With petrol.

      The customers not noticing that the machine no longer reeked quite as bad, and had silver highlights on black, rather than light brown and dark brown, was rather annoying. Even back when I was a heavy smoker, I noticed tar stains on things FFS.

      After that point, I advised My People that I'd be declining such jobs in future. I've minimal self respect, but Christ on a bike, I've got *some*.

      Anon, because that story will ring a major bell for a few people, and I like my plausible deniability - but I'm pretty sure everyone who has worked in home computer repair has seen 'that' computer at some point in their career.

  28. David_H

    Pop charts

    Long ago I looked after the terminals used to gather the UK record charts (while they were still independently compiled). The barcode wands were regularly returned dented from being used as drum sticks, the integrated keyboards (Epson PX4's) often smelt of spilt Coca-Cola and dope, but the worst was one that "had just stopped working."

    Brown a sticky, with an ooze of semi-liquid coming from the air vents. For some reason (!) I was suspicious and rang the shop. The answer machine message said "closed due to the flood from the sewer!"

  29. Kubla Cant


    the green screen was just a screen, for a VT100 terminal

    AFAIK, the VT100 only ever had a black-and-white screen. If the screen was green it was either a VT100-compatible or possibly a VT220.

    1. elDog

      Re: VT100

      Ah, jeez. You had to go and inject some reality in here. The rest of us were just getting up to speed.

    2. Roland6 Silver badge

      Re: VT100

      cough cough!

      Google is your friend - I remember using green screen VT100's and orange screen VT220's

  30. uncle sjohie

    Until a few years ago, we had our own chemical lab to do the testing of soil and water samples our field crews collected. We had to renew those off white Compaq cases in the xylene/toluene lab every two years, because the air inside the room ate away at the plastics, and corroded the metals in the connectors. the people working in those rooms had full respirator masks of course.

  31. David Neil

    Sausage skinf factory

    Did a support contract for a large maker of sausage skins on the outskirts of Glasgow. Cow hides in one end, casings and the like out the other.

    PC on the shop floor, covered in a film of collagen, opening the case was like watching a egg pod in Aliens. They replaced 8 of these all over the shop floor with new PCs, I sometimes spare a thought for the poor sod who has to maintain them now

  32. Joe User

    This sort of thing is old news

    Missing Novell server discovered after four years

    12 Apr 2001

    A uni near me walled up a NetWare server and then tracked it down via a cable that disappeared into the wall.

  33. GX5000

    Is it dead Jim ?

    Got asked to look at server rack, more rather it was a rushed installation in a funeral parlor.

    When we got to the corridor they opened a locked closet the size of a dumb waiter.

    One App server under an Applications, cubed up like a mini Vault.

    The warm breeze that escaped that small space brought forth many pungent smells I will not go into here. Their Router was just thrown in with the default password and the wifi was free and easy.

    I explained that servers needed a little more room and air conditioning in the summer.

    They scoffed, but ended moving the lot to the back of the owner's office, which did indeed have AC.

    I never did go back but charged them extra from burning myself off the damn things.

  34. ckm5

    US gov't web server....

    In 1995-96 I built a site for serving out some aggregate data from a US gov't agency. It was built on Linux and people would upload data to it to be available to the public. I left the government contractor in the late 90's to move out West and the server was still running.

    It turns out that it ran for another 10 years, under a desk. It finally died when a maintenance worker pulled the plug and it wouldn't restart. Funny enough, they are still using the same UI I designed in 1995, but it is now running on Windows....

  35. noominy.noom

    I've seen this exact tale in an IBM marketing piece. IBM used to collect stories about the AS400 and decided to publish a few. The one like this farm server was identical except it was a a bowling alley that had been remodeled several times. Presumably this server was boxed in by drywall but still running the business software. These stories appeared in IBM literature in the early 2000s time frame.

  36. lotus49

    A UK financial institution...

    ...I did some work for as a consultant had a similar experience. There was a server running OS/2 (this was a few years ago but even then it was old) but no-one knew where it was. It ran a really specialised piece of software that had performed its function perfectly for years so it was left to its own devices.

    Eventually we decided to find it for DR purposes and we had to work out which ethernet cable belonged to it and follow it back. This server was also in a wall void although it wasn't very dusty so it wasn't too filthy. We moved it and shortly afterwards, it was replaced. I wish we'd left it where it was and it could have been a sort of computing time capsule.

  37. Robert D Bank

    with early IBM mainframes we'd also have 'issues'. They were to big to hide but still some interesting things would happen despite being in a controlled environment.

    First one I worked on in 1976 was a 360/40 (48k main memory) for a well known soap manufacturer. As an operator one duty was to vacumn the computer room, but as I learnt the hard way, don't fire up the vacumn within about 10ft or the CPU would die. Literally a whole panel of red checklights (no screens in those days, only a golf-ball typewriter). Same machine, which had carbon core memory and valves, no silicon, had some sort of pressured air cooling. Sitting there one night shift and this hissing noise starts, quickly followed by the check lights all going red and it dying. Searched around, opened a side panel and tracked down this square metal block, some sort of air valve had come off a plastic tube which continued hissing. Managed to push it back on to the plastic tube, set the dials to the boot address and it booted (IMPL'd) fine. This happened more than once..couldn't fess up to the engineers what I'd done. Eventually had to wind some wire around it to keep it in place. Also had to replace hammer fuses on the impact printer at times because we'd print 6 part lineflow which would just kill the printer. It was that or wait 2hrs for the engineer to arrive. No contest, despite the dangers of high voltage power supplies.

    Next one was a popular phonograhic company (look it up). Sitting in the machine room one night with the trusty IBM 370/125 and next thing within seconds the entire machine room is filled with choking white smoke. Didn't even have time to power it off. Turns out it was a large capacitor had decided to burn out. Hate to think of how many PCB's I consumed from that smoke. Same day had to swap one of the 3340 hard-drives (they're look like the starship enterprise), but the switch that prevents you opening the cabinet while the disk is still spinning (think 16-17inch disk) wasn't working, so I proceeded to lift the sealed unit disk off the drive and it just takes off due to the gyroscopic effect. Nearly broke my wrist before I could get it on the floor and hold it down 'til it stopped spinning.

  38. This post has been deleted by its author

  39. Anonymous Coward
    Anonymous Coward

    Routine cleaning in the BBC

    Only vaguely related but inspired by the stories of filthy computers...

    When I joined the BBC as a studio engineer in 1990, I was told there used to be a an area where circuit boards from all sorts of bit of kit were cleaned occasionally - or maybe I should say washed because IIRC they used water and then just dried them carefully.

    I know if anyone spilt coke* or coffee in a control desk we simple removed any cards and washed them asap with water. Coke seems to be somewhat corrosive.

    I was also told that people used to smoke in control rooms - must have been lovely being couped up for hours while someone chain smoked next to you....

    (*I know it was broadcasting but not that kind of coke)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like