'The computer was sitting in a puddle of mud, with water up to the motherboard' Each Friday, The Register shares another instalment of On Call, our weekly tale of epic tech support efforts. A few weeks back On Call featured servers stuck down a mineshaft and we asked readers to detail the dirtiest places they've been asked to work. With the imminent Christmas break slowing the pace and quality of news, …
Yes, I've worked in a couple of places with water collection pans over equipment funnelling water to a drain.
One was due to water leakage through decrepit concrete.
The other was in a new building, but the (heavily lagged) liquid Nitrogen pipes to environmental ovens caused so much condensation as to require the water collection above each oven and associated control and test rigs.
I once got sent to a print works in *redacted* because the elderly Macintosh Quadra 800 that was running their printer had died. This was back in about 1998, and even back then the Quadra was state of the ark. Don’t imagine a laser printer either - it was running a printing press for the local newspaper (amongst other things), just sitting there, in a cabinet, next to the press, hooked up to a network and receiving the papers for printing from the newer, fancier, Macs in the nice clean offices next door.
I don’t know what the presses for big newspapers like The Guardian or Financial Times are like, but this place was only superficially clean. It looked okay, but behind every panel, under every item, paper dust was thick like snow. I’m amazed the place hadn’t burned down (or blown up). I had a sneaky feeling, even before opening it up, that I knew why that old Quadra had packed its bags and quit.
…and yes, sure enough, when I pulled it from its dusty cocoon and opened the panels I discovered it was quite literally packed with dust. The fans had pulled the dust in, the charge on the components had made sure it stayed. And one day, it got so packed that the fans gave up, and even the minimal cooling that they provided to the thickly blanketed components was taken away… and the computer shut down. And refused to start up again.
Amazingly, all it took to repair the computer was a new power supply and a vacuuming out of the computer. And, in fact, the old PSU worked fine once I got it back to the Apple Centre where I worked, pulled it apart, cleaned it, and reassembled it.
As far as I know, that printer is still there. I wonder if they’ve started cleaning the place?
On a different note, I studied engineering at Uni back in the late eighties/early nineties and we did our CAD work in nice clean offices on Sun workstations. When it came to the CAM part, the designs were downloaded into filthy old Apple IIs which were hooked up to the machines responsible for making our creations. Lathes and so forth. Whilst they had plenty of cooling, there was also oil and general factory grime around - and those old Apples bore the brunt. And to the best of my knowledge, none of them faltered (although I have my suspicions that they keyboards probably didn’t work, then again though, they didn’t have to)
That probably worked pretty well. It has been decades since I have been under the hood of an Apple ][+ or //e.... I know they didn't have CPU or other chip fans. Pretty sure the power supplies didn't have fans either. The exterior might have been soiled, but they probably maintained a cleaner interior compared to modern day kit.
A pint to the //c that is still lurking in my attic.
In late '90s, working for a Mom-n-Pop Apple/HP repair shop, we had a standing order to beg/grab/snorfle any Macintosh IIcx systems we came across in our daily on-site repairs (mostly schools and school related offices).
We had a manufacturing client running hyperspecific CAD/CAM cards that supposedly only ran in IIcx. I have no idea how much the boss made from each machine but profit had to be nice as most schools/offices were glad for us to clean up old kit from their stash closets.
Working tech support at a small private collage (with private beach), I had a student tower brought in at end of year; wouldn't start. Machine was from the stoner dorm. Opening the case, everything was coated at least half an inch of ash and resin.
So glad that job did not entail drug testing.
I've been to a few presses in my career and they vary from the spotless, typically bigger, pros to the not so spotless ones where everything is covered in paper dust and its advisable to wear something over your face if you're tasked with anything involving disturbing said dust built up on various bits of kit.
Not sure how bad paper dust is for people - probably not brilliant.
However a clean printing floor with a shiny well-maintained press running full tilt is something to behold.
Worst ones I've come across were PCs on a factory floor near dozens of CNC machines. These were leased Dell Optiplex machines, and after 3 years of sitting there in that environment, they were completely coated - outside and in with a thick layer of oil. Every single surface and component had a sticky, slimy coating to it. Before boxing them up for return to the lease company, we used to wrap them in a couple of bin-liners, just to protect our hands and clothes.
The thing is, we had a full list of all the various issues with a return PC that could incur a penalty charge from the lease company. You know, the usual deep scratches, broken casing, faulty/missing components and all that. Yet there was nothing in the list that covered dirt or oil. Hence the lease company used to receive PCs that were bordering on being a biohazard, yet we never received any form of penalty payments for sending back the machines in that state...
The three years of lease payment probably means the Dells were paid for twice over or more so no trouble.
If the leasing company were to be difficult you would probably consider purchasing refurbished ex lease and ewasting them when they expire - the larger cost could be the disposal. I guess they wouldn't rust or the fan bearings lack lubricant :)
We have the dreaded laptop in the plating tank. which I've mentioned several times in the past.
Now the usual procedue is to call over the PFY and get her to fish it out. sadly this gets an answer of 'go fornicate with thyself' , kids these days... back in my day, we'd be made to swim to bottom with no breathing gear for tuppence a fortnight.*
So its out with the long rubber gloves and go on a fishing expedition, allow said laptop to drain and ask if the power was on when it fell in. if it was, then everything inside will be plated, and theres no point even trying to save the HDD... if not.... we may have a chance of saving the HDD, but usually no.... so we rinse the solution off, and say our goodbyes to our beloved laptop and let the forklift give it a dignified send off** (especially if thats the 2nd one this month and the boss is getting testy)
*senile ramblings cut short
**goes crack quite nicely under one of the wheels...... then its making up a reason it was on the floor in the loading bay...... over to the PFY for this
Into the plating tank?!! What were they doing, balancing it on the edge of the tank as it was the only "magic" spot in the factory which could get a wireless signal, then using both hands to type an update into their Faecesbook page, followed by the "oops" moment when they typed too vigourously?
One of the lathes backs onto the plating area..... and it needs to talk to the laptop. however , the bright spark who designed said lathe put the RS232 commport on the back of said lathe.
So its much easier to plug the cable into the machine, leave the laptop set for upload around the back, then hit upload at the machine, rather than follow the sign thats been made up saying "DONT TAKE THE LAPTOP AROUND THE BACK SO IT CAN BE KNOCKED INTO THE PLATING TANK".
And then the plater lifts the lip of the plating tank, hits the laptop and splosh... fizzle.
.
.
Followed by cursing and screaming and finally a BbzzzzzzzzzzzzzzzzzZAKKK! noise.
The cattle prod is then returned to the shelf marked "Staff education tool"
Hmm ... perhaps an RS232C extension cable could be attached to the com port on the lathe and the connection "sealed" with a metal cage. Then, the extension cable can run around the back and side of the lathe so the connector is at the front of the lathe. And, if there's space, perhaps your machine maintenance tech could build a little wooden table attached to the side of the lathe, to hold the laptop. It would have a wooden "back" so that the laptop could not be knocked backwards off the shelf.
(Icon for designing things so that the "easiest thing for users to do" is the desired-by-techies thing for them to do.
I was involved in a world wide games event, and so there were lot of temporary facilities used.
One was under a main arena. Because of the thousands of extra people who attended,the toilet facilities could not cope,and there was a sewerage leak into the area where the server's were stored.
The rate of ingress was slow - they reckoned that they had a week before it got into the servers - fortunately the games only lasted another 2 days.
The fall back plan was a taller table and electricity cables tied to the ceiling.
I used to look after the small Epson PX-4 laptops that were use to collect the UK Pop Charts (while it was still independent and not run by the industry, for the industry).
I had a laptop exchanged (we used to send a replacement and the courier would replace the new laptop with the broken and then return the broken one to us.)
I once received a laptop back in a sealed bag, that had evidence of a brown leakage, but generally looked pretty clean. On investigation, this was because the shop had diligently cleaned the outside, including keyboard, but it still did not work. The shop had been flooded with raw sewerage, but they were trying to avoid a bill!
That's even more common now, especially since the pandemic and so many orgs switching rapidly to laptops and still have some level of WFH. When I get called to a faulty laptop and it's suspiciously clean, my spidey senses immediately trigger and I wonder just what form of liquid, slime, ooze or gunk might be in there.
The silly thing is, if the user fessed up in the first place, it would be cheaper and with less come-back on them. By attempting to clean it and reporting "I don't know, it just stopped working", they inevitably get caught out in a lie and there's a charge for a service visit that ends up being chargeable as "non-warranty".
I have posted this before but as the IT manager we got all kinds of accidental damage.
One of our electrical Engineers had just had a new baby boy. He was filling in his overtime sheet on his laptop. His wife realising Engineer junior needed a new nappy removed his current one and handed him to his father while she located a new one. Cold air hit the babies undercarriage and the laptop keyboard got an impromptu wash.
He phoned me up and we laughed for about 30 minutes. I salvaged spare parts from a similar but broken laptop. Posted the replacement parts to him so he could dismantle and replace while cleaning & drying everything. We accelerated his laptop replacement and sent the broken to the recycling vendor.
The smell of a dye house is unique. I have been in many over the years in the UK and US and they all had the same aroma.
One particular place in the US was dyeing sheets for major hotel chains. The whole place had a 1 inch layer of cotton fibres covering everything. In harder to reach places it was even thicker. It looked very pretty - like a thick layer of snow, but did nothing for the ventilation of the panels that housed the electronics.
The "prettiest" one I came across was the small machine room for a semi-rural museum. It was just a small room that had a tiny window to outside that was kept open a crack all year round.
One day I walked in to find a thick layer of colourful dead butterflies covering every inch of every surface. There must have been thousands of the poor things.
They had similar overnight infestations of ladybirds too, over the years. I'm not sure what it was about the room that attracted them.
The warmth? I accidentally filled my parents' bathroom with ladybirds a couple of years ago when visiting them at Christmas. I opened the window slightly before showering, to avoid steaming the room up, unaware that there were dozens of ladybirds enjoying the warmth and possibly humidity escaping through the cracks in the frame.
I once went to decommission a site that was used to train drug detection dogs.
I took all our gear back to our office so they can be wiped clean and one of them, a network switch, was making a strange noise. I popped the screws and lifted the lid and found every space packed in dog hair. The noise was coming from the fan struggling to breathe!
PLC's by there very definition end up often in filthy areas, and I've seen a few in coal mines, chemical works etc which were caked in various solid products
The worst was Tate and Lyle in Silvertown, London. Its was where the raw sugar was offloaded to be processed. we had a couple of NT servers, we had to maintain and the entire site was caked in a sort of mucky brown layer of sugar, so much that your fillings would start to spontaneously explode after about 30 minutes
Horrible place, I felt really sorry for anyone who had to work there, but it probably put you off sugar for life
Heh. I have spent a lot of time in the Caribbean. The stuff that gets to England has been cleaned up and pre-processed. That means that most of the rats, roaches, lizards, ants, etc. have been removed and some stuff converted into molasses, rum, and other sugar products… Note that Bukra Massa, in his ineffable idiocy, was deathly afraid of snakes, and imported mongooses from India. Problem one: on several islands, including Jamaica, there are no native venomous snakes. (Some vipers did make it to Jamaica, aboard inter-island shipping) There were lots of boas. On those which do have native venomous snakes, they’re mostly fer-de-lance and other pit vipers. Pit vipers are optimized to hunt small mammals. Like mongooses. Mongooses are highly resistant to cobra venom, but not at all to viper venom. Mongooses also hunt lizards and small mammals, such as rats. Lizards hunt roaches. Vipers hunt rats and, thanks to the generosity of Bukra Massa, mongooses. So there were mongoose and snake bits in the sugar as well. Sugar factories are very interesting (and smelly) places, best viewed from a distance. I found that five miles was close enough. There’s nothing like being downwind from a few thousand tons of molasses.
There’s nothing like being downwind from a few thousand tons of molasses.
Unless it's being *under* a good bit of it, as happened in Boston MA, in the early 1900s. And which is why it's really important to make sure your metal storage tanks are built of quality steel, and designed for the purpose.
I'll see your molasses tsunami and raise you the London Beer Flood of 1814... A 22ft high wooden vat burst, causing others to burst and leading to a 15ft high wave that demolished the back wall, two houses, damaged two more and killed eight.
https://en.m.wikipedia.org/wiki/London_Beer_Flood
Back when computers were larger, heavier, and more-expensive, it was more-likely that the manufacturer's environmental requirements for said computers (typically, mini-computers) would be followed. (More-likely followed, but not always followed.) Modern desktop PCs are relatively small, lightweight, and cheap, so the average drone doesn't think twice about stuffing them into an unventilated closet, with no surge suppressor, and plugged into the same outlet as the photocopying machine or laser printer.
One of the longest, gunkiest computer clean-up jobs I had was working on a Data General Nova 4 minicomputer I was given ("Please, just haul it away and it's yours."). It had been doing auto-parts catalog database and ordering duties in an automobile repair shop. Doing them in the shop, on the shop floor. The interior of the rackmount cabinet, the card cage, and the circuit boards all were covered with a grease/dust combination. I cleaned the circuit boards and backplane in a shallow pan of gasoline owned by a mechanic friend (it was at his house, not mine!), and used isopropyl alcohol and many rolls of paper towels and Q-tips on the interior of the rackmount box and in the slots on the squirrel-wheel fan.
(Icon for "gunk in the computer.")
...I worked in a medium sized college. It was of the kind where undergraduate students actually lived on site, and The "IT Department" such as it was was located in what had once been a professors apartment on the ground floor of one of the accommodation blocks.
The server room was located in what had originally been the shower block for that floor when the building was put up back in the 60s before it was re-fitted so that each student room had their own toilet/shower room somewhere in the late 90s. This was actually a remarkably good idea, because it meant that the floor contained a high capacity drain, and the room had huge amounts of ventilation that was perfect for putting a nice big AC unit up to, and any run-off could simply go down the hole in the middle of the floor. It also meant that should the fire suppression system ever actually go off, it wouldn't flood.
This setup all worked out very nicely until one day the accumulation of hair and other... waste products... produced by showering students blocked up the drainage pipes and it was discovered that the retro-fit job hadn't been of the highest quality.
Everything drained in series, and once one blocked up, everything "up stream" of that would block up too. This wasn't immediately obvious to the students in question, because the shower drainage pipes ran through the drop ceiling of the floor below and snaked back and forth so they could cover all the rooms. This meant that there was actually a pretty large storage capacity in all the pipework.
I think you can see where this is going. Over the course of a weekend all the students upstairs were happily showering away and that large pipe was filling up with water as the blockage at the far end prevented it from draining. Eventually water started coming back up through the floor and into the shower cubicles - but we're dealing with undergraduate students here, and they're used to living in crappy accommodation where that sort of thing happens, so most of them just ignored it.
Then the pipe burst.
Directly over the server room.
...and about three days worth of accumulated filthy waste water got dumped out straight onto the top of a rack full of hardware that all immediately went "Bang"
The company I work for has one of its largest hubs for residential customers in a European capital city in a back room at a metro station. It's essentially just off the platform, next to the back exit. There's a small alcove just outside our door where people frequently go to relieve themselves on the way home from the pub. You're always told to bring thick gloves for opening the door to that site, both for risk of contamination and also because some of the more annoyed field techs occasionally hook up a battery contraption to the door to make it slightly electrified as a deterrent.
A friend had a job to paint the toilets at a bookies in Kilburn. He got fed up with people ignoring the sign on the door and barging in, so he locked the door.
He later discovered the shop's clientele had got around the problem by sideling up close to the carpet-covered counter and... erm, watering the carpet.
Small garden centre, had an absolute tank of a dell machine in it. Machine had Fan error - press f5 to continue for years. User happily pressed F5 every day and it just carried on running.
Fast forward two years. Garden centre is being shut down. Inspect the machine and discover a mouse nest inside. It had chewed through the CD and fan cables and used them to line its nest. A pair of us very carefully dusted it out above a bin, but did not find the offending stowaway. Good thing it didn't chew through the high voltage on the PSU!
Thankfully, I only saw a couple things, but the shop I used to work for had a few machines come in that were non-repairable; one of them was absolutely covered inside and out with cigarette smoke and tar- apparently, the house it was in had several adults who all had a 2-carton/data habit and no fresh air exchange. (and/or the ashtray was directly in front of the poor computer's air intake). The bench tech absolutely refused to touch it, and the extended warranty company that used us for service calls agreed with us and refused to cover any work done on it.
My direct "that's gross" stories are as such:
1. Performing preventive maintenance on base model workstations on the 'dirty intake" line of a garment/uniforms management company- the machines were all on wooden blocks to keep them from rusting to the floor, as they power-washed the floor on a weekly basis. the outsides of the machines had a fair amount of splash dirt on them.
2. The server that had a mouse nest in it. thankfully, by the time I got there to haul it in for work, the occupants had all been evicted, but it took a couple days for the smell to leave the bed of the truck I used.
3. The OfficeJet all-in-one that got enough use that the holding area from where the nozzles dumped during the cleaning/priming cycles had overflowed. IIRC, they ended up replacing that unit straight up, presumably with a unit that had a higher duty cycle.
4. No actual dirt, but I decommisoned a dial-up point of presence when I worked for [ISP] in a rural part of the state that was literally located in a former bathroom- the equipment was perched on the capped and sealed floor ring where the toilet used to be. It was a nice little day trip for me, at least.
5. after the [ISP] job, I worked for a mental health organization that had a site that literally had their office's switch gear and internet connection sitting in a boxed in area constructed out of ceiling tiles and grid... in an active bathroom.
6. When I was in the support team at my current employer, there were a few times were I had to clean out equipment that was placed next to coffee machines, or located in the bars. one had a network jack next to where one of the bar soda guns was holstered, and as expected, the patch cable was a sticky, stinky mess along with the jack which had also failed. (the coffee machine one was a thermal printer that had gotten a cup of joe spilled onto/into it. Thankfully, we have spares, so a spare was chucked into service and I spent a couple hours with a bottle of isopropyl, q-tips, and paper towels refurbishing the poor thing.)
Back in the 90's, I was the PFY at a site which manufactured industrial fans. This included the offices as well as lots of big, dirty machines for melting down metal and bashing it into various different shapes.
My first task was to go around every PC on the site (about 300 of them), and assess each one for a memory upgrade. This involved:
- evicting the user from their PC
- shutting it down
- opening the lid (including moving whatever monitors / detritus was sitting on top)
- looking to see how many memory slots were empty/filled
- put it all back how it was before I arrived
This was the ideal task to upset every single PC user on site in my first week!
Because of the environment, I found lots of PCs which were completely filled with swarf and other dirt. Surprisingly, only 3 of them failed to come back on after my 'assessment'. I do remember that the PCs running the foundry were the worst.
I can atest that foundry\casting PC's are much abused, I used to visit one when I worked for a company that had a support contract for one, when time came around for renewal I discouraged us from renewing.
You could walk in at 8.30am in the morning freshly showered in the last hour & walk out at 8.45am having only seen the factory offices & feeling grimy with a metallic aftertaste on your mouth & throat that stayed with you all day.
I did a memory audit on a bunch of machines (Software tool reported on what was inside each one) for upgrade, a few months later Monday Morning we were swamped with tickets locking up PC's & BSOD's for the same floor.
Deskside support staff (As usual) had taken the plum weekends overtime for themselves & as usual applied no thought to what they were doing & leaving the IT Service Centre in the wooden hut in the car park to resolve their screw ups on Monday by not matching the memory in the slots or populating the correct banks.
A merry morning was spent populating alternate machines with all the old sticks or a couple of the new ones to bring them all up to 4Gb IIRC.
Once called to look at a server in a Disney store, a floor standing unit in the admin office next to the stockroom.
When the cover was removed all the free space was filled with fluff and fibres from the plush toys. Once the fluff was removed I discovered a pink residue around the fan vents and some parts of the chassis.
When I pointed this out to the staff they explained that someone had accidentally set off the security dye in a cash box.
Several staff had to go home covered in pink dye and the office had been redecorated.
Have worked in a couple of logistics operations where PC's gradually ingest and build up an internal layer of concrete dust (bonus points for not having a sealed floor) and cardboard fibres.
Nothing beat the efforts of humans though.
My last employer had a couple of staff, that if I ever had to check/fix anything on their PC, it was likely to be a biohazard so I went in every time with a brand-new keyboard and mouse. Was once asked to check a desk phone for one of the offenders where the keys were sticking down, took one look and said 'nope - I'm not even touching that' and gave them some wipes. And how people manage to get food deposits on a monitor which is also out of arms reach is beyond me.
Lastly, some of the stuff that came back in for return or repair during the last few years of remote work... headsets pretty much go straight in the bin and everything else almost needs to be pressure washed and fumigated.
ps - if there's ever an ON CALL:UNDER THE BUS edition... let me know...
Used to get dispatches for $DEVICE OFFLINE in various branches of a big-box store. Wireless access points weren't too bad to deal with, though some of their IDFs were a she-dog to reach. Often, the failed unit was one of the WAPs with external antennas outside the building, with RF cables through a wall penetration to the WAP located in the (theoretically) warm-and-dry interior of the building. The WAP would be mounted on or under the structural steel. I would find the network jack corroded or burned up from water intrusion collecting in it, whether from condensation or from an actual roof leak, and getting sizzled by the PoE power.
For the life of me, though, I can't fathom why Cisco thought it a good idea to put the jacks on the TOP edge of the WAP, when mounted with the logo right-side-up, so that the jack would collect water that ran down the cable. (It's not just Cisco, though; their competition seem to do the same thing. A subtle form of planned obsolescence, perhaps?)
I would try to remount the WAP sideways, or make a plastic-bag tent to prevent recurrence.
The absolute worst calls were for failed connections to scales in the deli. The LAN cables run under the customer-facing display/cooler cases. There was at least one jack-and-plug located under the case, perhaps more if the scale had been moved from its original location or the cable had been serviced previously. Access to under the case requires lying on the floor and removing a toe kick plate, which often showed evidence of attempts to silicone-caulk it to the floor. The caulk was not usually intact, and in any event was not intact after I removed the access plate. There was usually a grey slime under the case, washed there from cleaning the tile floor. Not a great place to be working. Wet environment, corrosion, etc., etc., the fault was often due to corrosion at the interface between the cat5 cable and the punchdown terminals on the jack. The repair itself was not so bad, just punching on a new jack and/or plug and replacing a patch cord. But the grey slime -- yuck!
In my younger days I worked a a company that provided systems to answering services (in the US). Sometimes we went to the offices and did "maintenance" and enhancements. Since the terminals were CRTs and their screens attracted smoke particles quite easily, one of the things we would do is apply a bit of window cleaner to the face of the terminal, and with a dust rag clean the terminal. After that we could ALWAYS get remarks that were similar to "WOW what did you do, it is so much better". Thankfully the main box (rack) was usually in an office where people didn't smoke too much, but the operators were worse than fireplaces.
I won't comment on the looks of the attendants but I will say that (and I'm making a generalization here) looks and voices were usually inverses.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
