BOFH: What's the Gnasher? Why, it's our heavy-duty macerator sewage pump

BOFH logo telephone with devil's horns "We need you now!" the Boss says, bursting into Mission Control early one morning. "Why?" the PFY asks. "There's a smell." "Where?" "Everywhere!" the Boss snaps. "But not here, though?" I ask. "I… No.  No, it's not." "To the basement, Robin!" I cry, as the PFY and the Boss follow …

[Paul Crawford]

Speical cards

We have all seen them, some of us have worked with them or (cough) designed them. But it is so common for companies to have no "plan B" for when they reach EOL or the ability to support them dries up.

I think in this case the PHB would be praying it dries up...

[Evil Auditor]

Re: Speical cards

And with designed I understand wire wrapping them.

[BebopWeBop]

Re: Speical cards

Hey we ran a wire wrapped pdp8 - bloody sight more reliable than the official gear

[Evil Auditor]

Re: Speical cards

Oh, there's nothing wrong with wire wrapping! Except if you have to trouble shoot/maintain/reverse engineer the thing. And no, I can't believe that any of you kept the correct schema AND still knew which one would belong to which board - after any number of years. Besides that, totally reliable.

[BugabooSue]

Re: Speical cards

Ahhh, the Joy of fault finding wire-wrapped computers and tracing THOUSANDS of identical white wires through multitudes of panel connectors...

I'm looking at you - Sperry 7000 Series.

When I was a PFG (Pimply-Faced Girly) in the late '70's, I was tasked with keeping this beast going.

Every few months it would refuse to play, and I'd shut it down to find one or two mains-powered cooler fans had given up the ghost.

Next would come the "Whisker Search" near where the fans had failed. Hours under a microscope looking for the 'tin whisker' that had shorted out a track, along with the associated eyestrain.

We also had a super-expensive EMI infrared battlefield scanner mounted in a mobile lab that would do the same thing. (Vehicle number 00WB10 if you know it?)

Good old MOD - bought the kit, but never wanted to pay for the upkeep...

Happy Days!! :)

[Captain Scarlet]

Re: Speical cards

Yup and always production machines someone brought by stealth that no-one knows about until it dies.

Replaced such a machine with an ISA card, to then get a frying because it didn't have Windows 98 on (No-one asked and I wanted it on our Windows Domain and network), so Windows 98 installed, all networking removed and its probably still there chugging away (Still no idea what the bleeding thing did).

[Anonymous Coward]

Overhead sewers

"building waste up the rising main to the council sewer"

Mind boggled

[Martin an gof]

Re: Overhead sewers

Mind boggled

The confusion may be because while the term actually (I believe) refers to any kind of pressurised pipework, common usage is that it only refers to potable water. (of course, in the case of the BoFH, he might actually be referring to the potable water...)

We had a similar thing at a previous place of work - public loos in the core of the building all fed by gravity into a box in the basement which contained two macerating sump pumps which mashed it all up and forced it up a 4" plastic pipe for several hundred yards until it met the main sewer under the car park.

The 4" pipe so badly installed that it sagged and wobbled about every time a pump came on, but at least the glue joints held. On opening day - a couple of weeks before I started - the pumps failed and a colleague was tasked with clearing the blockage.

Advice from the M&E contractors (who were wisely not on site that day) was to open an inspection elbow, which said colleague had to stand on a ladder to reach.

Folklore has it that his screams - as about 60ft of human waste suddenly took a left-turn at the elbow - were audible without the aid of his radio and I can absolutely confirm that despite a "professional clean up" there were still bits of loo roll stuck under awkward bits of construction when I left that employment some two years after the incident.

Hi Matt, if you're listening!

M.

(Oh, Magna Science Adventure Centre in Rotherham in case you're interested)

[pavel.petrman]

Re: Overhead sewers

Is it just me or does this previous place of employment of yours feature repeatedly in your comments under the BOFH, On Call and Who, me? articles? Keep them coming, please. One day I will visit the science adventure centre thinking about all the adventures you related here.

[Anonymous Coward]

Re: Overhead sewers

In multistory buildings, waste is often fed into a basement collection "area" before being macerated and pumped back up into the council sewer.

This design allows better control around airflow issues, speed of material and feeding/backflow into shared sewage systems.

[Anonymous Coward]

Re: Overhead sewers

... but boy oh boy are you literally in the shit when the pumps fail for whatever reason..

[Anonymous Coward]

Re: Overhead sewers

Yes but sewers are generally underfoot. An overhead sewer is something to be feared.

[Mark 85]

Re: Overhead sewers

In the sub basements and sub-sub basements, they're well below the sewer lines leading to the street connection. Same for incoming fresh water. The pipe goes down and then the water is pumped up to the top floor. Pipes then coming down feed the building services.

Fresh out of the military and waiting for the new college sessions to start I worked as a maintenance man in a 15 story office building that was 50 years old. A feed pipe broke late one Friday night (more like was dissolved as it was cast iron) and flooded the building. It was impossible to get to the sub and sub-sub basements when I came in Monday morning. Queue up frantic phone calls to turn off the water (city) and electric power (it all came into the basement) and then finding pumps to get the water out. Building was closed for about 2 weeks while the water was pumped out and pipes replaced. Then there was the damage to offices that had been flooded. Good times... NOT.

[Johndoe888]

Re: Overhead sewers

Some years ago a plumber friend had the job of installing a macerator toilet in the basement of an Indian restaurant, that evening the boss of the restaurant thought the job was finished and decided to err test it :(

The pipe was only loosely connected prior to fixing it to the wall and finishing everything, of course the inevitable happened and the pipe blew off turning it into a pre digested curry shower !

[Dave Bell]

Re: Overhead sewers

I used to live in a village which was affected by one of those. The next village had the tiny processing site replaced by a couple of miles of pipe, running by the road over a couple of hills and connected to the local unpressurised system. Used a holding tank and a macerator pump, and when the pump started the holding tank couple be well ripe. We got the stench.

After three years, the solution was to extend the pressurised pipe direct to the local sewage works, another mile.

[A K Stiles]

It's all fun and games...

...until the macerator pump dies, then some fule has to get their hands dirty. Very, very dirty.

Mine's the one with the box of extra long vinyl gloves in the pocket.

[phuzz]

Re: It's all fun and games...

"extra long vinyl gloves"

The shoulder-length ones that vets use?

[I ain't Spartacus]

Re: It's all fun and games...

The problem with those long gloves is that they've never quite long enough to also cover your nose...

[Rich 11]

Re: It's all fun and games...

You need a body condom.

[I ain't Spartacus]

Re: It's all fun and games...

Naked Gun style.

[Mr Sceptical]

Re: It's all fun and games...

I'd only go in there with a full Outbreak/Contaigon/Chernobyl hazmat suit on!

Actually, make that a Black Mesa environment suit, God knows what might emerge from the sewer line. Where's my crowbar?

[khjohansen]

The Arch-BOFH?

.. Selling archaic hardware at inflated prices .. Would this be the BOFH from when the present BOFH was a mere PFY?

[Wellyboot]

Re: The Arch-BOFH?

Thats how Craggy Island does IT support!

[iron]

Re: The Arch-BOFH?

That would be an ecumenical matter!

[Anonymous Coward]

Re: The Arch-BOFH?

Feck! Arse! Girls!

[A K Stiles]

Re: The Arch-BOFH?

I love my brick!

[Sgt_Oddball]

Colour me shocked...

They used something as common as an IBM compatible rather than something more esoteric like an Amiga A1000.

Also surprised the boss didn't enquire as to if it could be replaced with a raspberryPI or some such.

At least he didn't mention cloud computing (or is that what happens when everything backs up and not in a way you can easily restore from).

[Charlie Clark]

Re: Colour me shocked...

Also surprised the boss didn't enquire as to if it could be replaced with a raspberryPI or some such.

Eh? this this the Boss, you realise?

And Amigas aren't esoteric, at least not for the time. A PC with a very custom board, ie. one that did all the work, was way more esoteric and completely unfixable.

[Wellyboot]

Re: Colour me shocked...

I think in many places you'll still find that "A PC with a very custom board, ie. one that did does all the work, was is way more esoteric and completely unfixable"

[Anonymous Coward]

Re: Colour me shocked...

Yes Apple makes them.

[Trygve Henriksen]

Re: Colour me shocked...

Yeah. Amigas are way too common, and not all that often used in these kind of systems.

I know that C64 was used in some Teletext systems, though. And because of the relatively simple HW and the 6502 instruction set it would have been easy to use it as the controller in a large HVAC system.

They even had a chained serial port with addressable nodes.

A PC,(Original, or XT) with a custom ISA card that may even have some sort of custom chip or programmable logic, where the documentation is... lacking... can be almost impossible to upgrade to a modern system.

This is actually a real issue. Many buildings have ancient controllers that only still works because of regular blood sacrifices, and they can't upgrade the HVAC systems because there's no way to update the controller to support new components. Because the original manufacturer no longer exists or will not touch it because it's so old that they no longer have documentation or anyone old enough to even remember it.

[Jemma]

Re: Colour me shocked...

You want esoteric - try a BBC B running a scanner in an NHS hospital about 12 years ago. I guess I shouldn't have been surprised but I have to say I was.

[Thrudd the Barbarian]

Re: Colour me shocked...

New city hall from Day one ran it's propriety HVAC control board on an Apple IIe computer system .

Only things that ever went on it were the FDD cards which wear an easy chip swap fix. Oh and the one time the power supply blew up when 600vac was on the ground line. Gotta love public service unions. At least nobody died, just a few tingles in inappropriate places.

[Anonymous Coward]

Re: Colour me shocked...

Well how about being called out for a BBCB of 30 years (aprox) ago running a machine processing aircraft turning blades, I have to say I was responsible - as a placement student. Luckily I have a good memory an my thesis - five similar machines with cloned ROMs were being used. Never thought that grant would come back and feed me. (Anon for obvious reasons)

[Anonymous Coward]

Re: Colour me shocked...

Cosworth used to use BBC B's to run its foundry mould injection stations, making all those lovely Cosorth engines for Ford, MB, Rolls Royce, Bugatti, FI, etc.

Re housed in a natty Stainless Steel Casing.

[Deimos]

Ah blood sacrifice

It’s amazing how much kit needs need bodily fluids properly sacrificed to either start their function or maintain it.

The whole of M$ original software was entirely powered by the various fluids of a dozen odorous nerds and as the

devil agreed to make the OS compulsory, more and more serious onanists were required.

The onanistic critical mass and sacrifice of said perma-virgins resulted in the “launch “ of Win 95. We all know what happened next.

If only a couple of million litres of linux holy water had been available at the right time?

[Grikath]

Re: Ah blood sacrifice

You mean the Linux that at the time had no programs suited for [job/function X] off the shelf, without any formal support or a billable company address for the software? Doubly so for esoteric hardware that had to be hardcoded to work with a single operating system and sometimes even hardware setup?

I know it's Beer o' Clock, but ......

[J. Cook]

Re: Colour me shocked...

... and the business doesn't want to pay the 5 or six digit sum (with the first digit of that being more than 5) for a forklift upgrade of the entire control system.

[A.P. Veening]

Re: Colour me shocked...

And don't forget, that all other digits, including those after the decimal separator are nines.

[Anonymous Coward]

Re: Colour me shocked...

The control hardware and software has to be updated over time, the HVAC units at work went from ancient local PID controllers to something more modern 3 or 4 times with a building control system and subsequent reducing of the onsite maintenance staff.

[Anonymous Coward]

Re: Colour me shocked...

"They used something as common as an IBM compatible rather than something more esoteric like an Amiga A1000."

Amiga 1000 27,500 units sold in Germany alone (http://www.amigahistory.plus.com/sales.html)

I suspect these cards likely sold a few thousand globally over 10+ years. Certainly the BMS systems I worked with in the 90's were pretty much site specific, only requiring an ISA slot and the specific software that matched the card to run although I admit BMS systems weren't my primary occupation.

All up there will be millions of these cards, but almost as many variations with dubious compatibility. The advent of MODBUS/BACnet (maybe others) was a godsend. No, really...

[James Dore]

Re: Colour me shocked...

I've found Acorn RiscPC's, an Archimedes, and an IBM RS6000 all doing this. Nothing so common as an Amiga though.

[Michael H.F. Wilkinson]

And again, the BOFH comes out smelling of roses

unlike the boss, most likely.

I wonder what cut of the 3.5 grand the BOFH gets

[Grikath]

Re: And again, the BOFH comes out smelling of roses

I wouldn't be surprised if the answer was "none".

Guys like the old geezer are worth their weight in gold when dealing with obsolete systems like that.

Plus that that 3.5 grand bill is far cheaper than having to refurbish the whole system with modern stuff, which incidentally also means Having To Do Actual Work and Risking Pesky Audits (both feeding off each other, risking a lack of available carpetry and wear and tear on shovels) and a missed opportunity to re-enforce the notion of quiescent compliance with the Boss for BOFH and PFY.

[Vaughtex]

ST225

When your first thought on seeing ST225 mentioned is "20Mb half height 5 1/4" SCSI, so it could be an NCR PC", you realise you've been in this game far far too long.

[drsolly]

Re: ST225

It wasn't SCSI. The ST225 (and I handled a lot of them) was MFM ST 412. And those drives were used in a lot of systems.

They had their own way of failing, but we developed a way to recover the data from a failed ST 225 called "magic finger". We didn't even need to open the drive.

[Petergwilson]

Re: ST225

Yep, defo MFM. I saw many many of these. Everyone used them. Seagate must have made a fortune.

When they 'failed' I used to whack them with a big screwdriver handle and listen to them spin up again. The look on the customers face was priceless.

[theblackhand]

Re: ST225

But did you sing "there's a fraction to much stiction"? Or is that just an NZ/Australian thing?

https://www.youtube.com/watch?v=gRI-Nr8S9bw

[Kiwi]

Re: ST225

The version I recall most is at https://www.youtube.com/watch?v=nP0v4lL_rZw

Interesting the line "Holding on to a bygone era" is used a couple of times :)

[VicMortimer]

Re: ST225

The ST225N was SCSI. I've probably got a few of them sitting around somewhere.

It was one of the drives used in the Apple Hard Disk 20SC. They didn't ever last very long, if you were lucky you got one with a Rodime drive, those lasted decades.

[Anonymous Coward]

Re: ST225

I may STILL posses a ST225, I remember replacing it with a massive 200MB MFM drive, but I usually keep the old drives as spares.

Potters off up the garden to "The Shed" clutching a bottle of "Super Bock".

[Adrian 4]

Re: ST225

I've heard that some people throw things away. Can't grok that.

[Chairman of the Bored]

Great, I'm now that consultant

Designing hardware is like having kids. It was fun at the time, but now you're supporting the end product until you die.

Somewhere in my vast collection of abandoned crap is an Intel PROM burner and a matching 386 machine. Every couple of years I have to resurrect it to modify and burn firmware for a 1980's vintage antenna pedestal or two - based on a Z80 - last upgraded in, oh, 1996. Customers think it's too risky to upgrade to something modern, and I think it's far too lucrative to push the point.

[Anonymous Coward]

Re: Great, I'm now that consultant

Talk about old crap, I still must have a UV EPROM eraser somewhere I built for erasing PSION data packs. Even better, I came across the interface I made for it to talk to a digital caliper. I have no idea where the software is now, but I still have the gear.

Nothing has really changed, people still come up with interesting requirements that need some creative problem solving, but now you use a smartphone if it has to be portable and Raspberry Pis and Arduinos otherwise.

And it's still fun to do IMHO :)

[Nunyabiznes]

Re: Great, I'm now that consultant

And in 30 years there will be some old graybeard on here complaining about RPis and Arduinos. "Yeah it was fun to program those little buggers and build them from scratch - but why don't they just let the AI control that system?!?"

[ma1010]

Re: Great, I'm now that consultant

Designing hardware is like having kids. It was fun at the time, but now you're supporting the end product until you die.

Unless you're some big corporation selling IOT crap. After 18 months or so, you just call it EOL and turn off the servers. Time to buy our new model!

[Chairman of the Bored]

Re: Great, I'm now that consultant

I guess a guy has to put food on the table somehow, so I know people have to engineer products that die quickly and deliberately. But I cannot see myself doing it, and for the most part I've been able to work with firms and organizations that have some commitment to quality. Been lucky, I guess!

[spold]

Re: Great, I'm now that consultant

Not that old... but still very old.... I was doing a network security scan for a steel company.

Please please please don't scan these IP addresses they said, we are not quite sure how it all works any more but the systems attached to them control the blast furnaces... molten metal and people issues.

[BebopWeBop]

Re: Great, I'm now that consultant

A source of much interesting and useful work? If a network scan is dangerous then WTF

[J. Cook]

Re: Great, I'm now that consultant

... that's when you split off the entire network for those systems into their own air-gapped network, with maybe a firewall to allow selective access to the operator machine/interface.

[Doctor Syntax]

Re: Great, I'm now that consultant

They probably daren't do that either.

[Anonymous Coward]

Re: Great, I'm now that consultant

re: steel mill. They're might be Allen-Bradley, first gen. They wouldn't even run on a LAN with more that 25 broadcasts per second....aargh, the memories...

[Vincent Ballard]

Re: Great, I'm now that consultant

And do you have a plan for what to do if you can't resurrect the PROM burner or the 386?

[Chairman of the Bored]

Re: Great, I'm now that consultant

Plan for obsolescence? Absolutely! We can upgrade to a Danaher Motion controller, maybe a GE Automation controller if the problem is big enough. For UK firms, I've heard great things about Motion Control Products, Ltd but haven't played with their toys yet.

[chivo243]

BOFH going down the crapper?

I'm just curious???

My coat with a map of London's sewers...

[Sgt_Oddball]

Re: BOFH going down the crapper?

No, just sending the boss round the bend...

Mines the used Sou'ester.. Hold the door will you?

[James Dore]

We've got a couple of these guys who appear whenever some ancient PC with proprietary software written in FORTRAN-77.SpecialVersion connected to massive multi-million-pound scientific research kit craps out. Usually a laser or something.

[Will Godfrey]

Easy Peasy

Bespoke ISA card?

Wait till you're faced with a paxolin breadboard wired point-to-point and crudely bolted to a decapitated card.

It was controlling ink levels on an ancient 6 colour printing press, and while we applied further bodges to get it working, the operators had to run up and down recharging the trays by hand. - all now happily scrapped.

[BebopWeBop]

Re: Easy Peasy

Well you win pint of the week!

[John Brown (no body)]

Re: Easy Peasy

of what? C, M, Y or K?

[Will Godfrey]

Re: Easy Peasy

That's only 4. Nos 5 & 6 are usually for 'specials' like metallic-esque inks.

[ma1010]

Thanks, Simon!

Always good to see a new BOFH.

[MadonnaC]

Reminds me of my previous place of employment, which had (as of a year ago, last i checked) a 386, running OS Warp, which held the only version of an imaging software which they had to use to browse the older manuals/diagrams.

I had a look at the software, and found that it used some sort of index to a large blob file, where it would extract the appropriate picture. Not friendly, and not worth what they wanted to pay to get it out of there into a group of image files.

[FeRDNYC]

That's OS/2 Warp! Remember, it's only half an OS.

[bpfh]

Which half?

The OS/2 command line half, the pre-emptive multitasking OS/2 presentation manager half, the MS-DOS compatibility half, the Windows 3.1 for OS/2 half or the virtual system half where you could boot yourself a Linux, QNX or a legacy version of DOS straight from the presentation manager from a boot disk image?

Been there, done that, still have some nightmares, but half an OS it was not.

Mines the one with the 1800 page OS/2 Warp 3 IBM Certification training manual in the pocket. Yes that one that’s pulling the coat hook out of the wall...

[Anonymous Coward]

Maybe half an OS but a big box of 3.5" diskettes, the PHB wouldn't pay for a CD drive.

[AndyMTB]

OS/2 !! - remember the tag-line "a better Windows than Windows". If only we'd have believed them....

[defiler]

Halfway there.

A better DOS than DOS, a better Windows than Windows.

And yes, I can testify to that. I had it as my daily driver when games were on DOS. Worked great until I got a Diamond Orchid for 3DFX tastiness. At least that's a card you can ditch, with recent versions of Dosbox.

[Doctor Syntax]

And I was sure it was going to be the bodies buried under the concrete that were smelling.

[corbpm]

Whats missing is the mention of the backhander for the £3.5K ISA card that came from the box of them bought at a "fire" sale previously for £20 the lot.

Must be worth a few pints.

[baud]

Well it's 3.5K also for the ISA repair guy, his knowledge, his stock of old cards and all that

[Prst. V.Jeltz]

I was sure it would end with:

"Thanks Bill , here's your half , see you next year..."

[FeRDNYC]

Could be worse

There was a story, a surprisingly few years back, about a housing complex in I think Boston, and the troubles they were having keeping their HVAC system operational. Turns out it was all controlled by custom software running on an Amiga A500. Good luck replacing that!

[Tom Paine]

Re: Could be worse

For £5000, I'll see what I can do.

*Type type*

https://www.ebay.co.uk/i/223650084253?chn=ps&norover=1&mkevt=1&mkrid=710-134428-41853-0&mkcid=2&itemid=223650084253&targetid=800764438998&device=m&mktype=pla&googleloc=1006886&poi=&campaignid=1506012229&mkgroupid=59698560162&rlsatarget=pla-800764438998&abcId=1139606&merchantid=119241892&gclid=EAIaIQobChMIsbKs_K7g5AIVk4XVCh3GnwRGEAQYAyABEgIXBPD_BwE

[Camilla Smythe]

Re: Could be worse

Here you go...

https://www.ebay.co.uk/i/223650084253

Much shorter when you remove the code resulting from your previous searches for but plugs plus no-one else gets tarred with the same feather.

[Prst. V.Jeltz]

Re: Could be worse

he's probly deliberately tarring readers in some sort of cookie stuffing associate marketing scam ...

[Anonymous Coward]

Re: Could be worse

Surely there is a wonderful Amiga emulator that can be adapted from running games to running HVAC software.

[NetBlackOps]

Re: Could be worse

Really depends on if the custom hardware is involved. If just the 8520's, should be meh.

[Richard 12]

Re: Could be worse

It's the hardware interface which is, well, hard.

In most cases there is a custom board (sometimes a one-off) which does much of the real work.

And that isn't on a sensible interface like TTL serial or CAN or ISA, it's some custom thing bitbanged on the GPIO and the timing really matters.

So you'd have to emulate that, too. Without access to a test jig, because the only known example is operating the building.

And make sure that your insurance is good, because you may end up in court a while later proving it wasn't your code wot dunnikin it.

BMS controllers are terrifying.

[mr_souter_Working]

Re: Could be worse

"Good luck replacing that!"

yeah, they're not getting my A500 - even if I don't have any disks for it anymore. (it still worked the last time I powered it on) - although, I suppose we all have our price......

[Herby]

Idea for ages...

1) Invent an ISA socket that works via USB.

1a) Or as an alternative, one that interfaces with the Raspberry PI.

2) Design..

3) Sell, profit...

[JulieM]

Re: Idea for ages...

PiSA probably isn't too crazy an idea. ISA is essentially just an address and data bus. You would just need bidirectional level-shifting to interface the 5V ISA to the 3V3 Raspberry Pi GPIO. The Raspberry Pi could then emulate a peripheral (e.g. by accepting commands for a disk controller and translating them to use its own storage) or the host PC driving a card.

[Will Godfrey]

Re: Idea for ages...

You're sacked!

They didn't want it resurrected, they wanted an excuse to scrap it. That's why they shorted the [REDACTED]

[Sgt_Oddball]

Re: Idea for ages...

Turns out such things already exist (who knew?)

like this one . Must be a pain getting the drivers to behave though.

[Anonymous Coward]

An unwell KSB Compacta sewage pump..... https://i.ytimg.com/vi/Wg3KTjcbABM/maxresdefault.jpg not one of my photos, although I have experienced similar, not always in a nice, clean basement though! These pumps typically have two motors / pumps which run alternately to balance the run hours, with the second pump kicking in should the effluent level continue to rise - all good, but does not account for someone locking off a 200mm gate valve downstream because of sewer repairs. Consequently, both pumps ran flat-out over a weekend, finally tripping out on thermal protection and triggering a red panel light on the archaic BMS. The site plumber went into the basement on monday morning to remove one of the pumps from the housing to remove the obstruction - didn't bother to shut off the upstream gate valves (because why would both pumps fail?), took the last flange bolt out and remembers wondering why the pump didn't feel particularly heavy - just as the chocolate fountain erupted with 30m head of pressure behind it......

Thankfully, I wasn't that plumber, but I did have to go into the basement to check the network cabinet wasn't contaminated shortly after the honey sucker had left.....

[John Geek]

Early Monday morning of my last week at work before retiring, I get a panic call from someone in corporate IT who'd heard I knew PostgreSQL. Apparently there was this PC in a wiring closet, running some sort of ancient linux and a bodge of proprietary crapware that used a Postgres database, it had been powered down to move it and change its UPS, and it wouldn't come back up.... It ran the card reader and turnstile for the garage security gate. They had the logins for it, so from home over the VPN I was able to log into the box, poke around, wow, this is some old stuff, Red Hat Linux 6, an app written in perl by a Japanese firm, and a PostgreSQL 7.3 database... Anyways, it took me about a half day to figure out that someone had changed a postgres configuration file over a year ago but never restarted the database or the box, and the change they made was invalid so postgres wouldn't start. logging had been disabled, so there was no error logs to debug, thats why it took me 4 hours instead of 30 mins. wave magic wand, turnstile works again.

[Will Godfrey]

Just a week later, and you could have charged them a fortune!

[Giovani Tapini]

Late comment

But I was in this situation once. Facilities guy say 'you know computers, right?" Shows me something disturbingly similar to the system in the story and asks if to can make it work.i genuinely had no idea. It used a scripting language to configure it that may as well have been based on that used by the predator. It got very hot in the office while waiting for what must have been the only guy in the country to rescue us from it...

[Anonymous Coward]

It worked out this time

I tell my son that sooner or later, the computer bubble will burst.

But everybody needs a plumber. Always. And there are keen benefits of plumbing over programming. A comparison:

***

When the programmer arrives, he's met with "What took you so long?"

When the plumber arrives, he hears, "We're so glad you could come so quickly."

***

When the programmer estimates the cost, they say, "Are you out of your mind?"

When the plumber tallies up the cost, they say, "Whatever it takes."

***

When the programmer finishes and hands over the invoice, the client says, "Send it in to accounts payable and expect payment in 60 days."

When the plumber finishes and writes up the charge, the customer says, "Do you take cash?"

***

And probably best of all, when the plumber goes home after a hard day at work, he can shower off the unpleasantness of the day.

Priceless.

[Joe W]

Re: It worked out this time

Mate, that was not funny.... (true things seldom are, unless happening to other people)