Developer wrote a critical app and forgot where it ran – until it stopped running With the weekend behind us, it's time to once again ask the question "Who, Me?" That's the name of The Register's Monday column in which we share reader-contributed confessions of making a mess with tech. This week, meet a reader we'll Regomize as "Sam" who in the late 2000s built an application for his colleagues that he …
I'm one step ahed of him. Or... half a step? The application runs on a VM. Admittedly, not in the production environment. Without having an entry as a "service" in the company I won't get that. Without a security audit and concept I won't get that, and without being on a production machine I won't pass said audit. The ITSec guy I'm working with is trying his best. Yes, genuine support, knowledgable concering the company rules and... how to.. ehm... yeah, circumvent some to make progress. I get the need to look at those things, and I also want to avoid this thing spinning out of hand and being all important all of a sudden, and all that without thinking about certain important things. At the moment it feels stupid, though. My ITSec-guy agrees, which makes the whole thing actually fun to work on. We are taking this (and our work on it) serious, but not overly so. Icon: how I feel about it... (a bit)
Man, auditing those apps sounds like an easy job.
"does it run in the production environment?"
"currently not, it first needs to pass this aud..."
"sorry that's unacceptable, audit failed"
I have no problems with the concept here, but i feel like validating in a twin test/acceptance environment should allow you to pass the audit, especially for new apps.
Something similar. I had written a web page that "collected" results from various different document management systems and presented them on one page. Just as a proof of concept/demo of what this new technology could do (back in the days when Bill Gates still didn't "get" the Internet). It was hosted on my office Linux system (that's a whole other story). The factory, the end users for this collection of documents were quite discontented with the current state of affairs. But getting any fixes through IT management would take many months. If approved at all.
My boss took the demo to his weekly factory staff meeting. Upon seeing the simplicity of the web system, a very highly placed manager on the production floor said, "We want this in production in two weeks."
With that managers pull, we got some space on a Sun server where we could build an NCSA httpd service (this before Apache) copy my Perl code over, have the IT department bless it and flip the "On" switch. Because the alternative was having the factory people pull documents through my desktop machine (which I suspected they had started to do on the day after the demo).
I wrote a tool for a utility that showed where outages were. It started off as something that ran on my desktop PC (only managers had laptops) but if I real work that needed all resources then I'd shut it down. Control room people would then ring up and want it started up if bad weather was coming. The manager eventually purloined a spare PC and the software ran happily in the control room.
Until it didn't. The software was hardcoded with my username to access a database and after I left the business and a year passed they finally cleaned up the database users. That killed the software.
It was only CxO discussions that got "my" account rehydrated while a fix was prepared, after IT swore black and blue that it wasn't possible to reactivate an account. Apparently the software was now deemed so important it was then ported to a redundant VM host with hot restart and all the fruit. The skunkworks project that ran in the background sure grew up!
Fortunately all this software is now well retired and real software to do the job is making the maps.
Didn't something like this happen to one of Microsoft's domain name registrations? Service stopped working. Tech support traced it back to the domain having been dropped from DNS. Tech support person called registrar and had them put the $35 annual fee on his personal card.
I know of a large silicon valley company where one engineer behind a spectacularly successful project became a real golden boy, VP, and had a lot of freedom in how he ran his team. Fed up with the commute he decided that it would be nicer to be in central San Francisco, but couldn't persuade facilities to provide space. His solution was just to unofficially rent some space on his company AmEx card. No-one seemed to notice until a few years later when the golden shine had worn off and he had quit. The accounting department were puzzled to get a large demand for back rent from a landlord for a property they knew nothing about. Said ex-golden boy hadn't bothered to cancel the building rental, but his AmEx card had been cancelled. Eventually the landlord noticed that he wasn't getting any money.
We have at least one service provider where I work that is actively hostile toward customers who don't/can't pay with a credit card. Every year when our maintenance renewal comes due, I have to go through an arduous multi-week process of requesting a quote and then repeatedly explaining (AGAIN) that my purchasing department is that of an Enormous State University™ that pays their bills with purchase orders rather than a credit card, and no, I don't have a lot of choice or influence about that.
At the time we picked them (4-5 years ago now), they were best in show for their niche, and from what I've seen that's not changed.
...HOWEVER...
The company was swallowed by a bigger fish in the interim, and their corporate overlords recently decreed (like 5 minutes after we paid our last annual support bill) that the service is going to be sunset in a couple years in favor of a component of their big-ticket ITSM framework. Which would be fine, except we're just about to complete a multi-year, 6-digit-budget project to overhaul how we do our customer support using a *different* big-ticket ITSM framework, and nobody is exactly chomping at the bit to chuck all that work at the last minute just for this, especially if our ITSM framework can do this (which it of course can, just not as nicely as what we use now).
Back in the good old days, I had to liaise with our Infection Control team who were told they had to have software to replace their two filing cabinets full of folders. (A short aside--people underestimate the utility of paper in a job where folk are required to nip off and audit a Ward for IC compliance at a moments notice. Irene could find the folder for a ward and be out of the office in about 20 seconds. Ward visit over, her notes were typed up by her secretary and stuck back in the file. The same job using their paperless system involved them hunting and pecking at a keyboard for 20 minutes till yhey'd found the file and last report, then printing that out to take down to the ward... Nightmare. We did try PDAs, but the screens proved to be two titchy for our Nurses to read...)
Anyway, the project included daily downloads from the labsystem to generate a textfile (using an SQL query) which was then imported into the fledgling Infection Control software. (another sidenote: we were all learning as we went along. I did the SQL, Computer Services managed the export of the file with a cron job and the IC team struggled to get to grips with with the commercial software which, lets face it was extremely specialised and cobbled together by a very small company who only had a rudimentary grasp of Infection Control issues to a specification dreamt up by a nurse somewhere who knew what they wanted but had no idea how to get a computer to achieve it. Oh, and it was a DOS-based system)
All of these steps required logging in to various bits of software to run the SQL on the Lab System and to access the server to export the file and the new server to import the file. The solution turned out to be fake users at each stage and for each server (no-one seemed to have heard of service accounts, I certainly hadn't. I fell into the IT role as a Biomedical Scientist who knew which end of an RS232 cable went into which socket (NOT the VGA socket--good grief!))
The trial actually worked (to everybodies surprise, personally I never thought Irene would get the hang of it...) and the system was rolled out and functioned for a couple of years, as various changes made to the IC software made it actually useful (Evolution in action)
Then Computer Services were audited and it was noted that they didn't have a security and access control person, so they appointed one. Their first job as they saw it was to tidy up user access to all their systems by getting rid of all these pesky fake logins. Cue complete system failure.
(Last sidenote, I promise: NHS computing especially round the edges of the major apps was a bit of a wild west. Our Apple //e and later our Windows 3.1 PCs were showing people what computers could do, while our Computer Services department had cut their teeth on Big servers storing data. It was all a bit tricky. And we were all flying by the seat of our pants.)
> "I had never transferred it to a production server. For years, it had been quietly running on my laptop, happily doing its job."
Sounds like he had a 5 year old work computer to continue working with. Yuck. That, or he made the app on day 200 of working for the employer, and ran it for .... years. 4+ years old computer?
I dunno about you, but when I swap computers I start from scratch. Maybe copy some (editor) configuration files across. Something like a service would have never made it.
Start... from... scratch...?????!!!!????!!???!
I NEVER EVER EVER EVER EVER do that. That's INCREDIBLY stupid.
I migrate EVERYTHING. Data loss is not fun, self imposed data loss is beyond idiotic.
Also, it's 2025, a 5+ year old computer is perfectly adequate for most users. They're just not that much faster than they used to be. I tell my clients when they buy new machines to expect a minimum of 5 years, 7 is likely, and 10 is not uncommon. Shit, I JUST migrated my data to a new MacBook this year, the 10 year old MacBook it replaced still works fine, I just wanted to hand it off to a user with less demanding needs and a 14 year old MacBook.
4+ years old can mean Zen 3 (my desktop was built in late 2020 and it's more that sufficient). Hell, I just bought a used laptop that's from 2015. It's not fast but perfectly usable for what I need (mostly cheap and good battery life, web surfing and light office work).
I've been issued with a laptop for 10-15 years. No issue as nowadays the laptop is mostly used to connect to a login server in the computer farm, from there to a terminal session opened on a queue machine for editing code etc then the "real work" is run as queue jobs from there ... and recently the "computer farm" is more likely to be Azure or AWS than any "local" machines.
Advantage of laptop is I can plug it in to a docking station with a couple of large monitors etc either at work or home and work in exactly the same way depending on where I'm working from now that WFH is considered normal.
Having done large scale pc deployments, people _seriously_ underestimate the cost of offering multiple options. Despite laptops being somewhat more costly than desktops, it's much cheaper to give everyone laptops than to give almost everyone laptops, and a few people desktops.
Or the more famous works on the machine that we lost behind a new wall
Actually not, but why would you not just license and use flexlm† (lmgrd) rather than reinvent the wheel?
The upside for your typical BoFH is that with a little skulduggery he or she can more or less easily temporarily circumvent flexlm based systems while licensing issues are being resolved (normally the boss supervising the PFY's kicking of heads in the accounts payable sewer.) Thieves honour ensured that the software vendor's rightful dues were quickly paid and the skulduggery removed.
If regomised Sam had lost or totalled his laptop and thereby losing the private halves of the key pairs he would have been in much deeper hole especially if the public halves of said key pairs were hard coded into his application.
It wouldn't be the first time I have fired up tcpdump to locate forgotten and lost services/servers that had finally gone AWOL paralysing critical applications.
When unencrypted services like FTP were being uncritically decommissioned without consultation by the usual security thespian suspects this wasn't an unusual occurrence. Try convincing these wizards that a jailed public anon-only FTP server running in a read only environment isn't a first class security threat (and that using a normal SSH service with sftp and keys isn't really an improvement in that particular situation.)
The same clowns firewalling the germane TCP ports on the enterprise licence servers is/was a pretty regular occurrence usually synchronised with the turnover of ITsec staff.
† Flexnet is a horse of a different colour which does seem to phone home like young Sam's.
why would you not just license and use flexlm† (lmgrd) rather than reinvent the wheel
Aargh, that brings back memories best forgotten. Clunky old stuff.
Back in the mid-90s I worked for a large HW+SW manufacturer. My group started to get support calls from customers, mostly in the far east, but we had no record of them having bought the software in question. They insisted that they had a legitimate copy. After much investigation we found that the sales guys were giving them the SW licences (just bits of paper with permission to run it) and copies of the media free, to sweeten their hardware deals on which they got commission. Customers thought it was all above board, but our SW group had no record of the "sale", so no revenue nor credit. Senior managers were muttering about closing our "unprofitable" team down due to low sales.
It was decided to add enforced licensing (to protect us from our own sales guys!) and FlexLM was chosen. I got the job of adding it to our products. Testing this meant generating test licences for lab systems, and eventually the licensing centre decided that they trusted me enough just to give me a copy of the tool that would create licences, on the understanding that I wouldn't abuse it (those were the days...). I only had keys for our product, of course.
Fast forward several years, customers who used these on critical systems were increasingly irritated by the licence software which was proving more trouble than it was worth. They had issues with changed hardware (replaced motherboard was a classic) necessitating new keys at short notice, network glitches blocking access to the lmgrd daemon, etc. The sales guys had been educated, managers had realised just how much money we were making with the software, and commission was now also paid on software sales. New versions of the product were released without FlexLM licensing, to a general sigh of relief. The licence centres were closed.
Many years later one large customer with an old system (in an industry that considered 10 years to be a short replacement cycle) had a problem, a replaced system with different ID, failed licence. They couldn't move to the new software version until they'd updated other components and hardware, and that update cycle was scheduled for a year away. Mass panic ensued as the system failed to restart. I got a phone call, did I by any chance know any way to bypass the licence scheme to get them running again?
A rummage in old project subdirectories turned up the lm generation tool which still ran on the current OS version, and I found old emails still had the instructions. A few minutes later a new licence was winging its way to them. I got beers from the field service team.
The motto of this is never throw old software away, you never know when you might need it. I still have it, somewhere, I think...
Anon cos I was supporting customers in actual green fields with their weaponry with allegedly perpetual licenses for one of the equipment management systems. A few years into support I got a call that several of the management workstations had stopped working. This turned out to be an expired license. I called the supplier of this already long in the tooth system and they had to get an old system out of bed to cut new keys. The so called perpetual license expired after five years. I set my calendar for another five years and right on time those strings expired too. Fortunately the old key cutting system was still going and got new keys. After that, I decided to retire the whole customer system a couple years later citing ever increasing risk that we couldn’t sustain it,
My colleague had a desktop PC under his desk. Attached was a post it note saying do not power off ...
Anyway, we moved offices and I enquired about that desktop. Thankfully someone from operations had done a check and then cloned it into a data centre. One of our old systems would have just stopped for end users.
Mid-2000, I was part of a big team of DC kit ops.
This then, 12 MdUSD corporation had a web server, like every one else. It was managed in a very silo way by a weird dude working on the same site as us.
No-one knew where the freaking web server was running but not many worried much.
We started to worry, one day, when the darn thing stopped entirely and the dude was nowhere to be seen, in vacation.
After one week of mgmt turmoil, down web site, and upon the weirdo's return, it all appeared clear:
- The darn thing was running in his office, under his feet, on a dusty carpet
- Of course, no backup (what's that ?) of any sort, no power redundancy, no RAID protection, all local storage etc ...
I think one week later, we re-platformed all the stuff on a proper server, in the DC !
I used to work for a company, self up all the network and servers etc, and then left.
I worked for another company for a few years, was made redundant and went back to the first company. Even though I had worked there, I wasn't allowed admin access. Until the day no one received any email at all and I was told to fix it. I was handed the documentation and discovered that it was the same documentation that I wrote before I left!
The issue was that my colleague had redirected all the emails for the company to go through his laptop. His excuse so he could check email orders. Anyway, he goes on holiday takes the laptop home... and his mother unplugs it to run the Hoover around, battery goes flat, no more emails.
I fixed the problem, everything works fine until a few weeks later and exactly the same thing happens again!!
I was there for 12 years the first time, lasted 3 months the second time - the reasons I left the first time were still there (actually the son of the MD was there as well, Dr Evil & mini me as they known), did try to make it work the second time round but jumped ship.
Years before the interwebs and computers small enough to carry I worked in an engineering firm in which one gentleman in the spares department "knew about stuff before the computer did".
When he retired the computer department redirected "his" reports to the departments where they belonged and Hey Pesto! The computer was supplying answers a day faster than before.
Yes, this chap had been directing everything to his desk and holding if for a day or more. This sort of nonsense was so rampant in the engineering world Ollie White talked about it in his video training on Material Requirements Planning. When I went freelance I would keep an ear out for tales of the person who knew more than the computer did, and slip a quiet word to the DPM.
That guy from the spares department also was the only person in that office with a cubicle. The other staff sat at desks in the otherwise open plan office.
When, after he retired, they removed the stacks of greenbar "lining" the walls of his cube, they discovered that the greenbar *was* the walls. He'd literally built himself a cube out of stacks of computer paper.
Used it to remote into another system that was outside of my LANA (US area-code registry) and would incur huge telecom bills if I dialed in directly.
Totally forgot about it since it was working so well.
Sold the townhouse to a nice couple. A few days later got a call from a US Government number asking about the hidden server and the extra phone line coming in. Turns out the couple worked for a 3-letter agency in Langley, Virginia.
I think that you meant LATA (Local Access and Transport Area). Not the same as a Numbering Plan Area (from whence "Area Code"). See <https://en.wikipedia.org/wiki/Local_Access_And_Transport_Area>.
FTA: "LATA boundaries tend to be drawn around markets, and not necessarily along existing state or area code borders. Some LATAs cross over state boundaries, such as those for the New York metropolitan area and Greenwich, Connecticut; Chicago, Illinois; Portland, Oregon; and areas between Maryland, Virginia, and West Virginia. Area codes and LATAs do not necessarily share boundaries; many LATAs exist in multiple area codes, and many area codes exist in multiple LATAs. "
That's a funny story that the me today would find impossible to do. The me of a couple decades ago could probably have done differently.
These days, I'm such a staunch advocate for co/cd and containerization it would be incredibly unlikely. Not to mention that I now actively avoid direct access to production servers.
Well not quite the same but national rail doesn't know what server runs it's now depreciated train alerts system. The one that sends texts when your train is running late. They shut down the web page that lets you set up the alerts ages ago and confirmed to me that they were shutting the system down, however I still received alerts until very recently when the alert expired. I even got reminder texts that my alert was due to expire with a link to a now non existent page. It's a shame they shut it down, it was a useful feature!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
