back to article Developer wrote a critical app and forgot where it ran – until it stopped running

With the weekend behind us, it's time to once again ask the question "Who, Me?" That's the name of The Register's Monday column in which we share reader-contributed confessions of making a mess with tech. This week, meet a reader we'll Regomize as "Sam" who in the late 2000s built an application for his colleagues that he …

  1. Joe W Silver badge
    Pirate

    Ah... deployment...

    I'm one step ahed of him. Or... half a step? The application runs on a VM. Admittedly, not in the production environment. Without having an entry as a "service" in the company I won't get that. Without a security audit and concept I won't get that, and without being on a production machine I won't pass said audit. The ITSec guy I'm working with is trying his best. Yes, genuine support, knowledgable concering the company rules and... how to.. ehm... yeah, circumvent some to make progress. I get the need to look at those things, and I also want to avoid this thing spinning out of hand and being all important all of a sudden, and all that without thinking about certain important things. At the moment it feels stupid, though. My ITSec-guy agrees, which makes the whole thing actually fun to work on. We are taking this (and our work on it) serious, but not overly so. Icon: how I feel about it... (a bit)

    1. SVD_NL Silver badge

      Re: Ah... deployment...

      Man, auditing those apps sounds like an easy job.

      "does it run in the production environment?"

      "currently not, it first needs to pass this aud..."

      "sorry that's unacceptable, audit failed"

      I have no problems with the concept here, but i feel like validating in a twin test/acceptance environment should allow you to pass the audit, especially for new apps.

    2. Paul Hovnanian Silver badge

      Re: Ah... deployment...

      Something similar. I had written a web page that "collected" results from various different document management systems and presented them on one page. Just as a proof of concept/demo of what this new technology could do (back in the days when Bill Gates still didn't "get" the Internet). It was hosted on my office Linux system (that's a whole other story). The factory, the end users for this collection of documents were quite discontented with the current state of affairs. But getting any fixes through IT management would take many months. If approved at all.

      My boss took the demo to his weekly factory staff meeting. Upon seeing the simplicity of the web system, a very highly placed manager on the production floor said, "We want this in production in two weeks."

      With that managers pull, we got some space on a Sun server where we could build an NCSA httpd service (this before Apache) copy my Perl code over, have the IT department bless it and flip the "On" switch. Because the alternative was having the factory people pull documents through my desktop machine (which I suspected they had started to do on the day after the demo).

  2. Dafyd Colquhoun

    Similar thing, but zombie user not laptop

    I wrote a tool for a utility that showed where outages were. It started off as something that ran on my desktop PC (only managers had laptops) but if I real work that needed all resources then I'd shut it down. Control room people would then ring up and want it started up if bad weather was coming. The manager eventually purloined a spare PC and the software ran happily in the control room.

    Until it didn't. The software was hardcoded with my username to access a database and after I left the business and a year passed they finally cleaned up the database users. That killed the software.

    It was only CxO discussions that got "my" account rehydrated while a fix was prepared, after IT swore black and blue that it wasn't possible to reactivate an account. Apparently the software was now deemed so important it was then ported to a redundant VM host with hot restart and all the fruit. The skunkworks project that ran in the background sure grew up!

    Fortunately all this software is now well retired and real software to do the job is making the maps.

    1. OldGeezer
      Happy

      Re: Similar thing, but zombie user not laptop

      On a lighter note, years ago I wrote an app to fill in deadbeat timesheets (100% allocated to the same job in the same location) and 6 months after I left I got a call asking why I was still submitting them. Ooopppps!

      1. SVD_NL Silver badge
        Devil

        Re: Similar thing, but zombie user not laptop

        The more important question here: did they continue to pay you for all those hours you "worked"?

    2. Csmy

      Re: Similar thing, but zombie user not laptop

      I know of one ftse100 company whose CI/CD pipeline will fall over spectacularly if a particular employee ever leaves.

      1. Korev Silver badge
        Facepalm

        Re: Similar thing, but zombie user not laptop

        I worked somewhere where the hosted Git repo went down for days because it automatically charged someone's credit card which was fine until he left...

        1. Mike007 Silver badge

          Re: Similar thing, but zombie user not laptop

          We have a main card that is used for most payments to suppliers. It expired in January. By the end of the year we will have found everything that needed updating with the new card details...

          1. anothercynic Silver badge
            Facepalm

            Re: Similar thing, but zombie user not laptop

            And I hope that all those things will have been documented, yes? YES?!

          2. Paul Hovnanian Silver badge

            Re: Similar thing, but zombie user not laptop

            Didn't something like this happen to one of Microsoft's domain name registrations? Service stopped working. Tech support traced it back to the domain having been dropped from DNS. Tech support person called registrar and had them put the $35 annual fee on his personal card.

        2. Anonymous Coward
          Anonymous Coward

          Re: Similar thing, but zombie user not laptop

          I know of a large silicon valley company where one engineer behind a spectacularly successful project became a real golden boy, VP, and had a lot of freedom in how he ran his team. Fed up with the commute he decided that it would be nicer to be in central San Francisco, but couldn't persuade facilities to provide space. His solution was just to unofficially rent some space on his company AmEx card. No-one seemed to notice until a few years later when the golden shine had worn off and he had quit. The accounting department were puzzled to get a large demand for back rent from a landlord for a property they knew nothing about. Said ex-golden boy hadn't bothered to cancel the building rental, but his AmEx card had been cancelled. Eventually the landlord noticed that he wasn't getting any money.

        3. AtomicWombat

          Re: Similar thing, but zombie user not laptop

          We have at least one service provider where I work that is actively hostile toward customers who don't/can't pay with a credit card. Every year when our maintenance renewal comes due, I have to go through an arduous multi-week process of requesting a quote and then repeatedly explaining (AGAIN) that my purchasing department is that of an Enormous State University™ that pays their bills with purchase orders rather than a credit card, and no, I don't have a lot of choice or influence about that.

          1. Jou (Mxyzptlk) Silver badge

            Re: Similar thing, but zombie user not laptop

            Does this service provider at least provide good service and get the toilets cleaned to sparkly level?

            1. AtomicWombat

              Re: Similar thing, but zombie user not laptop

              At the time we picked them (4-5 years ago now), they were best in show for their niche, and from what I've seen that's not changed.

              ...HOWEVER...

              The company was swallowed by a bigger fish in the interim, and their corporate overlords recently decreed (like 5 minutes after we paid our last annual support bill) that the service is going to be sunset in a couple years in favor of a component of their big-ticket ITSM framework. Which would be fine, except we're just about to complete a multi-year, 6-digit-budget project to overhaul how we do our customer support using a *different* big-ticket ITSM framework, and nobody is exactly chomping at the bit to chuck all that work at the last minute just for this, especially if our ITSM framework can do this (which it of course can, just not as nicely as what we use now).

      2. KittenHuffer Silver badge
        Coat

        Re: Similar thing, but zombie user not laptop

        Yeah, that would be TSLA, and the user would be EMvicePres!

        ---------> Mine's the one with ehd DOdGEy lining!

        1. Someone Else Silver badge
          Coat

          Re: Similar thing, but zombie user not laptop

          ..or the DOG-E eared pages?

    3. John 110
      Boffin

      Re: Similar thing, but zombie user not laptop

      Back in the good old days, I had to liaise with our Infection Control team who were told they had to have software to replace their two filing cabinets full of folders. (A short aside--people underestimate the utility of paper in a job where folk are required to nip off and audit a Ward for IC compliance at a moments notice. Irene could find the folder for a ward and be out of the office in about 20 seconds. Ward visit over, her notes were typed up by her secretary and stuck back in the file. The same job using their paperless system involved them hunting and pecking at a keyboard for 20 minutes till yhey'd found the file and last report, then printing that out to take down to the ward... Nightmare. We did try PDAs, but the screens proved to be two titchy for our Nurses to read...)

      Anyway, the project included daily downloads from the labsystem to generate a textfile (using an SQL query) which was then imported into the fledgling Infection Control software. (another sidenote: we were all learning as we went along. I did the SQL, Computer Services managed the export of the file with a cron job and the IC team struggled to get to grips with with the commercial software which, lets face it was extremely specialised and cobbled together by a very small company who only had a rudimentary grasp of Infection Control issues to a specification dreamt up by a nurse somewhere who knew what they wanted but had no idea how to get a computer to achieve it. Oh, and it was a DOS-based system)

      All of these steps required logging in to various bits of software to run the SQL on the Lab System and to access the server to export the file and the new server to import the file. The solution turned out to be fake users at each stage and for each server (no-one seemed to have heard of service accounts, I certainly hadn't. I fell into the IT role as a Biomedical Scientist who knew which end of an RS232 cable went into which socket (NOT the VGA socket--good grief!))

      The trial actually worked (to everybodies surprise, personally I never thought Irene would get the hang of it...) and the system was rolled out and functioned for a couple of years, as various changes made to the IC software made it actually useful (Evolution in action)

      Then Computer Services were audited and it was noted that they didn't have a security and access control person, so they appointed one. Their first job as they saw it was to tidy up user access to all their systems by getting rid of all these pesky fake logins. Cue complete system failure.

      (Last sidenote, I promise: NHS computing especially round the edges of the major apps was a bit of a wild west. Our Apple //e and later our Windows 3.1 PCs were showing people what computers could do, while our Computer Services department had cut their teeth on Big servers storing data. It was all a bit tricky. And we were all flying by the seat of our pants.)

  3. Korev Silver badge
    Coat

    I guess it should now be called a Lapstop after that...

  4. Anonymous Coward
    Anonymous Coward

    No Lifecycle Upgrades

    > "I had never transferred it to a production server. For years, it had been quietly running on my laptop, happily doing its job."

    Sounds like he had a 5 year old work computer to continue working with. Yuck. That, or he made the app on day 200 of working for the employer, and ran it for .... years. 4+ years old computer?

    I dunno about you, but when I swap computers I start from scratch. Maybe copy some (editor) configuration files across. Something like a service would have never made it.

    1. Anonymous Coward
      Anonymous Coward

      Re: No Lifecycle Upgrades

      Start... from... scratch...?????!!!!????!!???!

      I NEVER EVER EVER EVER EVER do that. That's INCREDIBLY stupid.

      I migrate EVERYTHING. Data loss is not fun, self imposed data loss is beyond idiotic.

      Also, it's 2025, a 5+ year old computer is perfectly adequate for most users. They're just not that much faster than they used to be. I tell my clients when they buy new machines to expect a minimum of 5 years, 7 is likely, and 10 is not uncommon. Shit, I JUST migrated my data to a new MacBook this year, the 10 year old MacBook it replaced still works fine, I just wanted to hand it off to a user with less demanding needs and a 14 year old MacBook.

    2. Anonymous Coward
      Anonymous Coward

      Re: No Lifecycle Upgrades

      4+ years old can mean Zen 3 (my desktop was built in late 2020 and it's more that sufficient). Hell, I just bought a used laptop that's from 2015. It's not fast but perfectly usable for what I need (mostly cheap and good battery life, web surfing and light office work).

  5. Michael H.F. Wilkinson Silver badge

    Serious OOPS!

    Not something I would want to have running on my laptop, especially if it is my development laptop. I am really surprised this issue didn't raise its head earlier. It suggests the laptop wasn't just left at work, but was also left running 24/7.

    1. find users who cut cat tail

      Re: Serious OOPS!

      That part surprises me to. I mean, if you leave it always plugged in at one place for several years why would you get a laptop? A workstation would be a much better deal.

      1. storner
        Pint

        Re: Serious OOPS!

        I'll bet you one of these -->

        that laptops are standard equipment, whereas a workstation is a special request requiring forms to be filled out, budgets to be allocated etc.

        1. Anonymous Coward
          Anonymous Coward

          Re: Serious OOPS!

          I've been issued with a laptop for 10-15 years. No issue as nowadays the laptop is mostly used to connect to a login server in the computer farm, from there to a terminal session opened on a queue machine for editing code etc then the "real work" is run as queue jobs from there ... and recently the "computer farm" is more likely to be Azure or AWS than any "local" machines.

          Advantage of laptop is I can plug it in to a docking station with a couple of large monitors etc either at work or home and work in exactly the same way depending on where I'm working from now that WFH is considered normal.

        2. Dave314159ggggdffsdds Silver badge

          Re: Serious OOPS!

          Having done large scale pc deployments, people _seriously_ underestimate the cost of offering multiple options. Despite laptops being somewhat more costly than desktops, it's much cheaper to give everyone laptops than to give almost everyone laptops, and a few people desktops.

      2. pirxhh

        Re: Serious OOPS!

        Laptops use less power and come with built-in UPS... until the battery inevitably dies, that is.

        1. mirachu Bronze badge

          Re: Serious OOPS!

          Old Thinkpads with swappable batteries <3

    2. Jou (Mxyzptlk) Silver badge

      Re: Serious OOPS!

      Oh, that is simple: The service was obviously only needed during work hours. And everyone had, probably, the same work hours as "Sam".

      1. Emir Al Weeq

        Re: Serious OOPS!

        I thought that, but it only really works if Sam was always first in, last out (a "stack" employee) and never took a holiday.

        > For years, it had been quietly running on my laptop

  6. mpi Silver badge
    Joke

    Love how this gives a whole new level to the old saying

    "Works on My Machine"

    1. Yet Another Anonymous coward Silver badge

      Re: Love how this gives a whole new level to the old saying

      Or the more famous works on the machine that we lost behind a new wall

      1. Dostoevsky Bronze badge
        Pint

        Re: Love how this gives a whole new level to the old saying

        A beautiful story. *sniffle, tear drops* A reminder of happier times. Thanks for bringing it up! Cheers --->

        1. Yet Another Anonymous coward Silver badge

          Re: Love how this gives a whole new level to the old saying

          It made me feel old as well

  7. Bebu sa Ware
    Windows

    Why Didn't They Ask...

    Actually not, but why would you not just license and use flexlm† (lmgrd) rather than reinvent the wheel?

    The upside for your typical BoFH is that with a little skulduggery he or she can more or less easily temporarily circumvent flexlm based systems while licensing issues are being resolved (normally the boss supervising the PFY's kicking of heads in the accounts payable sewer.) Thieves honour ensured that the software vendor's rightful dues were quickly paid and the skulduggery removed.

    If regomised Sam had lost or totalled his laptop and thereby losing the private halves of the key pairs he would have been in much deeper hole especially if the public halves of said key pairs were hard coded into his application.

    It wouldn't be the first time I have fired up tcpdump to locate forgotten and lost services/servers that had finally gone AWOL paralysing critical applications.

    When unencrypted services like FTP were being uncritically decommissioned without consultation by the usual security thespian suspects this wasn't an unusual occurrence. Try convincing these wizards that a jailed public anon-only FTP server running in a read only environment isn't a first class security threat (and that using a normal SSH service with sftp and keys isn't really an improvement in that particular situation.)

    The same clowns firewalling the germane TCP ports on the enterprise licence servers is/was a pretty regular occurrence usually synchronised with the turnover of ITsec staff.

    † Flexnet is a horse of a different colour which does seem to phone home like young Sam's.

    1. imanidiot Silver badge

      Re: Why Didn't They Ask...

      The basic assumption to losing the private keys would be that "Sam" was competent enough to have backups of those keys somewhere other than his laptop. I know, dangerous assumption.

    2. Anonymous Coward
      Anonymous Coward

      Re: Why Didn't They Ask...

      why would you not just license and use flexlm† (lmgrd) rather than reinvent the wheel

      Aargh, that brings back memories best forgotten. Clunky old stuff.

      Back in the mid-90s I worked for a large HW+SW manufacturer. My group started to get support calls from customers, mostly in the far east, but we had no record of them having bought the software in question. They insisted that they had a legitimate copy. After much investigation we found that the sales guys were giving them the SW licences (just bits of paper with permission to run it) and copies of the media free, to sweeten their hardware deals on which they got commission. Customers thought it was all above board, but our SW group had no record of the "sale", so no revenue nor credit. Senior managers were muttering about closing our "unprofitable" team down due to low sales.

      It was decided to add enforced licensing (to protect us from our own sales guys!) and FlexLM was chosen. I got the job of adding it to our products. Testing this meant generating test licences for lab systems, and eventually the licensing centre decided that they trusted me enough just to give me a copy of the tool that would create licences, on the understanding that I wouldn't abuse it (those were the days...). I only had keys for our product, of course.

      Fast forward several years, customers who used these on critical systems were increasingly irritated by the licence software which was proving more trouble than it was worth. They had issues with changed hardware (replaced motherboard was a classic) necessitating new keys at short notice, network glitches blocking access to the lmgrd daemon, etc. The sales guys had been educated, managers had realised just how much money we were making with the software, and commission was now also paid on software sales. New versions of the product were released without FlexLM licensing, to a general sigh of relief. The licence centres were closed.

      Many years later one large customer with an old system (in an industry that considered 10 years to be a short replacement cycle) had a problem, a replaced system with different ID, failed licence. They couldn't move to the new software version until they'd updated other components and hardware, and that update cycle was scheduled for a year away. Mass panic ensued as the system failed to restart. I got a phone call, did I by any chance know any way to bypass the licence scheme to get them running again?

      A rummage in old project subdirectories turned up the lm generation tool which still ran on the current OS version, and I found old emails still had the instructions. A few minutes later a new licence was winging its way to them. I got beers from the field service team.

      The motto of this is never throw old software away, you never know when you might need it. I still have it, somewhere, I think...

      1. Anonymous Coward
        Anonymous Coward

        Re: Why Didn't They Ask...

        Anon cos I was supporting customers in actual green fields with their weaponry with allegedly perpetual licenses for one of the equipment management systems. A few years into support I got a call that several of the management workstations had stopped working. This turned out to be an expired license. I called the supplier of this already long in the tooth system and they had to get an old system out of bed to cut new keys. The so called perpetual license expired after five years. I set my calendar for another five years and right on time those strings expired too. Fortunately the old key cutting system was still going and got new keys. After that, I decided to retire the whole customer system a couple years later citing ever increasing risk that we couldn’t sustain it,

  8. Anonymous Coward
    Anonymous Coward

    Server under the desk..

    My colleague had a desktop PC under his desk. Attached was a post it note saying do not power off ...

    Anyway, we moved offices and I enquired about that desktop. Thankfully someone from operations had done a check and then cloned it into a data centre. One of our old systems would have just stopped for end users.

  9. Anonymous Coward
    Anonymous Coward

    web server mid-2000

    Mid-2000, I was part of a big team of DC kit ops.

    This then, 12 MdUSD corporation had a web server, like every one else. It was managed in a very silo way by a weird dude working on the same site as us.

    No-one knew where the freaking web server was running but not many worried much.

    We started to worry, one day, when the darn thing stopped entirely and the dude was nowhere to be seen, in vacation.

    After one week of mgmt turmoil, down web site, and upon the weirdo's return, it all appeared clear:

    - The darn thing was running in his office, under his feet, on a dusty carpet

    - Of course, no backup (what's that ?) of any sort, no power redundancy, no RAID protection, all local storage etc ...

    I think one week later, we re-platformed all the stuff on a proper server, in the DC !

  10. Blackjack Silver badge

    Amazing the laptop never overheated or was turned off at the wrong time.

  11. Anonymous Coward
    Anonymous Coward

    I used to work for a company, self up all the network and servers etc, and then left.

    I worked for another company for a few years, was made redundant and went back to the first company. Even though I had worked there, I wasn't allowed admin access. Until the day no one received any email at all and I was told to fix it. I was handed the documentation and discovered that it was the same documentation that I wrote before I left!

    The issue was that my colleague had redirected all the emails for the company to go through his laptop. His excuse so he could check email orders. Anyway, he goes on holiday takes the laptop home... and his mother unplugs it to run the Hoover around, battery goes flat, no more emails.

    I fixed the problem, everything works fine until a few weeks later and exactly the same thing happens again!!

    I was there for 12 years the first time, lasted 3 months the second time - the reasons I left the first time were still there (actually the son of the MD was there as well, Dr Evil & mini me as they known), did try to make it work the second time round but jumped ship.

    1. Stevie

      His excuse so he could check email orders

      Years before the interwebs and computers small enough to carry I worked in an engineering firm in which one gentleman in the spares department "knew about stuff before the computer did".

      When he retired the computer department redirected "his" reports to the departments where they belonged and Hey Pesto! The computer was supplying answers a day faster than before.

      Yes, this chap had been directing everything to his desk and holding if for a day or more. This sort of nonsense was so rampant in the engineering world Ollie White talked about it in his video training on Material Requirements Planning. When I went freelance I would keep an ear out for tales of the person who knew more than the computer did, and slip a quiet word to the DPM.

      That guy from the spares department also was the only person in that office with a cubicle. The other staff sat at desks in the otherwise open plan office.

      When, after he retired, they removed the stacks of greenbar "lining" the walls of his cube, they discovered that the greenbar *was* the walls. He'd literally built himself a cube out of stacks of computer paper.

  12. Jou (Mxyzptlk) Silver badge

    Finally a real "Who me" again!

    And not one from before the year 2000 :D. A good gem!

  13. elDog

    Buried an old PC running a proxy server in a wall in my townhouse.

    Used it to remote into another system that was outside of my LANA (US area-code registry) and would incur huge telecom bills if I dialed in directly.

    Totally forgot about it since it was working so well.

    Sold the townhouse to a nice couple. A few days later got a call from a US Government number asking about the hidden server and the extra phone line coming in. Turns out the couple worked for a 3-letter agency in Langley, Virginia.

    1. A.P. Veening Silver badge
      Joke

      Re: Buried an old PC running a proxy server in a wall in my townhouse.

      Turns out the couple worked for a 3-letter agency in Langley, Virginia.

      There is No Such Agency.

      1. Antron Argaiv Silver badge
        Thumb Up

        Re: Buried an old PC running a proxy server in a wall in my townhouse.

        Langley is the home of the Culinary Institute of America. You're thinking of Fort Meade, MD.

    2. NITS

      Re: Buried an old PC running a proxy server in a wall in my townhouse.

      I think that you meant LATA (Local Access and Transport Area). Not the same as a Numbering Plan Area (from whence "Area Code"). See <https://en.wikipedia.org/wiki/Local_Access_And_Transport_Area>.

      FTA: "LATA boundaries tend to be drawn around markets, and not necessarily along existing state or area code borders. Some LATAs cross over state boundaries, such as those for the New York metropolitan area and Greenwich, Connecticut; Chicago, Illinois; Portland, Oregon; and areas between Maryland, Virginia, and West Virginia. Area codes and LATAs do not necessarily share boundaries; many LATAs exist in multiple area codes, and many area codes exist in multiple LATAs. "

      1. elDog

        Re: Buried an old PC running a proxy server in a wall in my townhouse.

        Right you are. I've stopped thinking about those ancient money-making gimmicks now the phone companies are draining our accounts with cell phones.

  14. Softsuits

    Undocument security update

    That is the usual excuse. Because time is money and five nines needs a trouble ticket solution. Paddy's Day and back to my Jamesons I go.

  15. tracker1

    That's a funny story that the me today would find impossible to do. The me of a couple decades ago could probably have done differently.

    These days, I'm such a staunch advocate for co/cd and containerization it would be incredibly unlikely. Not to mention that I now actively avoid direct access to production servers.

  16. Gazzat5

    Well not quite the same but national rail doesn't know what server runs it's now depreciated train alerts system. The one that sends texts when your train is running late. They shut down the web page that lets you set up the alerts ages ago and confirmed to me that they were shutting the system down, however I still received alerts until very recently when the alert expired. I even got reminder texts that my alert was due to expire with a link to a now non existent page. It's a shame they shut it down, it was a useful feature!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like