back to article That thing you were utterly sure would never happen? Yeah, well, guess what …

A Who, Me? moment last week introduced a much-needed laugh into the lives of at least a few end users, after a bit of testing in production had all too familiar consequences. Cuvva is an app-based car insurance service. It is a handy thing for some — rather than signing up for an annual policy, a customer can use its app to …

  1. lansalot

    Ah, will you not have a cup of tea father...

    My go-to test strings are the Father-Ted favourites, "arse", "feck", "drink" and "girls".

    Two of which once popped up in front of some bigwigs I was demoing something to. Wasn't the last two....

    1. Korev Silver badge
      Windows

      Re: Ah, will you not have a cup of tea father...

      That would be an ecumenical matter

      1. Anonymous Coward
        Anonymous Coward

        Re: Ah, will you not have a cup of tea father...

        On a Father Ted tangent - I once asked an Irish woman about Father Ted, (I was curious how it was accepted over there). Apparently it was very popular in NI as it was very realistic and true to life....

        Personally I think she meant that it was stereotypical - but was too drunk. She then told me that she loved me, then threw up over her car.

        1. Anonymous Coward
          Anonymous Coward

          Re: Ah, will you not have a cup of tea father...

          I can confirm - but usually the real ladies manage to puke in their handbags.......

          Then go fishing for the keys.

          1. Anonymous Coward
            Anonymous Coward

            Re: Ah, will you not have a cup of tea father...

            Being a non-drinker, I often end up as the designated driver.

            She was very apologetic, but as I pointed out - it was her car not mine....

    2. Zarno
      Mushroom

      Re: Ah, will you not have a cup of tea father...

      My go-to prefixes for things and kludges that go bump in the night and never should exit are variations on "Nuke". Such as "NukeMeNow", "Nuke_After(x)Days", etc.

      Relatively tame, and not the same clenchable as "Dirty_Blooming_Hack" or "Total Inability To Just Operate Beautifully" if it squeaks out into the world.

      I must admit, I've had a few slip through when I've handed a project over and forgot I wouldn't be the one finishing it.

      Icon because I'll be admiring the fusing ball of hydrogen in a few hours.

      1. Yet Another Anonymous coward Silver badge

        Re: Ah, will you not have a cup of tea father...

        I always use "dialog with stakeholder", "lessons have been learned", "we are innovating for your pleasure" or "leveraging synergy"

        That way if there is a bug and the message goes out to a million users I don't have to apologise, or even fix it.

        1. veti Silver badge

          Re: Ah, will you not have a cup of tea father...

          You don't "use" those phrases, you can only utilise them. You need those four extra letters to distract from the vacuity of the other words.

  2. Anonymous Coward
    Anonymous Coward

    We had a little workshop test environment set up with with some rather fruity login banners.

    One day a field engineer was on a panic job and couldn't get hold of a vital part... but he knew where there was one just laying about and borrowed it without asking.

    Unfortunately the IT Manager at the Christian society where he tried to install it was not pleased with what he saw as the kit powered up

    A number of signs went up in the workshop... "Don't borrow without asking first" and, more importantly, "Erase/Default programmable devices before reuse"

    1. John Brown (no body) Silver badge

      Better would have been "no matter what you are working on, assume a customer will see it". Single sign, single policy, and no mater the rush, safe to take without checking first.

    2. Anonymous Coward
      Anonymous Coward

      Had a Apple Mac II (cx? - small one, had NuBus slots, so 199x???) that I used. Every now and again I would into work and found that the sales woman had taken it out on the road for demos at customer sites.

      Until I set the startup screen to be a page 3 model, and it was never touched after that.....

    3. Anonymous Coward
      Anonymous Coward

      I worked in a IT shop

      Back in the day we used to make beige boxes.

      This is early internet.

      One of the engineers assembling the machines was downloading some 'adult material' for 'personal use' in the workshop - it had a T line - speedo for his dodgy gifs....

      Well, his download folder was on a partition on the imaging machine.

      Another engineer innocently (we think) set up a new image for the HDs going out to a customer...

      He included the folder.

      When it came to light what had happened, we had to send out a 'driver' disk to 200 users with instructions to URGENTLY patch their machine, by booting from the floppy. It silently deleted the partition on the box, while putting random 'progress' bars on the screen.....

      Bullet dodged with customers - Boses did find out. Engineer moved on.

  3. WanderingHaggis

    Done that to the wrong mailing list. Being a bit bored I had a fake startrek type message. Most receivers were OK but not everyone. Development definitely needs to be air gapped from the world.

    1. J.G.Harston Silver badge

      Star wars...

      Anybody remember the Dynix library information system, that if you confused it in a certain manner gave off things like The system cannae take it, captain.

  4. Noodle
    Devil

    The Devil Makes Work for Idle Hands

    .. And also for bored programmers testing things, as we found at the company I worked for in the late 90s. We generated mailing address data that was printed on labels for catalogs and such. One client was very surprised to receive a sample catalog addressed to "Satan, 666 Devil Street" with various other occult references attached.

    Fortunately they saw the funny side.

    1. Flocke Kroes Silver badge

      Re: The Devil Makes Work for Idle Hands

      UK post codes do not begin with an X so those are safe to use in your test database - except XM4 5HQ.

    2. Potty Professor
      Facepalm

      Re: The Devil Makes Work for Idle Hands

      When I started working at the local pharmacy as their delivery driver as a part time job in my retirement, I was given a delivery for Mrs. Jane Seymour, 8, Henry Street. It took me quite a while to get the joke, so when I returned to base, I deposited the parcel in the "No Reply" box and said nothing. The Branch Manager was "Not Amused" when she found it there, and a series of questions were asked, resulting in a stern warning that further jokes were to be run past her before deploying.

  5. Chairman of the Bored

    Probably apocryphal yet amusing

    Back when I worked for the worlds largest operator of grey-painted ships and nuclear subs, we had a senior developer who told a story (*):

    A certain early build of a surface ship combat system had a training mode that would simulate threats to provide operators with realistic displays they might see on a bad day. When under antiship missile attack, when the raid impacted the defended point an ASCII pic of a donkey would show up on console, along with the phrase "kiss your ass goodbye". That somehow made it onto a platform and the CO was ... Surprisingly amused.

    (*) What's the difference between a fairy tale and a sea tale? The fairy tale starts with "Once upon a time", and the sea tale begins with "There I was..."

  6. Anonymous Coward
    Anonymous Coward

    Ooops...

    many years ago, back when PCs were a twinkle in someone's eye... I had a DCL script running so that when a certain user logged in it posted messages at random times to the status line on their terminal (with their consent... haven't we all been there? no? you must be too young).

    This was, I reasoned in my innocent youth, safe because the machine to which the terminal was connected was classified and people never ever share usernames and passwords on that machine... until the project manager used that username/password for a customer demonstration and a quote from a certain songster by the name of Ian Anderson popped into the status line... mentioning candles and dinner. The usual user of the account was an attractive member of the fair(er) sex...

    Somehow it was my problem for running the code, not theirs for illegally using someone else's account details to login. Sigh.

    I survived the verbal hosing, a sadder and wiser man!

    1. Pascal Monett Silver badge

      And the lesson learned is ?

      Never give anyone outside the dev team anything else but a guest login.

      Carefully crafted to allow exactly what is supposed to be done, and nothing else.

      1. My other car WAS an IAV Stryker
        Black Helicopters

        Re: Guest Login

        It was mentioned the system connected to was classified.

        On a classified system there is No. Such. Thing. as a "guest login" PERIOD.

        One may learn this the normal, tedious way -- proper training when becoming "cleared", and the whole process is not exactly easy but also not "hard" -- or the wrong way (allowed said access), which certainly counts as the "hard" way, since it will mostly likely end with doing "hard time" in prison.

    2. heyrick Silver badge

      Re: Ooops...

      "Somehow it was my problem for running the code, not theirs for illegally using someone else's account details to login."

      You should have spun it as a honeypot. The message was entirely intentional and would be reported up the chain, exactly as expected, to expose the one using the classified machine in an illegal manner. Explain yourself.

  7. ColinPa

    Sometimes production is the only place to test it

    I remember going to a customer who seemed a little distracted. They had to make a change to production that night without being able to test it. Several large financial organisations had to make the same change at the same time on the same day so data could be shared/transmitted. Backing out would be almost impossible.

    One of the key benefits of the fix was to make a change, so any future changes could be backed out.

    Next morning it was smiles all round. With the months of planning and the what-if scenarios, the 5 minute change had all worked.

    I still remember that you can sometimes cannot test, and some fixes have to go straight to production.

    1. Pascal Monett Silver badge

      Re: With the months of planning and the what-if scenarios

      And that's the key point : months of planning.

      Most customers that call me in want me to solve a given problem NOW. They don't have a test server, because that is a cost center.

      So I end up doing the job as cautiously as I can, asking all the questions I can think of before committing code to a production server.

      Even then, sometimes things go wrong. Generally it's when someone comes in and says "this thing is broken now", and I most of the time I can answer "you never told me about that".

      The rest of the time it's an edge case nobody was aware of.

      But I basically have to code a test script that modifies nothing, test the hell out of it and, when I'm satisfied it does the job, then I port the code to the actual production script.

      I've had my share of stomach butterflies, but so far so good.

      1. My-Handle

        Re: With the months of planning and the what-if scenarios

        Done that as well.

        I usually invest a significant period of time in creating a "panic button" process (be it a script, backup, or a process for people to follow) designed specifically to return the target system to a stable state in the event that everything goes tits-up.

        The funny thing is, I very quickly discovered that the very people who come in shouting "IT'S BROKEN!! IT'S A DISASTER" are the very people who will decline when I offer to push the revert button. Usually with an associated "it's not that bad, it just needs fixing" mumble.

  8. Anonymous Coward
    Anonymous Coward

    Sounds like agile at it's best

    One of the side effects of agile that people seem to pick up along the way is that it skips the planning and all those costs and pesky timeline impacts. it favors action and making changes with the assumption that you can make more changes to clean things up.

    1. yetanotheraoc Silver badge

      Re: Sounds like agile at it's best

      "making changes with the assumption that you can make more changes to clean things up"

      If changes create problems what do more changes create?

      1. GroovyLama

        Re: Sounds like agile at it's best

        A healthy pipeline of work for the next sprint!

      2. Anonymous Coward
        Anonymous Coward

        Re: Sounds like agile at it's best

        "If changes create problems what do more changes create?"

        I once had a calendar with large pictures on a theme of things going wrong. One was given the honour of permanent display.

        "Every solution breeds a new calamity".

        IIRC the picture was a classic "It's alive!" Frankenstein scene.

  9. Robert Moore
    Thumb Up

    Possibly the smartest words ever spoken.

    "If you make a mistake as public as this," he said, "it's better to lean into it rather than cover it up and attempt to shift blame."

    I learned this very early in my career with a mistake that was WAY too big to even try to cover up. I came clean to the bosses, explained how and why it happened. How it had been corrected, and what I had learned. Then waited to get escorted to the door. Didn't happen, stayed at that job for over 7 years after that.

    Learned my lesson that day. Own your mistakes.

    1. G.Y.

      Hoare Re: Possibly the smartest words ever spoken.

      Tony Hoare has a para on this in his Turing award lecture

    2. stiine Silver badge
      Mushroom

      Re: Possibly the smartest words ever spoken.

      It also helps if you report the problem before your customers have a chance to report the problem*.

      * - makes no difference if you've taken the phone system offline...

    3. Anonymous Coward
      Anonymous Coward

      Re: Possibly the smartest words ever spoken.

      Go on, Robert - tell us the big mistake.

      Or do we have to wait for a future "Who, me?" article?

  10. stevebp

    Back in the days

    When I started my course in Computer Studies back in the mid-'80s, the administrators of the PDP-11 system that was running, didn't really understand much UNIX, if at all. However, I was learning fast, as I'd made friends with a 3rd year UNIX geek, who taught me, among other things, how to crack passwords, create trojan logins and clear a room full of people if the computer was too slow to use, so that I could get my coursework finished. Of course I never used this knowledge...

    So one day I was browsing through all the UNIX source code libraries that they helpfully left online and unprotected, to try and understand the scripts and pick up some more tips. I came across the source code for "wall" - or for the uninitiated "write all". I copied it, set it to execute and ran it with a message (I forget what that was, but I'm sure it wasn't rude). It then proceeded to write my test message to every terminal in the uni. An hour later, one of the admins sat down next to my 3rd yr pal and complained to him that "Someone managed to write a system message to everyone and we don't know how they did it!". Of course, I kept quiet, but it was rather amusing to witness the consequences of my deed

    1. Major N

      Re: Are you sure?

      Back when I was still learning things, my school had an old Novell/DOS network, that included a messaging program that would send a message to any user or terminal ID, which would dutifully pop up to that user/terminal with the sender's userID and terminal prepended. Of course, it wasn't long before someone had a hacked version of said messaging exe that didnt attach the sender details, and it made the rounds. I of course named the exe the same as the original and set the execution paths so that it would use this version, but look like it was using the original...

      One kid was amazed at the messages I was securely sending, and asked how I did it, so I showed him, neglecting to mention that I was using an anonymous one. He promptly sent a message out, and I did not see who to.... until 30 seconds later, when the head of IT marched in, grabbed him literally by the ear, and frogmarched him out of the computer suite. Seems he'd decided to test it by sending a message to the head of it proclaiming he 'smelled like cow pats' or something like that, and the original program had dutifully prepended his actual username and terminal ID. He was banned from the computer suite for the rest of the school year....

  11. Robert Carnegie Silver badge

    You could send a test message which is in the form of an advertisement... but that gets you into trouble as well.

    Or... just send the name of the app itself, as the message.

    1. ThomH

      Ed Balls.

  12. GroovyLama
    Trollface

    Testing in production, the fun way

    A Previous employer, was helping roll out our latest OCS platform for a Telco based in mainland Europe...

    Someone wanted me to test that the OCS integration with the SMS-C was fully operational, but the configuration for the standard alerts (low balance, top up completed etc) hadn't been deployed yet.

    The system was essentially being rolled out as a Greenfield deployment first, before migrating the rest of the customer base from a legacy system. So it left us in the weird state of having a Production system which had some real customers, but mostly test accounts to finalise things with.

    I had one potential way I could inject in a message to send, the fact it wasn't fully logged by our system was just a coincidence, honest guvnor.

    So I took one of the clients standard messages from customer care and "personalised" it for someone from the client side who had recently come back from holiday, informing him that his roaming usage was very high and he needs to contact customer care immediately to discuss how he will pay for it. It was great because he had been a) boasting about the great holiday he had while we were all toiling away and b) he did take a test SIM with him and generate a lot of valid roaming traffic on his travels for testing.

    The look on his face was priceless, what was even better was watching him walk around and show it to 10-15 other people asking if they knew anything about it. They didn't (only 2 of us knew) but everyone agreed it looked like a legit message, since it came from the official number etc.

    We had to fess up when we found him in the middle of calling customer care about it!

    Unfortunately the client couldn't accept the prank as a formal test of the SMS-C integration, so I had to repeat the test with valid configuration once it has been rolled out.

  13. Anonymous Coward
    Anonymous Coward

    The main console of an online TP system suddenly printed "I am a ghost and have changed your password". Sure enough a bored programmer had found a way to change people's passwords - and had done a trial as a prank on the in-house admin. Unfortunately the code went into an unexpected loop - continuously changing the admin password and printing the message.

    The system had to be taken off-line in order to fix it. As this was a very sensitive time in customer service relations - the programmer was summarily fired.

  14. Anonymous Coward
    Anonymous Coward

    Testing a supermarket checkout update on-site - the engineer swiped his own credit card though the till for a "dummy" transaction for something like £9999.99.

    Unfortunately the system processed it as a real transaction. After that the engineers were issued with special test cards.

    1. Strahd Ivarius Silver badge

      When testing a new POS system at a luxury retailer, we discovered when integrating a system provided by Visa that:

      - the cards provided were real ones

      - their spending limit was about £30

      The cheapest item that could be bought was £70, so we had to resort to create gift cards or £25...

      The previous system could be setup to use a test environment where it was possible to go up to £15,000 (a far more realistic amount for the retailer)

  15. Anonymous Coward
    Anonymous Coward

    When it's the higher ups "testing in production", the f**k ups reach even higher...

    This is a real story when a "testing in production" bungle reached almost the highest level:

    More than a decade ago, in one unnamed European country, the politics and serious newspapers were back then divided, very roughly similar to UK's Conservatives and Labour and supporting press (with something like their own little version of Boris included). Let's just say the parties didn't exactly like each other and it ran deeply personal, the very senior newspaper staff included.

    Enter one big conservative, business-oriented newspaper I was then working at in some unrelated role (oops, the sins of youth!)

    One day, the big Editor-in-Chief (or perhaps the one directly below him, the memory gets hazy) of the whole online edition decided to test the Very Important "Breaking News" SMS notification system for paying subscribers.

    As what probably must have seemed as a hilarious inside joke to all the involved (because what could ever go wrong, right?), the internal test message was, very roughly translated:

    "XXX is an ar*ehead!" - where XXX was the name of then leader of the main opposition Labour-like party, just previous year's Prime Minister (by now very unpopular even among the younger social-minded voters, due to some of his decisions, but that's another thing)

    You probably can guess the rest... By some unknown mistake (like pressing Enter, I'd wager), the test SMS was sent as Breaking News to all the business subscribers. There is no Recall in SMS.

    Needless to say, the opposition party leader was royally pi**ed, even the Labour-inclined young voters (who mostly turned to the Greens by then, putting them in the Parliament) were laughing out loud and some slogan T-shirt & sticker makers probably made quite the fortune...

    Anonymous Coward, obviously.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like