back to article User worked with wrong app for two weeks, then complained to IT that data had gone missing

Welcome again to On-Call, the Friday feature in which we help Reg readers to recount times when they were asked to fix problems that should never have happened. This week, meet “Terrence” who told us about his time working as a sysadmin for local government. “My main area of work was the system used for revenues and benefits …

  1. macjules

    By any chance ...

    Would this council be Lambeth? Was called to rescue a very impressive Drupal 7 clusterf*ck some years ago which used the Color [sic] module to do exactly that - change environment theme colours. This was a council that also failed to comprehend what SSL is, as all their forms were unprotected.

    1. Anonymous Coward
      Anonymous Coward

      Re: By any chance ...

      that'll be COLOUR ;o)

      1. This post has been deleted by its author

        1. Scroticus Canis
          Headmaster

          Re: By any chance ... Whoosh!

          Simon you forgot the obligatory Whoosh in your response title :)

    2. wolfetone Silver badge

      Re: By any chance ...

      "Was called to rescue a very impressive Drupal 7 clusterf*ck some years ago"

      I'm glad you followed "very impressive Drupal 7" with "clusterf*ck", as there is nothing impressive about Drupal.

      1. TRT

        Re: nothing impressive about Drupal

        Oh, I don't know. Some clusterf*cks can be incredibly impressive.

  2. Terry 6 Silver badge

    TBH

    Sorry, it doesn't sound to me like the user was wrong, nor a " cretin". Users, unless particularly tech aware, are not tuned in to the finer points of tech. The "cretin" is the person who allowed the test version to be "customised" to look like the live version.

    Think what happens when you log into windows safe mode. Absolutely no mistake. It tells you. A test version should have the words "test version" clearly imprinted round the border, or something of that sort. And if it was customisable, then there should have bloody well been a limited range of colours, maybe even striped or something. It's not rocket science. It's not even unique to IT. People do dummy runs of all sorts of processes, and they make it clear that this is a test run.

    1. Lee D Silver badge

      Re: TBH

      Nope.

      Colouration is fine.

      The cretin is the person who didn't splash TEST SYSTEM over every dialog box and window title to ensure that no matter what colour was chosen it couldn't be confused with the live system.

      I'd even put in a big scary warning when you load it up or close it down.

      The user isn't at fault, as such, but not rebooting your computer because "it takes a long time "means the IT team are rubbish or the user is paranoid (reboots on my systems are on the order of 20-30 seconds and I don't do anything particularly special, and only a handful are actually SSD).

      If you have colouration options, those also NEED TO BE TESTED don't forget.

      But there's no reason not to have big annoying warnings everywhere to discourage you from using the testing version for anything serious.

      1. Dave K

        Re: TBH

        To be fair, we don't know if it did. It's quite possible the app came up with a huge warning about being a test version when launched, but as the user had changed the colours then been to lunch, and had never restarted the app for 2 weeks, this could have been missed.

        I agree that locking the colours of the test app would have been a good idea though.

        1. Bronek Kozicki

          Re: TBH

          Actually locking the color scheme of LIVE version would be even more important - imagine the mayhem if a user decided that TEST version scheme is so nice, he wants it on LIVE one. Then few weeks down the line proceeds to use LIVE as if it was TEST, because it looks like one.

          1. Dave 126 Silver badge

            Re: TBH

            There's been a couple of cases of stage actors being seriously injured because a rubber prop knife has been substituted for a real knife.

            The product design solution would be for all rubber prop knives to have a clear textured coating on the handle so that they can be identified by touch.

            1. ElReg!comments!Pierre

              Re: rubber prop knives

              I don't buy that for half a second. Rubber prop knives and real ones don't feel the same at all, they don't even wheight the same at all. It's a bit like saying kids get routinely injured by baseball bats in swimming pool because of confusion with foam noodles. Perhaps we should add a clear textured coating to foam noodles, so that they can be differentiated from baseball bats by touch?

              1. larokus

                Re: rubber prop knives

                lolololololol @ omg made my morning

              2. This post has been deleted by its author

            2. Uffish

              Re: "substituted for"

              I think you owe me some compensation for the mental effort expended in imagining a realistic scenario where an actor could be seriously injured by the use of a rubber knife instead of a real knife.

              P.S. Where is the 'Elf and Safey' icon when you need it?

              1. 's water music

                Re: "substituted for"

                imagining a realistic scenario where an actor could be seriously injured by the use of a rubber knife instead of a real knife

                Act 1

                Scene 1

                A deserted street.

                Enter Alice and Bob, both carrying fearsome looking knives

                Alice: Have at you bob, yer bastard.

                Bob: Bring it on.

                Bob and Alice circle one another and then simultaneously lunge forward with their knives clashing the blades

                FX: spark shower and metallic clang Boing sound.

                They both fall, fatally wounded hilariously knocked out by their own rebounding blade.

                Curtain.

                1. Jamie Jones Silver badge
                  Happy

                  Re: "substituted for"

                  Enter Alice and Bob, both carrying fearsome looking knives

                  ..... and a secret message they need to share without Eve and her machete muscling in on the action.

              2. katrinab Silver badge
                Headmaster

                Re: "substituted for"

                Use this icon for elf'n'safety

                1. Kiwi
                  Thumb Up

                  Re: "substituted for"

                  Use this icon for elf'n'safety

                  Even the "pedantic grammar nazi" bit fits.

                2. Swarthy
                  Boffin

                  Re: "substituted for"

                  Use this icon for elf'n'safety
                  I prefer this one -->

                  Or maybe the Big Brother...

            3. Anonymous Coward
              Anonymous Coward

              Re: TBH

              > There's been a couple of cases of stage actors being seriously injured because a rubber prop knife has been substituted for a real knife.

              Not just knives: https://en.wikipedia.org/wiki/Brandon_Lee#Death

            4. Anonymous Coward
              Anonymous Coward

              Re: TBH

              In murder mystery stories maybe (IIRC one of James Runcie's Sidney Chambers Grantchester stories revolves around that scenario), but [citation needed] for that happening in real life.

              And can I be the 94th person to point out that the original sentence suggests that the injury was caused by a rubber knife, which had been switched into the place of what was expected to be a real one?

            5. Anonymous Coward
              Anonymous Coward

              Re: TBH

              There's been a couple of cases of stage actors being seriously injured because a rubber prop knife has been substituted for a real knife.

              I certainly wouldn't act in a script that calls for me being stabbed and the usual policy is to use real knives.

              Did you mean "a rubber prop knife has been substituted with a real knife?"

            6. Glenturret Single Malt

              Re: TBH

              1. See reference to "sic" above.

              2. I think you will find that the damage was caused when a real knife was substituted for a prop knife.

          2. Anonymous Coward
            Anonymous Coward

            Re: TBH

            Worked for a firm that had agreed to give a software firm some real world testing for free. This was years ago when XP was still relatively fresh and 95 was being mothballed. We provided a computer and one of their people installed the new Windows version of the software. The previous version had been DOS and whilst it was competent at doing what it was supposed to it was still DOS. Therefore the Windows version was seen as a major step forward in that respect. Functionality like copy and paste would become available that were absent from the previous version. More colours, a better GUI, and the ability to use a mouse were also among the top of the hoped for features. People were encouraged to use the machine and say what they thought of the new system which was supposed to be fun. We made sure that no one had any illusions that this was a live machine which would do anything to any active systems in the business. The monitor (a big chunky CRT) was emblazoned with large notices in eye wateringly bright neon pink telling people that this was beta software and the machine was not connected.

            The fact that it was a Windows program not DOS should have been a bigger clue but you can't really take chances we thought. Anyway second person to sit down wants to know about backing up his work and follows the on screen help instructions to do so. He suddenly announces he's found a problem it will only back up to a floppy drive. Incredulous we all go over to have a look and find he's correct. It's weird because they don't appear to have expanded on the DOS version. This also only allowed back up to the A: (i.e. floppy) drive and in the windows version we were expecting more. There isn't a floppy drive on that machine as it's brand new and we didn't order new ones with floppy drives. One is then hastily retrofitted but it was clear in our opinion that all they had done was the minimum to port the thing over from DOS. They hadn't made many of the expected changes and we told them what we thought of their first attempt! When it went wrong twice once catastrophically during our testing we said "No thanks and left it at that".

            No one managed to confuse the system for the live one but we were alert to the possibility even with the wildly different GUI, the fact you could use a mouse, and the fact that the machine was in a corner away from everything else giving people clues.

            1. Terry 6 Silver badge

              Re: TBH

              I think in this thread we need to distinguish between testing a newly minted bit of programming and trying out a bought-in package. I've done both,over the years. In the first it's about saying what works or doesn't. In the second it's more about whether the package is suitable for us. You can have a modified "test" version if it's still in Beta or earlier,and worry about interface details when the actual working code and basic interface is sorted out. I can't thin of an example where a new version of software, using the same data, would be so radically different it would need live user testing. But the query is, are we looking at an early version to see if it works in real life,or a later version to test interface detail (font, colours, etc ).

        2. DJSpuddyLizard

          Re: TBH

          I agree that locking the colours of the test app would have been a good idea though.

          But then how do you test that users CAN change the colour scheme and it stays consistently changed??

          No, I agree that warning text "TEST MODE" or something would be better.

      2. Prst. V.Jeltz Silver badge

        Re: TBH

        "(reboots on my systems are on the order of 20-30 seconds and I don't do anything particularly special, and only a handful are actually SSD)"

        I call bullshit / rose coloured glasses / wishful thinking / best case scenario

        1. kain preacher

          Re: TBH

          I've worked in places that a reboot takes 3 minutes or longer. You have scripts and other corporate crap running before you get to some thing useful.

          1. Anonymous Coward
            Anonymous Coward

            I've worked in places that a reboot takes 3 minutes or longer.

            I don't reboot unless absolutely necessary because its going to be 15 - 20 minutes before I'm back in.

            That's the normal around here. Everyone has a policy they want to push out or another app that scans the HD and a separate login for encryption and on and on.

        2. Andy Taylor

          Re: TBH

          One of my friends worked for a company that insisted his (Windows NT) machine be switched off each evening and powered back on again the following morning, I forget the reason why.

          His daily timesheet had 15 minutes at each end of the working day - 15 minutes for power up and login, another 15 minutes for logout and shutdown. This was not because of some minimum time accounting period, but because that's actually how long the process took.

      3. richardcox13

        Re: TBH

        > The cretin is the person who didn't splash TEST SYSTEM over every dialog box and window title to ensure that no matter what colour was chosen it couldn't be confused with the live system.

        The bigger the difference between test and production the greater the chance of bugs being missed in that difference. Any layout changes in web apps definitely fall into that category.

        1. Dan 55 Silver badge

          Re: TBH

          The bigger the difference between test and production the greater the chance of bugs being missed in that difference. Any layout changes in web apps definitely fall into that category.

          You can't set aside a space on the dialog or web page and fill it with "TEST VERSION" or "LIVE VERSION" (or just spaces)?

      4. phuzz Silver badge

        Re: TBH

        "reboots on my systems are on the order of 20-30 seconds"

        Oooo, look who's got an actual budget! I'll bet all the machines you're responsible for are actually from this century, you posh git you.

        1. Anonymous Coward
          Anonymous Coward

          Re: TBH

          " the order of 20-30 seconds"

          Probly got no AV or GPO or sccm client, or logon script or any of that other tedious corporate stuff that slows us down :P

        2. verno
          Coffee/keyboard

          Re: TBH

          @phuzz I really, really wish I could upvote that moe than once!

    2. goldcd

      I have some sympathy with your view

      But if the user has the wherewithal to start playing with the colour palette of the system, then I think maybe they are at least partially at fault..

      ..Still, at least it wasn't the other way around.

      1. Mog_X

        Re: I have some sympathy with your view

        To be fair to the user, there are legitimate reasons for the colour schemes to be changed - red/green colour blindness and other visual impairments can make some screens difficult to see.

        But as other people have said, the absence of a 'Test Environment' label on the screen is the real problem.

    3. oiseau

      Re: TBH

      " ... the person who allowed the test version to be "customised" to look like the live version."

      Hmmm ...

      Not really a 'cretin'.

      It would seem that this fellow just had a temporary lack of (essential) foresight.

      Far too many times we tend to think of the end user as having the same type/sort/brand/flavour (if any at all) of common sense we have.

      And that is a big (BIG) mistake which you usually end up paying for.

      They don't, so you just have to play safe and make up for it in advance.

      An option for the end user to change something/anything in the test version?

      No. It should *not* have been there.

      I mean, you don't allow third parties to meddle with anything in your test system, you have to have full control of it. Otherwise, how do you know what's going on?

      Have a good week-end.

    4. Blotto Silver badge

      Re: TBH

      @Terry 6

      you are correct, but most people's test system is just an identical implementation of the live system with dummy data.

      Most COTS systems don't provide the ability to customise the system as you describe and most IT shops won't have the ability to add that customisation either.

      The best thing they can do is ensure no test data gets on the live system and that they have documented processes that ensure separation.

      At the end of the day, real data was added to the test system, provided it was suitably scrubbed and stayed onsite (in country or just not on the net) then the data was contained.

      It would be upto the software vendor to provide a facility to brand the test system as a test system.

    5. saabpilot

      Re: TBH

      So TRUE

      When will people learn: If anything can go wrong, IT WILL!

      The person that didn't allow for that possibility in the first place is the CRETIN, not the user. They are users for a reason, otherwise they would be developers.

  3. Anonymous Coward
    Anonymous Coward

    Ped-ANT-ic typo in title

    'User worked with wrong app for two weeks, them complained to IT that data had gone missing'

    Typo pendant spots massive ants in title!

    Yes, yes, yes, m and n keystrokes are next to each other..

    1. Anonymous Coward Silver badge
      Headmaster

      Re: Ped-ANT-ic typo in title

      You picked that up, but missed the errant 'rit'

      from “an irritate manager...

      1. Anonymous Coward
        Anonymous Coward

        Re: Ped-ANT-ic typo in title

        or 'irritated manager'

        1. Prst. V.Jeltz Silver badge
          Headmaster

          Re: Ped-ANT-ic typo in title

          "never shut down their system since it took to long to reboot"

          1. This post has been deleted by its author

            1. IsJustabloke
              Trollface

              Re: Ped-ANT-ic typo in title

              "Is it Terrence or Terence?"

              If it's the one I know it's actually spelt T.W.A.T

            2. Alistair
              Coat

              Re: Ped-ANT-ic typo in title

              @Symon

              That would be Auntie Terrie.

              *cough*

    2. The Indomitable Gall

      Re: Ped-ANT-ic typo in title

      " Typo pendant spots massive ants in title! "

      Where can I get a typo pendant? My girlfriend's a massive geek and always complains that I never buy her any jewellery....

      1. Barry Rueger

        Re: Ped-ANT-ic typo in title

        Typo isn't jewelry, but you could take her for a lovely vacation in Kentucky.

    3. ssharwood

      Re: Ped-ANT-ic typo in title

      Damn. Busy day and I missed it. [Slaps self sharply, slinks into corner]

      1. Jamie Jones Silver badge
        Happy

        Re: Ped-ANT-ic typo in title

        Damn. Busy day and I missed it. [Slaps self sharply, slinks into corner]

        I'm sure I'm not the only one who read that three times looking for typos!

  4. Solarflare

    So wait, you allowed a Test version to languish on a system after the testing was complete?

    1. Dave K

      Testing is complete?

      Testing is never complete. This emplies that the software that is deployed is perfect and will never need to be patched or updated.

      In the real world, the vendor will continue to release patches from time to time to fix issues, and over time will release new versions of the software whilst dropping support for old versions. For any mission-critical software, it's important that every single patch and update is tested before being deployed - especially as lots of ERP apps have various customisations added for different customers, and you need to ensure that the patches/updates don't interfere with anything important.

      For this reason, you always need a test version to be available to a specially chosen group of staff.

      1. A K Stiles
        Headmaster

        Re: Testing is complete?

        <cough> implies</cough>

      2. Prst. V.Jeltz Silver badge
        Trollface

        Re: Testing is complete?

        @ Dave

        Testing is never complete.

        For any mission-critical software, it's important that every single patch and update is tested before being deployed

        soooo, given that testing is never complete , how do you complete your testing of every patch?

        Hmm , so thats why testers get paid so much , often more than the developers it seems!

        Mission impossible?

        1. Dave K

          Re: Testing is complete?

          I was meaning more that the process of testing is never complete. Yes of course you release a patch to the test system, get people's feedback and approval before pushing it to the live system. However, chances are within a few weeks there'll be another patch or update to implement.

          Also, a lot of places use the test version for training and experimentation, seeing as it allows new users to play around with the program and learn the ropes without the risk of modifying crucial live data.

          Overall, for these reasons, plus the fact that your test group are usually more experienced and key users means that it's rare to keep taking the test system down unless you're doing some active maintenance on it.

    2. Korev Silver badge

      If the test system was never shutdown then deleting it might not have any effect.

      1. Solarflare

        If the instance is shut down from the server side then the user should get a nice little error message as it cannot connect. They then know something is wrong and either:

        1) Work out that they had forgotten to move back to the live system, hopefully before entering any real data into the test system in the first place

        2) Blame IT and say that something is broken, which when investigated will quickly lead pretty much the exact scenario minus the two weeks of live data in a test system.

        I'm not exhonorating the user of any blame, I am simply pointing out that by shutting the test instance down properly when testing is not actually needed, it would have prevented this problem. Continue to downvote away if you don't like it.

        1. Anonymous Coward
          Anonymous Coward

          Right Application, Wrong Mainframe

          I used to perform large mainframe installations, where these were replacing previous generation machines the customer would usually pay for my team to perform the initial system set up and migration activities and for us to train there staff in things like peripheral set up and comms config to remote sites. We would also hand hold through parallel running of payrolls etc until sign off by the customer. On one occasion an issue was raised as the hardware engineer was physically decommissioning the old mainframe. One remote site was missing all the payroll data for several hundred employees. It turned out the site staff forgot to transfer the link to one site to the new machine. I did offer to try and recover the data from the old disks but this proved to be impossible as the dusk cabinets gad already been disconnected, removed from the data centre and were now upside down in a skip having been pushed off the loading bay We had raised our concerns about the attention to detail of the site team and suggested we managed. All migration activities but the customer did not want to pay for highly priced consultants to manage a set of checklists.

  5. jake Silver badge

    See where it says " Tips and corrections" at the bottom of the article?

    What do you suppose that might be for, AC?

    Granted, ElReg could make it a trifle more obvious that it's a clickable link ...

    1. Prst. V.Jeltz Silver badge
      Pint

      Re: See where it says " Tips and corrections" at the bottom of the article?

      Why would anyone use that when we can "lightheartedly rib" the author in the comment section?

      1. Dan 55 Silver badge
        Coat

        Re: See where it says " Tips and corrections" at the bottom of the article?

        "Lightheartedly ribbed for their pleasure".

      2. Rusty 1

        Re: See where it says " Tips and corrections" at the bottom of the article?

        Don't you mean stabbed in the back with a rubber dagger?

      3. ssharwood

        Re: See where it says " Tips and corrections" at the bottom of the article?

        Speaking as the author, a gentle ribbing is fine. Sometimes readers sent rather mean and rude notes as if to say we simply should not ever err.

        FWIW the SFO and LON offices have a production desk to check for mistakes. Here in Oz we just try our very hardest.

        S.

  6. jake Silver badge

    In my experience ...

    ... allowing ordinary users to interface with test systems in the comfort of their own desks is asking for trouble.

    1. Anonymous Coward
      Anonymous Coward

      Re: In my experience ...

      However it is entirely necessary unless you have public sector size budgets. Hang on wasn't this....

      1. Terry 6 Silver badge

        Re: In my experience ...

        Public sector sized budgets.....

        Really! The public sector might, in total, be big, even very, very big. But that doesn't mean there is a penny to spare. Do you not read the news? Have you been living in a cave? No council spends more on back office work than is less than it needs.

    2. Prst. V.Jeltz Silver badge

      Re: In my experience ...

      "... allowing ordinary users to interface with test systems in the comfort of their own desks is asking for trouble."

      I disagree. For one - getting users to help you is tricky , so getting them to relocate makes it harder.

      2) it makes a more realistic simulation , as seen in this case

      I think in an ideal world the users wouldnt even know they were testing - you'd just ask them at them of the day " any issues today?"

    3. Anonymous Coward
      Anonymous Coward

      Re: In my experience ...

      "Ordinary users" are the experts at real world use. Expert testers have different skills and focus & developers are frequently completely clueless about use or what needs testing. The number of unused features in most software is a big clue about that difference.

      1. ITS Retired

        Re: In my experience ...

        Not enough people think this way. Too often, if the end user is having problems, the blame is on the end user, instead of the programmer or tester, where it actually belongs.

        I've stood behind users with a clip board, during program development, watching where the user was having trouble. Fix the program once up front, during development, so you don't have to fix each new user to the program. Cuts training time significantly.

    4. Alistair
      Windows

      Re: In my experience ...

      @jake

      I keep all the ordinary users in the middle drawer of their own desks. Keeps things tidy around the office.

      I only let the *weird* ones out.

  7. A K Stiles

    different colours for test and live

    As a developer especially, I make sure my terminal windows for dev / test / live are different colours so I can be sure I'm not accidentally corrupting anything I shouldn't. (and I aim to not have the test / live terminals open at all if I'm not specifically doing something on them).

    In a previous role I've seen a sys-admin accidentally clear down a live database when they thought they were doing it to a development environment. Took about an hour and a half to restore from the overnight backup and then reapply the lost morning's work from the database journals. Admin thought they were out the door but the management took the sensible approach of "you didn't intend to do it? no, good. You won't do it again, will you? No, didn't think so." and implemented a policy of colour coding across everyone who had admin access to live - If it's red, REALLY don't break it!

    1. Korev Silver badge

      Re: different colours for test and live

      Didn't Plusnet delete rather a lot of email when a sysadmin typed rm -rf in the wrong shell?

  8. Anonymous Coward
    Anonymous Coward

    Test / Dev

    My wife used to work for a sizeable construction company, which had a monolithic application-of-everything. And yes, they had a dev copy and a live copy.

    Then the live copy broke. And after ten days or so they decided it couldn't be repaired and pressed the dev copy into service with restored data. And never replaced the dev system. The poor developers having to tweak a substantial Oracle-based platform in-flight. :-/

    1. DuchessofDukeStreet

      Re: Test / Dev

      Your wife and I must be ex-colleagues. Those ten days were "interesting" to say the least, as were the couple of weeks and months that followed. There were backup tapes in place, but as the live system had actually been corrupt and broken for several days without anyone realising, none of them could be used - it was debatable if anyone would actually have known how to restore anyway.

      We all learnt a lot about DR and BCP - and the difference.

  9. David Roberts
    Pint

    Modifying test system?

    IMHO once you do more than the tiniest of changes to a test system (such as in this case using a valid configuration option present on the live system to change screen colours) you are well down the road to invalidating the testing.

    A variant of the common user support question "Did you change anything?" with the usual response of "No." followed by "Yes, but that isn't important.".

    If you start adding code to make it obvious that it is a test system then you compromise the strategy of taking an exact copy of the live system then kicking the living shit out of it.

    Or building a new version, testing, then flipping it to be the live system.

    This problem reads like either a user communication error because nobody though a tester would change the colour scheme, or more likely the result of the kind of lunch break that results in the morning's work disappearing irretrievably into the mists of time such that two weeks down the line there is still no memory of ever having tested anything. Of course the user could just be irredeemably thick. Still, you would have thought the question "How is the testing going?" followed by the answer "What testing?" might have raised a warning flag.

    Beer - the obvious culprit. (Not from personal experience, you understand.)

    1. Doctor Syntax Silver badge

      Re: Modifying test system?

      Having something like a configurable background as part of the original design is fair enough. It doesn't touch any of the code responsible for doing the actual work. Just don't make it user configurable.

      In fact I've seen something similar where there were a number of production systems sharing a lot of common code and hence user interfaces but operating on different databases. The background was specified in the database so that the users would always be aware of what they were working on.

      But the same gig underlined the point about making sure that the test system tests the actual code that will be live. My client was a subcontractor processing data from other subcontractors and it was one of several where the data feed was to be XML so there was a bunch of systems sharing common code for handling that. On one contract upstream wasn't ready to generate XML when testing was due to start and wanted to send fixed width files instead (the data wasn't very complex so in this instance XML was overkill).

      Fair enough, we had to have some end-to-end testing in place to keep to schedule. I wrote a front end, in fact a two stage front end, which converted fixed width to CSV and CSV to XML, all parametrised and set up to generate the XML to the project's schema with both steps being trivial to implement and based on the in-house class hierarchy, etc. This enabled our test system to use the eventual live code to do the XML import and as a by-product provided modules to allow the client to import fixed width or CSV data should this be a requirement for a future contract.

      My client's development manager - yup, development manager! - couldn't understand why I didn't rip out the entire XML processing code, which was a large part of the entire custom code, and implant a completely new fixed width file processing code just for testing. In fact, it was the stress of dealing with that particular manager's bad decision making that persuaded me retirement time had arrived.

  10. Anonymous Coward
    Anonymous Coward

    Anyone that thinks locking colour schemes to differentiate between Prod / Non-prod systems needs to move away from their keyboard until they understand UI design in respect of people with visual impairment / colour blindness.

    1. Korev Silver badge

      I agree entirely.

      When I see vendors' new software, I always ask about colour blind users, I've yet to see one where they've actually thought about it.

    2. teebie

      Or have the colour schemes be distinguishable to everyone - black text on white background for live, white text on black background for test.

      1. Anonymous Coward
        Anonymous Coward

        And thinking about this a bit more, if a user can blindly use a test system without realising it then this implies the reference data has been copied from production, or it's a copy of production data being used for testing. Either is a breach of the DPA.

        1. Kiwi
          Holmes

          And thinking about this a bit more, if a user can blindly use a test system without realising it then this implies the reference data has been copied from production, or it's a copy of production data being used for testing. Either is a breach of the DPA.

          I can think of a number of situations where a user could be putting data into a test system without knowing it, eg they're responsible for creating accounts for new customers.

          Not sure how using real data would be a breach of the DPA, but the DPA doesn't exist everywhere - can't recall if TFA mentioned the country but it's not necessarily the UK.

          Using realistic data can be necessary in some cases. For a start, as mentioned above real users break things in ways the greatest devs and testers in the world cannot imagine, and real data (especially on large databases) can throw up things that no amount of test data would reveal.

          As someone who is very much into privacy esp in technology, I have no issues whatsoever with devs using real data (not live data) for their test systems, so long as they make a decent effort to prevent that data being leaked or abused. The more realistic the data (both in content and volume), the better chance of problems being found and fixed in the test environment rather than in live systems.

  11. BagOfSpanners

    Changing the colour scheme is not enough

    Using a different colour scheme between test and production is not enough.

    There should have been text on every screen saying "TEST".

    The configuration file that controls all this should be encrypted, to stop power-users trying to convert test installations to production installations.

    There should also be a set of test logins that won't work at all in production.

    Even then, sooner or later test users will find a way to insert embarrassing test data eg. "Mr Mickey Mouse, 69 Big Bottom Road" into production. It's one of the things that keeps me awake at night.

    At least in this case, the production data was being entered into a test environment, which is less likely to result in newpaper headlines than test data going in to a production environment.

    1. Anonymous Coward
      Anonymous Coward

      Re: Changing the colour scheme is not enough

      Down voted because you think "Mr Cock Womble" in a test system rates more highly than my details is some test database that could end up god knows where* or that the papers would even cover the former

      *like a web based product demo

      I don't wish to be insulting but are you a developer for CSC[HCSPEFGH or whatever they are this week]/Capita/Serco/nPower?

      1. BeakUpBottom

        Re: Changing the colour scheme is not enough

        Called DXC* now, at least the bit I spent some time working** for ...

        And you are correct, test data in the production system is irritating but nearly always mostly harmless, real data in the test system ... >>> shudder <<<. Someone in compliance (or the regulator) finds out that happened and you'll have to reclassify your test system & network and everything it touches until it's been sanitised convincingly (ie, never).

        * CSC (Cutting Serious Corners) merged with HPE (Horrible Projects Executed) to become DXC. Best I could come up with was Deploys eXcruciating Crap

        * OK, juggling my bollocks waiting for the opportunity to do something meaningful between endless meetings about meetings about meetings about why nothing ever gets done.

  12. John 110

    Label as TEST

    I think you guys are assuming distinguishing between systems with banners is even possible.

    I've been involved in the implementation of three LIMS and two electronic request systems, all of them have literally *no way* to splash "TEST" over every dialog box.

    The Test (or Demo) systems are essentially different instances of the same database, running on the same server. In the case of the LIMS the version you get depends on your username.

    So there is no way to tell visually which system you are in apart from a four letter code on the menu page, up in the corner.

    Welcome to the wonderful world of low volume high throughput software.

    1. Korev Silver badge

      Re: Label as TEST

      When rolling out your own applications, you can make all the colour and GUI changes you like; if you're rolling out a 3rd party application then there is often no way of doing it.

    2. GlenP Silver badge

      Re: Label as TEST

      Exactly the same with our ERP system, except it's just the database name and URL that differ.

      We don't generally have users in the test system, but it's there partly for training/experimenting as well as testing.

  13. Hello to Jason Isaacs
    Facepalm

    Customisation isn't necessarily the enemy

    Colour is never enough anyway because perception.

    Easy to say now it's more recognised in app design but only using colour to distinguish between states is never enough.

    1. I ain't Spartacus Gold badge

      Re: Customisation isn't necessarily the enemy

      Ah yes! The manual on many wireless routers that says, "the LED will change from red to green when it is working".

      Great. Except I can't fucking tell them apart! Now to be fair, I've not got it too bad. I can tell them apart if they're both next to each other and lit, and then do a compare and contrast with the others to slowly work it out. But a power light will tell me the fucking thing is on, then a light that isn't on will do fine to tell me other functions aren't.

      BT's Homehub isn't too bad. It's got a retina-bleedingly bright blue LED for broadband link working, and a flashing red one for when it isn't.

      1. Steve Aubrey

        Re: Customisation isn't necessarily the enemy

        I have a CD player with a red light indicating that power is off.

        Turn power on, and the red light disappears.

        I tend to prefer 1:1 correspondence - a light on the unit indicates power is applied.

        But wait, you say - what if the power is out in the building? Naaah, I'd rather use alternate means for checking that situation, and leave a power light meaning the unit has power.

        1. JulieM Silver badge

          Re: Customisation isn't necessarily the enemy

          The light is to let you know that the appliance has its switch installed on the secondary side of the power supply, and your electricity meter is still counting down. If it's one of those eye-stabbing mega-bright blue LEDs, it might be to remind you to switch the appliance off at the wall socket, just to avoid assaulting your retinas.

    2. Anonymous Coward
      Anonymous Coward

      Re: Customisation isn't necessarily the enemy

      Normally with my users after they've finished testing I reset their access to the test system precisely because I've had this issue in the past.

  14. BeakUpBottom

    Well, half right

    OK, the user was being a dick, but if you are going to have them test on their own PCs, and all you warn them with is a colour scheme they can customise this is an entirely predictable outcome.

    What was it? the guy's first deployment?

  15. trevorde Silver badge

    We told you so...

    Worked for a company many years ago and we released a beta of our product with strict/large disclaimer that it should not be used in a production environment etc, etc, etc. Of course, some of our users were keen to try all the new, shiny features and used it in a production environment... When we released it, tech support immediately got calls about not being able to open files (created in the beta) and possibly losing months of work. We had changed the file format between beta and release...

  16. InNY
    Pint

    "since it took to long to reboot"

    Shouldn't that statement be "it took too long"?

    A statement about a reboot is not a preposition nor an infinitive. Though, the experience of a reboot is frequently an infinitive experience...

    I do agree with the many others' commenting, the user in this case is not the cretin; the IT department is, for the many reasons already explained by those same commenters.

  17. Dan 55 Silver badge
    Meh

    Every day's a professional day

    His manager started talking about possible unpaid overtime to fix the problem.

    I've never had an IT job when I've been paid overtime. If my overtime weren't free to PMs I bet that would concentrate a few minds.

  18. The March Hare

    Deja vu

    I know this! it's a UNI... sorry. sorry.

    Actually, I work with a system that does exactly this: a revsnbens system, live and test instances (how else to you experiment with patches/training/customisation?) and colour changes to the interface for differentiation.

    the colour changes are per user so you have to hope the user doesn't get creative and the point of having Live and Test systems is that they should be identical in operation (but maybe not in function) - otherwise you are NOT testing reality.

    Also, it's not the function of IT to manage operational staff - the system belongs to them, they understand it better than IT and they set the rules of use - we are just there to clear up the mess when the crash happens.

    I feel sad now.

    Want to go home.

    but at least it's Friday!

  19. Anonymous Coward
    Anonymous Coward

    Test applications on a Production domain ?

    Isn't the correct way of doing it to have a completely separate Test domain ? Not always possible due to budget so we get around the "not knowing if you're in Test or not" issue by using Test laptops to connect to the Test systems that have Test credentials - still might not have prevented the sequence of events in the article, but at least screen locks force a password to unlock - you'd hope they notice at that point :)

  20. Anonymous Coward
    Anonymous Coward

    When the differences ARE the feedback

    Recognising that users might mistake Test for Live, I emblazoned a 'Test' watermark across the background.

    Initial feedback was that application looked rubbish, but it took a fair amount of coaxing to identify that what people didn't like was 'Test' being all over the background.

    To be fair, it was quite distracting!

  21. Packet

    My pet peeve is with mis-configured corporate systems that take too long to boot up.

    I've been around the block for many years now and have yet to see an optimized AD based logon script design.

    It boggles the mind that our wannabe AD BoFh's (more like superlusers) do not create optimized AD designs and desktop login scripts /startup processes that are fast.

    Naturally the rest of the organization hates the IT dept with an undeniable reason - add to that the typical arrogance of an AD wannabe BoFh who forgets that the purpose of IT is to support the business and not to create his own little fiefdom from which he cannot be fired

    1. Anonymous Custard Silver badge
      Trollface

      Oh I dunno, in the past here (where we used to have such things, thankfully now sorted out by SSD's and shouting at various idiots that these things should be tools to help us perform our jobs, not to stop us doing so) they used to be a good excuse for an extended coffee break first thing in the morning whilst they booted up.

      Or if a Windows Update got itself mixed into the mire, either a good discussion about last night's telly or (if the inevitable reboot could be held off long enough) a nice long lunch break whilst everything updated, then resync'd, all the various scripts ran and generally a small pile of dropped bits ended up underneath the CPU.

  22. Anonymous Coward
    Windows

    Defending the user...a little

    You could argue that the user was being pretty sensible They logged on to a system, noticed that the environment was not what they expected, and proceeded to make it so. Is that any different from you or me logging on to a shell, seeing that the prompt is stupid and hurriedly setting PS1 to fix it?

    Where the user went wrong is the arse-covering, face-saving behavior that followed the realization that they had been using the test system.

    My pro-tip for test systems is to know who is using them at all times, and regularly talk to them. It means my environment is controllable (aha! so this bug only hits when 2^8+1 users are logged in, how suspicious), it allows me to get good feedback and it nips any "I was using the wrong system" in the bud.

  23. Mark Manderson

    in our test system on our ERP solution, we simply make the users as made up names, and ensure that live users accounts are denied access to the Test Dbs and subsystems, to counter end user stupidity.

    My favourites have to be Test McTesty (mine) and Tony Fitzpatrik and Patrik Fitztony, we also have Errol Flynn & Kenny Adabanjo, got to keep yourself grinning somehow in a test env right? I wanted Pat McGroin but boss thought that was pushing it a bit (boo!)

    ofc Ive reused this idea on our local domain, a Mr Test McTesty as my auth AD sys test user.....all good and well until trying to explain to our US owners why.......just no sense of fun those yanks :) they wanted it changed to my normal name but with test added to my surname....doh thats boring!

    1. Anonymous Coward
      Anonymous Coward

      With the first DB system* I worked with here decades ago I entered the cast and characters of Star Trek as the test data.

      That was also the system where I started by building the new system with all the demo's colours stripped out, plain black text on plain white background, and the manager complained: "put it all back as black text on red background". What, make it completely invisible? Are you kidding?

      *Think: small North Yorkshire village vs DOS-crunched filename vs what I worked on

  24. JulieM Silver badge

    Damned if you do, damned if you don't

    If you don't allow the test system to be as fully customisable as the live system -- especially if it uses the database to store the style information -- then your test isn't realistic enough. A namespace clash could result in odd things happening if, for instance, you changed a font or colour while a record was displayed on screen then saved that record -- only now it was corrupted with the colour or font in one of the data fields, but it will only appear there when that record is next read from the database.

    This is only ever going to show up on a test with real, live users, because they usually underestimate how delicately to treat anything.

    My current favourite method (I'm working on a database-backed web app right now) is to have one script that determines whether to use the real or fake data by means of regular expression matches on the URL by which it was invoked. That way, the obvious indicator is in the browser's address bar; and I only have to worry about it not 500-ing, not about keeping two copies almost identical save for a single include statement. (And, yes, I've missed it, and ended up doing an UPDATE on a record that wasn't there when a user saved a record they had loaded just before I overwrote the script with a new version that was still working on the wrong database. But that's what we have a general log for. If you add up all the CPU cycles that have been spent on logging over the years, it's nothing compared to the effort that might otherwise have been necessary).

    Also, MariaDB in Ubuntu's default configuration includes the name of the current database in the prompt. When you have two databases with identical table names and structures, this is a definite improvement over MySQL.

  25. russmichaels

    A tale of 2 sites

    I had a similar story.

    I had a client who for some bizarre had 2 copies of their site offshorepipe.co.uk and offshorepipe.com with different hosts.

    The client kept making changes on the .co.uk site and then complaining to me that she couldn't see the changes because she was looking at the .com site. It took me some time to figure this out as I didn't originally know about the 2nd site, and she would keep telling me she was logging in at the .com site, when in fact she wasn't. It took a screen share to figure it out.

    This was a recurring issue and happened every single time she updated the site, and each time I had to explain the cause of the problem and that the changes would only show up on the site she changed. I kept telling her that she did not need 2 copies of the site and to just get rid of one of them, and point both domains at the same site, but she never listened. Eventually, they went out of business.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like