back to article Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?

Welcome, dear reader, to another installation of Who, Me? in which we recount for your schadenfreude indulgence tales of Regizens who create havoc – either by their own design or others – but mostly emerge unscathed. This week's hero, whom we shall Regomize as "Sedgwick," worked as a systems engineer in the R&D department of a …

  1. Oglethorpe

    Apologies for the lack of detail, I wasn't present in the room but did see the effects. One Thursday, there was a lot of grumbling at the end of the day when an unfortunate IT engineer took down the main data centre by accidentally hitting the big red button, causing people to lose unsaved work. The following Monday morning, the same thing happened again, with similar grumbles.

    I later learned that the second incident happened *because* the engineer in question had fitted the big red button with a cover. They were proudly showing it off to their boss by sharply hitting said cover, which hadn't been properly fitted and gave way just enough to trigger the button again.

    1. Admiral Grace Hopper

      Father Dougal

      Accidents are one thing, but there are some people who do this sort of thing deliberately, as told in this tale of our own Father Dougal

    2. b0llchit Silver badge
      Go

      Ah yes, Big Red Buttons...

      A gift that never seizes giving.

      icon should have been red, but the red ones try to stop one

      1. Stuart Castle Silver badge

        Don't knock the big red button..

        In the Uni computer lab I used to work in, there was a big red button on the wall that killed the power to the student computers. I *think* it was there from when the room hosted several large, and potentially dangerous machines, it's just when the electricians installed the new electrical mains, they ran them through the same BRB.

        Now, when closing the lab, I always gave warnings half an hour before closing, then ten minutes, 5 and finally 2, so I'd argue they did get fair warning.

        95% of students would leave after the first or second warning. TBH, by the time you got to the third, you were down to the hardcore students who wouldn't leave no matter how many warnings you gave.

        After a few weeks, I changed the final warning to "If you don't pack up and leave within 2 minutes, I'll go to the big red button, hit it, and you'll lose any work that isn't saved". Weirdly, no one ever tested me on that..

        1. breakfast Silver badge
          Coat

          "Don't knock it" is very much big red button 101.

          1. Anonymous Custard Silver badge
            Trollface

            And knowing the state of some computers used by students, I'd wonder about using the term "dangerous machines" in the past tense...

      2. KittenHuffer Silver badge
        Headmaster

        Warning: Language nazi at work!

        It never ceases to amaze me when people use a phonetically similiar but incorrect word in a sentence.

        --------> Why does he look so much like Mark Labbett?

        1. David Nash Silver badge
          Meh

          Re: Warning: Language nazi at work!

          Thanks for clarifying the meaning...I just skipped over it, a little confused. Well it is Monday morning.

        2. David Nash Silver badge
          Happy

          Re: Warning: Language nazi at work!

          See also https://forums.theregister.com/forum/all/2022/11/07/who_me/#c_4562836

        3. that one in the corner Silver badge

          Re: Warning: Language nazi at work!

          You've been playing - Cat Phrase! [1]

          We'll be right back after these important messages.

          [1] (c) Dave Gorman

        4. b0llchit Silver badge
          Headmaster

          Re: Warning: Language nazi at work!

          Pedantic thinker(er) and professional cynic at work here...

          The seizes and ceases change is not without thought. The "gift" is the BRB and translate "seize" into "capture", "conquer" or "confiscate"; then read the sentence again.

        5. NXM

          Re: Warning: Language nazi at work!

          When doing my electronic engineering degree several centuries ago (that's how it feels from this distance, anyway) it became apparent that a lot of my fellow students were affected by dyslexia to some extent. I felt they'd deliberately chosen the course with the fewest essays, as it was maths-based. I've met many intelligent people who couldn't write a comprehensible sentence but could easily speak it.

          I think the quality of English here is excellent!

      3. Treefella

        I worked as a remote support tech that supplied biometric till systems for schools. For weeks we had the end of day process go offline at the same time on Friday so the school in question couldn't close there trading week and devs started to panic,(what fun)

        Turned out that the woodworking shops closed at that exact time and turned off all the electric with a big red button, which some genius had wired the till power supply

  2. Pete 2 Silver badge

    Working as it's supposed to

    > the latest revision of the system had introduced a bug

    After all, gotta keep the support staff in jobs.

  3. ColinPa Silver badge

    This security software has great quality..

    I remember working at a customer site, and by law they had to have some security software installed. The manufacturer was really pleased to quote their defect rate ... single figure defects per year, installed on thousands of systems.

    During my visit an audit was taking place. I heard the conversation went a bit like this.

    Auditor:Do you have this security software installed

    Customer: Yes.

    Auditor: Is it enabled

    Customer:Not as such

    Auditor:Please may I see it working, and that it stops someone without authorisation from doing something.

    Customer: Not during production hours

    They turned on the security system on a test system, and the performance dropped significantly, there were abends and other errors, it did not scale to hundreds of users, it just about did what it said on the box (after a lot of complex setup).

    A short while later the Government updated the law to say the software had to be installed, enabled, and working.

    The result was so many defects and problems were found the manufacturer had to withdraw it as it needed a redesign.

    1. Fonant
      Holmes

      Re: This security software has great quality..

      Aha, the unbreakable truth: apparently-bug-free software just isn't being used.

      Bug reports are a Good Thing for software that people find useful: it's a sign that people are using it (and quite probably for things you didn't envisage)!

    2. This post has been deleted by its author

    3. Jou (Mxyzptlk) Silver badge

      Re: This security software has great quality..

      Sounds Symantec to me...

  4. Anonymous Coward
    Anonymous Coward

    Nothing so drastic, but I did have to travel to a customer site to set up a room for on-site training. It was a last minute request (only about a day's notice). I was informed that everything was ready. It had to be done on a Friday, and this was on the Isle-Of-Wight. The company shut up shop early in the afternoon.

    So I get there and start my work - only to find the requirements weren't met. They were missing software. Queue people rushing around trying to

    a) Blame someone else

    b) Find someone who could authorize the software purchase so late in their working "day"

    Luckily I am covered (I simply produced the statement from them that all was ready and the requirements were met), and they managed to find someone willing to cough up for the software. Install done just in time, and I got a trip down memory lane (had to ride on the chain ferry - haven't done that in decades).

    1. J.G.Harston Silver badge

      Was it a long queue?

    2. NXM

      chain ferry

      Can you give us a link to that ferry's website?

      1. PRR Bronze badge

        Re: chain ferry

        >> chain ferry

        > Can you give us a link to that ferry's website?

        Shirley, you do not have Google/Bing/Duck ??? "chain ferry isle wight"

        https://en.wikipedia.org/wiki/Cowes_Floating_Bridge

        https://en.wikipedia.org/wiki/Cable_ferry

      2. Norman Nescio Silver badge

        Re: chain ferry

        I guess if a link breaks, they have an emergency rod or pole they lug along they can use to keep the ferry under way, taking care not to break any wildlife laws by disturbing any perch as they punt along. Of course, if the ferry is any size, it's an 'ellish job.

        1. John Brown (no body) Silver badge
          Coat

          Re: chain ferry

          Nah, they upgraded to an unbreakable block chain.

  5. Stuart Castle Silver badge

    I realise this isn't always feasible, but this is why I normally advise people, if they are doing development work on a system, to have a test install of the system running somewhere, probably with slightly out of date data.

    A system I'm working on at the moment runs on two VMs. One for the production system, and one for the test system. The test system uses a copy of the database from the main system that is about 24 hours behind. We do all our development, testing and demonstrations on the test system, safe in the knowledge that if the company hosting the systems is doing their job (and generally, they do) properly, our data is safe.

    That said, another system I'm working on, we didn't pay for a test/development server, so I've learned to be *very* careful on it..

    1. A____B

      Agreed - but would go further...

      Make sure it's easy to distinguish between the systems (e.g. change the backdrop image and/or colour schemes, fonts ... for the test system).

      I've seen an ohnosecond panic when the penny drops that the change was made on the wrong system -- fortunately nothing was lost apart from the demonstrator's** dignity

      Typically I used some hideous colour scheme, one that nobody (apart from Lawrence Llewellyn Bowen) would sanction for delivery to users, on test systems. Taste doesn't matter if you're making it blindingly obvious whether it's "safe to play" or not.

      ** Thankfully not me ! :)

      1. Anonymous Coward
        Anonymous Coward

        Make sure it's easy to distinguish between the systems (e.g. change the backdrop image and/or colour schemes, fonts ... for the test system)..

        This.

        That's why my user level terminal has a blue background. If not blue, danger. I don't set a red colour for a higher privilege level because sometimes that may not work or not be configured so I rather fail safe. Also, my working prompt is different.

        As for dev and test, we give those their own VMs on networks that can't even reach production. Might be a tad OTT, but we prefer several layers over guesswork.

        1. J.G.Harston Silver badge

          Ditto, when logged on as Admin the background is plastered with the word 'Admin' dozens of times. With the kerning and justification just ever so slightly annoyingly off.

          1. Arthur the cat Silver badge

            With the kerning and justification just ever so slightly annoyingly off.

            "If you really hate someone, teach them about kerning."

            Can't remember who said it, but it's true.

            1. Anonymous Coward
              Anonymous Coward

              bad kerning

              The line comes from xkcd:

              https://xkcd.com/1015/

        2. Strahd Ivarius Silver badge
          Trollface

          As long as you don't use Pantone colors, you should be fine

      2. Anonymous Coward
        Anonymous Coward

        Limited colour schemes

        Not always easy.

        I work on a system for customers and I do change the colours and typically have the same set for the instances I am on. Often I have limited access to their prod instance too.

        A recent customer had their prod instance set to my "dev" scheme and they had granted me full admin. I didn't do anything wrong, but occasionally would go to look for a record / update and wonder why it was not there and then remember "Oh I am working for acme" and this is NOT their dev instance.

        forget it having words of "Prod" / "Dev" in its title, I mean who look at that when you have nice colours to identify

    2. Jonathan Richards 1
      Go

      Test system +1

      Excellent idea, if you can get the funding for it. It may be easier to do so if you can arrange for the Test system also to be part of the Disaster Recovery/Business Continuity plan. Of course, if Production and Test are just two VMs running on the same bare metal, then it'll be difficult to be convincing about the DR Plan.

  6. Evil Auditor Silver badge

    "ohnosecond"

    Reminds me of the flight instructor who bet his students that it is not possible to retract the landing gear while on the ground. One Saab 340 was totalled in the process.

    1. I Am Spartacus

      Oh please - send the link to the You Tube video of this one!

      1. Evil Auditor Silver badge

        That was, unfortunately, many years before Youtube.

        But here you'll find the Investigation Report, see section 1.5.1.4.2 Unintentional retraction of the landing gear on the ground on page 24. And yes, that report was not about the unintentional retraction but about the unintentional touching the ground with sudden deceleration done about ten years later by the same pilot.

        1. Benegesserict Cumbersomberbatch

          That document is very, very, very Swiss. Up to and including the Helvetica.

          1. Evil Auditor Silver badge
            Thumb Up

            A Swiss authority investigating a Swiss pilot flying a plane of a Swiss airline into the soil of a Swiss hill... But Helvetica, I believe, it is not. Although I just found that this font is no longer installed in my work computer and hence cannot verify.

    2. that one in the corner Silver badge

      Someone hadn't seen "No Highway (in the Sky)" (1951) or the original 1948 Nevil Shute book.

    3. NXM

      landing gear

      There was a crash in Pakistan a couple of years ago. The pilots we're having a big discussion about covid while coming in to land and only realised they'd forgotten to put the gear down when the engines scraped the runway.

      They pulled up and went round again, but when they ran the engines up on the second approach, both failed and the plane landed on a load of shanty houses. There were only two survivors.

      1. Norman Nescio Silver badge

        Re: landing gear

        That's Pakistan International Airlines Flight 8303.

        PPRUNE thread, currently 84 pages.

    4. Arthur the cat Silver badge

      Some years back a friend of mine simply forgot to lower the landing gear of his plane, in spite of being a very experienced pilot. He apparently had just enough time to think "plane seems a bit low without touching down" when the shower of sparks started.

      1. An_Old_Dog Silver badge

        Checklists!

        There are good reasons why checklists are used extensively in aviation.

        Why didn't your friend use theirs?

        1. Evil Auditor Silver badge

          Re: Checklists!

          Why didn't your friend use theirs?

          Moderately experienced pilots tend to be at highest risk of jumping the checklist.They are certainly no beginners anymore and develop an overconfidence: "I know what I do!"

          What it was in this case, I certainly do not know but I really wanted to reply to a dog replying to a cat.

          1. Arthur the cat Silver badge

            Re: Checklists!

            My friend was also horribly prone to cycles of massively overwork followed by a collapse. I suspect it was during the latter part of one of those cycles.

            I really wanted to reply to a dog replying to a cat.

            Shouldn't you have been a horse or similar sized animal rather than an auditor?

            1. Evil Auditor Silver badge

              Re: Checklists!

              On the interwebs no one knows you're a horse. Until now, that is.

  7. I Am Spartacus
    Mushroom

    A near miss

    Some timeago I worked for a software development company that had a product to archive data from relational databases. It worked by running a search for candicate records to be archived and then found all the releated data. So, you could archive customers who had not ordered anything for five years, and it would also find all the invoices, payments, shipping, correspondence that related to the customer(s).

    As the tech person I always insisted that the client test their scripts on a test sustem before running it on production.

    One client proudly proclaimed that they didn't need a test system. Nor a development system. They just ran production. So I asked how they did development of the application and specifically new database tables (etc). The answer was "Oh, Malcolm here just gets on the main console and keys the changes directly in to the database. On production. And if it goes wrong, we are front page news next day. Ha!" We were out of there so fast as I explained to the samesman that no amount of profit was worth the risk that if anything did go wrong we were bound to be the fall guys.

    Oh, yes. The system? HMRC's tax computers.

    1. Anonymous Coward
      Anonymous Coward

      Re: A near miss

      Which "tax computers"? There are many systems in HMRC, most of which had discrete Live, Clone and Dev environments while I was working there, most particularly those involved in the assessment or collection of tax, such as COP, BROCS, CESA. I wouldn't be surprised to learn that there was a system somewhere in the estate that ran on the basis you describe, but I would be surprised if it was one the main revenue systems.

      (Unless it was from the former Customs and Exercise part, some of the things that were done in Southend were tantamount to piracy).

      1. I Am Spartacus

        Re: A near miss

        It was run by a large, internation, computer services team. I prefer not to expose them as my desire to end up on the wrong end of a defamation law suit is pretty low! This was decades ago, in the mid 1990's. And yes, if I recall, it was Customs and Excise. But it wasn't Southend we visited.

        I am sure that they are a lot more professional now. This would all have been refreshed as part of Making Tax Digital. I mean, it must be so, mustn't it!

        1. Anonymous Coward
          Anonymous Coward

          Re: A near miss

          "customs and excise computer". It's all clear now. Anyone ex-Inland Revenue was very clear on the difference between taxes and customs and excises. One was calculated and collected with precise accounting, a practice dating back to the Domesday book, while the other was gathered by freebooting Johnny-and-Jilly-Come-Lately privateers with a more swashbuckling approach.

          1. Flightmode

            Re: A near miss

            You just know that someone uttered the phrase "We want to be the Spotify of the tariffs industry" along the way.

  8. tfewster
    Facepalm

    Ready, fire, aim

    > for the demonstration he also created a dummy high-level process...

    > Sedgwick issued the command to kill the main data capture coordinating process...

    Maybe something was skipped in the retelling? Why set up a dummy process and then not demonstrate with that?

    1. Data Mangler

      Re: Ready, fire, aim

      I think it's iffy writing. If you carry on to the next single line paragraph you'll find that he didn't do the sensible thing.

    2. Robert Carnegie Silver badge

      Re: Ready, fire, aim

      I'm guessing that it went something like:

      > list processes | find "important"

      important business process do not stop this

      > start lowlevel process "kick me"

      > sudo start highlevel process "importent process - for demonstration"

      > list processes | " find "kick me" | kill

      Killed "kick me"

      > list processes | find "important" | kill

      Killed "important business process do not stop this"

      Edit... or maybe that was the "bug", an auto spell check??

      > list processes | find "impotent" | kill

      Did you mean: important

      Killed "important business process do not stop this"

  9. Jou (Mxyzptlk) Silver badge

    Nice story!

    The typical "This cannot work because I don't have the rights" - applied to Active Directory, Fileservices, Webservices, Volumes, you name it.

    Luckily the AD was new enough to have a recycle bin, and it was activated. The fileserver "delete test" was stopped fast enough to use robocopy to get the missing data out of the snapshot. The volume only contained replica-VMs which are on standby when the main server room would be burning or otherwise dead. The Webservice kill is not my experience, but I've seen others doing that. The older you are, the longer such lists.

    A worthy article for Who-Me.

  10. This post has been deleted by its author

  11. Boris the Cockroach Silver badge

    The ohnosecond

    is my friend... it keeps me warm at night with the high blood pressure, the tremors and the cold sweats.

    Especially when the ohnosecond is not the time to think "Oh my whats gone wrong?" but more like "&%^*$^ 7Kg of tool spinning at 6000 RPM is about to exit the machine and play merry hell with my running legs...."

    Deleting data sounds so much safer... if just as expensive in the end

  12. DS999 Silver badge

    Never perform a demonstration for the peanut gallery

    Without testing it first. Preferably during off hours.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like