back to article BOFH: You. Wouldn't. Put. A. Test. Machine. Into. Production. Without. Telling. Us.

BOFH logo telephone with devil's horns A lesson I learned a long time ago was: never let a user buy a server. It's not a hard lesson to remember because of the frequent booster-shots for the rule whenever some purchasing person decides to sidestep their purchasing restrictions by "renaming" what they're buying. … So we've …

Page:

  1. Maverick

    awesome, simply awesome

    1. b0llchit Silver badge
      Pint

      So much fun. And tragic because it is so true.

      A test device or temporary fix are always so permanent and mission critical. Time to open a window on the top floor to show the beancounter the beautiful view.

      1. Anonymous Coward
        Anonymous Coward

        But...but.... So often systems work fine in a test environment and only have problems when in production.

        So makes complete sense to cut out the test environment and save time, no?

        1. Stork

          I worked at the IT arm of a major logistics outfit where they in the late 90es had made a proof of concept of an access system for customers, based on certificates. No concern had been given to more than one user at a time, and as tradition demanded it had been coded by two people under the influence of cold pizza, warm Coke(the drink) and coffee.

          For years it ran in production and never caused trouble. As my dad used to say, many things actually go well.

          1. Insert sadsack pun here
            Trollface

            "As my dad used to say, many things actually go well."

            Wow, that's a coincidence, my granddad actually used to say something similar right up to the day he was squashed by an elephant. RIP, Lucky.

      2. Blackjack Silver badge

        Oh no, that would be getting him easy.

        After all he not only lied to the BOFH but to the company.

        Meaning they can get him in legal trouble and sue him to death.

        And then he will jump out himself.

        1. Hot Diggity

          Justice must move quickly in your part of the world.

          1. Blackjack Silver badge

            It doesn't matter if the trial takes a decade, with that kind of reputation no one would everr4 hire him again.

  2. Anonymous Custard Silver badge
    Headmaster

    Testing 1 2 3

    Ah yes, the beta/test equipment - only slightly less permanent than the temporary fix from the last On Call...

    1. Aladdin Sane Silver badge

      Re: Testing 1 2 3

      Would that be more or less permanent than temporary portacabins in schools?

      1. Whiskers

        Re: Testing 1 2 3

        Portacabins? In my day we had Nissen huts.

        1. Ahab Returns

          Re: Testing 1 2 3

          Nissen huts? Luxury....

          1. Will Godfrey Silver badge
            Facepalm

            Re: Testing 1 2 3

            Indeed!

            We had cotton tents...

            With no fly sheet...

            Nor a groundsheet

            1. Montreal Sean

              Re: Testing 1 2 3

              Pure luxury! We had lean-tos made from the sticks we could gather up at start of term!

              1. TripDadAu

                Re: Testing 1 2 3

                Lean-tos! POSH!

                In my day we all had to shelter in the lee of an empty crisp packet! And we were grateful.

                1. Aussie Doc Bronze badge
                  Pint

                  Re: Testing 1 2 3

                  You 'ad crisps???

              2. John Brown (no body) Silver badge

                Re: Testing 1 2 3

                "Pure luxury! We had lean-tos made from the sticks we could gather up at start of term!"

                We had a lean-to too. It was quite big. More of a lean-three really.

          2. Doctor Syntax Silver badge

            Re: Testing 1 2 3

            Anderson shelters?

        2. Chris G Silver badge

          Re: Testing 1 2 3

          We had a two level air raid shelter for storing the sports equipment.

          You could hear things moving and breathing in the lower level.

          1. Aladdin Sane Silver badge

            Re: Testing 1 2 3

            The dwarves delved too greedily and too deep. You know what they awoke in the darkness

            1. Chris G Silver badge

              Re: Testing 1 2 3

              Not dwarves but sixth formers who kept a stash of naughty magazines down there.

              1. earl grey
                Facepalm

                Re: Testing 1 2 3

                i thought for sure you were going to say orcs.

                1. Grikath
                  Devil

                  Re: Testing 1 2 3

                  Is there a difference?

          2. John Brown (no body) Silver badge

            Re: Testing 1 2 3

            "You could hear things moving and breathing in the lower level."

            That was the sports master and sports mistress! Jolly hockey sticks ladies!

      2. Zarno Silver badge
        Coffee/keyboard

        Re: Testing 1 2 3

        Read that as portaloos...

        1. rg287 Silver badge

          Re: Testing 1 2 3

          My primary school had temporary portaloos (okay, the ones in a portacabin, not blue boxes).

          They were there until they built a new school down the road. Was never sure what the pupils did before the portacabin...

          1. Mark 85 Silver badge

            Re: Testing 1 2 3

            Was never sure what the pupils did before the portacabin...

            Old military style slit trenches out back with pile of dirt next to it and shovel?

            1. Rich 11 Silver badge

              Re: Testing 1 2 3

              You had a shovel?

          2. Potty Professor
            Boffin

            Re: Testing 1 2 3

            I was extremely disappointed when I moved up to Senior School in 1959, the main school building was a venerable old institution, all marble and polished mahogany, but the first and second years were located three miles away in a dilapidated Victorian brick pile. The classrooms weren't too bad, but to get from one to another, you had to walk around the quad, which was open to the sky (and the elements). OK, I could live with that, but the bogs were in another shanty in the playground, there was at least a roof over the thrones, but no doors on the cubicles. The Piss Stone, however, was out in the open, so you always got soaked through if you had a pee if it was raining. Luckily, we had a new school built some three miles in the opposite direction and I was one of the first intake of third years, the luxury of inside plumbing! A few years after I graduated in 1967, the new school was bulldozed because it was built primarily of Asbestos, I wonder how many of my year are still alive?

            1. a pressbutton

              Re: Testing 1 2 3

              When they refurbed my school, they found a very large amount of mercury under the chem lab floor.

              Who knew that 80y of broken thermometers could add up to ?

              Never mind the asbestos.

      3. Stoneshop Silver badge

        Re: Testing 1 2 3

        Would that be more or less permanent than temporary portacabins in schools?

        Those self-destruct even without 20mm (or any other size) drills poking holes in them. Although I've worked at a government department housed in emergency barracks from the late 1950 or maybe early 1960s, that were still doing reasonably well forty years on.

        Temporary fixes left to themselves tend to be even longer-lived.

        1. Anonymous Coward
          Anonymous Coward

          Re: Testing 1 2 3

          The temporary buildings at DWP in Lytham St Annes were built on sand. the complex was reputed to have the longest corridor in Europe and looking down it you could see where the joins' between buildings were by the changes in slope. it was like a gentle roller coaster. It was also where Ernie the random number generator for Premium Bonds was located.

        2. John Brown (no body) Silver badge

          Re: Testing 1 2 3

          "Although I've worked at a government department housed in emergency barracks from the late 1950 or maybe early 1960s, that were still doing reasonably well forty years on."

          Sounds like the DSS at Longbenton on Tyneside. All replaced now with modern office buildings. Although now I come to think of it, that might have been an old wartime pre-fab hospital originally. The corridors followed the contours of the land. Weird place to walk around in.

      4. qwerty360

        Re: Testing 1 2 3

        More, but only because 'temporary' portacabins in schools have a slightly higher rate of critical failure (the entire system catching fire and burning to the ground/flooding/{insert disaster wiping out everything in the area})

        1. Rich 11 Silver badge

          Re: Testing 1 2 3

          {insert disaster wiping out everything in the area}

          The riot after losing the under-16s county rugby final. Again.

        2. rototype
          Headmaster

          Re: Testing 1 2 3

          Not sure the ones at our school would have caught fire, they were too soggy. They were starting to rot (after 3 years the window frames were like blotting paper) and fall down on their own. Mind, we'd not have said no to a nice fire in one in the winter - were b****y cold in them.

  3. astfgl
    Facepalm

    Test into prod?

    The test database is indeed on the test server. It worked flawlessly, so they put the prod database on there as well. And didn't tell anyone. Despite there being an identical prod server provisioned and ready to go.

    Then there was the prod database on the test webapp server along with the test webapp, the test database on the test database server, the prod webapp on the prod webapp server, and the prod database server doing nothing.

    Good thing we backup our test servers.

    1. ColinPa

      Re: Test into prod?

      I did a review of a customer's set up where they had copied test database into production, and renamed the files at the same time - so there could be no confusion.

      "are the production data backed up?"

      "of course" ( eye rolling)

      "please show me"

      "ahh - we have been backing up the test data for 2 years - and not production - we forgot to update the backup job"

      1. Potemkine! Silver badge

        Re: Test into prod?

        Next funny question: when did you test to restore from backups?

    2. I ain't Spartacus Gold badge

      Re: Test into prod?

      Good thing we backup our test servers.

      Is not what the BoFH would say at all!

      No. The BoFH would get out his rubber mallet and explain in simple terms why test servers weren't backed up and everything was lost. Only after the collective departemental managment had worked through the first two stages of grief (denial and anger) and reached the bargaining stage might it emerge that the BoFH had in fact backed up the test servers.

      Or he'd actually say that by using quantum state-analysis on the drives using an electron microspcope supported by neutrino beams from the Large Hadron Collider he might be able to recover the data. But obviously that would cost, as it would require equipment, a trip to Switzerland, a permit from the Office for Nuclear Regulation and a couple of hazmat suits. A quick skiing trip (and a heavily modified expenses claim) later, and... Your data has been recovered.

      Obviously backing up test servers is only for the purposes of malice and finding incriminating data, but it's always useful to keep copies of everything. You just shouldn't always tell people that straight away...

    3. This post has been deleted by its author

    4. Doctor Syntax Silver badge

      Re: Test into prod?

      "Good thing we backup our test servers."

      But don't tell the users until you've let them sweat a bit.

    5. Anonymous Coward
      Anonymous Coward

      Re: Test into prod?

      I once picked up a project where the team were complaining about performance. It turns out that they had never been migrated from the single CPU machine used for prof of concept testing to the massive High Availability Cluster purchased to provide the test and production environments.

      Ironically they project had been running long enough that the physical cluster was about to be superseded by a large Hyper V farm. In the next couple of months they went from a 4 GB single CPU environment to 32 CPU's and about 128 Gb of memory. Once in production they consumed have the resources of a large server farm.

  4. crosenblum

    You always need a mallet...

    Especially the one, where it doesn't show the damage it creates.

    Perhaps an electircally charged one.

    Thank you for a great Friday morning.

    Always gives me a smile and a minute of peace vs the chaos of idiots and managers.

    1. Paul Crawford Silver badge

      Re: You always need a mallet...

      Yes, and ideally a Thor one for proper smiting:

      https://www.thorhammer.com/

      1. RyokuMas Silver badge
        Thumb Up

        Re: You always need a mallet...

        Thor's hammer... percussive maintenance device and over-voltage cattleprod all rolled into one... the dream tool of any Bastard, surely?

        1. Boris the Cockroach Silver badge
          Devil

          Re: You always need a mallet...

          Now that gives this crazy engineer a jolly good idea.

          Need some decent caps that will fit inside the handle... reinforcing tube (dont want the handle breaking in service) cunningly hidden switch, wires drilled through to copper side and a battery able to handle repeat charges.

          If it works, then I sense a great business opportunity...

          1. quxinot Silver badge

            Re: You always need a mallet...

            Good application for old tech.

            Lead-acid battery, clearly, positioned appropriately.

            1. John Brown (no body) Silver badge

              Re: You always need a mallet...

              Oh yeah! A decently sized car battery would be ideal. Lots of mass and designed for rapid discharge! (The more muscular can try it with a truck battery, although I suspect the nerd community may be a bit lacking in people of that stature)

              1. Anonymous Coward
                Anonymous Coward

                Re: You always need a mallet...

                Motorcycle battery.. Provides the needed jolt, lightweight enough, easily recharged, and also made for high discharge.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021