back to article BOFH: You. Wouldn't. Put. A. Test. Machine. Into. Production. Without. Telling. Us.

BOFH logo telephone with devil's horns A lesson I learned a long time ago was: never let a user buy a server. It's not a hard lesson to remember because of the frequent booster-shots for the rule whenever some purchasing person decides to sidestep their purchasing restrictions by "renaming" what they're buying. … So we've …

  1. Maverick

    awesome, simply awesome

    1. b0llchit Silver badge
      Pint

      So much fun. And tragic because it is so true.

      A test device or temporary fix are always so permanent and mission critical. Time to open a window on the top floor to show the beancounter the beautiful view.

      1. Anonymous Coward
        Anonymous Coward

        But...but.... So often systems work fine in a test environment and only have problems when in production.

        So makes complete sense to cut out the test environment and save time, no?

        1. Stork

          I worked at the IT arm of a major logistics outfit where they in the late 90es had made a proof of concept of an access system for customers, based on certificates. No concern had been given to more than one user at a time, and as tradition demanded it had been coded by two people under the influence of cold pizza, warm Coke(the drink) and coffee.

          For years it ran in production and never caused trouble. As my dad used to say, many things actually go well.

          1. Insert sadsack pun here
            Trollface

            "As my dad used to say, many things actually go well."

            Wow, that's a coincidence, my granddad actually used to say something similar right up to the day he was squashed by an elephant. RIP, Lucky.

      2. Blackjack Silver badge

        Oh no, that would be getting him easy.

        After all he not only lied to the BOFH but to the company.

        Meaning they can get him in legal trouble and sue him to death.

        And then he will jump out himself.

        1. Hot Diggity

          Justice must move quickly in your part of the world.

          1. Blackjack Silver badge

            It doesn't matter if the trial takes a decade, with that kind of reputation no one would everr4 hire him again.

  2. Anonymous Custard Silver badge
    Headmaster

    Testing 1 2 3

    Ah yes, the beta/test equipment - only slightly less permanent than the temporary fix from the last On Call...

    1. Aladdin Sane

      Re: Testing 1 2 3

      Would that be more or less permanent than temporary portacabins in schools?

      1. Whiskers

        Re: Testing 1 2 3

        Portacabins? In my day we had Nissen huts.

        1. Ahab Returns

          Re: Testing 1 2 3

          Nissen huts? Luxury....

          1. Will Godfrey Silver badge
            Facepalm

            Re: Testing 1 2 3

            Indeed!

            We had cotton tents...

            With no fly sheet...

            Nor a groundsheet

            1. Montreal Sean

              Re: Testing 1 2 3

              Pure luxury! We had lean-tos made from the sticks we could gather up at start of term!

              1. TripDadAu

                Re: Testing 1 2 3

                Lean-tos! POSH!

                In my day we all had to shelter in the lee of an empty crisp packet! And we were grateful.

                1. Aussie Doc
                  Pint

                  Re: Testing 1 2 3

                  You 'ad crisps???

                  1. BobTheIntern

                    Re: Testing 1 2 3

                    You were lucky!

              2. John Brown (no body) Silver badge

                Re: Testing 1 2 3

                "Pure luxury! We had lean-tos made from the sticks we could gather up at start of term!"

                We had a lean-to too. It was quite big. More of a lean-three really.

          2. Doctor Syntax Silver badge

            Re: Testing 1 2 3

            Anderson shelters?

        2. Chris G

          Re: Testing 1 2 3

          We had a two level air raid shelter for storing the sports equipment.

          You could hear things moving and breathing in the lower level.

          1. Aladdin Sane

            Re: Testing 1 2 3

            The dwarves delved too greedily and too deep. You know what they awoke in the darkness

            1. Chris G

              Re: Testing 1 2 3

              Not dwarves but sixth formers who kept a stash of naughty magazines down there.

              1. earl grey
                Facepalm

                Re: Testing 1 2 3

                i thought for sure you were going to say orcs.

                1. Grikath
                  Devil

                  Re: Testing 1 2 3

                  Is there a difference?

          2. John Brown (no body) Silver badge

            Re: Testing 1 2 3

            "You could hear things moving and breathing in the lower level."

            That was the sports master and sports mistress! Jolly hockey sticks ladies!

      2. Zarno
        Coffee/keyboard

        Re: Testing 1 2 3

        Read that as portaloos...

        1. rg287 Silver badge

          Re: Testing 1 2 3

          My primary school had temporary portaloos (okay, the ones in a portacabin, not blue boxes).

          They were there until they built a new school down the road. Was never sure what the pupils did before the portacabin...

          1. Mark 85

            Re: Testing 1 2 3

            Was never sure what the pupils did before the portacabin...

            Old military style slit trenches out back with pile of dirt next to it and shovel?

            1. Rich 11

              Re: Testing 1 2 3

              You had a shovel?

          2. Potty Professor
            Boffin

            Re: Testing 1 2 3

            I was extremely disappointed when I moved up to Senior School in 1959, the main school building was a venerable old institution, all marble and polished mahogany, but the first and second years were located three miles away in a dilapidated Victorian brick pile. The classrooms weren't too bad, but to get from one to another, you had to walk around the quad, which was open to the sky (and the elements). OK, I could live with that, but the bogs were in another shanty in the playground, there was at least a roof over the thrones, but no doors on the cubicles. The Piss Stone, however, was out in the open, so you always got soaked through if you had a pee if it was raining. Luckily, we had a new school built some three miles in the opposite direction and I was one of the first intake of third years, the luxury of inside plumbing! A few years after I graduated in 1967, the new school was bulldozed because it was built primarily of Asbestos, I wonder how many of my year are still alive?

            1. a pressbutton

              Re: Testing 1 2 3

              When they refurbed my school, they found a very large amount of mercury under the chem lab floor.

              Who knew that 80y of broken thermometers could add up to ?

              Never mind the asbestos.

      3. Stoneshop

        Re: Testing 1 2 3

        Would that be more or less permanent than temporary portacabins in schools?

        Those self-destruct even without 20mm (or any other size) drills poking holes in them. Although I've worked at a government department housed in emergency barracks from the late 1950 or maybe early 1960s, that were still doing reasonably well forty years on.

        Temporary fixes left to themselves tend to be even longer-lived.

        1. Anonymous Coward
          Anonymous Coward

          Re: Testing 1 2 3

          The temporary buildings at DWP in Lytham St Annes were built on sand. the complex was reputed to have the longest corridor in Europe and looking down it you could see where the joins' between buildings were by the changes in slope. it was like a gentle roller coaster. It was also where Ernie the random number generator for Premium Bonds was located.

        2. John Brown (no body) Silver badge

          Re: Testing 1 2 3

          "Although I've worked at a government department housed in emergency barracks from the late 1950 or maybe early 1960s, that were still doing reasonably well forty years on."

          Sounds like the DSS at Longbenton on Tyneside. All replaced now with modern office buildings. Although now I come to think of it, that might have been an old wartime pre-fab hospital originally. The corridors followed the contours of the land. Weird place to walk around in.

      4. qwerty360

        Re: Testing 1 2 3

        More, but only because 'temporary' portacabins in schools have a slightly higher rate of critical failure (the entire system catching fire and burning to the ground/flooding/{insert disaster wiping out everything in the area})

        1. Rich 11

          Re: Testing 1 2 3

          {insert disaster wiping out everything in the area}

          The riot after losing the under-16s county rugby final. Again.

        2. rototype
          Headmaster

          Re: Testing 1 2 3

          Not sure the ones at our school would have caught fire, they were too soggy. They were starting to rot (after 3 years the window frames were like blotting paper) and fall down on their own. Mind, we'd not have said no to a nice fire in one in the winter - were b****y cold in them.

  3. astfgl
    Facepalm

    Test into prod?

    The test database is indeed on the test server. It worked flawlessly, so they put the prod database on there as well. And didn't tell anyone. Despite there being an identical prod server provisioned and ready to go.

    Then there was the prod database on the test webapp server along with the test webapp, the test database on the test database server, the prod webapp on the prod webapp server, and the prod database server doing nothing.

    Good thing we backup our test servers.

    1. ColinPa Silver badge

      Re: Test into prod?

      I did a review of a customer's set up where they had copied test database into production, and renamed the files at the same time - so there could be no confusion.

      "are the production data backed up?"

      "of course" ( eye rolling)

      "please show me"

      "ahh - we have been backing up the test data for 2 years - and not production - we forgot to update the backup job"

      1. Potemkine! Silver badge

        Re: Test into prod?

        Next funny question: when did you test to restore from backups?

    2. I ain't Spartacus Gold badge

      Re: Test into prod?

      Good thing we backup our test servers.

      Is not what the BoFH would say at all!

      No. The BoFH would get out his rubber mallet and explain in simple terms why test servers weren't backed up and everything was lost. Only after the collective departemental managment had worked through the first two stages of grief (denial and anger) and reached the bargaining stage might it emerge that the BoFH had in fact backed up the test servers.

      Or he'd actually say that by using quantum state-analysis on the drives using an electron microspcope supported by neutrino beams from the Large Hadron Collider he might be able to recover the data. But obviously that would cost, as it would require equipment, a trip to Switzerland, a permit from the Office for Nuclear Regulation and a couple of hazmat suits. A quick skiing trip (and a heavily modified expenses claim) later, and... Your data has been recovered.

      Obviously backing up test servers is only for the purposes of malice and finding incriminating data, but it's always useful to keep copies of everything. You just shouldn't always tell people that straight away...

    3. This post has been deleted by its author

    4. Doctor Syntax Silver badge

      Re: Test into prod?

      "Good thing we backup our test servers."

      But don't tell the users until you've let them sweat a bit.

    5. Anonymous Coward
      Anonymous Coward

      Re: Test into prod?

      I once picked up a project where the team were complaining about performance. It turns out that they had never been migrated from the single CPU machine used for prof of concept testing to the massive High Availability Cluster purchased to provide the test and production environments.

      Ironically they project had been running long enough that the physical cluster was about to be superseded by a large Hyper V farm. In the next couple of months they went from a 4 GB single CPU environment to 32 CPU's and about 128 Gb of memory. Once in production they consumed have the resources of a large server farm.

  4. crosenblum

    You always need a mallet...

    Especially the one, where it doesn't show the damage it creates.

    Perhaps an electircally charged one.

    Thank you for a great Friday morning.

    Always gives me a smile and a minute of peace vs the chaos of idiots and managers.

    1. Paul Crawford Silver badge

      Re: You always need a mallet...

      Yes, and ideally a Thor one for proper smiting:

      https://www.thorhammer.com/

      1. RyokuMas
        Thumb Up

        Re: You always need a mallet...

        Thor's hammer... percussive maintenance device and over-voltage cattleprod all rolled into one... the dream tool of any Bastard, surely?

        1. Boris the Cockroach Silver badge
          Devil

          Re: You always need a mallet...

          Now that gives this crazy engineer a jolly good idea.

          Need some decent caps that will fit inside the handle... reinforcing tube (dont want the handle breaking in service) cunningly hidden switch, wires drilled through to copper side and a battery able to handle repeat charges.

          If it works, then I sense a great business opportunity...

          1. quxinot

            Re: You always need a mallet...

            Good application for old tech.

            Lead-acid battery, clearly, positioned appropriately.

            1. John Brown (no body) Silver badge

              Re: You always need a mallet...

              Oh yeah! A decently sized car battery would be ideal. Lots of mass and designed for rapid discharge! (The more muscular can try it with a truck battery, although I suspect the nerd community may be a bit lacking in people of that stature)

              1. Anonymous Coward
                Anonymous Coward

                Re: You always need a mallet...

                Motorcycle battery.. Provides the needed jolt, lightweight enough, easily recharged, and also made for high discharge.

  5. Dwarf

    Brilliant and so true.

  6. OGShakes

    Been there and done that

    10 year old server that had no production databases on it, it was simply alive out of luck and strictly only used for Dev/Test monkey business when trying ideas for a strictly hypothetical things. 7 years ago it had been the physical box used for production databases, these had long since gone else where in to a cluster split over 2 data centres. This server was turned off and removed from the rack due to a building move, 12 hours later data refreshes between production CRM and Finance systems started to fail and no one knew why...

  7. mikeHingley

    A 24x7 3yearr NBD - Avast!!!!

    Was I supposed to read that in a pirate voice - because I totally did.

  8. Terry 6 Silver badge

    "It's only a 40-minute wait time," he says happily.

    "Yes," the PFY says darkly. "That's how they get you. Forty minutes now, then two songs, a message about how important your call is and how they have an unprecedented volume of callers at the moment, then your time's extended to 60 minutes, then two more songs, a message about using their web site, then your time's extended to 85 minutes, two more songs, a message, two hours and ten minutes …"

    This is such a thing that it's become the stuff of satire.Yet still it goes on!

    1. DCdave

      and we all know that it finishes with the call being cut at exactly closing time for their hotline (or before if they really don't give a rat's behind).

    2. Terry 6 Silver badge

      Addendum. I'd typed the above as I embarked on a telephonic journey to ask Allianz when they're going to resolve sorting out the damage to my car, hit by their insured while parked outside my own front door . And I had the above nonsense, plus being cut off while on hold, plus being passed around three different departments and somehow my call being forwarded/redirected to my own insurer - (which started with a message "we do not recognise that phone number."..etc)so that I thought I was still talking to Allianz, and was rather surprised to be asked the claim reference for a fourth time - only to be told "That's not one of ours" when I'd used it three times previously on this same call!!!!

      I'm sure there are plenty of others as bad(?) but I won't be using Allianz for insurance any time soon.

      1. Swarthy

        I have seen the bumper sticker "If you're insured by Allstate, please don't hit me"

        I'll assume there are ones for Allianz on your side of the pond.

        1. jtaylor

          "If you're insured by Allstate, please don't hit me"

          That's wonderful! Years ago, my parents had AllState auto insurance. They were hit by a red light runner, and filed a claim. AllState called to get information and certain forms. When my parents failed to return the declaration where they admitted fault, AllState harassed them repeatedly during dinner time. My parents eventually discovered that AllState had insured the other driver too, and were defending themselves against both claims. Once that came to light, the matter was resolved damn quick.

        2. Terry 6 Silver badge

          Sadly we don't do bumper stickers that much here. It's a very USA sort of thing.

          1. Mike Pellatt

            I'm making an exception for the Devon flag with "cream first" across the bottom that I saw recently, as soon as I can find one in a rip-off-the-grockles shop.

      2. Anonymous Coward Silver badge
        Meh

        As a general rule, that's up to your own insurance company to sort out and they then bill the other party's insurance.

        1. Terry 6 Silver badge

          Which is what is happening - now. The idiot I spoke to in my insurance company previously said I needed to call the other one. I don't have very great regard for any insurance company tbh.

      3. Anonymous Coward
        Anonymous Coward

        We had [anon for legal reasons] as insurer, and they tried delaying the payback for a claim for so long that it would end up in court so they could then offer to pay only 70% of it, which is apparently deemed acceptable enough for a court to OK it. Nice way to shave 30% off a bill..

        Unfortunately for them, another department of theirs already settled part of the claim (as it was in two parts), so when the division haggling with my lawyer offered 70% of the total as full and final, I told her to accept. Our acceptance of that claim ensured they paid out 120% after deducting the costs for our lawyer. As they stated full and final and accepted our agreement, there was no way for them to go back on it so they basically ended up paying us 20% over claim (which were actual expenses incurred) which was a nice bit of compensation for trying to game us.

        We got lucky, the fact that they could potentially fully legal get away with paying only 70% was not a fun thing to discover and is worth knowing for every insured user.

        1. Ken Hagan Gold badge

          Does that mean it is OK to pay only 70% of the premium then. That's worth knowing. Thanks for the info!

        2. Potty Professor
          Devil

          I was stationary in a queue of traffic, all waiting to turn right across the oncoming stream of traffic, when I was hit from behind by a young lunatic driving his mother's car. The impact pushed my old Range Rover forwards into the back of another car, and the result was two severely damaged cars. My car was, however, only slightly damaged, front and rear bumpers bent and two of the four overriders bent out of shape. My insurance company decided that it would be easier to write off all three vehicles, and paid me out the full agreed valuation (£2000), and sold the car back to me for essentially scrap value. A quick search on ebay produced the necessary overriders, and a big hammer, a concrete slab, and a block of oak sorted the two bumpers, so I made a nice little profit out of that eejit's mistake. I smelt alcohol on his breath and counted seven louts climbing out of the five seater car, so I don't know what fate befell him. His mother wouldn't have been too pleased, either, which was worse, the penalty or the tongue lashing at home?

    3. Doctor Syntax Silver badge

      As of today my soon to be ex-bank are still using covid as an excuse.

    4. PM from Hell

      So how old are you

      Am I the only person who remembers 'Word Perfect Radio'? I had to have my phone budget increased massively to pay for the hours of being on hold to support lines on the west coast. My team also all ended up with executive speaker phones so they could carry on working on other things while on-hold. This was after several incidents were instances where having been on-hold for over an hour they didn't manage to respond quickly enough when the p[hone was finally put through to support and were hanged up on.

  9. Jay 2

    I once worked at a Deathstar logo'd telo and the telo networks team decided to have a nice videowall in their new office. So far so good. They went ahead and purchased the 6 rear projection TVs etc and also some 1U beige box that would somehow allow computery stuff to appear on said screen. At no point during the look/purchase process did they ask the sys admins if/how this box could fit in with the HP-UX workstations that ran all the monitoring apps etc. So you can probably guess what happened...

    "We've bought this setup and now you must make it work"

    One of my colleagues got the fun job of wrangling loads of X11 stuff to get it to work. The funniest thing is that whilst you could get an X window with the app on the 6 screens, the icons didn't scale so they would still be *very* small in the very corner of the window.

    Overall throughout my career there have been far too many instances of "test" solutions suddenly morphing into prod. And the users still never quite get why we get so irate about it.

    1. ColinPa Silver badge

      "it only took us an hour to get it working in test - why is it taking you so long to set it up in production?"

      "did you have security, authentication, authorisation and logging enabled"

      "no"

      "I rest my case"

    2. Alan W. Rateliff, II
      Mushroom

      "We've bought this setup and now you must make it work"

      The frequently recurring nightmare of IT workers everywhere.

      1. Giles C Silver badge

        A few years ago someone bought a load of chromecast adaptors for large tvs in the office, back then it would not connect to a Cisco wireless access point or any form of corporate WiFi.

        Two weeks later and they got sent back, only to be told it works at home….

      2. Terry 6 Silver badge

        Common experience in schools too. Both IT and other stuff. Usually stuff obtained by the kind of Senior Managers who like to go with the herd and do what the other school up the road is doing.

        "This is the new scheme/product that we've spent all our budget on for the next three years. So you have to use it".

        Without consulting the specialist/informed staff in their own school- who know it's a pile of crap, which is why they didn't already recommended it. And who probably already know that the school up the road obtained it the same way- and have already proved that it's utter crap and expensive waste of money.

    3. Paul Smith

      "We've bought this setup and now you must make it work"

      My preferred reply:

      "I hope you kept the receipt."

      1. Ken Hagan Gold badge

        Re: "We've bought this setup and now you must make it work"

        Failing that, I hope you kept the packaging. Try mitigating your loss by flogging it on eBay.

      2. Grikath

        Re: "We've bought this setup and now you must make it work"

        Not really...

        "No problem.. What's the support ticket number?"

        "Huh?!! What do you need that for?"

        "Well... *Your* lot insisted all our hours should be tied to job tickets so they'd be billable internally against your budgets... So without a ticket number I can't verify my hours and all that.."

        "Ummmmmmm... Can't you make an exception?"

        "Sure, I'm allowed to do work outside hours as a contractor.. but... *sucking teeth* ... I'll send you an extimate for that tomorrow, ok?"

        1. A.P. Veening Silver badge

          Re: "We've bought this setup and now you must make it work"

          I'll send you an extimate for that

          Extimate, is that an extortionate estimate?

          1. John Brown (no body) Silver badge

            Re: "We've bought this setup and now you must make it work"

            "Extimate, is that an extortionate estimate?"

            And the rest! The BOFH would never go so low as only "extortionate"

    4. Anonymous Coward
      Anonymous Coward

      We never had that problem at the Atlanta Works.

  10. TeeCee Gold badge

    Ah, yes.

    ProductionTest Servers.

    I recall a long-running project finally coming to fruition. Many, many hours had been invested in test, migrating data from the old, production, system and then with the users filling in the gaps in the new, more puissant database.

    Then into testing, fixing, parallel running, yadda, yadda, you get the picture.

    Come the completion of this herculean effort, it was all deemed ready for live. Now there's only one of this thing, so there's no benefit whatsoever in doing a clean install to a new, production server. The test server is entirely adequate for the production tasks and so it is decided to put the test server live.

    Huge success, trebles all round and party into the night.

    Then the project close process runs and the Data Centre decommissions the project's allocated test servers.....(!) Oops.

    What's the other big difference between a production and test server? Production servers have a backup schedule....(!!) Double oops and now they're fucked.

    1. Doctor Syntax Silver badge

      Re: Ah, yes.

      I suppose the old production servers were also decommissioned and their backups destroyed to avoid any risk of data leakage.

  11. John 110
    Mushroom

    The guy's here...

    "The guy's here with the new analyser and he needs a network connection"

    "Well we'll need a node livened by IT, that usually take a couple of days if they're not busy rolling out COVID testing gear"

    "Could you not phone Denise and ask her nicely?"

    "I could've if you'd asked me yesterday, or last week would've been better. When did you know we were getting this?"

    "Weeellll........."

    1. MiguelC Silver badge

      Re: The guy's here...

      "I've forwarded you an email with the new regulatory requisites to be enforced starting next month"

      "One month to analyse, develop, test and deliver in production, are you nuts?"

      "The regulatory body just told us, it's mandatory, huge fines, yadda yadda"

      "Then why is the original email's date from six month ago?"

      "....."

      1. Stoneshop
        Facepalm

        Re: The guy's here...

        You got an email? Luxury.

        Computer: "Your password has expired and needs to be changed. Please enter old password."

        Colleague: tappitytappitytatptap.

        Computer: "Please enter new password."

        Colleague: taptaptaptappetytap.

        Computer: "New password not accepted. Minimum password length is 20 characters"

        Colleague:: #^$^*&#^*&^*^*@*&^@*&*()#IU#!!!

        Apparently this is a new chapter in the ever-ongoing play in the security theatre, which, as usual, gets distributed through screen prompts, not via messages one can read in advance. 20 characters will have people writing them down. Which is fine if you keep whatever it's written down on well away from the computer which it applies to, but we all know that won't happen if it's the work laptop you have to take with you when working from home.

        1. quxinot

          Re: The guy's here...

          >Computer: "New password not accepted. Minimum password length is 20 characters"

          >Colleague:: #^$^*&#^*&^*^*@*&^@*&*()#IU#!!!

          Computer to Colleague: Good try, but a number is required.

          1. A.P. Veening Silver badge

            Re: The guy's here...

            Computer to Colleague: Good try, but a number is and a lower case character are required.

            FTFY ;)

            1. stiine Silver badge

              Re: The guy's here...

              Sorry too many repeated characters.

              1. arachnoid2

                Colleague:: #^$^*&#^*&^*^*@*&^@*&*()#IU#!!!

                I'm sorry those characters are in our excluded word list, please try again.

                You have used 9 out of 10 attempts before you will be locked out of the system for 24 hours.

        2. Dante Alighieri

          requirements not met

          or my recent try - new password not acceptable to system.

          Only no hint of what the new requirements are in the password change dialogue.

          Nor on the intranet

          Nor a help ticket to Helldesk (including suggesting putting the info into the reject prompt) x2

          Another shiny day in the NHS...

        3. Anonymous Coward
          Anonymous Coward

          Re: The guy's here...

          20 characters is a decent password length. Just use an easy to remember phrase like "correct battery horse staple". Make sure you turn off complexity requirements and regular password rotation as well as per the latest nist password recommendations.

          Or, instead of a phrase you could just use an older password you can remember multiple times to meet the length requirements.

          Oh, and 2fa everything as well.

          1. Dyspeptic Curmudgeon

            Re: The guy's here...

            Retirednow. For many years the Teraview on-line real estate /title software required a new password every 2 months or so.

            For years I updated the password by going from 123Thisisme to 124Thisisme. When the damn system rejected my password I just backed up a few numbers and tried again, re-iteratively. No post-it notes required. Ended up at 152.... iirc. But I can't remember whether I started with 1 or 100!

            1. Anonymous Coward
              Anonymous Coward

              Re: The guy's here...

              And this is the problem with forced password rotation and the reason for the current nist guidelines. People just come up with variations on a predictable theme or write it down. The guy responsible for the old nist guidelines that recommended rotation has freely admitted that it is the worst mistake of his career.

              When it comes to passwords, length matters (cue Frankie Howerd style comments).

              Or just trust in Randall:

              https://xkcd.com/936/

              1. Terry 6 Silver badge

                Re: The guy's here...

                It's worse in schools and colleges. Because over a holiday all the staff will have forgotten their password. And/or the ones they had will have expired. So they'll arrive in their classroom or the staff workroom all at the same time, bright and early ahead of the new term, which will be before the IT dept, LA IT support etc have even finished their kippers. So it would be mid-morning before they could all get back online, and they still have to try and remember the new p/w.. They quickly learn to choose ones they can work out and change to one they can still work out. (Maybe, say, Summer21 followed by Winter21 followed by Summer22 and so on)

                1. Joe W Silver badge

                  Re: The guy's here...

                  OK, I'll admit: I am that guy. Not a teacher, so no six or eight week long breaks from work. However, after a three week holiday, hiking or whatever, being really very relaxed an rested I cannot remember my password. Usually I try to remember setting it to my pre-holiday-pretty-secure-password that I don't use elsewhere, and which I can sort of remember (after two tries). Other times I have to ring IT support (and apologise for being stupid).

                  If you can remember the password, clearly your holidays were too short. Good thing I use a password manager for everything else but the main work account! (Yeah, we also have the "change all four weeks" rule in place... eeedjits).

                  1. rototype

                    Re: The guy's here...

                    If you were at our place they'd say 'Hey, no worries, means you had a great holiday and a good break which is exactly what holidays are for' - Our management look at holidays as time to turn off, refresh and unwind so you're fresh when you get back to work again, even if it does mean IT need to reset your forgotten passwords. Did you guess a large proportion of our company are in The Netherlands...

          2. Stoneshop
            Facepalm

            Re: The guy's here...

            Make sure you turn off complexity requirements and regular password rotation as well as per the latest nist password recommendations.

            "You"? We don't have any say in those matters. In so far as we have control over password requirements it's only for one specific legacy platform we manage that already sits behind a further two logins: the work laptop's and a 2FA login on a virtual machine. Only from that VM can you get to those systems, and when an oukaze from Security was sent out that password life times on that VM and everything behind it had to be set at 3 months, we went "It's now at 30 days. Let's just happily ignore this silliness.". The password I'm using for another platform behind that VM is now some 30 months old, as every time I try to change it I get an error thrown back at me, and as the login keeps working I don't bother any more as that one's a SEP.

            The 20-character length BTW is for that 2FA-secured login.

          3. Doctor Syntax Silver badge

            Re: The guy's here...

            Just use an easy to remember phrase like "correct battery horse staple".

            Like?

            1. Mike Pellatt

              Re: The guy's here...

              I'm tempted to use that somewhere and save it in my browser, just to see if Google or Microsoft have spotted it in a list of stolen passwords anywhere

          4. anothercynic Silver badge

            Re: The guy's here...

            It boggles the mind to see how many Internet-facing systems these days still rely/insist on passwords of suspiciously low length (20 characters? Nahhhh, 8 must do, thank you). And there are quite a few large names behind those systems.

          5. AlanDouglas

            Re: The guy's here...

            Problem is that there are too many different ideas about password strength, and too many amateur implementations. How many passwords in the wild start with a capital letter and end with a number (1) and a 'special' character !

            Also, despite all this nonsense about correctbatteryhorsestaple being the most secure password, I have been using this for about 5 years and still get hacked about once every week;

        4. Terry 6 Silver badge

          Re: The guy's here...

          In fact we all know that this is what Post-it notes were invented for.

        5. Anonymous Coward
          Anonymous Coward

          Re: The guy's here...

          The minute the "stay at home" order was made, my organisation went from 60 day changes with 13 no-repeats (but easily defeated by the This123IsMe This124IsMe gambit) to, essentially, no expiry at all.

          Possibly something to do with the rather convoluted manual process for resetting forgotten passwords, which meant lodging personal information with IT or relying on IT being able to confirm your request via a colleague in the same office, which would have been difficult to do with everyone at home.

          Anon because they don't show any signs yet of reverting and I don't want to prompt them.

          Oddly enough, this way is probably more secure.

          1. Scene it all

            Re: The guy's here...

            My important passwords (like the master password to the thing that remembers all the other passwords) are just simple expressions like "nice to meet you". I even write them down in English to remind myself what they are.

            What I do NOT write down is:

            1. Which obscure and possibly dead language to translate it into.

            2. Which Leet-ish transformation to then apply.

            1. Caver_Dave Silver badge
              Angel

              Re: The guy's here...

              A colleague recently returned from foreign climbs, is working through all the place names he visited on his travels as the main part of each password. 3 months worth of travel and 3 different character sets should keep him going for a while.

        6. Recluse

          Re: The guy's here...

          Many years ago, on logging onto a website and being advised that I was required to change my password (and as a recent convert to the joys of the password database) I rather smugly decided to update my password to a super secure (so I thought) 30 digit one (system specified a minimum length of 10 characters).

          Auto generated new password and cut/pasted into requisite field, which was duly accepted and confirmed updated. Completed my business on site and logged out and happily went on my way.

          Next time I came to logon my password was rejected as invalid - impossible I thought as its recorded in a database and was copied/pasted.

          So I used the reset password link and updated my password once again, however next time I came to logon again history was repeated - updated password invalid.

          It was only much later I discovered that said system not only had a minimum length but also a maximum field length ... which I consistently exceeded, presumably it arbitrarily shortening my input.

          That will teach me to be smug (would still wish to shoot the developer - where was the error checking?)

          1. sanwin

            It is almost as if the company does not test things properly before unleashing them on the world.

            I had that problem several years back with a certain company in Redmond. The website insisted on a minimum length pw and I used 20 characters. It apparently accepted this but subsequent login attempts failed.

            I finally sussed the problem when, for a change, I had headphones plugged in to the PC and heard a beep when I typed the 17th character! No indication other than that beep that there was an upper limit. I complained and was not surprised to get no response, however the restriction was removed some time later.

            In a large office who has speakers attached to their PC?

  12. Boris the Cockroach Silver badge

    Whenever

    I see the mangler, and hear the word 'test' I know its time to make myself scarce..... "just repairing this hose" I cry out from my position under robot #4 knowing full well no one can see what I'm really doing (browsing el-reg on my phone) and waiting patiently until the mangler gets bored and wanders off to bother someone else.

    Why do I adopt this cynical attitude? because too many times said mangler has read/been given something new and whizzy that will make things 10 times faster at 1/2 the price(according to the salesman who actually getting desperate to shift the things before they ruin his monthly sales target),and I spend a good hour making the adjustments before finding out exactly why the tools are so difficult to shift (especially when broken and jammed in a robots grippers).....

    Which leads me on to the main theme of my arguement... a thor #2 copper/hide mallet.... when its in my hand and I'm about to argue with said mangler/salesman.....

    Wheres the mallet icon?

    1. Anonymous Coward
      Anonymous Coward

      Re: Whenever

      "a thor #2 copper/hide mallet.... when its in my hand and I'm about to argue with said mangler/salesman"

      If a salesweasel is within hammer-striking distance, you've already lost.

      1. Stoneshop
        Devil

        Re: Whenever

        If a salesweasel is within hammer-striking distance, you've already lost.

        If that's what you're thinking, you're holding him wrong.

        Yes, they're slippery. But there are tools to deal with that.

        1. rototype

          Re: Whenever

          Personally I prefer a 3lb Forge Hammer (Yes, I do carry one in the car - officially it's to aid in changing tyres), when a user has a tricky problem I often offer it to them, you should see their faces when I actually bring it into the office!

  13. Pirate Dave Silver badge
    Pirate

    "a rubber mallet and a Bible – with the instruction that if the Bible doesn't work he's to hit himself with the mallet."

    It's gems like this that make the El Reg subscription price worth every penny.

    This book doesn't have any answers!

    1. ColinPa Silver badge

      Bible and amallet

      Strangely enough this sounds like what doctors used to recommend for a ganglion cyst. My grandmother had one, and the doctor said hit it with a bible or other large book. That didnt work, so next time he hit it with a rubber hammer. This worked!

      1. Anonymous Coward
        Happy

        Re: Bible and amallet

        I remember hearing that one many times when I was younger!

        Apparently, "Back in 1743, the German anatomist Lorenz Heister (who invented the word tracheotomy) listed the treatment options for ganglia – wiping the lump with saliva; taking a bullet that had killed a wild animal (ideally a stag) and strapping it to the cyst; or even rubbing it with the hand of a dead man." (Source: BBC)

        But it goes on to say, "[In a 1972 study] Some tried the bible-bashing method: at first the fluid dissipated and the bump reduced; but recurrence rates were between 22% and 64%."

        The usual method these days is to drain with a needle, but there is no perfect method as it can still return. But most of them disappear within about 6 years (according to that article).

  14. Anonymous South African Coward Silver badge
    Pint

    Lovely start to the weekend!

    Off to the pub I go! YAY!!!

  15. DJV Silver badge
    Coffee/keyboard

    Hah! New keyboard required!

    "Honestly, who could have guessed that boron-coated drill would have gone straight through the lid, the hard drive and the bottom of the server," I ask.

    1. Anonymous Coward
      Happy

      Re: Hah! New keyboard required!

      As an aside, I've got to put some 16mm holes through an external wall of my house to feed some ethernet cables through for a WAP and some external security cameras into a PoE switch.

      I'm not looking forward to it, since I'm sure it won't go through as easily as it did through the black box in the BOFH.

      1. Richard 12 Silver badge

        Re: Hah! New keyboard required!

        That's a big hole for a small cable

        1. Anonymous Coward
          Anonymous Coward

          Re: Hah! New keyboard required!

          I know. Small cable, bloody big plug on either end.

          Having said that, weather-proof cable needed to be solid copper, and that's not the thinnest ethernet cabling in the world, either.

          If you use bare cable, you can get away with 5-6mm, but I didn't want to do that.

          I was hoping I might be able to get all three cables through a single hole if I threaded them one at a time, but I doubt it.

          1. Martin an gof Silver badge

            Re: Hah! New keyboard required!

            Having done quite a lot of this sort of thing recently, pulling RJ45 plugs through an external wall is not to be done lightly, especially if it's a traditional (UK) block/brick construction with a cavity (usually insulated). For a start you will get brick dust in the contacts, which not only scratches things up but also attracts moisture, possibly resulting in a concrete-y mess, but the little tongue also gets in the way, particularly if you have to pull the cable back for any reason.

            Or you could put a length of conduit through the hole. Standard plastic conduit is 20mm (so you'd need a larger hole) but very cheap. Make sure it slopes very slightly down to the outside. You can also use conduit accessories to make a neater job.

            Gaffa tape around the cable end helps when pulling, but I would always - where possible - go for pulling an unterminated cable, chopping a couple of inches off to get a clean end and then terminating it once it's through the hole.

            Without conduit, use a stiff wire, garden cane or similar to get through the holes and gaffa tape the first cable to the end (network cable should go through a short length of 20mm conduit just by pushing it). Get one cable through first, then gaffa the second on to the first, and the third on to the other two and pull them through. The trick is to avoid step-changes by careful use of the gaffa and by pulling all three cables together so they aren't rubbing against each other. You can adjust lengths once all three are through and you've separated them.

            Seal the hole as well as you can both inside and out. Expanding foam is good, and external rated cable shouldn't be damaged by it. Draughts will ruin your house insulation efficiency.

            If your house is timber frame, by drilling through the walls you will have punctured the vapour barrier. It's possible that over time this could allow moisture through the cavity to damage the internal leaf.

            Have fun :-)

            M.

            1. Anonymous Coward
              Happy

              Re: Hah! New keyboard required!

              It's brick cavity.

              My plan was to use rolled up overhead chart plastic as the feed conduit - should be able to roll it very small diameter, then let it unroll once inserted without losing any of the bore diameter. And I have silicone sealant at the ready along with a tonne of cable clips and cable ties.

            2. A.P. Veening Silver badge

              Re: Hah! New keyboard required!

              but I would always - where possible - go for pulling an unterminated cable, chopping a couple of inches off to get a clean end and then terminating it once it's through the hole.

              AMEN! to that, it isn't that hard to put RJ45 plugs on a cable, just make sure you follow standards in the sequence of the wires (and really push them as far as possible).

              1. Joe W Silver badge

                Re: Hah! New keyboard required!

                At least that stuff is soft (if it is what is called ytong here, a quite porous concrete based building block). Redoing a house atm, my Boschhammer is my new best toy. (yeah, the certified electrician will connect the stuff, the wife insisted). Network[*] is my job though, and I need to chisel out a channel (well, two, two storeys each plus a bit, plus three floors / ceilings) through some cs bricks for that. This is not much fun after the first few feet...

                [*] if the last months have taught me one thing, it is that wifi absolutely sucks, as the neighbours are too stupid to be allowed to configure it. Everybody cranks up the power to maximum, and in a densely populated neghbourhood this is madness, like a whole kindergarten shouting at each other in a gym. So none of us has great (or even "good", nor "reliable") wifi.

                I'll put two network sockets into each bedroom. Some rooms will have a third (not yet connected) cable, some have a reasonably straight path to the basement so I can add some wiring later on.

                1. Martin an gof Silver badge

                  Re: Hah! New keyboard required!

                  At least that stuff is soft

                  Our current house has an inner leaf of concrete blocks similar to these. It's reasonably dense but not very hard. I have some "nice" bits and my battery drill makes lovely holes through the plaster and block without any trouble and without needing to use the noisy hammer action - makes the holes more accurate too.

                  my Boschhammer is my new best toy.

                  Our last house - a 1920s-built ex-council house - was built with engineering brick.

                  My battery drill with the "nice" bits would make a hole big enough for a red Rawl plug, but only if you used the hammer setting and were willing to expend half an afternoon to drill the half a dozen holes needed for the shelf brackets (and boil the kettle part-way through while the battery was recharging).

                  My mains drill (with the same bits) was only fractionally faster but a lot noisier.

                  My cheap SDS from B&Q would do the job but the hole would be so out-of-true that you needed bookends to stop the books falling off. (and the drill was heavy)

                  As a Part-P registered electrician (back then, not now) I invested in a nice 110V Makita SDS. Like a hot knife through butter, as the saying goes. Not much heavier than the mains drill, it is now 18 years old and still going strong - I used it to core out 125mm holes for 110mm soil pipe just a couple of months ago.

                  The other problem in that house was that the "plaster" and the mortar between the bricks was a cement and coal dust mix, so every single hole had to be carefully dust-collected or you'd get permanent stains on your carpet.

                  For channelling, there are specially-shaped bits for an SDS drill, which work well if the wall isn't too hard, but don't last forever (I broke a couple!). You can also get dedicated tools such as this model by Titan which isn't exactly a premium brand but would probably last long enough for your job :-)

                  As for WiFi, I know a bloke who lives in a high-density modern estate and was struggling with this when he first installed WiFi. We talked about how it worked and about the problems and solutions and his solution was simple - he went and had a cuppa with each of his neighbours in turn, and offered to "tune" their WiFi for them. Just by moving the channels used by half a dozen access points, everyone had faster, more stable connections, and he made a few new friends, which isn't a bad thing when you are rapidly approaching your eighties.

                  The problem with network sockets in bedrooms (and I have installed a pair to each bedroom in this house, though they aren't wired up yet) is that typical bedroom devices - phones, tablets and maybe a laptop - don't particularly want to be wired in. Also I'm a firm believer in at leat trying to persuade the children to put their phones out of reach in the bedroom, we don't allow TVs in bedrooms and the household computers live in the study most of the time.

                  M.

                  1. rototype

                    Re: Hah! New keyboard required!

                    I wouldn't knock the Titan branded stuff, I have a cheap (£50 when I bought it 18 months ago) 1500W SDS drill that's performed like an absolute trooper, recently drilling/chopping through 6" thick very solid concrete without an issue and is still as good as the day I bought it. In fact the only casualty of the entire concrete removal process was a 22mm (7/8") 450mm (18") long drill that sheared the SDS tang off.

                    OK, it's heavy but makes beautiful holes quickly and if necessary you can turn down the force for more delicte jobs.

                    1. hoola Silver badge

                      Re: Hah! New keyboard required!

                      A similar experience here. I was doing some proper building work and needed to break up some concrete and drill some holes.

                      I found a Hyundai break/drill thingy on Fleabay for about £50.

                      It will go through absolutely anything (with a lot of noise), as long as you don't need precision.

                      I tried precision once to put a thin phone wire through, ended up with a hole about 1" in diameter with a 3/8" SDS bit.

                      The only thing we did learn (and this is not the fault of the breaker), if you are using the chisel bits to break things up, always make sure you have spares so when it is jammed in the bricks or concrete, you can then get it out!

                2. Dafyd Colquhoun
                  Stop

                  Lucky you're not in Australia

                  Great to hear about all the people from the civilised world that are allowed to install their own network cabling. Over-regulated Australia decided that was far too dangerous, and so Registered Cablers are a thing. You can run the Cat5/Cat6 around the skirting all you like, but going into the wall or roof cavity is a big no-no without the right ticket.

                  Bit like DIY electrical. Queensland Government is looking to ban the sale of plugs, sockets, light switches etc in hardware stores. New Zealand Government publishes a guide on safe DIY electrical. Both places have the same wiring rules (AS/NZS 3000)!

              2. Anonymous Coward
                Anonymous Coward

                Re: Hah! New keyboard required!

                Actually, after Martin's comment above, I took another look at the possibilities and I'm picking up a 450mm x 8mm SDS bit from Screwfix tomorrow, and I'm just awaiting confirmation from an Ebay seller that the professional EZ ethernet terminators they supply can accommodate my cable width.

                Then it's out with the cable cutter and in with the much more amenable smaller hole size.

                1. Martin an gof Silver badge

                  Re: Hah! New keyboard required!

                  You'll have a couple of problems with that method, but not insurmountable.

                  First, 450mm is a long bit, especially when it's a relatively small diameter. It will wobble about like no-one's business, especially if you try to put too much pressure on it as you start the hole - I'd start without hammer for the first couple of mm - but it might be worth drilling a pilot, or a partial pilot, with a shorter bit first, at least for the first leaf you are drilling through.

                  Secondly you will probably "spall" on the outside. Not a lot you can do about this (try putting as little pressure on as possible as you reach the end of the hole) and some materials or finishes are better than others.

                  Lastly, you'll have fun pushing a stick through that lot in order to tape the network cable on the end - it will inevitably be not-quite-straight, and crossing the cavity it will be difficult to guide it into the second hole. You could think about using the SDS bit itself, but you will need to attach the wires to the "drilling" end in order to avoid the greasy "chuck" end, which firstly won't like being taped, and secondly will collect dust as it goes through the hole, possibly risking damage to the chuck next time you use it.

                  Best of luck!

                  M.

                  1. Anonymous Coward
                    Happy

                    Re: Hah! New keyboard required!

                    My biggest concern is that the pebbledash on the outer wall will fall off! Probably unfounded, but a concern, but a much lesser one with an 8mm bit.

                    I've weighed up the options of drilling outside-in (up a ladder with the long bit) or inside-out (seated comfortably to lean on the drill as necessary with less risk of falling off and ending up in A&E). I'll try the latter, I think.

                    One way or another, I'll get it through (and I mean the drill and the cable).

                    If nothing else, if the 8mm option becomes terminally problematic, at least the hole will serve as a pilot for the 16mm bit I already have.

                    1. Martin an gof Silver badge

                      Re: Hah! New keyboard required!

                      Our external is pebbledash too (or a close relative). Feel very carefully for the drill exiting the block and entering the 25mm or so of render which holds the pebbledash, and take it very slowly from there, but you will still end up with a bit of a spall rather than a neat 8mm hole.

                      Larger holes - where you have to use a core drill (I've done a lot of pipework recently) - are much easier in this regard because you can keep drilling until the pilot dril pokes through (use a "spotter" if necessary - small children with loud voices or PMR radios are ideal) and then finish the hole from outside in. It doesn't matter if the pilot spalls as you're cutting that bit out, and just doing the last 25mm or so from up a ladder isn't so bad.

                      It's also amazing what kind of damage can be covered up with a bit of the correct colour paint.

                      Note that Sky installers have long since given up on making neat holes, and use this sort of thing to cover up their mess - until the silicone they use to glue them on fails, anyway.

                      M.

                  2. Potty Professor
                    Boffin

                    Re: Hah! New keyboard required!

                    I had a BT lineman come to fit a new drop cable and through the wall. He drilled from the inside out with a hammer drill, and burst the outer surface of the brick facing off before running the cable through, so I couldn't repair the face if the brick. More recently, I have drilled several holes to install phone cables, TV aerials, outside weatherproof sockets, and a 15mm water pipe to an outside tap. In each case I have started the hole from the outside in, (after careful measurement) drilling through the mortar at the intersection of the horizontal and a vertical joint, before continuing through the internal cinder block and plaster face. In no case did I damage the plaster, there was no need to use hammer action as the mortar is considerably softer than the brick, and by reducing the speed and pressure as you get near the inside, it doesn't burst the plaster off but wears its way through.

          2. arachnoid2

            Shirley.........

            Small cable, bloody big butt plug on either end.

    2. Anonymous Coward
      Anonymous Coward

      Re: Hah! New keyboard required!

      Echoes of an old limerick...

      There once was a Bishop called Densill,

      Whose prick was as thin as a pencil,

      It went through an actress,

      Sheet, blanket and mattress,

      And shattered the bedroom utensil.

  16. rob@bofh.co.nz

    Hmmm never quite had any contact with any random number devices for beancounters, but have played with some curious devices that all had the prefix BIDxxxx

    And some of them get Stroppy when you move them and they are switched on…

    And then the people get stroppy with you for moving them whist switched on, and then you point out what flaming idiot wanted to put them in a Land Rover and test it on a off road exercise

    Ok, so I rolled the landrover… it was a a valid test!….. kinda…

    1. swm

      I have a friend who developed military hardware. They would take his electronics, shoot it full of holes or blow it up and then return it so he could see what went wrong.

    2. ricardian

      Terrifying when stuff you used to use is now in a museum! https://www.cryptomuseum.com/crypto/uk/bid610/index.htm.

      Then you realise that it was nearly 60 years ago!

    3. The Oncoming Scorn Silver badge
      Coat

      Rollover data?

  17. amanfromMars 1 Silver badge

    Grow a Pair and be Rewarded with Fortunes Reaping and Sowing Whirlwinds and Dervishes and Daemons

    Nothing ventured, nothing gained is who dares win wins territory if one knows what one is doing and where it will lead to with nary a care about the scary worries of lairy others. 'Tis pretty much exactly the same in Pioneering IT circles too.

    Has nobody ever told you that to throw caution to the winds is very liberating and unbelievably rewarding whenever done extra especially well ‽ .

    1. Anonymous Coward
      Anonymous Coward

      Re: Grow a Pair and be Rewarded with title too long...

      I work in an entity where one team are focussed on repetitive production with manglers keen on process control and hierarchy, and one team are focussed on fast-moving reactive work with tight deadlines, in a collegiate structure, with managers encouraging innovative solutions and for moving fast, breaking things, and asking forgiveness after if needed. Guess which team I work in? Yeah, the one that delivers and develops, and yeah, I thank my lucky stars for my boss and my team, cos I'd have left the other team long ago because I would have been stifled!

  18. Nunyabiznes

    This week

    I got to watch another department explain that the test environment they had wasn't supposed to have live data on it. The department putting said data onto the system had been told repeatedly not to use it for production because it was a test system - please just test it so we know it is fit for purpose, no live data please.

    Of course it now has 3+TB (!) of prod data on it and when it was switched off panic ensued. Upshot was that the test environment is now production - even though it has some issues that caused the decision to scrap it in the first place.

    I would be more outraged, but the testing department did the same thing to us more than once in the past, so karma.

    1. Fred Daggy Silver badge
      Facepalm

      Re: This week

      Do remember that test servers get shutdown overnight and all weekends, also Xmas and Easter holidays. Because, cost savings, remember.

      Devs in another timezone? no problem, just schedule to meet their needs. Done early enough it gets the message through, mostly.

      1. Doctor Syntax Silver badge

        Re: This week

        Don't forget to test the procedure for replacing all the user IDs.

      2. Caver_Dave Silver badge
        Facepalm

        Re: This week

        Ottawa team employ a Mumbai team to help with some work that has to be run on the Ottawa servers. Said servers are disconnected from the net for backup each night!

  19. Anonymous Coward
    Anonymous Coward

    Ah yes...

    Pour encourager les autres

  20. earl grey
    Pint

    the only PROD you ever need

    is something with a fresh battery to use on the two-legged beef jerky who brought the problem to you.

  21. Andy 68

    IT'S NOT PRODUCTION

    "IT'S NOT PRODUCTION!" he snaps back with some urgency

    This... so much.

    From a frazzled Tech Support

  22. blah@blag.com

    Proper Laugh Out Loud

    "with the instruction that if the Bible doesn't work he's to hit himself with the mallet"

  23. J27

    Oh man, back when I worked in IT if someone tried this the SLA for non-production systems was 1 business week. So anything I didn't feel like doing or was created by a hard to deal with user would just get left to that last day.

    So I wouldn't have even contacted the beancounter until nearly a week after the request came in. And good luck getting anything escalated because the help desk was outsourced to another company who didn't pay their staff enough to care. In fact, the help desk was so bad that 95% of the tasks came in via a web application that was incapable of bothering me at all!

    Did I mention this was a cable company?

    1. Joe W Silver badge

      sounds like O2

      no, wait, you cannot submit stuff through the web to them. They don't even have a customer-facing email address (in my part of the world).

  24. Fullstop

    Hmmmm

    I guess it's time for the bean counters vacation. I heard he is flying TWA...tragic workplace accident.

  25. Brad16800
    Mushroom

    Test server work you just push it to a production server right?

    simples :)

  26. Dramoth
    Mushroom

    Moving dev into production without testing it

    That would be like moving the entire development environment into production without testing it... or backing up production. And then only doing a cursory test on the production server.

    And not informing the rest of the dev team you're doing this... on a Wednesday afternoon after everyone else has gone home.

    And not being around the next morning when your 80 billion dollar US partners UK offices call up asking why the production server is broken.

    What's even more amazing is that the culprit didn't even get the sack... just got a stern talking to from the MD of the company and sent into the devs office with instructions to fix it all ASAP!

  27. Medixstiff

    It's even better when your Devs hard code path's, credentials and give themselves Admin rights for the test environment then don't go through their code and come back with "It works in Dev" when they roll it out to Prod and the inevitable starts happening.

    Too bad none of the Dev teams I've worked with got bollocked by Manglement when it all went Total Inability To Support Usual Production.

    No the Ops team would get that privilege and the Devs never apologized afterwards of course.

  28. John70

    At least it not under someone's desk.

  29. Jesse 2

    Damn, I feel old....

    Simon,

    I just realized I've been reading to column since 1998. I love the way things have evolved over the last TWO DECADES. I hope you never stop writing the series, as I look forward to it, and also I thank it for bringing me to The Register and Situation in general.

    Don't you think it's time for a new PFY? He's not so young anymore......

    Cheers!

  30. arachnoid2
    Facepalm

    He's not so young anymore.....

    Hey, we'll not have any of that ageism nevou jiggery pokery feeding into the workplace...........

  31. staringatclouds

    Reminds me of the time we upgraded our SPARC box with an extra processor

    Then after we'd forgotten all about it the auditors turn up and we had a conversation along the lines of

    A: "SPARC Processor, part number blablabla, could I see it please ?"

    Me: "You do know that's a PROCESSOR not an actual computer don't you"

    A: "Yes, it says so on the ticket here SPARC Processor, can I see it?"

    Me: "You see that cream coloured biscuit tin shaped box sitting on top of the other cream coloured biscuit tin shaped boxes over there?"

    A: "Yes"

    Me: "That's the companies main financial database server, it's in that, I'd have to turn it off & disassemble it a bit, do you still want to see it?"

    A: "Yes please"

    Me: "OK I'll have to send out an email to tell the people in the UK, US, Australia & Switzerland that we're going to shut down the finance server for a hardware audit, it may take them a while to get back to me, would you like a cup of tea while we wait?"

    A: "... erm .... can I make a phone call?"

    I know we really needed a redundant system for such a vital function, but no bugger would pay for it, this was back in Windows 95 days, possibly early NT days, a while ago anyway

    1. Anonymous Coward
      Anonymous Coward

      I once had a ancient CD writer just when things came on the markey, I think it was 0.1 x speed and used special caddies to hold the CDs. It cost more than some arbritrary threshold so was a asset to be treasured and looked after for all time (or at least 10 years). We had to let the auditors know where this was long after it was decommisioned and consigned to a dark dusty cupboard because throwing it away would have been too much paperwork.

  32. hoola Silver badge

    It happens, again and again.

    We see this on a regular basis when developers request a VM in Dev/Test then "forget" to tell anyone that they are now using it to provide the live service. If we are very lucky they might go back and request a new VM for Dev/Test that has the same name as the one they have started using for live. Then it becomes apparent before the inevitable disaster.

    What usually happens is everything pootles along nicely until we decide so do something with the virtualization stuff like turning off Dev/Test. Then all hell breaks loose because this critical system is down. We all look at each other and say that all the Live VMs are running, no issues. The panic then escalates until someone finally gives us the correct server name and bingo:

    It is in Dev/Test and has been turned off according to change blah blah and notification blah blah, both of which nobody read.

    Much chuntering and meetings then ensue as blame is suitably packaged up and apportioned to everyone except the idiots who made a test VM live.

    Rinse and repeat about once every 6 months.

  33. perlcat

    Speaking of putting dev stuff where it doesn't belong

    I had a gig where I was setting up a system for a major financial player. They had four large, critical production systems with "DEV" in their system names, because they kept putting production data in the test setups and never made plans to migrate onto actual prod. When I was younger, I might have been shocked, but now, I just laugh, and wait for things to fail. IAPTS.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like