back to article It's the day before the grand opening but we need a firmware update. It'll be fine

Before one can organise a piss-up in a brewery, one must first get the brewery started. Something a Register reader found difficult in today's Who, Me? Our tale takes us back to the '80s, the decade of fun. Our brave reader – let's call him "Gareth" – was heading up a team building a state-of-the-art Brewery Process Block. It …

Page:

  1. WanderingHaggis
    Stop

    Friday rule

    Touch nothing unless you have no alternative.

    1. spireite Bronze badge
      Coat

      Re: Friday rule

      I've generally found that 'touch nothing no matter what' is a better mandate, unless in he club at 2am....

    2. Korev Silver badge
      Coat

      Re: Friday rule

      He hops they won't do it again...

      1. John Brown (no body) Silver badge

        Re: Friday rule

        It sounds like they barley scrapped thought! It could have all gone fizz bang, but he kept a cool head.

      2. ShadowSystems
        Pint

        Re: Friday rule

        Oh man that pun was bad. The yeast you could do is buy us all a pint to make up for it. =-)p

        *Hands out tankards to help swallow this tripe*

        1. Korev Silver badge
          Coat

          Re: Friday rule

          Oh man that pun was bad.

          Sorry, was I being a pain in the glass?

          1. jake Silver badge

            Re: Friday rule

            Of course! You made a right mash of it. Wort did you expect to happen?

      3. Anonymous Coward
        Anonymous Coward

        Re: Friday rule

        Water you think will happen if you do an update before a big presentation?

        (someone already got barley, hops, and yeast, I figured I'd complete the set).

        1. David 132 Silver badge

          Re: Friday rule

          Yes, there's a maltitude of puns here. As the number grows ever-lager, I am starting to get bitter.

          1. Will Godfrey Silver badge
            Happy

            Re: Friday rule

            I'd lager not take a chance on that either.

    3. chivo243 Silver badge
      Pint

      Re: Friday rule

      There's an unwritten rule in the IT world about Friday afternoon deployments?? Unwritten my ass! I wrote that in many memos!! It only took one overnighter on a Friday to figure that one out, no support after 5pm or on weekends! So happy to be out of full time IT now!

      Beer icon has probably been used to death, but what the hell!

      1. Bob the Skutter

        Re: Friday rule

        Depends on the system as to what time do IT Deployments

        Our IT department have done deployments at 2am on Sunday mornings. With full regression testing during the day to allow for rollback in case things go wrong so that system is working on Monday.

        When you have automated order and billing systems have to do when quietest. As have to put hold on through put

        1. Anonymous Coward
          Anonymous Coward

          Re: Friday rule

          I agree but 9.999999/10 don’t touch anything on the last day before a weekend/holiday.

          My last WAN migration for a government department was planned for a Friday after close of business. 3 months before the change I was told I could have the whole WAN off line from Friday 6pm through Monday 7am.

          The week before the change I was told I couldn’t start before 8pm on Friday and multiple sites needed the WAN from Saturday 7am as loads of staff across multiple sites where doing overtime.

          Due to good planning we where done all 30 odd WAN routers by midnight including 3 dc’s, everything tested and no dramas in the morning. Yes the new WAN was pre staged, connected & live, requiring cutting routes over with the benefit of a not too difficult rollback if needed.

          Of course I was logged in at 6 to double check and no reported issues at all during the day.

          They never normally go that smoothly, I got the sense that people where wondering what the fuss I was making about having a weekend of downtime was about.

          A classic example of perceived crying wolf all because of meticulous planning, huge amount of pre preparation & faultless execution resulting in a positive outcome.

          1. Spanners Silver badge
            Flame

            Re: Friday rule

            A classic example of perceived crying wolf all because of meticulous planning, huge amount of pre preparation & faultless execution resulting in a positive outcome.

            Pretty much like the "millennium bug " then! I still come across the profoundly uninformed/stupid voicing their opinion that it was all a fuss over nothing.

            If we are allowed to prepare we get "well that was a waste of (over)time then. If we are not, we get blamed. C'est la vie!*

            *If said with a Scottish accent, that phrase is pronounced "' 's a lavvy" which to those from the remote south means "it is a toilet". Both the French and us are referring to the same life!

            1. jake Silver badge

              Re: Friday rule

              In the years leading up to 2000, I got paid an awful lot of money re-certifying stuff that had been certified to be Y2K compliant some 10-20 years earlier. Same for the embedded guys & gals. By the time 2000 came around, most of the hard work was close to a decade in the past ... the re-certification was pure management bullshit, so they could be seen as doing something ... anything! ... useful during the peak of the dot-bomb bubble.

              Look for similar bullshit/misdirection during the end of the first UNIX epoch in 2038 ... despite the fact that all of the important systems that would be affected either already have been, or can easily be modified, making to so-called"problem" non-existent.

            2. Terry 6 Silver badge

              Re: Friday rule

              I agree, and not being an engineer I wouldn't dare not. But there's a caveat. A lot of perfectly good, non-system critical, non-networked devices were "millennium-proofed" at significant cumulative cost. Or simply mothballed/dumped. There was no rational risk-assessment performed on these devices.

              PCs ( some quite elderly) used for word processing did not need any of that. What was the worst possible outcome? They'd fail to boot in January 2000. Improbable, but manageable. No different to when a cmos battery fails.

              And I did take one or two out of mothballing a few years later - they were useful for taking off-site for kids to use etc. Worked perfectly of course. (Though the CMOS battery failed, in at least one - might have been coincidence).

            3. Anonymous Coward
              Anonymous Coward

              Re: Friday rule

              I seem to remember some 'Y2K compliant' card scanners fell over a few days before Y2K because the merchant system added a couple of days to one of the fields to cover time to allow the transaction to clear but that field hadn't been tested properly. It had been tested for the Y2K rollover and would all work properly once Y2K had kicked in, but in the last few days of the century it either overflowed or was flagged as 100 years past

    4. anothercynic Silver badge

      Re: Friday rule

      Ditto just before Easter, or Christmas, or Hannukah, or Diwali, or Eid. Or, worse, just before a change freeze!

      Just... don't. It's not worth it!

    5. Anonymous Coward
      Anonymous Coward

      Re: Friday rule

      "Read-only Friday" is our rule.

    6. Bruce Ordway

      Re: Friday rule

      >>Touch nothing unless you have no alternative.

      And at one site I remember the presidents motto was... "doing nothing is never an option".

    7. martyn.hare

      Re: Friday rule

      That’s odd. I tend to do all my maintenance on Fridays so I have all weekend to abuse a certain Redmond ISV’s business critical priority breakfixes in the event of a big issue…

    8. MachDiamond Silver badge

      Re: Friday rule

      "Touch nothing unless you have no alternative."

      Or it's getting close to the holidays and you could use a weekend's worth of OT to pay for gifts. Provided, of course, you are eligible for overtime.

  2. Pascal Monett Silver badge

    They had it running

    Then they had to go for a firmware upgrade. With no apparent good reason.

    Here's an idea : you fully test that the system can do what you need it do the next day, then you make up your mind on the firmware upgrade.

    If the test had did badly, then yes, you upgrade, but if the test went well, you don't touch a damn thing until after the event.

    Oh well, experience is what you get when things break unexpectedly - especially when it's your fault.

    1. Richard 12 Silver badge

      Re: They had it running

      I recall discovering that a major event was about to happen just after I started updating the system.

      I discovered this when a security barrier and nice guard suddenly appeared, blocking my route back to the control room.

      Fortunately he let me enter, and I was able to complete the updates and bring the system online half an hour before the Spice Girls came on to announce they were re-forming (again).

      I don't know whether I could have saved the world from this fate by being a little slower or making a mistake, but sometimes I wonder...

  3. ColinPa Silver badge

    Windows upgrade in process

    I remember going to a presentation of a great new technology and just after the demo started Windows update kicked in. The download activity was enough to impact the network connection, and instead of a slick demo it was a stuttering display. The audience offered suggestions on how to kill the update, but the marketing person did not want to go off script.

    1. spireite Bronze badge

      Re: Windows upgrade in process

      The root cause for this is almost always one ting in my experience....

      The presentation is usually carriend out on a machine who's ONLY job is in a meeting room.

      A few years back, we would have a meeting every Thursday (you see where this is going) - the only meeting ever in that room @ 9am

      .....and at 9:05am, it would dutifully start applying software patches pushed out to it by whatever patcher we employed.

      It took a month before someone realised it was beneficial to move the meeting, but switch the kit on well in advance - or the day before to patch in advance

      1. A.P. Veening Silver badge

        Re: Windows upgrade in process

        It took a month before someone realised it was beneficial to move the meeting, but switch the kit on well in advance - or the day before to patch in advance

        That month sounds long, but patch Tuesday comes only once a month (usually).

        1. Potemkine! Silver badge

          Re: Windows upgrade in process

          There isn't Windows only to patch, but all the apps. Our patcher runs everyday, and generally pushes updates 2 or 3 times a week on some computers.

          1. Not Yb

            Re: Windows upgrade in process

            One of many reasons the "continuous update" model is annoying. So many updates, so little time.

            Have a laptop that only gets turned on once or twice every year. Hours of required updates to get everything happy.

      2. Dinanziame Silver badge
        Unhappy

        Re: Windows upgrade in process

        I managed to have my laptop install a Windows update and reboot — by default to Linux — during the twenty minutes coffee break after I had made it ready for my presentation.

      3. Stuart Castle Silver badge

        Re: Windows upgrade in process

        We generally have a good record on updates. That's probably something to do with the fact we use system center to restrict them to running between 5pm and 9am, and have it set to wake up any sleeping machines during the night to check for updates, as well as power up all machines at 8am.

        We also have an "at risk" period from 7am to 9am on Tuesday, which we use to make changes to production servers (after testing, of course).

        The system is far from perfect, and we do get caught out, but we rarely have a situation where someone is giving an important presentation and the machine restarts to install updates.

    2. Korev Silver badge
      FAIL

      Re: Windows upgrade in process

      Back in the Windows XP days, my company had a standard image with a tool that would checksum c:\program[sic] files, c:\windows etc to make sure that all patches had been applied and nothing had been changed.

      This wasn't a bad idea in principle, but it was configured to run every four hours and in the days of single core Pentium 4s and hard discs this meant your laptop turned into a paperweight. This actually killed a number of high-profile presentations and generally made it hard to use your computer

      1. keithpeter Silver badge
        Windows

        Re: Windows upgrade in process

        In the education industry, the interaction between options baked into standard operating system images, users with limited rights unable to change option settings, and power management hints to external monitors (e.g. screen projectors) causes a never ending stream of multilayered fun.

        Add in OS updates and separate application auto-updates for bonus laughs.

    3. Rich 11 Silver badge
      Alien

      Re: Windows upgrade in process

      The most embarrassing example of this I've seen in my life was when the auto-updates began an hour or so after I'd set up the sentry guns in the access tunnels. Thirty-nine light-year lag. Sheesh.

    4. Anonymous Coward
      Anonymous Coward

      Re: Windows upgrade in process

      "a great new technology and just after the demo started Windows update kicked in"

      A great new technology and Windows are mutually exclusive.

    5. Arthur the cat Silver badge
      Unhappy

      Re: Windows upgrade in process

      just after the demo started Windows update kicked in

      Wasn't there a case a couple of years back when Windows Update kicked in in the middle of open heart surgery?

      1. First Light Silver badge

        Re: Windows upgrade in process

        Because of an antivirus scan.

        https://www.techdirt.com/articles/20160506/07161334360/heart-surgery-stalled-five-minutes-thanks-to-errant-anti-virus-scan.shtml

        1. Arthur the cat Silver badge

          Re: Windows upgrade in process

          Thanks.

          Marginally better, though god knows what would have happened if it had started getting false positives.

      2. R Soul

        Re: Windows upgrade in process

        Did Clippy pop up to say "you seem to be doing a mitral valve replacement. need any help with that?"

      3. imanidiot Silver badge

        Re: Windows upgrade in process

        Surprised a computer inside a surgery room would even have an internet connection. I know atleast one hospital in NL has the computers running the surgery equipment and such all airgapped with only a single network link to a single computer outside the OR. Only way to get data onto them is through that computer outside the OR (Which is ofcourse loaded to the gills with software to prevent any nasties getting in and with a very strict firewall to the cables going to the OR equipment. Also decreases the risk of contamination as data carriers don't need to be brought into the clean environment.

      4. TheRealRoland
        Coat

        Re: Windows upgrade in process

        Now that would get someone's heart rate up...

    6. LDS Silver badge
      Facepalm

      "just after the demo started Windows update kicked in"

      Ah, the lazy sysadmin that refuses to understand group policies and WSUS groups...

      1. Anonymous Coward
        Anonymous Coward

        Re: "just after the demo started Windows update kicked in"

        Fairly sure said lazy admin works for Microsoft demo team these days.

    7. Anonymous Coward
      Anonymous Coward

      Re: Windows upgrade in process

      A few years ago a (semi?)pro basketball team had to forfeit a game due to windows update. IIRC, the laptop that ran the arena scoreboard had failed just before game time. They actually had a spare laptop on hand to run the scoreboard, but when they plugged it in, it insisted on performing a windows update.

      Unfortunately, the backup was fairly far out of date with updates, and they ended up taking a very long time to complete.

      Granted, modern scoreboards have a lot of fluff for entertainment value, but they also provide the basic scorekeeping and timekeeping functions that are mandated by the league. Per league rules, the home team was responsible for providing the scoreboard hardware, and if the hardware failed, they could only hold the start time for a limited time (40 minutes, I think) before the result was a forfeit.

      1. jake Silver badge

        Re: Windows upgrade in process

        After reading this post (and the rest in this thread), again I have to ask how many billion man-hours has Windows cost the Corporate world over the last 20 years? How many perfectly good dollars has that translated to? And it's getting worse from what I can see ... so why on Earth do the Corporate Lawyers allow the silly clusterfuck into the building at all?

        More to the point, how much is Windows contributing to the global recession?

        Answers on a postcard, please ...

      2. Stork Silver badge

        Re: Windows upgrade in process

        I remember reading that story. As I remember it, it was even more bitter for the home team as they had given the opponents a bit of time as they were arriving delayed.

    8. Doctor Syntax Silver badge

      Re: Windows upgrade in process

      I'd call that a realistic demonstration.

    9. Antron Argaiv Silver badge

      Re: Windows upgrade in process

      Second only to Adobe Upgrades. Enabled by default and feel the need to pop up a bloody huge red box covering the screen. Designed to occur whenever a presentation is being made.

      Which is why I refuse to use Acrobat Reader. First thing I removed on my corporate-issued laptop.

  4. vogon00

    why is it that people forget flow control?

    Every component of interlinked systems that I have every worked on (Including the human-driven ones!) always involves flow control somewhere. In-band or OOB, it's always present.

    IMHO, flow control make thing work smoothly, and 'bottleneck' restrictions are easier to identify, as long as you bother to have flow on/off assertions visible somewhere. You can then tweak the setup as required to improve things.

    Also, if one is dealing with something synchronous you need to 'Think Synch'...

    1. John Brown (no body) Silver badge

      Re: why is it that people forget flow control?

      To be fair, they probably did have flow control and monitoring of it. It was, after all, a brewery.

      And as head brewer, Maud Dibb said, The Brew Must Flow!

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022