back to article Admins report Hyper-V and domain controller issues after first Patch Tuesday of 2022

Microsoft's first Patch Tuesday of 2022 has, for some folk, broken Hyper-V and sent domain controllers into boot loops. A Register reader got in touch concerning KB5009624, which they said "breaks hypervisors running on WS2012R2." "I'm currently dealing with this right now and it's a hassle," our reader said. "After several …

  1. LDS Silver badge

    Did MS retired the updates?

    I can't see them in Windows Update now.

    1. ShadowSystems Silver badge
      Pint

      Re: Did MS retired the updates?

      No, they retired (*Cough*Fired*Cough*) their entire QA testing department to prevent any "features" from preventing the swift deployment of such awesomely produced code.

      *Hands you a pint*

      Drink up, we'll be shloshed before this is over...

      1. Anonymous Coward
        Anonymous Coward

        Re: Did MS retired the updates?

        "No, they retired (*Cough*Fired*Cough*) their entire QA testing department " HR later went to their office to tell them the bad news, however they couldn't be found. It later transpired, there was no QA or testing department, the employees were canned years ago to save costs.

  2. Potemkine! Silver badge

    Woohoo! I'm part of the winners

    Oh, bugger.

  3. keith_w
    Mushroom

    Whilst not condoning MS's issuance of patches that do these nasty things, I have to wonder why they were installed without extensive testing.

    Icon chosen because there's nothing like nuking your running system with untested patches.

    1. Mixedbag

      reality

      Very few have the time and resources to run a test environment that replicates everything in production.

      Many don't have any resources for testing unless it's code their company has created or commissioned.

      Sure, you don't deploy to every box at once and you start with the most trivial but those aren't going to be DC's and HyperV hosts so you'd not find the issue in a low pain fashion.

      We expect for the sums of money that we are paying to MS that they do a reasonable degree of testing.

      In this case, it's clear they didn't. The scenarios that seem to result in the patch's breaking fundamental windows components are not rare edge cases.

      1. Anonymous Coward
        Anonymous Coward

        Re: reality

        So even on a test environment the DC issue didn't fire at install time or on the initial re-boot, it apparently causes the machine to crash after startup and restart periodically.

        So you would need to let it run in your test lab long enough to crash and reboot a few times. Likely most organizations can't devote that deep a testing for every patch run, and it slipped by some as a result. The hyper V thing is more glaring, but showed up about when you would expect for tiered deployments.

        Storage backends and Hypervisor Hosts are always some of the last things we get to patch, bout because the impact of a bad patch is high, but also because of the need to down so much of the rest of the deployment to get to them.

        So the patches get applied like reverse growth rings, even in testing. If the bug only shows up on the equipment in those last, inner tiers, than we get what we saw with this months patches.

        Instead of pointing fingers at each other and our test environments, we need to start pointing finger back at M$. Yes they need to raise their QC game, but they also need to unbundle the individual fixes so we can roll back a single issue w/o having to remove every fix from that month.

        The attackers are already retroengineering the patch they issued, and can weaponize any of those exploits in a couple of days. Several of those were auto-exploits from the preview pane level nasty. That level of exploit should be released as a spot fix separate from the roll up to allow the most serious threats to be addressed even in the event of a problem with the monthly roll-up patch.

        1. Roland6 Silver badge

          Re: reality

          >"we need to start pointing finger back at M$. Yes they need to raise their QC game, but they also need to unbundle the individual fixes so we can roll back a single issue w/o having to remove every fix from that month.

          The attackers are already retroengineering the patch they issued, and can weaponize any of those exploits in a couple of days."

          This is a big problem that MS et al need to address. Events like this mean that more people will delay in implementing server updates, so we have both an increased window and population at risk of being victims of the retroengineering attackers. Unbundling to increase fix granularity will be helpful, as will up-todate security suites.

          1. J. Cook Silver badge
            Alien

            Re: reality

            Now, Now, let's not let common sense cloud our thinking! the real question we should be asking is "why haven't you moved all of your business data up to The Cloud(tm) and let MS worry about the infrastructure instead? /sarcasm

            There's a reason why we lag a month behind in patches, and this crap is one of them.

            1. J. Cook Silver badge

              Re: reality

              ... and I take a week off from the rock face, and we got hit by that bug.

              Pass me some brown sauce, this crow needs it.

    2. UCAP Silver badge
      Mushroom

      They are being tested - by anyone who installed the patches.

    3. Vince

      If only it was that simple - some of the issues aren't really noticeable until there is reasonable demand and load and it's pretty hard to simulate real world user demand in test labs and gets harder continually as everything expects to be online and really working all the time.

      1. Fred Daggy Bronze badge

        Except Microsoft themselves must have a pretty good setup, just to support their internal staff. Do they eat their own dogfood? Testing patches "at home" would at least give some soft of indication. Or do they run everything out of AWS?

    4. Roland6 Silver badge

      > I have to wonder why they were installed without extensive testing.

      MS want their customers off (pre Azure) on-premise Windows Server and on Azure subscription on-premise...

      1. Anonymous Coward
        Anonymous Coward

        fallout from old, bad decisions

        This is still the long shadow of the dissolution of the "Trustworthy Computing Initiative" inside Microsoft. While that program certainly needed further internal reforms, it gave a high level voice inside the company to address quality issues. That team was dissolved and spread around to other teams, with the (now proven flawed) notion that the team would be able to spread their skills and knowledge to all of the other teams.

        Instead they ended up as junior members of established groups with no formal or soft power to effect change within those new teams. Most were ignored and many moved on, but the impact goes on and on. Worse, even if the essential parts of that program were implemented today, it would take a decade with mop and bucket to catch up with the cleanup.

  4. chivo243 Silver badge
    Go

    Text message today

    My replacement thanked me for passing on my philosophy of waiting a week, checking El Reg and various other IT news sources for complaints about WU.

    1. gryphon

      Re: Text message today

      Really depends on your risk appetite since MS hotfixes are known to often have problems.

      But that sort of thing should be brought out during the change management process if it is anything more than a box ticking exercise.

      And presumably everybody has pre-release environments to test these things on? :-)

      1. Potemkine! Silver badge

        Re: Text message today

        And presumably everybody has pre-release environments to test these things on

        For servers yes, for hypervisors no. We aren't rich enough to buy hypervisors just to make tests once a month.

  5. Pascal Monett Silver badge

    "if it was aware of the problems"

    Not until enough beta testers customers had voiced their dissatisfaction.

  6. Richard Jones 1
    Facepalm

    At The Basic Level

    Today after the snafu, sorry update, Firefox was not playing well, but a second restart cleared my issue. Small beer for me and compared with bigger issues elsewhere it pales into the landscape.

    Do new problems count as a feature update?

  7. katrinab Silver badge
    Flame

    I got hit by the ReFS one on on Server 2022.

    I took one of the drives out and tried to mount it on a freshly updated Windows 10 desktop. It didn't work there either.

    (Fortunately) it is a backup copy, I still have the original, on a zfs file system.

    1. Roland6 Silver badge

      Cascade of problems

      P*ss because some client admin decided the Hyper-V host and associated VMs should get updates auto installed so calls started around 3pm about loss of service.

      Result on WS2012R2:

      1. Lost Hyper-V due to update - but simple to resolve.

      2. Restarted VMs were in process of installing updates...

      The Domain Controller boot loop was a bugger to resolve - getting the DC VM sufficiently stable so that KB5009624 could be removed.

      Seems Exchange 2013 (on WS2012R2) had difficulty digesting the updates, seeming stuck in a reboot loop for a few hours.

      Other WS2012 VM's seem happy...

  8. Howard Sway Silver badge
    Facepalm

    Windows Resilient File System (ReFS) volumes are left inaccessible after the update

    Well, it isn't very resilient then is it?

    Calling it that was just asking for trouble.

    1. Anonymous Coward
      Anonymous Coward

      ReFS

      While this started off in the right direction, it appears to be a dead duck. The biggest red flag was when they released the new format and the disk repair and recovery tools were missing.

      If the people who built a filesystem didn't support fixing/recovering it from the beginning, it will probably fail as a project (A) and (B) take your data with it. Not because the lack of developer supported tools will make the project fail on their own, but because the "field of dreams" model doesn't work for low level disk formats, and that clearly, the team already failed at basic use case/requirements analysis if they put the horse before the cart and released the file system without repair/resize/snapshot/convert back parts already sorted out.

  9. moctad-tlhd

    Looks like the same is happening for Windows Server 2012 Hyper-V only with KB 5009586.

    https://support.microsoft.com/en-gb/topic/january-11-2022-kb5009586-monthly-rollup-9541f57c-89b0-48d6-ade2-31609678ce9b

    Uninstall update with the below.

    wusa /uninstall /kb:5009586

  10. Anonymous South African Coward Silver badge
    Thumb Down

    And this morning I've sent out an email detailing this incident, and impressed the need that we really look at switching away from Windows as this will be a risk we cannot ignore.

    Imagine your production (critical) going poof after applying a not-so-kosher update... Been there, done that, don't want any repeats kthanxbai.

    Or worse still ReFS showing raw, and somebody thinking a quickie format will resolve things... (one poster above commented that there isn't tools or recovery software available for ReFS, so what's the use of it then?)

    Even more so if it is a remote server in a remote location, and somebody need to be sent out to do the gefingerpoken at the server in order to fix things... most of us most certainly do not need extra fun and games of this sort.

  11. DoctorPaul

    Goes all the way to Win 7!

    Still running Win 7 here (plus Mint of course) and getting ESU updates (we all know how don't we?).

    Security rollup for .NET arrived this week, seemed to install OK. Then I tried to load a movie for editing in AVIdemux 2.7 - file opened, indexing took place, then instead of displaying the first frame the application just crashed.

    Tried a clean install of v2.8 and the problem persisted. Rolled back the .NET update and everything is fine - $deity knows what M$ did there.

    1. J. Cook Silver badge
      Trollface

      Re: Goes all the way to Win 7!

      What, .NET updates break software? INCONCEIVABLE! .NET is PERFECT!

      .. sorry, I can't continue, I'm laughing too hard.

      Especially when installing a .NET patch breaks little things like, say, Exchange, and the only fix for it is to install the latest version of .NET so that you can install the latest CU, and then PRAY that it all works.

      :wanders off grumbling::

  12. TempestQuad

    KB5009595

    KB5009595 seems to break Hyper-V 2012 as well.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022