back to article Microsoft patches the patch that broke VPNs, Hyper-V, and left servers in boot loops

Microsoft has patched the patch that broke chunks of Windows and emitted fixes for a Patch Tuesday cock-up that left servers rebooting and VPNs disconnected. There was a time when out-of-band updates from Microsoft were considered a rarity. Not so much these days. On the receiving end of the company's attention were Windows …

  1. ThatOne Silver badge

    Sounds familiar

    "The goal is not to provide the best service, but to provide the only service"

    Reacher Gilt, CEO The Clacks

    1. b0llchit Silver badge
      Devil

      Re: Sounds familiar

      That is covered and finalized with the "extinguish" part from embrace, extend, extinguish.

  2. Clausewitz 4.0
    Devil

    Use Linux

    Use Linux.

    If there is a real need to use windows, install it in a virtual machine inside Linux.

    Remember to take a snapshot before any windows update.

    Borked update? Just restore the snapshot

    1. djvrs

      Re: Use Linux

      I can understand your point about restoring from a snapshot, but easier said than done when you're 12-24hrs in before the issue is realised on a Domain Controller and things have changed moved along, and restoring to a previous time will cause more issues than just uninstalling the patches.....

      1. Anonymous Coward
        Anonymous Coward

        Yeah, DONT try a DC snapshot restore

        Regardless of the backend, M$ is crystal clear that restoring from a snapshot is a great way to hose your active directory. The AD synchronization code is was built by the clinically insane, and includes what is essentially a one way incrementing counter/timestamp. If you have the mandatory secondary DC, and the two have synced since the snapshot, when you restore the DC it will start on the old values and the sync code breaks.

        Congratulations! You have won Active Directory database corruption! Please proceed to obscure technet articles to discover the joy of trying to clean it up line by line at the command line and the joy of typing GUID's in manually. Have a nice day! Or just tie a noose for yourself out of the nearest cat 5 cable.

        I'm close to replacing our DC's with a SAMBA server just so that I can have a safe and sane restore capability that isn't dependent on code that was deranged 30 years ago and hasn't really changes since.

        1. Anonymous Coward
          Anonymous Coward

          Re: Yeah, DONT try a DC snapshot restore

          Since server 2012 you can restore from a snapshot as long as you are running an appropriate hypervisor that supports the msDS-GenerationID attribute, which is pretty much any recent version of any of the mainstream hypervisors.

          https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-ada2/11b969e5-180c-4c16-89b4-2a2bf3317660

          1. david 12 Silver badge

            Re: Yeah, DONT try a DC snapshot restore

            But when you do that, what happens to the Workstation Passwords? Last I looked ---

            MS domain computers are validated by a (binary) password, (the process is exactly the same as user login), which is replaced/updated every month (or at a configured interval). Once the client password is updated, it doesn't match your snapshot from last month -- which might have been 2 minutes ago. Not a disaster, but all the client machines have to be re-joined to the domain.

            1. martyn.hare
              Happy

              Get yer burflags here!

              You do have at least TWO domain controllers, right? Set burflags appropriately and roll back all but the one you want to be authoritative! Problem sorted.

              Or you could consider migrating to a less “agile” platform like RHEL, Debian or Ubuntu where stuff generally doesn’t randomly break when receiving patches.

            2. ScottK

              Re: Yeah, DONT try a DC snapshot restore

              "But when you do that, what happens to the Workstation Passwords?"

              Nothing. Everything keeps on working as normal. If the generation ID is incorrect as the result of a snapshot restore, it will just do a non-authoritative restore from another DC and keep on working.

              If you do something silly like only have one DC, or snapshot restore all your DCs at the same time you can probably get yourself into trouble, but otherwise nothing to see here.

              More details here if you are interested:

              https://blogs.msmvps.com/acefekay/tag/msds-generationid-attribute/

            3. Anonymous Coward
              Anonymous Coward

              Re: Yeah, DONT try a DC snapshot restore

              You don't have to rejoin to the domain if you manage to break the trust relationship. Just logon with a local admin account and use Powershell or NETDOM to reset the password (or run the Powershell/Netdom command using your remote management tool):

              Reset-ComputerMachinePassword -Server DomainController -Credential DomainAdmin

              Netdom resetpwd /Server:DomainController /UserD:DomainAdmin /PasswordD:Password

  3. DoctorPaul

    Quite a blast radius!

    I installed KB5009719 (.NET security rollup) on my Win7 box last week and it broke the AVIdemux video software somehow.

    Upgrade from v2.7 to v2.8 didn't help, but rolling back the update fixed things.

    If only I could get Media Companion to run via Wine (developer says it can't be done) then I could jump ship to Mint full time and be done with this. Oh, and on another box where I'm having to do a reinstall, the copy of Office 2010 that I paid for refuses to activate - WTF? It may be out of support, but I bloody paid for it and I sure as hell won't be taking out a 365 (ymmv) subscription as an alternative.

    1. Clausewitz 4.0
      Devil

      Re: Quite a blast radius!

      You can try to pay Wine developers to fix the Media Companion. I am sure some will happily jump aboard.

      For office, can libreoffice do the trick?

      1. DoctorPaul

        Re: Quite a blast radius!

        Will be looking at LibreOffice "real soon now".

        I finally retired last year, so no longer need to exchange docs / spreadsheets with corporate clients, which is why I had to stick to Office.

        1. mikus

          Re: Quite a blast radius!

          I've used only linux for almost 20 years now, and in IT consulting have to exchange files regularly with clients that of course only use MS Office. OpenOffice worked well enough mostly, and now with current LIbreOffice have almost no issues with import/export of native docx/xlsx files. Of course MS Office supports native odt and ods files for Word/Excel native import, which works well too, and so does even 365 online word/excel.

          Give it a try, I think you'll be pleasantly surprised these days. The great equalizer is export to pdf if nothing else, which I tend to do for formal docs anyways.

    2. fidodogbreath

      Re: Quite a blast radius!

      365 (ymmv)

      I see what you did there...

    3. Ramis101

      Re: Quite a blast radius!

      Yep,It also breaks ZoneAlarm when running on 2k8r2. Great Joy!

    4. X5-332960073452
      Meh

      Re: Quite a blast radius!

      If you have applied service pack 2 to Office 2010, you can't activate using the applications.

      You need to enter the serial number in the application, then from an elevated (admin) command prompt

      CD "C:\Program Files (x86)\Microsoft Office\Office14"

      cscript ospp.vbs /act

  4. This post has been deleted by its author

    1. fidodogbreath

      Re: Weekend Ruined

      How will Microsoft compensate me for the lost hours last weekend undoing their c0ckup

      The same way they always do, of course.

  5. Anonymous Coward
    Anonymous Coward

    Here's your updates for 2012, 2012 R2 & Win 8.1

    Got to catalogue shopping though..

    https://support.microsoft.com/en-gb/topic/kb5010797-out-of-band-update-for-windows-server-2012-january-17-2022-1f14f497-8404-404d-8d78-0c962c9e486d

    https://support.microsoft.com/en-gb/topic/kb5010794-out-of-band-update-for-windows-8-1-and-windows-server-2012-r2-january-17-2022-a92500fb-f227-400e-b70e-f7dd50386fd3

  6. Pascal Monett Silver badge

    Seems like it is time to train a new generation

    About fifteen years ago, Borkzilla was famous for teaching us to wait for v1.1 - it looks like today's new generation of admins need to update their training courses.

    NEVER install a Borkzilla patch the day it comes out. Wait a few days. Find out what the feedback is.

    It's incredible that people just blindly go and update their business-critical software when Borkzilla has publicly recognized that it has no more Q&A department.

    HELLO ?! YOU ARE THE TESTERS.

    If you want to risk your network, go ahead, but don't come back griping about how the latest "patch" borked your network.

    Live and learn.

    1. MrReynolds2U

      Re: Seems like it is time to train a new generation

      Lovely sentiment. I used to wait too but unfortunately Win10 loves to just install updates and restart on it's own. Even where I've taken measures to prevent it, I'll come in and find my office PC has restarted on a whim.

      So for your average punter, they are doomed to suffer this fate until the year of the Linux desktop (are we still expecting it to happen?)

      1. herman

        Re: Seems like it is time to train a new generation

        You missed the year of the Linux desktop. It was somewhere around 2010

    2. DCdave
      Flame

      Re: Seems like it is time to train a new generation

      This one took a little too long though. Patches issued on Tuesday, tested, no problems seen in our environment, updates pushed to next test machines over the weekend. Discover on Monday there's an out-of-band patch that compromises your testing strategy, even if no adverse affects were actually seen.

  7. AndrueC Silver badge

    My first thought on reading the notification was not 'Will it fix it?' It was 'I wonder what else they broke this time around?'

    I'm beginning to think there was too much Sherry sloshing around MS over Christmas. The most recent VS 2022 Preview broke several things as well.

  8. ShadowSystems

    An ElReg prediction came true...

    My son is a teacher at an elementary/middle (4th~6th grades) school & has been venting his frustrations about just how bad Win10 has been.

    "Win 10 ate my homework" is no longer a potential excuse, it's a *legit* reason for students not to have an assignment, teachers to be unable to grade/issue assignments, or even to create lesson plans in the first place.

    His school has an IT department, but it's one guy for the physical network & one for the software. Keeping all 1,000+ machines updated means that *one* person is being driven to either an early retirement or possibly an early grave. Having all of those freshly updated machines simultaneously fall over for the exact same supposed MS "fix" means that poor sob is probably crying into his beer at the mere thought of having to re-patch all those just-patched machines.

    I felt vindicated on one hand, sad as hell on the other, when the boy said "Now I understand why you're still using Win7, Dad. This ((Win10)) shit sucks balls."

    =-(

    1. Anonymous Coward
      Anonymous Coward

      Re: An ElReg prediction came true...

      'but it's one guy for the physical network & one for the software'

      Sounds like overspend for a school, I'm sure it will be fixed.

      1. John_3_16
        Facepalm

        Re: An ElReg prediction came true...

        Just in time for the next update that destroys more than it fixes. Job security for some; insanity for the rest. Always waiting 7-10 days for my Home Premium Win7 updates. Hoping borks are noticed & updates updated before I download & install.

        Yes. Security be damned until then. My own security is great. No negative results from waiting for last 15 years. And no change in my zero trust of M$ in those years either. Will use Win7 until I absolutely cannot. Then will convert 100% to Linux/Ubuntu. Already running & using as my backup for backups to M$...

    2. Is it turned on?

      Re: An ElReg prediction came true...

      Sounds like the school is doing it the hard way. Before COVID we had 400+ school owned and managed Microshaft machines, and run a WSUS server to control the updates. No problems.

      Since COVID we've gone all BYOD, and blocked the BYOD's from doing updates via our WiFi to preserve bandwidth on the internet line. To be honest in 18 months I haven't seen any Update issues on those devices, the biggest problem is the kids installing iffy VPN solutions to get around daddy's blocks on certain sites, and the pc not reconnecting to the school network on Monday morning. - The windows 10 Network reset button is your friend here :-)

  9. Anonymous Coward
    Anonymous Coward

    Server 2019

    The OOB update for server 2019, which seems to have been released a bit later than the others is kb5010791

    https://support.microsoft.com/en-us/topic/january-18-2022-kb5010791-os-build-17763-2458-out-of-band-43697313-d8e0-4918-b6df-7f64d4d9a8cd

  10. Brad16800

    Glad I wait a week before doing updates. I know the security updates are important but i'd rather let someone else find the issues first.

    We have Hyper-V so this would have been a little bit annoying.

  11. ecofeco Silver badge

    YO! We heard you like patches!

    So we patched a patched for your other patch!

    That's right! We pimped your patch!

  12. david 12 Silver badge

    Setting up a Windows VPN is irritating at the best of times. Doing so in the middle of a patch fail added another day to the process.

  13. Dinanziame Silver badge
    Happy

    I think the most embarrassing issue recently was Outlook breaking at the start of the year because 2201010001 could not be stored into a signed 32-bit integer...

    1. John_3_16
      Facepalm

      2201010001 could not be stored into a signed 32-bit integer...

      Guess everyone must bow to progress eventually. My Win7 is 32/64 bit with 64 bit in charge. Beats using a legal pad for all of that data storage. 1Tb SSD with 16 gigs of ram. Trying to outrun M$ OSes as hard as possible. Linux/Ubuntu are my future. NOT M$ 10 or 11... Too old & tired to deal with anything M$ calls "new & better".

  14. Anonymous South African Coward Bronze badge

    Consistency is key to cuss-tomer satisfaction...

    ...the company's approach to testing has thus far remained reassuringly consistent.

    I am thinking a bit ahead, and warned our TQM and Dev department - we may have to look at the feasibility of going 100% over to Linux for our products, as we can standardize on a specific Linux distro, and lock it down the way we want it...

    ...and only allow patches on a strictly controlled basis.

    This update issue will cause some major grief sooner or later...

    Heck, even NT4 with SP6 is looking better and better, we never had any issues with NT4SP6 back in the day... but that is our very, very very last resort.

  15. fredesmite2

    They don't test SHIT at microsoft

    User's deserve everything they paid for

    NOTHING

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like