Patch blues-day: Microsoft yanks code after some PCs are rendered super secure (and unbootable) following update A bunch of PCs running the wares of Sophos or Avast have been freezing or failing to start following the installation of patches emitted by Microsoft on 9 April. The afflicted are those running Windows 8.1, 7, Server 2008 R2 and Server 2012. Avast for Business and Cloudcare have been hit by the problem, as have PCs running …
Honestly? That's pretty much spot on these days for corporate PCs...
1. Patch Tuesday happens, making a bunch of new updates available on the MS Website.
2. Our WSUS server syncs with the new list of updates sometime early Wednesday Morning (4am ish)
3. Client PCs sync with WSUS sometime on Wednesday, and tell it what they need from the new list of updates. WSUS then goes off and actually downloads all the needed updates in the background over the course of the day from the MS Website onto one of its local disk drives.
4. Client PCs sync to WSUS again every ~6 hours. At some point whenever WSUS has their requested updates available, the clients will start to download them. Group Policy determines whether the local user of the machine is actually permitted to install them or not (which kicks in at midday - so realistically this will be 12pm Thursday or Friday depending on the number of updates needing downloaded), otherwise they wait for them to be manually installed by IT staff during an agreed downtime window.
If at any time there's any argy-bargy noticed on the interwebs relating to botched updates, those updates get declined on WSUS before Thursday midday; so they never actually hit the client PCs.
It's literally a built-in time delayed "stop those updates!" killswitch.
This week we actually got told about the AV Crashing Issue on Thursday at ~11am, and I had a Decline in place on WSUS by 11:15. And a lucky escape it was too, since we're one of the poor fools whose higher-ups have mandated the use of Sophos Enterprise...
EXCEPT. Microsoft will know these products well. They aren't obscure. So they should have been tested for.
Over the years their has always been the excuse of "well we can't possibly test every possible configuration of software/hardware...."
And it's always been at least partially bullshit. Because they screw up on standard configurations that most users are likely to have.
"these programs which hook undocumented API calls"
back in the 90's, MS got sued over having undocumented API calls that Word and other MS applications leveraged in order to "not crash all of the time". I had just started doing windows development for Windows 3.0 and ran into some of these problems with 'GlobalLock' etc. which could occasionally UAE when you had a bad handle, not necessarily your fault either.
The solution, for me, was to validate the param by using the undocumented 'GlobalHandleNoRIP()' and fail gracefully. At the next dev conference, Microsoft introduced the 'ToolHelp' library [to assist with things they'd been doing internally, cough cough] and DOCUMENT all of those undocumented calls...
[they also separated the applications from the OS as separate 'business units' and complied with 'no information between OS and applications units that is not disclosed to EVERYONE']
So - with respect to 'undocumented calls' - which ones are those again? Microsoft may be forced to DOCUMENT them...
Useless advice - are we now supposed to keep parallel, identical computers running? What about non-standard software that's been installed?
I don't know if we should blame MS or the AV vendors, but either way you can't have umpteen test PCs.
Useless advice - are we now supposed to keep parallel, identical computers running?
I do. They're not identical in terms of content or scale but they are identical in installed software and general configuration. They're close enough to be a canary in the coal mine, for my software development QA as well as for vendor patches and upgrades. I accept it might not be achievable in everyone's environment, and certainly not even in all of mine all of the time, but I always aim to get such an arrangement agreed in the original project plans and budgeted for. If the project manager or the beancounters turn it down and something subsequently goes wrong, my arse is covered.
I do that too. They are pleasure to work with because they boot really quickly so testing is simple, let me build the updates myself (if I wish to) and the license is not only free, they also let me see and tinker with the sources or build options.
Oh ... but that gives the game away, doesn't it? It's all Linux. Smug, me?
Yeah, smug you. ;-)
It's fine if you've got the option, but if the business requires you to work with Windows servers (or whatever) then you've got to work within those requirements and solve problems within (as well as with) those requirements.
It's fine if you've got the option, but if an idiot requires you to work with Windows servers (or whatever) then you've got to work within those requirements and solve problems within (as well as with) those requirements.
TFTFY
We have a bunch of "test" PCs, they get updates and patches ahead of the hoi polloi. They are single PCs in key departments. If the updates don't cause problems, they are rolled out to the rest of the machines.
Stop installing patches on the day of release automatically.
Set up a small set of machines, in a separate WSUS group, apply them there, test them there, then roll out to larger and larger groups as you get more confidence in them.
Seriously, "auto-update as soon as something is released" is the stupidest thing I ever heard.
Home users, sure, I understand that they don't have the nous, but even so a "I know what I'm doing" button should always be present.
You cannot forcibly inflict updates on any group of people, though, unless they've CONFIRMED that they have a backup ready and in place. That MS think they can just force it anyway, without even telling the user, and just assume they've been backing up is RIDICULOUS. Hell, even System Restore is pretty much a waste of time that I don't know of anyone ever successfully using to any decent extent.
In this case, System Restore has been the saving grace for us, as the only way back for 20 plus machines.
A symptom of the problem which I've not seen mentioned elsewhere was that on a full boot, after CTRL-ALT-DEL no login box showed up, so users could not get into their machines at all.
Booting in safe mode, and doing a system restore to a restore point prior to the upgrade has been how we've fixed the issue.
If you have Windows server, you have WSUS. No need for third-party (though third-party often does it more nicely).
There's no excuse, one you have more than a handful of computers, for not using WSUS for exactly what it's intended for - scheduling updates, revoking them, and centralising all Windows Update downloads in your organisation.
Although you are correct I think that only distracts from the problems caused by MS updates. MS's new(ish) rolling update system does not seem to fit for purpose, but I'm sure if saves MS more than enough money to continue contributing to Senators and Congresspeople as they find expedient.
I'll bet those contributions don't come cheap.
On one of our Windows 10 systems, running Cisco AMP, after the Tuesday patches went on we get a yellow '!' on the Windows Defender shield icon. When you look, it says "Virus and threat protection status unavailable, open Cisco AMP for Endpoints for information." The link it provides is to the AMP Connector, which won't open. Opening AMP directly, it thinks things are fine. I'd turn on AMP Connector Debugging if I could find where the log file for this is!
Anyway, something seems off in the Connector interface.
Before, we had all those Windows Update patches to avoid that every knowledgeable person had a list of. That worked not very well, so Microsoft is on to the next round of the same, but with a different excuse.
The goal remains the same : get them all up to Slurp Edition.
And who do you think would be sued when the machine gets a virus? Sure, the EULA might protect Microsoft...but then again, an average judge and jury might understand "Microsoft silently / remotely turned off the antivirus and the victim got a virus.". It's not rocket science. Even worse that victory might start undermining the other aspects of the EULAs....probably safer just to brick the machine, everyone understands updates break things, right?
people update to prod in patch tuesday week? Jesus no 1 month behind MS always. SCCM is a bulky beast but serves a purpose.
And just Avast and Sophos affected, hmm doesnt sound like a real windows issue to me, symantc/Webroot rocking here and in global corp space, sounds more like Soph&A not following MS best practise for such tools, yet its Ms fault. ;o)
I run neither Sophos or Avast but my Windows 7 64 bit setup was rendered unbootable by the April 9 update, forcing me to restore using the Monday system image backup. I disabled Windows update, figuring that this update was another attempt to force me to use the Windows 10 Spyware OS. As of now, I am preparing to move to Linux to get away from Micro$oft's BS.
Are we sure it's not straight from the movie script ?
"an unfortunate user has downloaded "the thing"
[ .. ]
And of course Windows 10, to which Microsoft would dearly like users to upgrade, is not affected by the borkening.
Thanks to all the Reg readers who got in touch about this.
We feel your pain. ® "
Windows 10 has also been affected by this (1803 & 1809), during testing I've had issues on 3 W10 devices, which magically worked normally again once the update had been uninstalled in Safe mode.
This was caught on the W7 devices on the first test device and reported to Avast (via Barracuda who now own Managed Workplace) and so hasn't gone any further.
Avast have pushed out a micro update which needs to be installed and have a restart before the update then stops causing problems.
Sophos look to be working around with exclusions.
Fun all round....
I shall no doubt be moaned at for taking so long on updates this month!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
