Microsoft's attempts to harden Kerberos authentication broke it on Windows Servers Microsoft is rolling out fixes for problems with the Kerberos network authentication protocol on Windows Server after it was broken by November Patch Tuesday updates. As we reported last week, updates released November 8 or later that were installed on Windows Server with the Domain Controller duties of managing network and …
Also time to change your KRBTGT password (twice, after allowing replication time to all DCs or 10 hours to be safe (krb token expiry interval)) if it hasn't been done since Server 2008 DCs were introduced in the environment. Helpful hint: the WhenCreated date of the "Read-only Domain Controllers" group tells you when. Otherwise KRBTGT won't have AES crypto keys.
The domain needs to be at least the 2008 DFL first. The safest method is to use the New-KrbtgtKeys.ps1 script by Microsoft, available on Github, since it does a single-item replication to all DCs that finishes in seconds in our environment.
Any other accounts (i.e. service accounts) with passwords that haven't been changed since then need to be rotated too. Yes, I got an unfortunate surprise when I queried our domains for the number of active accounts in that boat.
The whole mission of these updates is to deprecate RC4, eventually. It'd be nice if MSFT didn't screw up any further updates, like they seem to have with every single Kerb/NTLM update since late last year.
Indeed - given that, IIRC, M$, in their wisdom, decided that +-5 minutes would be OK for MS Kerberos.... other OSs want much tighter control than that and back in the day had to put in mods to cope with the M$ Kerboros bodge.
Yup bodge - reason? well that would be because M$ allow their RTC to be affected by system interrupts so could not ensure it would be updated in a timely fashion. Unfortunately this is still true (at least in Windows server 2016) - get a VM or two cranked up to 100% CPU for an extended duration, and your Hyper-V host clock will measurably drift, which can make things "interesting".
Really. I don't see in the Kerberos v5 spec where it states the 300s clock skew is MS-specific: https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-admin/Clock-Skew.html. Also, like other OSes, the acceptable skew for Kerberos auth is customisable in Windows, down to 60s.
And I don't care what OS you're running; you're not configuring a Kerberos environment correctly if you're not using an NTP service to coordinate your system times. As MSFT doco specifically states should be done when setting up an AD environment, complete with a time service embedded into all Active Directory implementations that all clients will synchronise to by default.
Up until Win 6.3, the acceptable time drift on clients was not defined. In practice, it was up to a couple of seconds (you could install third-party higher-precision NTP clients). More recent versions of Windows allow you to specify max time drift from 1s down to 1ms (the latter with expected constraints - no more than 4 hops and 0.1ms network latency between client and stratum 4 or higher time source, avg daily CPU time <80%)
In any case, if you have Windows machines in a domain hierarchy getting time from VM hosts, you've done it wrong. I totally accept that Windows/AD is not perfect, but if you're going to snipe at it, at least get your facts correct.
The computing environment is extremely complex thee days, so many updates making changes to "fix" problems but in the environment, while they will normally "work" there's the potential for new issues - caused by the computing interactions between the "fixed" components and other interacting features. So this is normal, and just needs to be fixed by "engineers trying to resolve the problem". Resolving a problem may work, but can result in a different broplem (sic) sometimes.
I would be more sympathetic if this situation wasn't entirely of Microsoft's own making.
They overcomplicate everything, introduce all new protocols, languages and tech stacks as often as Google introduces new messaging systems, and needlessly over-integrate systems that should never have been integrated, so they can drive more lock-in.
And us morons keep handing them ever more cash so they can keep doing what they're doing cause we prefer the devil we know.
Well, this selection of morons left that boat quite a while back - no more MS software.
Now it would be ungraceful for us to laugh at all those victims but, bwaha, hahaha, snort, it's, hahaha, haha..hard to control. Hihi. Sorry.
:)
No, I'm sorry, Kerberos is a core technology in the AD stack. While MSFT making moves to tighten up their entire auth structure is commendable, it is only in the last year - in fact, since November last year - that they have released so many updates that have screwed up one or other component of AD auth (NTLM/Kerberos - LDAP stuff has been fine).
This is *entirely* due to whatever testing model they have implemented most recently, which I understand is essentially relying on the Insider program for "regression testing".
I expect some pain while we ensure that legacy accounts get remediated prior to force-deprecating RC4, and the documented staged approach to do that via these updates is fine. But you cannot break core services simply because you can't be bothered regression-testing - it is not a new concept, and nor is it an obsolete one.
AD security has always been highly complex and has undergone significant changes in the last 20 years, mostly without too much disruption. But this past year has been more like the mid-90s days of bad NT4 service packs. It's really not acceptable, unless you subscribe to the view that this is a deliberate ploy to force everyone into cloud, which frankly seems increasingly likely to me.
Of course, debacles like these mean that our clients are even more reluctant to do even basic timely updates, let alone migrate if it makes sense for their workloads (which, frankly, it doesn't always).
Having broken it, why are they making it extra hassle to install the update? I.e. you have to do it manually and it isn't pushed out through any of the update channels (WSUS, etc).
Also in the past week, we have the latest version of Outlook Android refusing to work on certain models of Samsung phone (MS being exeptionally unhelpful and keep closing cases - forum posts indicate that this is the response to other people raising it too). Still not found a workable solution to this yet so having to buy a different model of phone for urgent issue requests, while we have a pile of one of the affected models sitting in the cupboard!
Plus, separately, authenticator randomly stopping working for some users, and the only solution I've found so far is to delete the authentication methods from the user's Azure account and get them to re-register the phone for authentication.
No idea if there's any relation, but hasn't MS very recently started forcefully disabling basic authentication all of their online services? It wouldn't be surprising at all if MS found out - after the fact - that internal components of their systems were still reliant on basic auth in some way and were getting borked.
Or I could just be spouting nonsense...
They do.
As far as I can tell, the aim is to pretend being secure by forcing you to entirely rely on their non-RFC mechanisms, mechanisms for which there is no third party assurance they haven't "accidentally" left themselves a neat little backdoor..
Yes, I don't trust them. Neither should you.
This isn't the first time they screwed up critical infrastructure software, and it won't be the last.
This reinforces update hesitancy because people cannot risk losing access, and forces them to risk compromise.
If this only happened occasionally, I could accept it was an honest mistake. But Microsoft does this routinely, which means these mistakes are entirely intentional and Microsoft is specifically trying to undermine computer security around the world.
I don't think they're deliberately trying to undermine computer security around the world, that's more an(other) unintended side effect.
They're just pumping stock by showing investors that their customers are so locked in they can be abused and exposed without any notable consequences for Microsoft.
It's a demonstrator.
The Privileged Attribute Certificate (PAC) is an extension to Kerberos tickets that contains useful information about a user’s privileges. This information is added to Kerberos tickets by a domain controller when a user authenticates within an Active Directory domain. When users use their Kerberos tickets to authenticate to other systems, the PAC can be read and used to determine their level of privileges without reaching out to the domain controller to query for that information (more on that to follow).
Now, admittedly I'm not exactly an expert in the inner workings of kerberos, but this bloody thing has caused me no end of trouble and looks like another example of MS screwing with something for their convenience without considering the consequences.
The last time there was a bug in the PAC, we got a samba update that "fixed" it by breaking all non MS kerberos instances, because it put in extra PAC checks and hard-required them to pass for authz to take place.... which of course they didn't. Because we're not running AD. And MIT Kerberos doesn't have the PAC attached to it to start with.
Kerberos doesn't have a notion of "groups" it is only an authentication mechanism. Authorisation is a different thing.
Kerb tickets are granted on application to the KDC and have a lifetime decreed by a policy (config file or GP or whatevs). This bunch of shysters: https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/decrypting-the-selection-of-supported-kerberos-encryption-types/ba-p/1628797 decree 10 hours for TGT.
Since I patched my DCs at work with a 1.5GB "Cumulative Update" a lot of kerb related auth snags have vanished. I use Arch (btw) on my personal laptop and PC and so does my wife. At the moment my hatred for MS knows nearly no bounds for wasting a lot more of my time.
Back in the day the same fuckwits used to make me jump through hoops involving getting autoexec.bat and himem.sys to get enough memory available to do useful stuff.
How do we go about suing for a lot of wasted time from a bloody monster 8)
Yeah. I understand the motive behind this – Kerberos makes the TGS (Ticket-Granting Service) the hub for all ticket requests aside from requesting a TGT, so TGSes become bottlenecks. An extension which lets some services avoid the TGS interaction makes scaling easier and can improve performance.
But, of course, messing with a security protocol is an excellent way to break some part of security, and that includes availability.
Heck, even Apple's stuff is better.
It's a good thing Microsoft spends so much time and money wining and dining those who can't tell a decent cost of ownership study from a screwdriver or they would be out on their ear already. If they spent half the amount of money they spend on *cough* "marketing" *cough* on security there would not be any problems, but why change the habits of literally decades?
It's a part of Microsoft's Azure/Microsoft 365 only stance...
The bloody error messages, when they are eventually tracked down, state that the Microsoft Azure connected/enabled device is not able to authenticate. Except neither the client nor the server system are in any way Azure enabled and the Microsoft reporting command "dsregcmd /status" confirms this.
In other words, it's yet more Microsoft shite that was only laughably tested against their own Azure/Microsoft 365 systems and nothing else at all.
We didn't see any Azure-related errors in our environments - we got the known error Event ID 14 in System log on DCs that had been patched without the fix yet and a problem account:
"While processing an AS request for target service krbtgt, the account [ACCOUNT] did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 1). The requested etypes : 18 17 23 24 -135 3. The accounts available etypes : 23 18 17. Changing or resetting the password of [ACCOUNT] will generate a proper key.
This was a service account where the "This account supports AES-256 encryption" option had been set. So I unchecked the option on the account and made sure that msDS-SupportedEncryptionTypes was set to 0. That fixed the problem for that account while we applied fixes to the DCs.
If you're getting Event 42 errors, that needs the KRBTGT password to be reset: https://dirteam.com/sander/2022/11/09/knowledgebase-you-experience-errors-with-event-id-42-and-source-kdcsvc-on-domain-controllers/
And don't worry, I too feel like this is part of MSFT's ongoing campaign to stealth-deprecate on-prem AD, but I wish they'd just accept that in the real world, some of us don't want to put ALL our eggs in the cloud basket. Especially if you're running critical public infrastructure.
This kind of thing makes it more difficult for IT teams to convince the nervous that no, we DO need to apply security updates regularly, and yes, we DO need to update these 20-year-old systems before they fall over, since no-one is supporting either the hardware or OS.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2023
