CrowdStrike Windows patchpocalypse could take weeks to fix, IT admins fear If you're an IT administrator with Windows boxes on your network, Friday can't have been a lot of fun. What's likely millions of systems were or still are stuck in blue-screen boot loop hell, mostly requiring manual intervention to fix. It's due to a broken file pushed out by CrowdStrike to Microsoft Windows systems, causing …
> If you're an IT administrator with Windows boxes on your network, Friday can't have been a lot of fun
Wait, do some people in IT actually enable auto-updates and carelessly assume they will work with zero testing? I thought that was just a joke. That is a terrible strategy in terms of resilience!
I actually don't blame Microsoft or Crowdstrike. I blame slackers who don't apply updates responsibly. Fun to see just how large a proportion of the IT industry are incompetent.
> could take weeks to fix, IT admins fear
The longer the better. It gives people more time to go back and learn their trade.
You're an idiot.
Given the choice between an occasional glitch and a zero-day, I'll take the glitch.
Turning auto updates for security patches on is the ONLY responsible choice in 2024.
In this case, the mistake was attempting to 'manage' the updates with a broken 3rd party tool. Don't do that. Just turn on the auto updates.
NONE of my clients were hit by this, because NONE of them use this crapass 'management' garbage software. If you feel like you absolutely must 'manage' something, stick to open source software.
> Given the choice between an occasional glitch and a zero-day, I'll take the glitch
You are an idiot.
You do understand updates don't need to be "automatic" to be installed right? haha.
Perhaps you don't know how to install updates properly? Hint: You don't go around all machines with a floppy disk anymore ;)
> stick to open source software
If you carelessly install updates (many of them are irrelevant I may add), even open-source won't save you.
It's not just a db update (though it appears it should have been) or even an application update, it's a file that is loaded into kernel space. Hence, it has the potential to root your devices. It has full control.
If you apply such an update blindly, from a third party, you're the idiot.
I wonder how many state-hackers have realised that the best way to take down our infrastructure is just to hack some third party supplier who hacks like you have given the skeleton keys to the whole kingdom.
I am not any of the above anonymous cowards, apart from the one you replied to.
I understand entirely how it works, but more importantly, how it *should* work.
Apologies for your lack of reading ability - thanks for incorrecting me though!
If you care to explain your reasoning, rather than spout obviously groundless insults, then I'll be happy to debunk your arguments piece by piece.
By the way, why are you now defending the product you slated in previous posts? That's a bit weird
I'd wager 85% of the times that auto updates are either enabled, or that updates are completely disabled(or perhaps the software version is obsolete and can't get updates). Which is worse? Most places likely lack the skills to perform proper testing, I know this from experience working for companies that have built their own software for 24 years now. Talking internal QA failures, which of course MS and obviously Crowdstrike has as well. The safest bet is to just delay the update by a bit see if others get bit by it first.
Problem is, for security software like this I suspect 95%+ is updated from "the cloud" anyway(and likely updates are checked for multiple times a day). Likely large numbers of systems running in isolated areas not connected to main corporate networks so no easy way to slip in some kind of intermediary update management platform. Not to mention remote employees who may almost never login to a VPN to do their work. I'm sure there are some systems that can work around those kinds of things but it adds more cost and complexity and more often than not the organizations don't want to pay for it(and may not have the staff skilled to handle it adding to more costs). Same can be said for going "multi cloud" (in the truest sense of the phrase), extra cost, complexity. I recall at the last org I was at they used Sophos for their IT security endpoint solution. I recall at one point I asked Sophos a question about something and they said something along the lines of, "do you know you don't have ransomware protection enabled? you just need to go into this setting and click this check box". The IT staff at the company never paid attention to some of the most basic things, which I think is the norm rather than the exception. After the "network engineer" quit I found out that he had not applied any software updates to their ASA firewalls in ~4 years and I counted 120+ known security vulnerabilities in it. He didn't ever put them on software support because "the devices never fail" .... .... ...
For me, I feel sorry for those folks impacted myself I don't have any real suggestions. Glad I don't really have to deal with corporate IT endpoints, my work has been on internet facing linux stuff for the past 21 years. Though I do deal with windows servers as well, just is a tiny fraction of my routine.
On my Windows 10 1809 LTSC VM that I use for work stuff I only apply updates there manually, by using local security policy or whatever it's called to disable the auto updates(apparently disabling the windows update service in Win10 wasn't sufficient like it was in Win7 which I used till late 2022). I get updates till 2029 I believe so don't have to worry about Win11 for a while, by 2029 even Win11 should be to a decent point of stability. I haven't had a known security incident on my home systems since the [STONED] virus in the early 90s. Though I did have AV software flag some malware in some pirated game stuff I did back in the late 90s(none if it appeared to be actually harmful as far as I could tell).
That and I moved my org out of the cloud 12 years ago, so I don't have to worry about that aspect of things either, my co-lo runs smooth as butter in their super conservative configurations.
> The safest bet is to just delay the update by a bit see if others get bit by it first.
Yep, it is as simple as that. First step is to turn off the compusive auto-updates and show a tiny bit of due diligence.
Let the 12 year old gamers be the first to test the random Windows updates. That is what they are there for!
>> The safest bet is to just delay the update by a bit see if others get bit by it first.
> Yep, it is as simple as that. First step is to turn off the compusive auto-updates and show a tiny bit of due diligence.
But it looks like this was a driver update, not a malware detection update. There are stories that companies which did configure a delay were still hit. What's the betting that CrowdStrike don't offer an option to delay updates to their drivers?
> There are stories that companies which did configure a delay were still hit. What's the betting that CrowdStrike don't offer an option to delay updates to their drivers?
This is basically the crux of the problem. As companies (IMO criminal) forcibly add software to our machines without our concent, it opens us up to epidemics.
The industry really is full of idiots for accepting this!
Whilst I agree with a lot of your points, the last one is irrelevant.
The actual endpoints/servers being in the cloud has no relevance to this scenario. Whether on-prem or cloud you can manage your updates in exactly the same way (e.g. wait a day, test on single device, dev, UAT, etc) whatever your strategy may be. So if your update strategy is poor, having your servers on-prem or in the cloud ain't gonna save you.
“ This is most definitely where cloud helps. Spin up a machine (small), test and then shutdown. You'll pay pennies for your testing, and even less (next yo nothing) when not testing.”
That’s a lot easier by knowing how the error showed. You can have completely different errors that wouldn’t be detected that way.
> > The safest bet is to just delay the update by a bit see if others get bit by it first.
With the previous solution we had we could and did do that for critical systems, if a definition update caused a false positive the distribution would be stopped until it was resolved. Now it is gone after higher ups decreed it and its cloud cloud cloud all the way......down..
Actually watched the video and fully agree with the sentiments, one glitch and the world will grind to a halt....and one day you WILL wake up to blue skies and all your cloud systems a memory....and what is your DR plan then?
Will the links have THIS STORY in writing too?
That's the goal, here: my phone has no headphone jack so 1) its shite and 2) I don't carry headphones, and thus on the bus or just when life's too short we don't want to die inside while someone speaks loudly and slowly. Written words allow us to take in info - and diagrams - in a way that ensures our fellow passengers doesn't hate us more than we do already.
In short, there's value. And when people spend so much effort to produce a story or article, why limit its reach?
-..
Nah it's a video/audio discussion by Reg staff. It's partly to show we are smart, normal, nice but sarcastic humans putting this site together, not some bots or humorless suits. You don't have to watch it; it complements our written coverage. Some people like to listen and don't have time to read through pages of text.
Their needs are as important as your needs, as important everyone else's needs. This is classic 'you can't please all the people all the time'. We're at peace with it.
C.
(BTW YouTube does auto-generate a transcript that replays in real-time, which you could follow on mute, tho it might not be to your liking.)
I appreciate the coverage. Both written and spoken.
I've often railed against the morphing of journalism into podcast chatter and u-bend narcissism, but this time we had about 5 written articles and one audio summary (i didn't watch the heads move) which is a good balance IMO, and it was a good summary.
CloudStrife (for that is what I will call them from now on) have perfectly demonstrated how there is "no silver bullet" for infosec, and they have provided me with a plentiful supply of schadenfreude for a Friday afternoon.
Who can afford to mix and match security across their systems? People usually buy in bulk, from one supplier, because you get a better price. Go with a few sites on supplier A and others on supplier B etc and you might not be able to afford the contracts. Only the biggest organizations could afford to do that.
And if you mean limit the numbers of customers per product then how will you enforce that? It's customer choice. If half the world chooses to use one supplier then there's little you can do about it. What are you going to do, issue quotas per supplier? Who will maintain that on a global scale?
"How affordable was it not doing that?"
Irrelevant. If you don't have the funds then it doesn't matter how much it might have saved you in the long run.
Many places just don't have the resources. The consequences might be expensive or they may just be salaried techs having to fix numerous machines after hours or instead of working on other projects. That costs 'nothing'.
If the legal system was just (haha) Cloudstrike would be liable for damages from every organisation and individual adversely affected by their incompetence.
However the chance of it happening is on a par with the chance of the US having an honest competent government.
Some lawyers will get richer but I bet that Cloudstrike will end up with nothing more than a slap on the wrist and perhaps a nominal fine equal to less than one weeks profits,
Just as with every software licence agreement I've ever read (dating back to the late '70s - I'm not old enough to have used software prior to that), I've no doubt there'd be a clause in there saying something to the effect of "you're only paying for the right to use this; we don't guarantee anything; if something bad happens, tough sh*t."
A clause in a license agreement that is contrary to law is null and void (and if they didn't make those clauses separable from the agreement that may make the entire agreement null and void).
In some palces in the world, there are legal requirements on implied warranties that can't be waived by contractual language.
So if it later transpires in court that CS didn't fuzz-test a kernel component that reads data files, predictably resulting in entire fleets of machines simultaneously becoming unbootable, that might count as negligence?
"you're only paying for the right to use this; we don't guarantee anything; if something bad happens, tough sh*t."
Given the impact of this lawyers are going to be looking vary carefully for ways round any such clauses. For instance if this bypasses any controls the customer might wish to make and pushes (or pulls) files in automatically then that might be caught by some provision such as the UK Computer Misuse Act.
The uS gov is competant, problem is you dont have the vision to understand who they are "helping"...
Often the answer in life everything is to ask who wins... when you answer that q, you will understand why the US gov does what it does...
Not a chance of them facing justice.
CS will file for Chapter 11 in a few days or if they are really desperate to escape, Chapter 7.
Then some mega corp will buy the ashes of the corp and bring the corpse back to life.
But IMHO anyone doing business after this fiasco with the current or future Cloudstrike company is an idiot. If the Suits on the 'C' Level insist, then walk away. There are other jobs.
Your reply seems to accept there's no other way to do things. Any reason why:
- Windows can't have a standardised update mechanism like Linux distros have?
- confidentiality is prized over integrity and availability to such an extent that airports, airlines, banks, businesses are knocked over worldwide and it's going to take weeks to sort out?
- this flaky design is necessary in the first place? Perhaps AVs should be relegated to userland and they get an API to call instead of messing round with the kernel in unexpected ways.
There are also perfectly good reasons why you don't want systems to fall back to dead, especially when they're collectively running a large part of the world's infrastructure including health care systems.
There are no two ways about it: this cannot be argued into having been an acceptable situation.
In the 19th century, we learned that a monoculture could result in a famine - albeit in that case it was potatoes and a virus (and a disastrous political environment of occupation) that caused the problem.
I’d argue that the same applies in IT. An infrastructural monoculture can lead to the rapid propagation of a problem (whether a failed update (as in this case) or malware), poor political decisions can make it worse (perhaps not in this case, but regulation of encryption might be an example), and the result is failure of global IT systems.
A better solution might be to ensure diversity of minimal underlying environments, including the hardware, with the required functionality being delivered through container solutions which can be re-spun up rapidly in the event of a problem or need to fall back.
As long as the minimal underlying OS is diverse then the likelihood of a global, or even companywide, outage is greatly diminished.
One major reason for our reliance on single providers is that all the functions are monetised and exclusive.
The wife may write her report on a Mac but her employer pays a subscription to MS and therefore the file has to be converted into a MS Word document for example.
Just to share an mp3 or a photograph with all our household machines involves third party programmes, several different bits of wire, memory cards and often reformatting.
Sharing across devices is hampered by the fact that none of the manufacturer will allow any other make to connect.
Magnify that across a whole planet and it means that one os becomes standard for business and one registry error cripples us all.
@jokerscrowbar
We have an easy solution for home network sharing (mix of Windows, Linux, Mac and then iPads and androids).
External HD plugged into router that is available over the network (use router to define access rights, not 100% ideal but is on isolated home network & nothing "important" on there as originals all elsewhere , it's just for easy availability) so photos, music etc. can be put there & anyone can access them at home
"A better solution might be to ensure diversity of minimal underlying environments"
How would you do that? Again, not everyone can afford to have multiple solutions within an organization. And between organizations, if suppliers A, B and C all provide solution X and B can supply a good standard of X at a lower cost then more people will use B. Which is exactly what happened with CrowdStrike. It wasn't the whole world that was affected and the majority of organizations have got back to some sort of running in hours.
Part of an update from Cloudstrike mentioned in a earlier Reg article
"Windows hosts which are brought online after 0527 UTC will also not be impacted
Hosts running Windows 7/2008 R2 are not impacted"
In other words hosts running Windows 10, 11 and servers after 2008 R2 that pulled the original update are impacted. So there is no way Cloudstrike could have tested the update they released to channel, because if the had they would have found the crash.
Could they not change the deploy to channel to take down the channel* so it is only visible internally, deploy the update, and then run up all the hosts the update is for. Then only if all hosts have updated without error, put the channel back up.
*I would expect they would take the channel down before they send updates to it.
Crowdstrike have also managed to do the same kind of thing previously on Debian, Rocky, and RHEL. So looks like they didn't learn from those three previous episodes and went on to bigger and better things and took out Windows.
What makes you think George is responsible ?
Crowdstrike have 100s or 1000s of software engineers and 100s or more managers.
George hasnt a fucking clue how to writ e a single line of code... he is many things, a bullshitter and a liar, but he is not an engineer.
No wonder this shit happens, morons like yourself cant even figure out how fucked up the software.
The elephant in the room is an OS that can't recover from a bad kernel-level driver.
If Windows can get as for as a blue screen, it can work out what was the faulting application, remove that .sys file from the startup, reboot and therefore continue running.
Yes I know the same thing can happen in Linux, but it doesn't need to either. This functionality should be standard in serious OSes. Maybe it is already in which case it should work better!
In 2024, an OS should be able to recover from a blue-screen all on its own (as long as it's not hardware, obviously).
In 2024, an OS should be able to recover from a blue-screen all on its own (as long as it's not hardware, obviously).
CrowdStrike marked their driver as "non-recoverable".
The reason the OS supports "non-recoverable" drivers is for hardware, obviously, and it's also used by kernel-level anti-malware protection.
Sense of Schadenfreude here, in that here we have an enterprise class employee spyware program. An external third party app effectively given root acces to the os kernel, a trainwreck waiting to happen. A wide open attack surface, just waitng for some hacker to find a way in, and you have to ask, is this some very subtle cyber attack ?.
Auto updates enabled and probably autoruns as well. Total lack of change control discipline, sloppy update verification at os level, so what does anyone expect ?. Total idiots, the lot of them...
I get to my local independent auto maintenance shop 15 minutes before it opens on Monday...To be first in line for my annual Massachusetts automobile inspection.
Hmmm no one else queued up. Lucky me!
....
Argv, Argc! (versus swearing). The state wide vehicle inspection system is CloudStruck.
No Windows at home, just Macs and Linux...But I was zapped by Redmond et all in anyway.
Sigh!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
