CrowdStrike shares sink as global IT outage savages systems worldwide CrowdStrike's share price is currently tanking amid a major global IT outage its leadership has attributed to a dodgy channel file. At the time of writing, the share price is down more than 19 percent as the security shop to some of the biggest organizations in the world continues to work through issues with its customers. …
When I searched google news for Crowdstrike a couple of hours go i came across this CrowdStrike stock could drop to $275 amid valuation concerns, analyst warns from yesterday - so what did Redburn Atlantic know?
The Reddit Crowdstrike thread has gone nuts with horror stories, poor BOFHs looking at 1000s of endpoints in boot loops. Glad we don't use it, bullet dodged even for our small footprint...
My condolences to those looking at RSI from entering zillions of bitlocker keys.
Had a call with a client; "can we reschedule to next week, we use Crowdstrike." = Weekend ruined
> My condolences to those looking at RSI from entering zillions of bitlocker keys.
Wait, the workaround trips BitLocker? Argh
So for some, CloudStrike has turned Microsoft into an inept but large ransomware gang?
Where's my BitLocker key? It's somewhere in AzureAD.. Which one is for this server? Err ..
To be fair, deleting excel.exe is probably a good measure to improve organisational efficiency. I was literally only just having a conversation with someone about why Excel should be destroyed with fire, and then the ashes launched into the sun (and only then because there aren't any convenient black holes close enough by to dispose of it there).
https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-if-you-use-that-spreadsheet/
Anyone who disagrees with me can go along for the trip. I'm sorry it has come to this, but it's for the good of humanity.
> I will now have nightmares about AI writing Excel spreadsheets.
Only now? How have you been sleeping peacefully for the last three days?
1. I retired in 2018
2. I have lots of other nightmares queuing up, so AI will have to wait for its turn.
3. I had a medium* operation on Monday which has pretty much wiped me out physically.
*Not major, but not minor either. A reasonably standard procedure, but involving a general anaesthetic and an overnight stay, plus I am not allowed to lift anything heavier than a dull kettle for 6 weeks.
Thanks, Doctor Syntax.
Sadly as I live alone I am the only one allowed to lift the kettle.
However, after spending yesterday afternoon and this morning at the SAU (Surgical Assessment Unit), I was pronounced fine-ish - my current latest discomfort will abate all by itself, in 3 to 4 weeks according to Dr Mahmoud (lovely warm hands, I mean really warm, a definite plus when you are as ticklish as I am).
Still not allowed to drive for a while though.
I certainly seem to have demonstrated that spreadsheet jockeys are not only extremely defensive of their previous Excel, but also completely lacking in a sense of humour.
In all seriousness, though, the number of times in my career where Excel turned out to be the right tool for the job can be counted on the fingers of one hand. It is always a "quick'n'dirty" stopgap measure that ends up being bent into something more, and which manages to be nothing other than an administrative overhead.
I want something with a Schwartzchild Radius to make sure there's no chance of information escape. You know, just to be on the safe side, just in case there's some massive solar prominence in the future that looks for all the world like it spells out "VLOOKUP" in the sky.
Have an up vote and a pint on me for the Schwartzchild Radius quote.
Made me laugh and God knows I need it just now with a seven month old Malinois puppy running me ragged.
Oh, and I'm very glad that I am retired and outside the blast zone for this balls-up.
> I want something with a Schwartzchild Radius to make sure there's no chance of information escape.
Sorry to be the bearer of bad news, but according to The Holographic Principle
>> the information content of all the objects that have fallen into the hole might be entirely contained in surface fluctuations of the event horizon.
One day, those fluctuations will be readable, imprinted on the Gravity Waves and propagated across the Universe.
Better to chuck it into the Sun and have it all drowned out by the thermal noise.
My understanding of the Holographic Principle is that it applies only within the event horizon; light, and thus information, can't escape the event horizon, but to an observer within it, information coming from any point within that sphere can appear to be indeterminable from that coming from a point on its surface, or something like that.
From an outside observer, the Holographic principle would surely only mean that it's impossible to tell whether the "surface" of the event horizon is black, or everything within it (ignoring for a moment that the curvature of space caused by the singularity would probably mean that all you see is a distorted view of what is behind it anyway).
As for information escaping an event horizon, there is a possible mechanism through Hawking Radiation, although how information might be exchanged between a particle falling into the black hole, and one escaping it, in any meaningful way, remains to be explained.
I remember McAfee deploying an update that basically removed a key boot file from all windows machines (around the 2000's).
I was working for Logica at the time. It bricked most of the PCs and laptops in the company.
Fortunately, I was out of the office and offline that day, so I missed the update. And I made damn sure that the issue had been corrected before I reconnected to the company network!
It depends on whether or not you take precautions. One might be to test before deploying, another might be to wait a day to see if any adverse reports roll in. I guess any Cloudstrike customers who adopted either approach won't be rolling it out today.
Is that even possible with CrowdStrike? I note that my company phone gets Teams updates over which I have no control whatsoever. It's not a managed phone, I had to install Teams myself from the app store. But every now and then I go to Teams and it tells me that an update has been installed and "we are getting things ready for you". It looks to me like Ms & Teams are by-passing both the app store and consent and forcibly updating the app whether I like it or not. Every other app on the phone, I get app store notifications when there are updates available and I get to choose if I install them.
There are these old-fashioned things called IT departments. As this seems to be a product aimed at big corporates there's a fair chance their customers still have them. Not guaranteed these days, but a fair chance. The IT department does the test and makes the decision on behalf of its users - and does the roll-out. I suppose they could still roll out something they know will bork all the workstations on the grounds that it will keep out ransomware but at least it becomes a deliberate choice.
No, not necessarily, but in that situation, I want the server admins to be able to get the fix, test it and choose to deploy it. A zero-day ransomware is a risk that might be worth taking for a few hours or a day if it's been shown the fix is a guaranteed clusterfuck. The customer gets to test and choose, not the vendor. I'm ok with auto-install as the default, but prefer a way to switch that off. Corporate used should be managed by corporate IT, not by the vendor. Vendors don't test as thoroughly as they should, we know that. In fact it's been demonstrated many times by borked "updates" being pushed out causing much expense and difficulties for the end users and admins. Looking at you MS, who not so long ago fucked up the core printing subsystem for many users.
Here's a thought. MS run two different update servers, one releasing updates a day after the other. You choose which one to default to, with possibly only the absolutely most vital patches being pushed out ASAP to all users. That way even home users get the option to only install "tested" updates because, as the Windows Insider Program demonstrates, there are plenty of people willing to be free MS QA testers.
Microsoft had a patch for the zero day print driver exploit months before it was released. If you had the special $$$$ contract you get those patches.
I know someone that got hit by it and WAS patched up to date and they withheld the patch.
Have you ever noticed how their cloud products seem to be have patches way before your on prem doesn’t?
They want that monthly cloud money.
If there's a new "feature" patch from MS that is badly written and contains a flaw, thus producing a future zero-day exploit, then you're fine with that being automatically rolled onto your device with no interaction? That knife cuts both ways.
The main thing that I hate about "SaaS" is that you have no control over when or how the software you use changes arbitrarily, often forcing unexpected downtime while it updates, or causing lost productivity while you re-learn it.
That option is being removed quickly.
Just as a test, fire up Wireshark and filter the background noise. Then load any application and watch it phone home and download changes.
Another thing to watch is how windows, without authenticating can copy files between system even without being on the same domain. Only needs to be on the same network. It is part of the windows update. If you can find the setting, you can turn it off till the next update and it will be dutifully turned back on.
are on Unix or Mainframes (remember them)!!
funny enough at £dayjob all the Mac users are able to carry on, with the exception that our files are stored on OneDrive/SharePoint due to IT locking our machines w/ JAMF so we're screwed too.
time to head to the garage to put new spark plugs in our very analogue Triumph car
Yes, 2038 is going to be really annoying for everybody still running Mac OS X 10.5 and earlier.
And 2040 is going to be really annoying for everybody still running Mac OS X 10.13 and earlier.
So... me (assuming I'm not dead yet) and other vintage Mac geeks. I mean, it's not that the machines won't boot, they just won't have correct dates. But most folks aren't likely to be running pre-2017 Mac OS in 2040 or pre-10.6 in 2038.
(It uses 64-bit dates now. So there's a Y292,277,026,596 problem. Humanity is unlikely to have to worry about it.)
Depends whether the absolute datetime matters.
Most embedded systems really don't care what the date is. They might have a clock but it's only 12 or 24 hours.
Lots of systems use the date purely for display purposes. So the displayed date will be the wrong year, and it'll get leap years wrong.
A few systems have watchdogs or other subsystems that require now > then. Those will crash as it rolls over, and be fine after reboot.
The nearest traffic lights to me - please bork them. Everyone agrees traffic flows much better when they're out of action
I don't think I'd want anything from any of McDonald's machines but from what I hear perhaps it would fix them.
Sewage systems, yes might be a problem.
But would any embedded systems old enough to have a 32-bit time_t still be working by then?
> Early reporting from national media organizations misattributed the IT issues to Microsoft
Isn't this always the case? Even going way back in the 90s people would blame Windows for constantly BSOD'ing when in reality they decided to get the cheapest eMachines computer they could buy, made with capacitors that decided for themselves if they wanted to work that day or not.
Sure, let's pretend that we bothered to read the comment before replying and it wasn't us completely missing the point:
That we all KNEW Windows would BSOD at the drop of a hat or ill-timed sneeze, so it became the habit to blame EVERY problem on Windows, EVEN when it was so blatant a hardware issue as cheap crapacitors.
I find it both amusing and ironic that on the main page of CrowdStrike's website they have a pic of some SciFi baddy (looks like it is taken from some game art) along with the prophetic words:
"62 minutes could bring your business down.
That’s the average time it takes an adversary to land and move laterally through your network. When your data, reputation, and revenue are at stake, trust the pioneer in adversary intelligence."
Oh how right they are!
https://www.crowdstrike.com/en-gb/
Perhaps they should have replaced "an adversary" with "us".
Yes the root cause is Crowdstrike. But let's be clear on this, the OS is Microsoft's. Why do we put up with an OS that can be felled by one program with a problem? Why can't the OS have a fallback to automatically roll back during a boot loop?
We've probably all seen Windows 10 or 11 fail to boot and then say "repairing" and more often than not just fail. This isn't really acceptable any longer. Microsoft needs to up their game and start making their OS reliable and resilient. It should not be possible for security software to break the system like it seems to have done for countless desktops and servers around the world.
It has to be possible for security software to break the system -- that's what it's supposed to do (in specific and limited ways) when it detects malware.
Without the ability to filter low-level system operations it can't detect and disable malware.
With that ability, any software defects can be disastrous.
I can only refer you to Joe Tidy, the BBC "Cyber correspondent". He says They have "god-like" access to all the inner workings of an IT system for obvious reasons..
Mind you, I don't think Joe is really a techie. His next paragraph tells us that "End Point Protection" programmes have to be able to monitor the inner workings of computers. Back in the last century it was not unknown for reactionary UK crusties to try to insist on spelling "program" that way, but I don't think I've seen it for at least 40 years.
"But who watches the BBC any more?"
People who want their news as un-slanted as possible and as verified as possible. Yes, that makes it a bit boring (I go elsewhere to find out the rumours they're not telling me), but it is the most reliable and unbiased source we've got. And you – yes you, spitting your tea at the screen and laughing with incredulity – I challenge you to point me to a more unbiased general news source in the UK. It's not perfect by any means, but it's a start.
I expect downvotes.
I might raise bloomberg news as an alternative relatively unbiased news source.
But it is slanted towards news for the financial sector and even more boring than the BBC...
Turns out people investing large amounts of money want unbiased news, though it does need to cover what bias is being thrown around as well...
(N.b. The BBC is biased; It is slightly biased towards the CURRENT UK government; I.e. it has spent the last 14 years being pro-tory; It will spend at least the next ~4 years pro labour. But it has far, far less bias than every other news source in the UK. Generally it doesn't take much looking to realise BOTH sides are regularly alleging that the BBC is biased, which is by far the strongest indicator that it isn't...)
I appreciate your sensible response, and I mostly agree! Ad Fontes Media believes that Bloomberg is very slightly more left and very slightly less reliable than the BBC, but there really isn't much in it at all.
This is when someone will chime in saying that Ad Fontes Media is itself biased. For the record, much like the BBC, it gets shouted at from all sides, which is probably a good sign. Speaking to this person: if you'd prefer, you can refer to Alex Jones's chart, and I won't argue with you because – having been shunned by peers and teachers alike at school due to your inability to relate to people on a personal level – you've probably got a gun and a tendency to angry outbursts. Which clearly means you're right.
> I'm going to assume this is deep trolling.
It would be nice to think so, but coming from someone who believes that Putin is the greatest leader and statesman of the 21st Century, we are faced with a TruFan here.
When I were but knee high to a grasshopper a program was the new fangled computer thingy, and a programme was something you got at a theatre laying out the various acts. Similary, a programme was also a collection of activities that made up overarching initituative or agenda.
I blame the left-pondians for adopting simplified English as usual.
A programme can be applied to the various parts of a circus show... so perhaps it might be useful for the US to relearn this one following their election.
I blame the left-pondians for adopting simplified English as usual.
No, just blame Noah Webster, a man who hated the British so much that he tried to distance American English written after the American War of Independence as far from standard "King's English" as possible. Hence all the spelling mistakes. :-)
"I blame the left-pondians for adopting simplified English as usual."
About 40-some years ago when taking Computer Studies at school aged 15-16, we were taught the American spellings for program and disk and encouraged to use them in relation to computers so as to distinguish the terms from the more everyday meanings in English. It seems entirely logical at the time. Then the Compact Disc came along and screwed up the logic!!
The last time I saw it written that way was when a school teacher incorrectly "corrected" the spelling in an essay I did about what we'd done at the weekend, when I was maybe 11 years old?
It was enlightening in the sense that it taught me that teachers don't necessarily know what they are talking about, and that you shouldn't blindly trust authority figures, especially when they venture outside of their sphere of expertise. Whilst I'm not advocating going full flat-earther with the advice to question everything, you should be able to question anything you randomly feel like questioning that day...
That is rather disingenuous.
Anyone remember the unleaded fuel additive debacle several years ago where drivers would pull into (typically a supermarket) petrol station, fill up, then find their car majorly malfunctioned?
Yes the engine stopped working, but that is only because of the crap additive in the fuel. You can't blame the manufacturer of the engine for that, they way you are blaming Microsoft for CrowdStrike's monumental f*ckup.
A car engine is not an OS. It's so uncommon for petrol to be 'faulty' in that way that there were no safeguards built in. But if it was a common occurrence to get bad fuel, that could cause catastrophic engine damage, then the manufacturers could, if they could be bothered, put some failsafe in.
I'm challenging the notion that OSes have to be able to be broken by anti-malware. Why? Why can't they be designed *better* so that the chances of them being broken by a faulty drive, or faulty antimalware, are **LOWER**?
It happens far too often that Windows gets into boot loops after an update. The underlying OS is so full of holes, so flaky and so capricious in its nature that this can happen at all. These problems don't happen *despite* safeguards, they happen *because* of lack of safeguards and well-designed code.
The elephant in the room is Microsoft's poor quality control (ok - and Crowdstrike's). We shouldn't stand for this intrinsic crappiness.
Okay, here's a better explanation of why you are wrong...
An operating system would be pretty useless if it could only run software provided by the provider of the OS (although MS would probably love this, I shouldn't give them ideas).
So, third-party software is allowed. For obvious reasons, it can't go near OS functions.
Similarly, the OS should be able to run on a wide variety of hardware (unless you are Apple). It should not be incumbent on the maker of the OS to provide drivers for that hardware, so third-party drivers are needed.
Those drivers need to have low-level access to parts of the OS, so the OS needs to trust them. This applies equally well to any general-purpose OS, not just Windows.
Sadly, software flaws exist. Again, it would be poor form to not allow third parties to provide defence against these.
Low-level system and antivirus software needs to interact with OS functions at a low level, so the OS needs to trust them. Again, this applies to any general-purpose OS.
So, you have a situation where the OS doesn't normally allow access to, or interference with, some of its functions, except where it has to, in a limited way, for trusted third parties.
CrowdStrike is (was?) in this instance a trusted third party. Now, trust in computing is a whole PhD thesis topic of its own, but your argument essentially boils down to a claim that MS is in the wrong for trusting CrowdStrike, whereas the more nuanced argument would be that CrowdStrike is in the wrong for not living up to the terms of that trust.
Agree 100%, I've been writing low-level system software for a different OS for around 35 years which can do basically anything possible on the hardware.
We test it extensively before allowing it out to customers, but ours is an environment where our customers also understand the risks.
We also run with the elevated privileges for precisely as long as is required to achieve whatever function we require. Thus far, we've never had a problem but I do ask our QA folks to always run with everything turned up to the max - better we find any problems before our customers do. We really do not want to find ourselves in a CrowdStrike-type situation; knocking over the entire OS is beyond the pale. Fortunately the one I work with is considerably more robust.
Yes, like that. If my laptop was like a non-smart phone, as in it can run the three programs that the manufacturer came up with, with the small subset of supported protocols that they chose to put in, and if I needed anything else at all I had to buy new hardware to get it, it would be a pretty bad laptop.
Are you running a proprietary "non-smartphone" OS on a general purpose computer, such as a server in a data centre, or a work laptop?
Nope, didn't think so. You might as well be complaining that you don't need a CNC milling machine to hang a picture on your wall, because you've got a perfectly good hammer to bang a nail in with. That, good sir, is a non sequitur, if nobody was suggesting that you use said machine to do that with.
On the other hand, you probably could use your Nokia 3210 as a hammer.
"You can't blame the manufacturer of the engine for that, they way you are blaming Microsoft for CrowdStrike's monumental f*ckup."
But M$ are to blame. Third party code should never be able to crash an OS. Or run on an OS which allows that. CrowdStrike's crapware wouldn't be necessary if M$ shipped a decent OS that wasn't riddled with security vulnerabilities. CrowdStrike's epic, epic fail is unforgivable They deserve to be sued into oblivion for the inconvenience and consequential losses they caused. [They cost me thousands in cancelled/rebooked flights and hotels this weekend.] However, that's just a side-effect of the underlying disease.
Nobody should be running anything important on Windows. Ever,
Then again, engines in cars can take a lot of crap fuel with no problems - see some 3rd world countries, so while you may alway get 95ron or above, in some places it is well under 90 - and the cars still pootle around.
a really old disel should be fine no matter what - remember, his first design was to run on peanut oil
One man's freedom fighter is another man's terrorist.
Please explain the difference between an issue with "system reliability" and a "vulnerability".
Is the ability to bypass the login prompt in WinXp a system reliability issue, but a backdoor in the Unix login command a vulnerability that was found later down the line?
Is it possible that people's judgement of where blame lies comes not from an objective assessment, but in a preconceived bias?
Could it be, just maybe, that all software can have flaws in it, and those 1s and 0s that come from evil capitalist Microsoftland are indistinguishable from those that come from the Glorious Socialist People's Republic of Linuxania? (reductio ad absurdum here is merely for effect, and no political commentary is implied or should be inferred.)
Please explain the difference between an issue with "system reliability" and a "vulnerability".
One is a subset of the other. A vulnerability is a susceptibility to an external - usually malicious occurrence. Other causes of system unreliability could be all sorts of things from inadequate memory provision upwards.
I'd argue that failing to incorporate handling of insufficient system memory into your design is a vulnerability, especially if it could be exploited, for example, to cause a denial-of-service attack.
This is my point, the terms "system reliability" and "vulnerability" are pretty vague, and mean different things to different folk, they're certainly not technical terms that carry a universal, agreed upon, and strictly defined meaning, and they carry a fair amount of subjective bias, especially in a field as complex as security.
Because kernel maintainers are constantly adding new hardware... they are new kernels and they are only installed once you choose the kernel you want
Also you have the option of what level of updates
you want....
But if you are happy patching once a week you have at it.....christ would hate todo a roll back on m$ ...
Can you restart the graphical interface without rebooting the whole machine? For that matter, does it brick the machine until you boot into safe mode and delete a file?
Not saying it's ok for any program to be able to crash the GUI, but still better than Windoze where a USB-to-serial adapter can bluescreen the whole machine. (Yes, even in 2024.)
Thats runbish.....
modern gliphs are size agnolstic there may be a limit but 96 ain't one of them ....
Just done it and again at 500 points ....
Check your system, install or raise an issue it's on your side not liber....
Also page size amount of pages that you can have in a document
"...Thats (sic) runbish (sic)..."
Important lesson I've frequently wished I could pound into somebody's head (hence icon, would be clue-by-four were such an icon available) : "Works on my machine" is not the same as "there's not a problem".
It does work on my machine, and am sure it does on yours, too. I'll bet it works properly on >99% of machines; it's hard to imagine a bug of that magnitude slipping through otherwise. (*)
The failure on five machines is interesting, though. Are there differences between them? What's the common factor explaining why Ian sees this repeatedly and most of the world doesn't see it at all?
(*) I say that, but... it's happened often that I've fixed bug X that I've heard about from one user, mention it in release notes, and then hear from a few dozen people who ran into exactly that bug, sometimes years back, but had assumed I already knew about it.
"Like the way my Linux Mint session is instantly terminated if I set a font to 96pt in Libre Office or Abiword and zoom in once?"
You keep saying that. I just entered A at 98pt Liberation Serif in LO Writer at zoom 100% and zoomed right up to 400% with no adverse effect. This is Devuan/KDE.
Maybe it's systemd? [Ducks]
More seriously, is it a specific font, every font or just some fonts?
Crowdstrike runs on MacOS and Linux too. Malware can get by as nothing more than a userspace app suffering a supply chain attack. Even if the OS and the user are both perfect, corporate IT still has a checkbox to tick that everyone has protection.
It's IT's fault for not worrying more about the capabilities of data retention and malware detection software. These products can consume $4000 worth of every computer's performance or cause mass spontaneous data loss.
"corporate IT still has a checkbox to tick that everyone has protection"
The box-ticking culture! How about, instead of ticking boxes, we start out recognising, evaluating and mitigating risks. Is there a possibility of a supply chain attack on the O/S or 3rd party S/W? How do we mitigate that? Could we test on a sacrificial machine? Should we use some sort of threat detection S/W? If so, is there a a threat of a s supply chain attack on it etc.?
Genuine question I do not know the answer to....
Will Linux already do what you suggest?
If not then the dig at Microsoft is irrelevant, that is just the general hatred on The Register of anything Windows with the inevitable solution of Linux and Open Source.
You're buried a long way down in the comments. What was the suggestion you're querying?
FWIW Linux kernel upgrades usually leave one or more old kernels in place. The user will get a few seconds grace to bypass the default boot into the most recent kernel.
Some distros will just leave the last one in place and delete the one older than that, some will leave all deletion to the user. But the presence of, at minimum, the kernel you were running immediately before the upgrade means that you can go back to what is expected to be a known good kernel.
Also, the manual boot options include booting any of the available kernels into what would be the equivalent of Windows safe mode in which the system is running single user without starting any more than an absolute minimum of services. It still wouldn't defend against a situation where a bad update affected something outside the kernel which was essential to booting single user because either old or new kernel would pick that up. There is also the possibility o manuallyf issuing parameters to the kernel at boot time. All in all, although no OS is fail-proof here is a great deal more defence in depth than Windows has."".
AIUI one of the issues with the present situation wasn't just that the update downloaded a corrupt data file but that CrowdStrike's SW did not simply reject it and carry on* but crashed and crashed in such a way that it then blocked the rest of boot. That's a double failure for which the corrupt file was only a trigger. This goes against everything we were taught years ago - that problems that can be caught and handled should be caught and handled.
And, of course, don't release an update on a Friday.
" it's evident from the recommended "just delete it" that it the file wasn't essential to normal operation
** I should add that my experience is based on SysV usage - systemd based systems may be less or more robust.
The previous kernels won't help when there is a driver update, since driver updates are applied to old initramfs images.
Single user mode is not useful because the system has to boot first, and that means loading the drivers. Even if this weren't the case, users would never be able to reach that point simply because single user mode would be disabled in any environment managed by competent administrators. Otherwise, anyone in proximity of those machines has root access to them, which is irresponsible.
I can't remember seeing any but the new initramfs being rebuilt. It would negate the whole point of keeping the old kernels available.
And single user here (Devuan) wants a root password to bring up a root shell for single user or a Ctrl-D to continue normal boot. What are you running?
Basically, no. If I put in a program which works at kernel level, configure that program to start early in the boot process, and then do something in that process which takes down the kernel, having a Linux kernel instead of an NT kernel won't prevent that from crashing the system nor from making the recovery process annoying. There are some differences meaning that I might not have to run at kernel level for the same purposes, and then maybe my mistake will happen at a higher level and the boot will complete, but there is no guarantee that this will happen. Linux gives the user the ability to run software with very elevated permissions, enough to cause serious faults if that software is badly written.
"Why can't the OS have a fallback to automatically roll back during a boot loop?"
Remember when it was easy to get Windows to go into safe mode if you got stuck in a boot loop? Now, the only way into safe mode from a PC that can't complete the boot process is to power cycle it three or four time in the hope that the early part of the boot process will realise "something is wrong" and go into repair mode, from which you can try to get into safe mode. I wonder why they made this simple process so difficult?
I wonder if this whole shitshow will cause MS to pause their long-term "OS as a Service" plan and let us stay with a local OS on a local PC and not be forced into "going cloud" with everything? (Luckily for me, I went BSD some years ago, the only Windows PCs are wife's laptop and my work laptop. Wifes data is all backed up to my BSD server and the work laptop, well, that's not my problem :-)
I think the naswer to that, dredging up some vague memories from a long time ago, is that Windows was supposed to become stable and self-healing.
Microsoft, being Microsoft, did things the Microsoft Way. Which, as ever, means taking away the old method, whether they need to or not, irrespective of whether the new one is any good- or even works.
Everywhere you start running into problems these days you need to "upgrade" ... using Windows 10? You need to recycle the computer and buy a new one, but if you are using a much older version of Windows then you are probably just reading posts on El Reg about today's problems.
If you are thinking about your problems these days then review the corporate profits everywhere that are appearing continually.
I was sitting in my GP surgery's waiting room a few weeks back, while I waited for my appointment (it was well past the time of the appointment, I was waiting for quite a long time), and while I was there, workers from Iron Mountain were loading huge trolleys of document archive boxes with names written on the side onto a lorry.
The conclusion I came to was that they were archiving all of the paper patient records. I do not know whether they had already, or were going to be scanned or what was happening to them since, but I know that they are no longer at the surgery.
I don't have a reason to call or visit the surgery, so I don't know how their systems are coping. They barely functioned at the best of times, so if they are affected, I think it will be bad!
Iron Mountain fires is a premium service to selected clients... Here in Argentina they managed to get rid of a lot of evidence killing about 10 people that were trying to put down the fire.
https://www.computerworld.com/article/1517192/iron-mountain-fire-in-buenos-aires-kills-9-destroys-corporate-records.html
(the Wikipedia entry is only in Spanish).
consider this:
Windows PC borked, in a constant loop of BSOD
MSFT/Crowdstrike release a fix. Fix cannot be applied remotely to BSOD machines!
Monday morning y'all form an orderly queue for the IT bods to manually update 100s of thousands of PC at every company.
Pickpocket alert on Monday!!!
Bosses might demand IT work all weekend fixing the users machines, but good luck getting the users to turn up over the weekend to get them fixed. Most corporate users have laptops these days. I suppose it depends on the proportion who take their work laptops home, but with most employers still operating a level of WfH, I suspect that's a large proportion.
Yep. The root cause is really Windows. It should be resilient enough to cope with one badly written update, even if it is security.
In 2024 we should not be putting up with an OS that can fail so easily. It's part of the enshittification of everything in IT. And Microsoft's always been at the heart of it.
Can you name an operating system that is guaranteed not to fall over when someone with system level access changes stuff?
What i'd like to know is how an update fucking up that many machines made it past CrowdStrike's QA and change control processes and what a "channel file" is.
Bound to be a bad character in a file
Had that myself recently where we copied some data from one saas app to another and all looked good, but data was not mapping as it should
Looking, all looked correct - field a matched field b - until I took one of these entries and into notepad++ where I then saw an odd space character. Sorted that.
We added a bit of error checking for standard characters on the revised system
Wasn't there a recent crash in the airlines where this happened recently with an upload to something like flight plans ?
> Bound to be a bad character in a file
We have been told what the dangerous file is called and what timestamps to look for.
Has anyone thought to keep a copy, rather than just deleting it as they were instructed?
Does it by any chance read "Step 7 - this file to be overwritten with the output from Step 6"?
"Can you name an operating system that is guaranteed not to fall over when someone with system level access changes stuff?"
No, but can you name one that's guaranteed not to fall over when it changes stuff itself without waiting for someone with system level access to do it?
> when it changes stuff itself without waiting for someone with system level access to do it?
The OS didn't change itself, it was third-party code that updated its own third-party self.
And the person with system level access who installed the third party code also explicitly gave that code the authority to update itself.
Whether that is a sane thing to have done is what you should be questioning: should any OS allow a system level user to enable automatic updates?
If not, how do IT get the road warriors' laptops to keep up to date?
No, Windows cannot be expected to cope with a badly written security patch. For CrowdStrike to do it's job it has to be deep into the sensitive part of Windows.
The problem is still Windows however because such services as CrowdStrike should not be required if Windows was safe and reliable.
> The problem is still Windows however because such services as CrowdStrike should not be required if Windows was safe and reliable.
And if all the users were safe and reliable and never executed anything that go bad, including making sure that they never allowed anything to update itself. Which could be possible, if the PC is locked down hard enough. And users don't badger IT to let one through "because I must have the latest version of Notepad++ all the time (and I have convinced enough upper management as well - or I am upper myself, get on with it, serf).
Except for all those programs that don't need to be installed to run, don't even need any new executables that IT haven't already vetted: anything written in a scripting language.
CrowdStrike *claims* to be able to detect anything suspicious, like *all* the accessible files on your PC being overwritten or vanishing (encrypted, for example). If CrowdStrike is not up to its own claims, any other software that could do that trick would have to have the same level of access.
The name for a computer that is *so* locked down is "calculator".
Wonder if Croudstrikes heuristics found somthing shonky in m$ telemetery code and it chucked a spanner in loop....
Could be a big lawsuit on the horizion .....
Share prices tanking, fingers being pointed andl what not...
Feel for the folks that have to clean this up though...
Yeah yeah im going.... beer cos it's a nice day and did no need for a coat.
From CrowdStrike's website https://www.crowdstrike.com/blog/statement-on-windows-sensor-update
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. "
This is not a security incident or cyberattack. "
Sure it is. It has pretty much the same impact as a very severe cyberattack. Security includes boring little details like business continuity and risk. But from the article-
CrowdStrike Falcon – the vendor's flagship EDR solution trusted by organizations the world over – is to blame.
I think that may end up becoming past tense. From roaming around the news sites etc, this seems to have had a huge impact and massive costs. I've never used Crowdstrike, don't really like their politics and have heard some rumors that they may have expanded too fast. But I'm curious. As a 'flagship' solution, does it allow customers to test updates before mass deploying them? So how much of this may be down to their customers just setting auto-update on all their endpoints and hoping for the best..
Sure, apart from the active attacker having copies of the data and continuing to do even more damage. Not all bad incidents are the same, and this is different from a cyberattack in several ways. That doesn't make it good, but it's akin to saying that a car crash is exactly the same thing as falling down the stairs, because the injuries you received are basically the same.
>> This is not a security incident or cyberattack
> active attacker having copies of the data and continuing to do even more damage. Not all bad incidents are the same, and this is different from a cyberattack in several ways.
The point is moot.
If it were an attack, it would be CrowdStrike's data that leaked, they would be the ones suffering the continued problems.
But for the people who have been affected, they are seeing *their* systems failing. Was this due to an attack on CrowdStrike or "merely" their incompetence? Have CrowdStrike's files been copied or are they still safely on their servers?
Who gives a damn?!
The Worldwide damage would be the same, all the same systems would have crashed.
Will people be more trusting of CrowdStrike from now on because it *wasn't* an attack? CrowdStrike's procedures are still broken, whether it was "Procedure A: lock system" or "Procedure B: send working updates".
Even if you say "well, at least the Secret Sauce that secures the update process hasn't escaped and can't be used by Bad People to infect all those systems next week" - well, even assuming that there *is* a secure process in place, CrowdStrike have a hell of a lot of work to do to convince the users to keep it installed - and making a brand new, more robust updater, with the chance of a controlled rollback, sounds like a good place to start regaining trust.
"If it were an attack, it would be CrowdStrike's data that leaked, they would be the ones suffering the continued problems."
Supply chain attacks don't work like that. If it had been one, and it wasn't, then customer data would be at risk.
"Was this due to an attack on CrowdStrike or "merely" their incompetence? Who gives a damn?!"
Me. If the data I'm responsible for has been copied to an attacker's systems, I need to start dealing with it, and I need to start doing that right now. If it hasn't, then someone else needs to clean up the systems, and I would likely pitch in to help. Depending on whether it's an attack or a malfunction, my next steps are different, the situation for the users and customers is different, the likelihood of substantial damage to my employer is different, so I care. If you work in any area related to this, you should care too.
The term "Blue Screen Of Death" was previously unknown to the BBC Radio 4 presenters on the 'Today' programme when I turned it on at 8:30 this morning (I'm recovering from an operation, so allowed a lie in). I don't recall exactly, but I'm sure I experienced my first BSOD some time in the last millennium.
Just goes to show how partitioned we are in our own experiences we can sometimes forget that other people know almost nothing of our experiences, as we know the same of theirs.
I hope everyone has a great weekend, and the mess is cleared up quickly. As it is Friday, maybe have one of these ----> and relax bit.
All the best!
I've had several BSODS a day for the last two weeks (& stupidly I hadn't made a habit of regular backup images on this machine - No crucial data was lost though), remedial action included two reinstalls's (With one HDD nuking), swapping the GFX card & playing with combinations of RAM in the banks, updating\rolling back drivers.
It seemed to be stable last night while I watched performance monitor like a hawk, maybe tomorrow I'll throw the other stick of RAM back in if it's stable tonight.
On my new desktop PC I don't think I've ever had a BSOD. But it runs a fairly vanilla configuration with Win 10.
On the other hand my work laptop BSOD's nearly every week due to some issue with Lenovo's drivers, or the latest changed made to VirtualBox to brick my VM's or the host OS. And unfortunately CE means we have to apply all patches to all software and fix problems later (at least that's what the IT guy tells me. I'm only a peon engineer.)
Or perhaps it's their job to explain what industry terms mean to the subset of listeners who haven't heard it before.
That often means asking the "stupid questions", so the listeners don't have to.
In this case likely the oldest and youngest don't know. If a six year old was sat with their grandfather at breakfast, it's quite likely neither of them knew!
Be glad of it. You're not an expert in everything.
BBQ last Friday, someone insisted on a round of room 101, I proposed Lemmings, not the furry ones, but people/companies that don't think for themselves, they don't use the best thing for the job on an as needed basis, they just copy the crowd and use what everyone else uses.
Today it's Cloudstrike. Other days microsoft, snowflake, solarwinds, Okta, jetBrains, MOVEit so many lemmings jumping off cliffs with all the other professionals?
Let's not kid ourselves this is a Microsoft problem, if windows was secure we wouldn't need this deluge of security software. An OS should be able to eject an errant driver on boot, whether it enables a network connection is an administrators decision.
This post has been deleted by its author
No, you would not be correct. Read again. It's not Defender. It wasn't pulled through Microsoft. The central fact, and one that's usually in the second paragraph of most stories, is that if CroudStrike was not installed, you don't have a problem.
I'm not sure if this is another attempt to find a reason why this is actually Microsoft's fault or not, but you have critical facts missing from your model.
Having worked for an anti-malware provider in a previous life, though not on the frontline, I can tell you that reputation is gold dust. It's true for any commercial organization of course, but particularly true in the antivirus world. I only hope that when the internal investigations happen, it won't be a witch-hunt in which one or two drones are fired. Mistakes happen but that's why we have quality control, right? And that's why you fund your test environments thoroughly, right? That's why there's someone on the board who will own this, right? We shall see.
Yes, OK, but that isn't sold through reputation or you wouldn't be able to give it away.
In my opinion that is sold through lobbying, bribing, price manipulation, blackmail and flat out lying, and then entangling the customer as soon as he or she has made but the tiniest step towards it. You know, like a cult.
They may be crap at writing decent software (although after a couple of decades of no improvement you could be forgiven for concluding that that must be deliberate), but they're damn good at flogging rubbish.
> or you wouldn't be able to give it away
Microsoft do try damn hard to give it away* yet some of us still stick to the older, still working, one, or don't even unwrap the User Catcher's lollipop in the first place.
*at least until you realise just how annoying it is to not be able to "personalise" away some of the "features", so now you need a product key
We use CrowdStrike, and I'm currently doing business in Johannesburg (and yes, the winter weather is somewhat friendlier than our summer). The group of companies I belong to uses CrowdStrike and has been badly affected. Unfortunately (or fortunately), the corporate Dell is poorly, so I'm using my backup* Surface Pro.
Anon because I don't want to risk inadvertently identifying the group of companies I work for.
* I've learned the hard way to never travel to Africa without a degree of redundancy.
Part of the blame is certainly on Crowdstrike: if their content update breaks Windows with such high probability (if the probability were low only some parts of the world would crash) how come their QA didn't catch this?
The other part may or may not be on Crowdstrike: do they offer a protocol and recommend a change procedure that includes staging and testing? If not it's on them. If yes, then it looks to me that hardly anyone in the whole world (OK, in the part thereof that uses both MSFT and CRWD, on the basis of the observed data) implements a reasonable change protocol.
Mind you, EDR/XDR products typically require admin level access to the target machine, without it it's kinda difficult to fight invaders off (the R=response part, at least). And security updates tend to be quite time-sensitive, but that should be handled by the change protocol, at least at the crash/no crash and boot/no boot level.
Why do I hear Sarah Conner in my head ....
"By the time Crowdshite became self-aware it had spread into millions of computer servers across the planet. Ordinary computers in office buildings, dorm rooms, everywhere. It was software in cyberspace. There was no system core."
Critical systems do not fail because a person makes a mistake, but because insufficient controls fail to prevent the mistake. Dr. Johannes Ullrich
As a 40-year Unix/.../Linux user, I am enjoying (the rare English sunshine and) the Schadenfreude of the disruption to M$ users.
It has taken a while for me today to extract what actually happened from all the stuff on the news reports.
So someone in this company made a blunder and triggered automatic download to millions of Very Important Computers across the Globe.
The thing that was downloaded was not itself malware, but it caused enormous disruption.
Of course the malware groups in certain significant countries and the autocrats of those countries are taking notes.
All they need to do is to infiltrate their Nasty into this company, which will obligingly install it in Very Important Computers across the Globe.
I am surprised that nobody here has commented on how frightening this is!
"In today’s modern galaxy, there is of course very little still held to be unspeakable. Many words and expressions which only a matter of decades ago were consider so distastefully explicit that were they merely to be breathed in public, the perpetrator would be shunned, barred from polite society, and in extreme cases, shot through the lungs, are now thought to be very healthy and proper, and their use in everyday speech is seen as evidence of a well-adjusted, relaxed and totally un****ed up personality.
So, for instance, when in a recent national speech, the financial minister of the royal world estate of Qualvista actually dared to say that due to one thing and another and the fact that no one had made any food for while and the king seemed to have died and most of the population had been on holiday now for over three years, the economy had now arrived at what he called M“one whole jujuflop situation,” everyone was so pleased he felt able to come out and say it that they quite failed to notice that their five thousand year old civilization had just collapsed overnight."
https://jujuflop.yule.org/jujuflop/
HHGTTG, a bit of predictive sci-fi from the late, great Douglas Adams, perhaps? (Now, just how many shoe shops are there in the High Street ...)
We have some machines hit with this. The workaround is simple - boot to safe mode and delete a particular file. Trouble is, with Dell machines, we can't get to the Windows boot options, only the Dell ones (normal boot, boot to hard drive which is still normal boot, diagnostics which shows the hardware is fine). Tried F8, F10, F11, F12, BIOS, shutting it down twice while Windows is loading, etc.
How DO you get to Win10 safe mode from outside of Windows on a Dell?
Older wiser heads set up legacy (F8) Windows boot menu on their home machines (bcdedit /set {default} bootmenupolicy legacy - Or something similar).
There's possibly a way for force recovery options by powering off 2 - 3 times while windows is loading.
https://community.spiceworks.com/t/windows-10-bootmenupolicy-legacy/662183
Should an OS really be so fragile that a 3rd party driver can take out the whole OS?
There is an argument to be made that the kernel should handle this better, e.g. perhaps kill the bad driver / process / alert 'all the things' but crucially stay up - especially in critical server roles.
This is reminiscent of the HP printer update that I've moaned about on here prevously.Said update having removed the previous software, bar one .dll which for reasons unknown it couldn't remove ( and which couldn't be removed manually) aborted the install with no option to skip the said .dll and continue round it. Leaving the printer unusable on that PC. And it was the exact same version .dll so didn't need to be deleted and replaced anyway.
You're quite correct.
However, have you seen the quality of code produced by most keyboard-monkeys?
Recent example: load and parse a JSON file. No checking if the file exists, if it's readable, or if it contains valid JSON. And in a language that has an easy to use exception system, and functions for the aforementioned checks.
The problem is "it works" 99.9% of the time. Until it doesn't.
'E's not pinin'! 'E's passed on! This falcon is no more! He has ceased to be! 'E's expired and gone to meet 'is maker! 'E's a stiff! Bereft of life, 'e rests in peace! If you hadn't nailed 'im to the perch 'e'd be pushing up the daisies! 'E's kicked the bucket, 'e's shuffled off 'is mortal coil, run down the curtain and joined the bleedin' choir invisible!! THIS IS AN EX-FALCON!!
Readers of a certain age will not need an explanation.
Kudos to the register who, prior to Williams taking over, used to be an IT publication for not only arriving to the party once it was over (there was nothing in here for the first few hours) but also failing to explain what every IT inclined person in the world asked himself when reading the news:
What in God's unholy name is a "channel" file?
PS: some nonsensical mumbling about Russia something Ior other included in the article as well, for added pathetic effect.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
