Financial institutions told to get their house in order before the next CrowdStrike strikes The UK's finance regulator is urging all institutions under its remit to better prepare for IT meltdowns like that of CrowdStrike in July. The Financial Conduct Authority (FCA) said issues at unregulated third parties were the leading cause of operational disruption within Blighty's financial institutions between 2022 and 2023 …
This illustrates the problem environment created by CrowdStrike and "supported" by Microsoft's requirement that everyone buys new computers (which would have seen the same problem).
A potential solution would be for companies to move to open source operating systems like Linux and employ a few hundred technical software employees to update their own operating system. A development and unique support environment that could be made private and difficult to hack. And the companies employees would test the company operating system world all the time to keep the company safe and working.
Continuously employing a few hundred local technical software experts would probably cost less than buying and installing new computers all the time.
I agree, but as a company, employing a lot of programmers to make the environment the company needs then you'd be relatively safe (e.g. for Delta all flights completely monitored within Delta) and not trying to sell the operating system outside your company. That's pretty much the world in the early computing days before malware appeared once users everywhere were running a fully accessible operating system - back then companies were only trying to use their private software - not sell it.
This post has been deleted by its author
"Continuously employing a few hundred local technical software experts would probably cost less than buying and installing new computers all the time."
I think you either underestimate how hard and expensive it is to have that size of OS development team or how often businesses are buying new computers.
Some businesses do cycle out their computers a lot more often than they need to, but when they, for example, replace them every four years, it's not because Microsoft made them. Windows lifetimes are longer than that. Even if they're going to have their own operating system, they'll still need to replace broken or outdated hardware. Relatively speaking, the additional cost of buying a new laptop twice as frequently as you need to is small in budgetary terms. Other businesses, even those that use Windows, hold on to equipment until it breaks. They might incur a cost if they buy extended Windows updates this time, but in many previous cases, they wouldn't have. If you're only calculating financial cost, I think your numbers are off.
Having such a team would enable them to have several technical benefits to justify the much higher financial costs, but it could also get them into more unpleasant situations. By hiring that many people, they would be able to make a lot of customization to an operating system. If they embraced open source, they could deliver a lot of that to the wider community and join forces with similar companies. I worry that doing that would run against your theory that it is helpful to have "a unique support environment that could be made private and difficult to hack", which I also disagree with. If they were less careful, however, it might just mean that they have created a lot of systems that it's difficult to manage because none of the industry-standard tools work on them, so if someone does hack them, they might not be able to find out or recover as quickly as someone who can have software that someone else wrote. Having too much custom software also makes it harder to add new employees, either normal employees or ones to this OS team, because they'll need to spend a lot of time being trained or self-learning all the tools available. When management inevitably wants to speed things up a bit, they're likely to leave necessary things unused. Meanwhile, their competitors who do not have a software team because they bought more standard software would be able to do things much cheaper. It's not unusual to have a situation where you could achieve great things by spending more than you need to, but in most cases, people don't choose to do that.
"A potential solution would be for companies to move to open source operating systems like Linux and employ a few hundred technical software employees to update their own operating system."
Could you explain this a little more
On the one hand you say "open source operating systems" in the plural but "like Linux". That in itself needs explaining as it appears contradictory.
But then it seems to be the company's own operating system. I suppose there's a philosophical view that an open source operating system is everyone's own but I doubt that's what you mean. It sounds contradictory.
Oh, hang on, I've just re-parsed that. Maybe you're telling me that you haven't the faintest idea about using Linux and imagine it must have a gross upgrade system like that of Windows you envisage alt he employees would be updating the OS on their own PCs and they'd have to be some sort of technical expert to do that. Ah, bless.
The Linux version of Crowdstrike had the same problem.
Windows was only at fault insofar as it allows third parties to install things in kernel mode. Linux also allows this. MacOS doesn't, and therefore didn't have this problem. There are some immutable Linux distros that don't allow it, and they wouldn't have had this problem, but they are not widely used.
"Windoze was only at fault insofar as it was completely Micro$loth's unforced error."
FTFY.
This isn't a Clown$hit issue. This is a Micro$loth issue that was exposed by Clown$hit.
Clown$hit didn't hack their way in using some unsupported method. They did it exactly as Micro$hit told them to do it. Sure, they're incompetent too, but only because the HORRIFIC Windoze architecture forced them to be.
A lot of them do use open source OSes for servers. Even the organisations listed here: the London Stock Exchange was mentioned and its trading systems run on Linux.
IMO an in-house OS is going too far for almost everyone, and a lot of big businesses do use in-house software.
The problem is with end user devices, and the common software that runs on them.
I also think a lot of issues arise from a tick box approach to security and reliability. People aim to meet sufficient requirements so they cannot be blamed if it goes wrong. They are a lot less concerned about whether a system is actually secure and reliable. In many ways Crowdstrike did what it was supposed to - the management (and IT staff) of the businesses affected cannot be blamed because its Crowdstrikes fault.
I've setting up active/standby clusters for enterprise and academics institutions for more than a decade. XenServer / Proxmox / VMWare / choose your solution.
The standby node can be offline, D-1 or D-2, or even D-7.
If you don't know how to setup this, you don't deserve to be in the business. And recovery takes about 1 hour, sometimes 5 minutes.
If you think a person who doesn't document, manage, and know the state of their server is getting things right, I'm worried about whatever servers you manage. Yes, it means the software running on that server was correct, but that's no miracle. That's the case for lots of software, but not crashing is the bare minimum, not the entire goal.
I've seen many disasters with that going on. Yes, the computer concerned has three different strains of malware on it. Yes, the customer data is left unencrypted and could have been copied by any or all of those strains. No, there are no logs of what was done. No, there are no backups if the hard drive in that box fails. No, the software has no version control for the data it's working with. No, we have no idea who to talk to if we need anything different than what we have needed so far. But the Python 2.4 script that processed the input is still getting input and producing output in the right format (Excel 2003 files), so it's perfect. Why do you want to do anything to that box?
I once worked at a place that somehow lost 6 servers. Still on-line, but could not get access nor knew their exact physical location.
Even more stunning was the admin not knowing how to geo-locate things. Turns out the servers were about 500 miles away in an old stand-by surplus office.
But did anyone thank me for finding them? Of effing course not.
What's the penalty if they don't? If the fines are lower than the cost of making real improvements, and potential financial losses are mitigated by insurance (either commercial or implied "too big to fail" type of protection from the government, what's the incentive for them to do anything but carry on as before?
If the people on top don't face real sanctions (if not criminal, something like clawing back their last five years of salary and bonuses) they will have their accountants just the numbers and decide the risk is manageable.
We've already working on DORA compliance where I work, and if translate what it actually demands it's basically what you ought to have in place anyway in operational resilience (the 'OR' part of DORA) and decent, exercised BCM. So a well managed company only needs a few tweaks, but in Germany the government got cute and made that compliance a personal responsibility for the financial industry which created a whole lot of entertainment from my perspective as suddenly Board members of a bank can no longer duck their responsibility if things go sideways, which resulted in somewhat of a panic. Personal responsibility and jail time? Thar's so not done in banking!
Anyway, I'm digressing. As far as I can tell you're basically looking at something that only needs a different name because of BREXIT :).
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
