Posts by Doctor Syntax 42030 publicly visible posts • joined 16 Jun 2014
"If you cannot code defensively to ensure third party services don't take down your product when they fail, you have no place as a software engineer or a software company."
Likewise if you can't code defensively to ensure your product isn't taken down by your own badly formatted data file.
The issue here is worse than that. It wasn't the kernel module itself that was replaced, it was a data file which triggered a bug that had been there all along. There was no good version to roll back to. It was entirely the responsibility of the kernel module or whatever it was to handle the bad data file.
"leaves you system running but with all the security turned off"
It's not a straightforward question. If the system is in some way compromised but apparently undamaged this is not good. However just crashing the system and making it unbootable deprives you of your best tool for diagnosis and repair - the system itself. The best course of action might be to boot into some sort of safe mode. The advised solution is indeed to do this although it doesn't seem ti be automatic. From what I've read here the limitation of this is that if Bootlocker is installed it would normally obtain the keys from the server but this isn't an option when booting to safe mode, they need to be entered manually which depends on there being a more accessible copy. It would obviously have helped if systems were able to obtain the server's copy of the keys when booting to safe mode. Of course if the server is also down because of this...
The issue, as always, is to try to anticipate problems at design time and set a goal of having as few problems as possible that can prevent the system getting into a state where it can at least act as a tool to help recover to the intended production state.
There are these old-fashioned things called IT departments. As this seems to be a product aimed at big corporates there's a fair chance their customers still have them. Not guaranteed these days, but a fair chance. The IT department does the test and makes the decision on behalf of its users - and does the roll-out. I suppose they could still roll out something they know will bork all the workstations on the grounds that it will keep out ransomware but at least it becomes a deliberate choice.
You're buried a long way down in the comments. What was the suggestion you're querying?
FWIW Linux kernel upgrades usually leave one or more old kernels in place. The user will get a few seconds grace to bypass the default boot into the most recent kernel.
Some distros will just leave the last one in place and delete the one older than that, some will leave all deletion to the user. But the presence of, at minimum, the kernel you were running immediately before the upgrade means that you can go back to what is expected to be a known good kernel.
Also, the manual boot options include booting any of the available kernels into what would be the equivalent of Windows safe mode in which the system is running single user without starting any more than an absolute minimum of services. It still wouldn't defend against a situation where a bad update affected something outside the kernel which was essential to booting single user because either old or new kernel would pick that up. There is also the possibility o manuallyf issuing parameters to the kernel at boot time. All in all, although no OS is fail-proof here is a great deal more defence in depth than Windows has."".
AIUI one of the issues with the present situation wasn't just that the update downloaded a corrupt data file but that CrowdStrike's SW did not simply reject it and carry on* but crashed and crashed in such a way that it then blocked the rest of boot. That's a double failure for which the corrupt file was only a trigger. This goes against everything we were taught years ago - that problems that can be caught and handled should be caught and handled.
And, of course, don't release an update on a Friday.
" it's evident from the recommended "just delete it" that it the file wasn't essential to normal operation
** I should add that my experience is based on SysV usage - systemd based systems may be less or more robust.
"Can you name an operating system that is guaranteed not to fall over when someone with system level access changes stuff?"
No, but can you name one that's guaranteed not to fall over when it changes stuff itself without waiting for someone with system level access to do it?
"corporate IT still has a checkbox to tick that everyone has protection"
The box-ticking culture! How about, instead of ticking boxes, we start out recognising, evaluating and mitigating risks. Is there a possibility of a supply chain attack on the O/S or 3rd party S/W? How do we mitigate that? Could we test on a sacrificial machine? Should we use some sort of threat detection S/W? If so, is there a a threat of a s supply chain attack on it etc.?
Please explain the difference between an issue with "system reliability" and a "vulnerability".
One is a subset of the other. A vulnerability is a susceptibility to an external - usually malicious occurrence. Other causes of system unreliability could be all sorts of things from inadequate memory provision upwards.
It depends on whether or not you take precautions. One might be to test before deploying, another might be to wait a day to see if any adverse reports roll in. I guess any Cloudstrike customers who adopted either approach won't be rolling it out today.
Is it too much to hope that when the dust settles legislators will start requiring that major infrastructure failures will, by statute, be followed up by an inquiry to determine what led up to the incident; decisions were made which impacted release of faulty S/W & so on. Bad decisions, especially those undocumented or made to cut costs would then lead to prosecution of those who made them.
Yes. If you have incremental backups you'll be fine. But my point, which I maybe didn't make plain, was that just restoring a pre-1900hrs backup will, on its own, take you back to the state of work then. It will take further action to recover subsequent transactions, whether by restoring incremental backups or by manual re-entry. If, however, data was entered, say from a website, and there was no separate backup of that then it will have been lost. Is there a chance of recovering information from email acknowledgements or did the restore overwrite outbound emails? The system may even start handing out order numbers duplicating those issued between backup and outage.
Restoring a backup image is only the start of getting back.
