back to article Too late now for canary test updates, says pension fund suing CrowdStrike

CrowdStrike, after suggesting canary testing as a way to ensure it avoids future blunders leading to global computer outages, has been sued in federal court by investors for not using a phased approach in rolling out updates to customers in the first place. In what will likely be one of many class-action complaints against the …

  1. xyz Silver badge

    Is it ok to say...

    Fry the fucks?

    1. BasicReality

      Re: Is it ok to say...

      The lawyers? Yeah, I'm all in for that.

  2. abend0c4 Silver badge

    Hurting investors

    It's possible they had some original involvement, but for most shareholders the likelihood is that they didn't "invest", but bought someone else's lottery stake. It is possible for shareholders to influence the behaviour of companies, but most of them seem to find that too much of an effort.

    1. John Brown (no body) Silver badge

      Re: Hurting investors

      Yeah, suing themselves! The shareholders are the owners :-)

      1. Doctor Syntax Silver badge

        Re: Hurting investors

        And for that you got a downvote?

        It's beyond belief that there are commentards who can't grasp that simple fact of company law. The word company refers to the company of people who have come together to own it by buying shares in it. Unfortunately some shareholders also seem to overlook this simple fact. Sue the manglement - that's reasonable - but otherwise congratulations to the lawyers who got the shareholders to pay them (the lawyers) to sue themselves (the shareholders) to maybe get some of their (the shareholders) own money less the cost of two sets of their own lawyers. Two sets? Of course, as plaintiffs they're paying to sue and as members of the company they're paying to defend themselves against themselves.

        1. EricB123 Silver badge

          Re: Hurting investors

          "And for that you got a downvote?"

          I guess Kurtz reads El Reg.

        2. Nematode Bronze badge

          Re: Hurting investors

          Doctor Syntax:

          Not really. A certain LLM says "Shareholders do not "own the company" in the traditional sense of direct ownership of its assets and operations. Instead, they own shares, which represent a claim on a portion of the company's profits and, in some cases, a vote in company matters.

          Here are the key aspects of what shareholders own and their rights:

          Equity Ownership: Shareholders own equity in the company, meaning they have a financial stake in its success or failure. The value of their shares fluctuates with the company's performance.

          Voting Rights: Typically, shareholders have the right to vote on important company matters, such as electing the board of directors, mergers, and other significant corporate actions.

          Dividends: Shareholders may receive dividends, which are portions of the company's profits distributed to shareholders. However, dividend payments are not guaranteed and depend on the company's profitability and policies.

          Residual Claim: In the event of liquidation, shareholders have a residual claim on the company's assets after all debts and other obligations have been paid. This means they are last in line to be paid.

          Limited Liability: Shareholders are not personally liable for the company's debts and liabilities beyond their investment in the shares.

          In summary, shareholders own shares in the company, which give them certain rights and a financial interest, but they do not own the company itself in the same way that an individual might own a private business. The company's management, under the oversight of the board of directors, is responsible for running the company and making operational decisions."

          Me again: Said rights are stated in the prospectus (usually for new issues) or other definition/offer document. If the company fails in its duty to run the company properly and according to its share offer statement, it's perfectly valid, and a common thing, that shareholders sue the company for misrepresnetation, misconduct, whatever. Of course, such action will probably affect the share value, hence shooting shareholders' own feet.

          1. I am the liquor

            Re: Hurting investors

            Well I guess if an LLM says it, it must be true, right?

            "Shareholders do not "own the company" in the traditional sense of direct ownership of its assets and operations."

            This sounds like the LLM is answering a different question. If you asked whether the shareholders directly own the assets and operations of the company, then clearly the answer would be no. The company owns the assets and operations of the company. If the question is whether the shareholders own the company, then yes, the shareholders, collectively, own the company.

            "they do not own the company itself in the same way that an individual might own a private business.

            Surely an "individual who owns a private business" is a shareholder who holds 100% of the shares.

        3. Michael Wojcik Silver badge

          Re: Hurting investors

          It's beyond belief that there are commentards who can't grasp that simple fact of company law.

          Perhaps because it's by no means that simple?

          "Own the company" is a largely meaningless phrase for a publicly-traded corporation, since "ownership" is a broad concept and only a handful of its possible aspects apply to shareholders (individually or en masse). So this objection to shareholder lawsuits is sophomoric at best.

          More importantly, it doesn't make economic sense. There are at least two sets of conditions under which a shareholder lawsuit is potentially valuable to the plaintiffs. First, if the value of the company has been irreversibly damaged, then a shareholder lawsuit may extract more value from the failing firm, for the plaintiffs, than they'd get from stock ownership. It could establish the plaintiffs as preferred creditors who get a larger share of the proceeds when the corporation assets are liquidated. So it can be a hedge against complete loss in the event of failure.

          Second, if a proper subset of shareholders sue and win, they may transfer wealth from other shareholders to themselves.

      2. Snowy Silver badge
        Coat

        Re: Hurting investors

        Only in the in simplest case but company's have liability insurance and they will pay out if the company loses?

        Rather than suing the company they are suing the insurance company.

        1. Jellied Eel Silver badge

          Re: Hurting investors

          Only in the in simplest case but company's have liability insurance and they will pay out if the company loses?

          Rather than suing the company they are suing the insurance company.

          Possibly not, or the insurers at least will try very hard to avoid any payout. It's possible to insure against fraud by employees, but harder with execs. Plus there are claims for gross negligence which generally are uninsurable. And this claim includes some of the execs, so possibly going after their DOI cover, but that again doesn't usually cover fraud or negligence. I get the feeling litigation is going to end up a very expensive way of answering a simple question "Did you test the release before deploying it? Yes/No?"

      3. Teal Bee

        Re: Hurting investors

        Shareholders are owners... of their shares. They don't have any involvement in the day to day operations of a publicly traded company.

        Publicly traded companies play by very strict rules as a condition of having access to the general public's money.

    2. Doctor Syntax Silver badge

      Re: Hurting investors

      The original investors in your sense would have done soe in hope of a return, either by dividends or by being able to sell their share of the company to others. The existence of those willing to by shares in the after-market are ultimately responsible for the willingness of the original investors to invest at all. And they themselves are now investors because they now own a slice of the original investment and because they were prepared to invest in it. It's unfortunate that they haven't grasped the fact that, together with the other shareholders they are members of the company they're suing. They're suing themselves.

      I suppose it's just possible that this is a vehicle to get some of their investment back before it's swallowed up by customers' suits.

      1. Jellied Eel Silver badge

        Re: Hurting investors

        It's unfortunate that they haven't grasped the fact that, together with the other shareholders they are members of the company they're suing.

        I don't think that's really a problem. These shareholders are also the trustees of a pension fund, so have a fiduciary duty to the members of the Plymouth County Retirement Association. Plus this is a class action, so Plymouth County becomes the lead plaintiff in that action because it has standing, ie has been harmed by the $120-ish fall in share price. But having been through a few Ch.11s, twice as an employee (not my fault) and then as an advisor..

        I think this is the usual defensive/opportunistic class action. Opportunistic because if it wins, the lawyers make bank. Then defensive because once the law suits start flying, like Delta's claim for $500m, the risk of Ch.11 bankruptcy proceedings increases. Then if CrowdStrike's forced into Ch.11, the ordinary shareholders are usually wiped out. So investors joining the class action might mean they get paid something because the litigation continues under Ch.11 proceedings, unless it's settled. Then if they win, the awards are protected and survive bankruptcy, or can get settled during those proceedings.

        But as you say, it's one of those slightly strange things when the litigation can trigger Ch.11, so if shareholders don't join in, they risk losing their investment but if they do, they risk destroying their investment as well. Especially as shareholders also have a fiduciary duty to the companies they own. I think this is just one of those strange things about the Ch.11 process.

        1. Brewster's Angle Grinder Silver badge

          Re: Hurting investors

          Yeah, they're getting their elbows out to fight for a bigger slice of the remaining pie.

        2. S4qFBxkFFg

          Re: Hurting investors

          "Especially as shareholders also have a fiduciary duty to the companies they own."

          I don't claim to be an expert in investment law, especially in the USA, but this seems odd. What exactly are the shareholders' duties to the company?

          1. Yankee Doodle Doofus Bronze badge

            Re: Hurting investors

            I could be wrong, but I think only majority shareholders have fiduciary duties, so in this case, there would be no duty owed.

          2. Jellied Eel Silver badge

            Re: Hurting investors

            I don't claim to be an expert in investment law, especially in the USA, but this seems odd. What exactly are the shareholders' duties to the company?

            Theoretically the same as any other fiduciary, ie act in the best interests of the company they own. From memory (and IANAL) there have been cases where shareholders have banded together and tried to vote changes that have benefitted those shareholders, but not the company or other shareholders. It's also where I think there can be problems with Ch.11. So the usual vulture capitalism, where debt holders may be secured creditors vs shareholders, but accumulate enough debt and trigger a default & Ch.11. Then shareholders get wiped, bond holders get their debt converted into new shares and take control of the company. Then often flog those shares as part of a merger and end up with more than they paid for the debt.

  3. A Non e-mouse Silver badge
    Unhappy

    It's a shame the only people that stand a chance of suing ClownStrike are the investors and not the affected customers.

    1. A.P. Veening Silver badge

      According to the article one of their customers (Delta Airlines) is suing them and Delta has a pretty good chance of succeeding, especially with a top lawyer.

      1. tfewster
        Facepalm

        In which case the Crowdstrike shareholders should be suing Delta for devaluing the Crowdstrike stock?

        If 99% of Deltas computers were disabled by a virus because they hadn't had AV updates pushed out to them during the canary phase, I expect they'd sue in that case as well.

        Vultures circling vultures (with all due respect to vultures).

      2. Julian Poyntz

        Down for so long

        What gets me is the inferences that Delta's systems were down for a long period of time. Makes me wonder if that is true, and indeed,what kind of DR / snap shotting they have (unless it is all tin).

        Maybe that inference is wrong and their problems are down to the sheer number of planes being in the wrong location - but I always that is a bit of an odd one. Surely they have replanning systems somewhere for this kind of scenario where they can "reset", tell the system where specific planes are and it can make best use of the planes to get the airline company running again (mostly)

      3. Charlie Clark Silver badge

        I suspect Delta's own suit is to deflect from their obvious own deficiencies. Not sure they have grounds either in law or in the T&Cs for a successful case. I guess we'll see.

        1. Michael Wojcik Silver badge

          Yes, it's hard to see how Crowdstrike wouldn't paint Delta's long recovery time — in contrast with that of most other victims — as evidence of Delta's own incompetence. I can see that playing well to a jury, if Crowdstrike go for a jury trial (though that'd be a gamble), and possibly to a judge as well.

          I think Delta's chances are overrated by some commentators, and frankly hiring Bois looks to me like posturing, a projection of strength in the hope of getting Crowdstrike to settle and deflect the attention of Delta's own shareholders and board. "We're so sure it's not our fault that we hired an expensive lawyer!" The best defense, and all of that.

        2. Madre O'Fender

          Delta's most obvious deficiency was being a MS-only shop, followed by being suckered into deploying CS everywhere.

      4. Michael Wojcik Silver badge

        Delta has a pretty good chance of succeeding

        Do they? I'm not so sure.

        I think Crowdstrike have demonstrated remarkable incompetence in this case, but actually winning this sort of suit has historically been a long shot. Crowdstrike might decide it's cheaper to settle, but I'm not making any bets.

    2. Madre O'Fender

      Yeah, it's a Falcon mess. Sorry, had to get that out there. Better out than in. But I don't expect I'm the first.

  4. Flywheel
    Facepalm

    WTF did I just read?

    "pushing changes to a small subset of users to see how it goes"

    Wot?! Can't they afford a test lab? Or maybe use the C-suite's personal computers then roll it out to the shareholders personal computers.that'll do it!

    1. spireite Silver badge

      Re: WTF did I just read?

      I suspect they may not get the chance

      It's highly likely they'll be sued out of existence, and the carcass picked up by one of the AV names.

      1. Charlie Clark Silver badge
        Stop

        Re: WTF did I just read?

        Under which laws will they be sued? And by whom? It's telling that the first class action we see is from investors, not customers, because software in the US is generally exempt from product liability.

        1. Michael Wojcik Silver badge

          Re: WTF did I just read?

          I wish whoever keeps downvoting this would explain what evidence there is for the probable success of a suit against Crowdstrike. All I've seen so far is a lot of unsubstantiated hope.

          People are angry at Crowdstrike, and they had patently terrible practices. Neither of those are slam-dunks when this gets into a courtroom.

        2. Zippy´s Sausage Factory

          Re: WTF did I just read?

          We don't know what was in the contract, and that is the important bit. We probably won't see that until discovery (and even then they'll probably not release it to the public). Big customers, remember, usually ask for specific contract terms that might now be turning round to bite CrowdStrike. Just because software is "generally" exempt doesn't mean that that general exemption necessarily applies in this specific instance.

    2. Anonymous Coward
      Anonymous Coward

      Re: WTF did I just read?

      They can spin up a Windows VM for testing, but the real cost would be hiring human QA testers to take responsibility for making the calls to delay a release. Other tech companies hire QA, but to maximize profit for the next financial quarter, you need to cut down on expenses like paying for employees. This is the strategy for optimizing for the short term over the long term.

    3. ecofeco Silver badge

      Re: WTF did I just read?

      I've said it before: to CxOs, I.T. departments should not even exist. After all, their vendors promised their services and products would be 100% trouble-free!

      This is not even snark, just pure sad, sad truth. And even worse, I've been prevented form holding some of our vendors feet to the fire on the contracted SLAs! Which told me obvious kickbacks were now obvious. And other fraudulent shenanigans between vendor and my employer.

    4. T. F. M. Reader

      Re: WTF did I just read?

      Can't they afford a test lab?

      I am guessing they probably had an additional flaw in their process/CI/CD/whatever, such as not verifying that what they are pushing to the world is the same thing their QA approved. At least check the hash or something?

      This is not instead of small yellow birds but in addition to them.

      1. Doctor Syntax Silver badge

        Re: WTF did I just read?

        I don't think checking the hash would necessarily work. Create a crap file, calculate its hash and all the hash will subsequently confirm is that it's still the same crap file. It needs actual validation of the contents of the file, for instance, look to see if the memory it's about to access is legal and reject it if log an error message instead of going ahead.

        1. yoganmahew

          Re: WTF did I just read?

          Read the comment again, the hash is to confirm that what's pushed is what QA approved.

          1. Brewster's Angle Grinder Silver badge

            Re: WTF did I just read?

            Their QA approved this. That's not in doubt.

            1. Michael Wojcik Silver badge

              Re: WTF did I just read?

              Exactly. They've already admitted that "QA", such as it was, of channel-file updates involved running them through an analysis program which looks for certain types of errors — but not, as it happens, all of them. It did not (and still does not, as far as we know) include actually testing the system.

              Crowdstrike have explained that they do not actually try to run their own software before pushing it out.

            2. Anonymous Coward
              Anonymous Coward

              Re: WTF did I just read?

              Existence of testing does not mean they have QA testers, because a type of partial testing called unit testing is done by developers. Developers write unit tests to test their code, which are typically incorporated into a stage of an automated DevOps pipeline that blocks the release if the unit tests fail. CrowdStrike's public statements indicate they don't have QA testers for the channel files, only unit testing.

              See:

              https://en.wikipedia.org/wiki/Unit_testing

              1. Teal Bee

                Re: WTF did I just read?

                They may have integration tests as well, which are written by the same developers who write the software and the unit tests.

                One can't generally expect a developer to write adversarial tests against their own code. If they knew which corner cases to test for, then they would have written the original code to account for these cases instead of wasting time writing extra tests.

                Unit tests are crucial when making changes to existing code, but they are useless in catching bugs. For that you need a QA team.

  5. Alex 72
    Coat

    I hate to defend Microsoft...

    Unless you are apple the diversity of hardware, bios, OS and other configurations make a test lab difficult even if you can afford it trying to ensure that you have every possible configuration represented and the updates that will be applied by other vendors like Microsoft if you are CrowdStrike or OEMs like HP, Dell, Lenovo... if you are Microsoft have been tested with the patch you are putting out is a large undertaking, and a large part of the reason even when hardware vendors are onside getting full compatibility for new hardware in free open source software maintained by volunteers can take time. So whilst Microsoft have dumped a number of dud releases on us they have not quite messed up like this and being the vendor of a whole OS and security product for the same OS their record doesn't look so bad compared to the Crowd now.

    Still given the over 660 million shareholders and a major fraction of the global population being customers, over 200 billion in revenue in 2023 one might expect Microsoft to keep improving.

    1. Anonymous Coward
      Anonymous Coward

      Re: I hate to defend Microsoft...

      Sorry, I disagree, you can't give Microsoft a pass like that, exactly BECAUSE they have the resources to do it right - all they do now is pass the burden to the customer.

      First off, hardware diversity is something that you can address by layering the software. It's exactly because they sought to give their own software a competitive advantage that they broke that layering, so issues there are their own fault.

      Secondly, their code is brutally insufficient, they got lazy through Intel using that to flog new, more powerful gear.

      Thirdly, good testing would have avoided the lack of functionality, missing features and other issues that bedevil their 'new' version of Outlook and Teams - no hardware dependency there, it's good old UI and software design. Instead, customers were forced to waste massive amount of staff time yet again working around problem created by practically every new MS software release ever.

      I've said it before and I'll say it again: if anyone was honest enough to also consider the OPEX expenditure associated with the use of Microsoft products they would not have gotten far.

      1. Anonymous Anti-ANC South African Coward Silver badge
        Coat

        Re: I hate to defend Microsoft...

        In contrast to Microslop code, OS/2 is still as snappy as ever.

        Sadly, it is more of a niche OS than a mainstream one.

        Seems the higher the windows version, the more bloat...

        Time to ditch the bloatedness for something more leaner and meaner...

        off to ye pubbe to drown my sorrows ====>

    2. Anonymous Coward
      Anonymous Coward

      Re: I hate to defend Microsoft...

      Then DON'T try to defend Micro$hit. This is ultimately entirely THEIR fault.

      They wrote an OS that allowed the ClownShit garbage to break booting at a low enough level that you needed a ButtLicker recovery key.

      They COULD have locked out rootkits. They tried to blame EU rules, but all they had to do was remove their own products from having access at that low a level - which they obviously should have done.

      They messed up, and messed up bad. It's not at all unreasonable to blame Micro$hit for bricking ALL of those computers.

      They don't improve. They get worse.

      1. Anonymous Coward
        Anonymous Coward

        Re: I hate to defend Microsoft...

        Christ, can you show me on the dolly where the nasty man hurt you?

        1. Anonymous Coward
          Anonymous Coward

          Re: I hate to defend Microsoft...

          Isn't it obvious.

          The wrists/hands and ankles where the nails were driven in, the wound in the side of the abdomen, and the scratches from the crown of thorns.

          Oh. "Christ" was an expletive. Sorry.

      2. Anonymous Coward
        Anonymous Coward

        Re: It's not at all unreasonable to blame Micro$hit for bricking ALL of those computers.

        It really is. They can't police every vendor and if they did you'd all be moaning about walled gardens and how you have to be free to do what you like with your own devices.

        Turn on a fucking dime you lot.

      3. VirtualizationGuy

        Re: I hate to defend Microsoft...

        I agree with the general thought, but Crowdstrike was only able to run at the kernel level because of a mandate by the EU. Admittedly, the kernel model is anticompetitive in that it only allows Microsoft created/approved services to run in ring 0 and does not have any other ring that can see all of the services that need protecting. Microsoft can and should be held responsible for that part of this issue.

        1. Charlie Clark Silver badge
          FAIL

          Re: I hate to defend Microsoft...

          No, the EU did not mandate kernel access, it stipulated merely that other vendors of antivirus software should have the same access as Microsoft's own products. There never was a need for kernel-level access but Microsoft was using it to give preferential treatment to its own products. Typical Microsoft and correctly slapped down.

          1. Michael Wojcik Silver badge

            Re: I hate to defend Microsoft...

            Anti-malware doesn't need to run detection at kernel (or some higher-privileged) level, but ultimately response has to — at least the portion that blocks hostile actions. Otherwise privileged user-mode programs could bypass it.

            And similarly it has to load as soon as possible at startup, or you have a race that could allow malware to bypass it then.

            Running in an intermediate ring (if Windows provided one) doesn't help, because it would still be unsafe to continue if there were a critical failure in the intermediate ring.

            Doing detector configuration in kernel mode is stupid, but that's on Crowdstrike. Nothing about Windows requires that. They could parse in user mode and then push valid rules down to the detector.

            They might even be able to do detection in user mode using ETW, as others have suggested, though I'm not entirely convinced that's feasible and comprehensive, or how well it would perform.

            1. gnasher729 Silver badge

              Re: I hate to defend Microsoft...

              There was no problem with malware detection. The problem was a stupid little bit of software crashing while reading a configuration file.

      4. Anonymous Coward
        Anonymous Coward

        Re: I hate to defend Microsoft...

        Here's a tip: avoid the really-not-cute replacing of the 'S' in Microsoft with a dollar sign and in general the name calling and stick to the facts in your argument and you could have the chance of a good debate.

        I know it's appealing to express yourself this way, but it creates two problems:

        - you don't establish a good basis for a discussion, but worse ..

        - .. this approach muddels your own thinking.

        You effectively radicalise yourself by expressing yourself in this way, and that creates a mental bias that is not going to do you much good. Keep an open mind.

        I too am absolutely no fan of Microsoft, but I also know there is absolutely no way I can steer people towards a better approach if I start off ranting at them with foam on my mouth - it will just make me look like an idiot. People who stay calm and avoid zealotry have far more impact (although I sometimes pretend to be one for the other side of a debate just to liven things up, but that's a conscious decision to either poke fun, or try out new arguments or generally be annoying :) ).

        Now try again.

      5. Anonymous Coward
        Anonymous Coward

        Re: I hate to defend Microsoft...

        Don't forget to sue Red Hat, they also got issues with CrowdStrike products earlier this year...

    3. Doctor Syntax Silver badge

      Re: I hate to defend Microsoft...

      Ditto but in this case it wasn't Microsoft's update, it was Crowdstrike's, delivered by Crowdstrike's channel, not Microsoft's. It was Crowdstrike's responsibility to test before release and theirs alone and doubly so because it was applied automatically so it would be difficult for customers to test for themselves.

      1. Anonymous Coward
        Anonymous Coward

        Re: I hate to defend Microsoft...

        Hmm - ish.

        On a tactical level it's Crowdstrike.

        On a strategic level it's Microsoft because it's their bad security that (a) made things like Crowdstrike necessary and (b) allowed Crowdstrike so deep access that it could kill the whole machine.

        Neither are without blame here IMHO, one for bad testing, the other for being the root cause.

        If we let the root cause get away with it this is pretty much guaranteed to happen again.

        1. Michael Wojcik Silver badge

          Re: I hate to defend Microsoft...

          (a) is poisoning the well. There are security issues with Windows, such as its large attack surface due to Microsoft dumping everything they think of into it, and legacy cruft like NTLM; but there are security issues with all OSes, and particularly with users. And Windows is a very large target. EDR would be necessary (under any reasonable threat model) for network-connected systems even if the Windows security situation were much better.

          (b) is nonsense. Any kernel-mode driver can BSOD, and the response function of EDR, at least for function blocking, has to run at a level more privileged than anything in user mode, or it could be bypassed by privileged user-mode programs.

          Microsoft potentially could have looked more closely at Crowdstrike's driver and told them that the architecture was unacceptable and rule parsing needed to happen in user space, and denied them WHDL approval until that happened. We can blame Microsoft for not putting more constraints on drivers in general; Linux does a somewhat better job of policing kernel-mode code, and Microsoft certainly has the resources to fund it.

          But it's sophomoric to suggest that the problem here was "Windows security sucks" (true but irrelevant) or that "Microsoft shouldn't let Crowdstrike run in kernel mode" (false).

    4. gnasher729 Silver badge

      Re: I hate to defend Microsoft...

      Known fact is that crowdstrike created a configuration file that made crowdstrike ‘s configuration file reader crash when it tried to read the file. Everything else was just a consequence of that crash.

      Such a reader should be written and reviewed to make sure that it’s not going to crash, whatever it tries to read. That’s how it works everywhere else. And that can be done without looking at every possible input. Actually it must be done _without_ looking at snh specific inputs.

  6. martinusher Silver badge

    Unfortunatly your canary is really a (dead) parrot

    Crowdstrike might have figured out that they really need to have organized testing but its really too late. This fiasco calls their entire software engineering capability into question -- until proved otherwise what they appear to be vending is a bright idea that was sloppily implemented backed by a first class marketing and sales effort.

    1. Woodnag

      Re: Unfortunatly your canary is really a (dead) parrot

      Considering the diversity of hardware setups around the world that crashed, the update was likely not tested on any real-life machine at all, just pushed out.

      1. Michael Wojcik Silver badge

        Re: Unfortunatly your canary is really a (dead) parrot

        Yes. There is no hardware configuration on which it wouldn't have crashed. It's fundamentally broken.

    2. Anonymous Anti-ANC South African Coward Silver badge
      Joke

      Re: Unfortunatly your canary is really a (dead) parrot

      Monty Python skit incoming...

      1. DoctorPaul

        Re: Unfortunatly your canary is really a (dead) parrot

        Beat you to it with my "dead falcon" comment last week:-)

    3. Doctor Syntax Silver badge

      Re: Unfortunatly your canary is really a (dead) parrot

      "what they appear to be vending is a bright idea that was sloppily implemented backed by a first class marketing and sales effort"

      Sadly yu could say this about too much of the industry..

      1. Michael Wojcik Silver badge

        Re: Unfortunatly your canary is really a (dead) parrot

        And most of the rest is terrible ideas that are sloppily implemented.

  7. jdiebdhidbsusbvwbsidnsoskebid Silver badge

    validated, tested, and certified

    Sounds like weasel words with built in get outs from crowdstrike.

    Validated: is it the right product? No reference to quality.

    Tested: anything can tested but what tests? (and did it even pass?)

    Certified: according to what?

    Could pass all those checks and still be a buggy thing that brings down global IT.

    1. Michael Wojcik Silver badge

      Re: validated, tested, and certified

      We know what tests. Crowdstrike have described the testing process. It's been discussed extensively. Maybe do a bit of research before asking the question?

      And, yes, it was insufficient and ill-conceived.

    2. Anonymous Coward
      Anonymous Coward

      Re: validated, tested, and certified

      I remember one old version of ISO certification document for quality, where it was asked to have a QA process in place, but nothing about following it...

  8. david1024

    Well.....

    There were/are configuration options that would have delayed the rollout and let the customers perform their own canary tests. Recommended or not, they were not exercised by the victims' IT teams. This is a good reason to TEST TEST TEST and stop firing the folks that would have prevented this from happening at your company! The vendor is not the one signing up for the risk here... it is the plaintiffs. I don't see lawsuits going much of anywhere. But maybe the point is only to bleed Crowd strike a little?

    1. Richard 12 Silver badge

      Re: Well.....

      It seems that those things did not, in fact, exist.

      It appears that it was only possible to delay engine changes.

      It was a data update that trashed the place, and it is pretty clear that said update had never been loaded into a running system anywhere before it got deployed to the world.

      Delta will probably win their lawsuit, the pension fund is quite likely to as well - they're both based on the same claims CrowdStrike made that have been proven false by events.

      Neither of them will get much actual money though, because CrowdStrike will be bankrupt.

      1. Michael Wojcik Silver badge

        Re: Well.....

        Right. The delay options didn't apply to channel-file updates.

      2. gnasher729 Silver badge

        Re: Well.....

        So the developer who created some new anti-malware code and modified the configuration file to enable his fix never used the configuration file reader to read that configuration file. There was a too supposed to test configuration files that was reportedly never run. And it is possible that the configuration file was 100% valid and it was just the reading software that crashed, so verifying the configuration file might not have found any problem.

  9. Manolo
    Flame

    If it's mission critical..

    ... don't run it on Windows.

  10. Inventor of the Marmite Laser Silver badge

    Can we have a popcorn icon?

    1. ecofeco Silver badge

      I second that!

      1. Anonymous Anti-ANC South African Coward Silver badge

        Thirded!

    2. Doctor Syntax Silver badge

      And a canary icon.

      1. DoctorPaul

        And a dead parrot

  11. BasicReality

    As always, bring in the lawyers.

    It was an accident. Shit happens. No matter the testing this stuff happens. I'm a developer for a large company. I've seen people make updates that worked fine in testing, but still broke the production environment. Testing environments are good and catch a lot of stuff, but they still simulate what actually happens in production. Sometimes stuff breaks. I hope they throw out all these lawsuits that are coming.

    1. Paul Crawford Silver badge

      It was not an "accident" but rather the result of incompetence at multiple levels - not testing properly to allow an borked update out, and not having any sort of phased deployment to catch that before the sh*t hitting the fan becomes a sewer hitting the fan.

      1. Strahd Ivarius Silver badge
        Devil

        It was a phased deployment: people in Asia got impacted before people in Europe, when they had CS Falcon on their laptop.

        Unfortunately, it was not the case for the servers...

        1. Falmari Silver badge
          Pint

          Good one :)

          Made me smile ---------------------------have a beer --------------------------->

    2. Anonymous Coward
      Anonymous Coward

      No, breaking one or two machines (or even 50) in your test ring is an accident. Making 8.5 million machines inoperable and causing 5.4billion dollars worth of damage is negligence, or to put it in more polite terms, a spectacular cock up.

      Lessons should be learned.

    3. Falmari Silver badge

      @BasicReality "I'm a developer for a large company. I've seen people make updates that worked fine in testing, but still broke the production environment."

      Only because the test plan failed to cover that production scenario.

      "Testing environments are good and catch a lot of stuff, but they still simulate what actually happens in production."

      Meaning you've "seen people make updates that worked fine in unit testing" and then deploy the updates to production without system testing the updates. What CrowdStrike did is no different, they deployed the update to channel without any system testing.

      If CrowdStrike had spun up some Win vms running Falcon before the channel went live that would test the upgrade on the system. Even a single vm would have blue screened when getting the update. Instead of 8.5 million machines.

      "Sometimes stuff breaks."

      And testing prevents stuff breaking being released.

    4. Anonymous Coward
      Anonymous Coward

      Shit doesn't "happen". There's always a cause though it may be hard to peel away the layers.

      There's a large industry of folk who chase down events. My current fave is the Buncefield incident (but only 'cos I was involved in a small way with some of the steps to mitigate against it happening again.)

    5. gnasher729 Silver badge

      There is no “production” environment for this software. It runs on random computers that people buy from random sources.

  12. Free treacle

    Not the kind of lawsuit I was expecting

    They only invested in the firm and are sad their investment depreciated, not impacted by the outage in some way? Stocks can go down as well as up, right? Maybe I just need a better broker...

    1. Doctor Syntax Silver badge

      Re: Not the kind of lawsuit I was expecting

      Not expecting shareholders to sue themselves? This is the US. It happens all the time.

    2. VirtualizationGuy

      Re: Not the kind of lawsuit I was expecting

      The lawsuit is for material misrepresentation. CrowdStrike's matching update process to their SEC reporting will require extreme pretzel logic to avoid a finding that they lied about how they do business. This matters because these statements are used by investors to determine the risk and future growth of a business.

      1. Nematode Bronze badge

        Re: Not the kind of lawsuit I was expecting

        Goodness, someone actually understands how shares work! Good point, well made.

  13. Anonymous Coward
    Anonymous Coward

    Three points

    1) Regarding this shareholder lawsuit, I don't expect it to actually get very far. Why? "There is no such thing as bad press." Given that my company tapped the brakes for about three days before going ahead to implementation, I expect their stock to be UP in three months BECAUSE of this incident.

    2) When you're talking about millions & millions of servers, you don't to canaries, you do rings. Each ring as ~5x as many servers as the one before. For your larger customers, you force some of their systems into the earlier rings. Basically, customers set the order for the rollout.

    3) Their password requirements contradict NIST. And they don't tell you, but "+" is not allowed. Which means that they are storing the password somewhere.

    Deep, deep cultural failures here.

    1. Anonymous Coward
      Anonymous Coward

      Re: Three points

      Wait, WHAT?

      Your company didn't have ClownShit before, ClownShit fucked up millions of computers, and NOW your company is installing ClownShit software?

      Find a new job. That's some serious industrial grade stupidity going on there, if they'll do something THAT stupid, they'll be going out of business soon. Get something else on your resume before it's tainted with whatever soon to be failed company you work for.

      1. Michael Wojcik Silver badge

        Re: Three points

        So much wishful thinking around the Crowdstrike disaster. Have any of you people paid attention to IT history?

        Companies fuck up hugely all the time. More often than not they get away with it. Sometimes they're rewarded.

        Being angry doesn't make you omniscient, and the IT industry is neither meritocratic nor just.

  14. Pascal Monett Silver badge

    "We believe this case lacks merit"

    Boilerplate response. Of course you do. Unfortunately for you, just about everybody else probably thinks is has merit, and I'm guessing the judge will too.

    That said, I think there should be a law stopping shareholders from sueing their own company. You're not happy with the company's performance ? Then sell your shares, take the loss and go invest in a better company.

    1. Richard 12 Silver badge

      Re: "We believe this case lacks merit"

      Companies are not permitted to deceive their (potential) shareholders. That's actual law, and an absolute necessity for a functioning market.

      However, you've only got standing if you took a loss because of a deception.

      So if shareholders cannot sue, then companies can lie with impunity - and the entire stock market collapses within about 39 seconds.

      1. Doctor Syntax Silver badge

        Re: "We believe this case lacks merit"

        "Company" is a collective word for people. Who are the people who comprise the company?

        The directors shouldn't deceive the shareholders. The manglement shouldn't deceive the shareholders. It makes no sense to say the shareholders shouldn't deceive themselves at least not collectively and in the legal sense.

      2. Will Godfrey Silver badge
        Meh

        Re: "We believe this case lacks merit"

        "So if shareholders cannot sue, then companies can lie with impunity - and the entire stock market collapses within about 39 seconds."

        I'm not entirely sure that would be a bad thing.

        1. John Riddoch

          Re: "We believe this case lacks merit"

          The underlying concept of the stock market is fine; it allows people to invest in large companies (too big to exist as sole trader/partnerships) with limited liability (the most you can lose is your investment) and lets the shareholder trade their investments. The issue is how it's been abused over the years by people extracting money from it, between shorting stocks, microsecond transactions, etc, etc.

          In terms of "not lying", most of the theory of capitalism and economics relies on "perfect information", which is why in practice it's horribly broken, but restricting the lies is intended to help.

      3. A.P. Veening Silver badge
        Coat

        Re: "We believe this case lacks merit"

        So if shareholders cannot sue, then companies can lie with impunity - and the entire stock market collapses within about 39 seconds.

        For obvious reasons I make that 42 seconds.

        Mine is the one with the HHGTTG in the pocket.

  15. nichomach
    Facepalm

    CS: "We believe this case lacks merit"

    Investors and clients: *point to millions of screen caps of goosed computers*

    CS: "We believe this case lacks sufficient merit..."

    1. Michael Wojcik Silver badge

      There is a vast gulf between "Crowdstrike sucks" and "Crowdstrike is legally liable".

      As others have pointed out, this case might (might) be viable on misrepresentation grounds — but holding a company liable for misrepresentation based on vague statements about quality and fitness for purpose is very difficult, at least in the US. Did Crowdstrike claim to investors that they perform system tests before pushing channel updates? Without a specific untrue claim, it will be difficult, particularly for investors, to prove misrepresentation.

      Note that there are no regulatory standards (outside certain narrow contexts) for things like software testing, so plaintiffs can't claim that they could reasonably infer particular practices on Crowdstrike's part. (Well, they could claim it, but they don't have any grounds to support it.)

      I'd love to see more precedent for liability in the software industry, but I'm not holding my breath.

      1. Falmari Silver badge
        Devil

        @Michael Wojcik "As others have pointed out, this case might (might) be viable on misrepresentation grounds"

        The filed lawsuit is 29 pages long, but the misrepresentation claim is section 3 on page 3.

        "3. Throughout the Class Period, Defendants (defined herein) repeatedly touted the efficacy of the Falcon platform while assuring investors that CrowdStrike’s technology was “validated, tested, and certified.” This complaint alleges that these statements were false and misleading because Defendants had failed to disclose that: (1) CrowdStrike had instituted deficient controls in its procedure for updating Falcon and was not properly testing updates to Falcon before rolling them out to customers; (2) this inadequate software testing created a substantial risk that an update to Falcon could cause major outages for a significant number of the Company’s customers; and (3) such outages could pose, and in fact ultimately created, substantial reputational harm and legal risk to CrowdStrike. As a result of these materially false and misleading statements and omissions, CrowdStrike stock traded at artificially high prices during the Class Period"

        In spite of CrowdStrike’s anti-threat update procedure the statements “validated, tested, and certified.” are still true. Also (3) such outages could pose,... was covered in the SEC filing quoted in the lawsuit. Also had CrowdStrike disclosed their anti-threat update procedure would CrowdStrike stock have traded at a lower value? Probably not.

        Finally “validated, tested, and certified.” is only mention twice in the lawsuit. The second mention is the only example of the statement being used, which was in an earnings call. For such an important point to the claim I would have expected more than a single example in the lawsuit..

  16. shawn.grinter

    Offshore

    I bet CrowdStrike regret offshoring their Development to India in Spring 2024 - how much money did that save you chaps......

    1. Michael Wojcik Silver badge

      Re: Offshore

      We currently have no evidence that had anything to do with this disaster. As far as we know, the inadequate testing regime was in place before the offshoring.

  17. Tubz Silver badge
    Trollface

    Crowdstrike "We believe this case lacks merit and we will vigorously defend the company", you have to laugh at these morons, blurting out the PR BS, they crashed the world but it's not our fault,

    1. Michael Strorm Silver badge

      Mandy Rice-Davies applies

      They don't believe that, no-one believes that they believe that, everyone- them and us- are in in the fact that PR will say that regardless if it's in their interests to do so, or that of those who- having fucked up- have to at least go through the motions of pretending that the case is defensible for their own sake.

      They know that we know that they know that it's bullshit, the game is played regardless.

    2. Michael Wojcik Silver badge

      Of course this is a boilerplate response from the legal team and no one pays it any attention (except Internet busybodies); but even taken at face value it's correct. They're saying the case has no merit, and that's a reasonable position to take, because it's very difficult to establish liability or misrepresentation in the software industry.

      They didn't say Crowdstrike weren't at fault for their screwup. They're talking only about the case at hand.

  18. harrys

    Bottom line is crowdstrike is a good peice of software and well supported, but....

    They made one stupid mistake (hubris)... and for that they should die

    like it or lump it..... Capitalist system is cruel and aggressive

    the underlying tech will survive on its own merits. it is the best of the bunch

    unless microsoft aquire it and integrate it into there absolutely shite joke competitive system

    because no matter how high quality and clean anything is, put it in a bowl of shite it will never come out clean :)

    1. Anonymous Coward
      Anonymous Coward

      Bottom line is crowdstrike [sic] is a good peice [sic] of software

      No, it really isn't. Parsing configuration in kernel mode is a stupid, lazy architectural decision. The parser is brittle, and error conditions are not checked for or handled adequately.

      The testing regime is patently terrible — and we have Crowdstrike's own description to base that on. The distribution regime is terrible, and configuration-file distribution ignores the package's own phased-deployment options.

      I'll also note, anecdotally, that I've been working for a Crowdstrike customer (thus the anon post) for a few months, and I've seen a couple of dozen tickets raised with the helpdesk because of huge performance issues or other adverse effects.

  19. Anonymous Coward
    Anonymous Coward

    Dave DeWalt and And George Kurtz: How It Started vs. How It's Going

    Dave and George are mates of old from McAfee.

    Dave publicly supports George on LinkedIn after Crowdstrike snafu

    Dave who also sits on the board at Delta Airlines: Delta sues Crowdstrike

  20. Rich 2 Silver badge

    We believe this case lacks merit….

    Looks to me like the case has quite a bit of merit

    I wonder how badly Clownstrike thinks it needs to fuck up before it considers such a case to have sufficient merit

    1. Michael Wojcik Silver badge

      Re: We believe this case lacks merit….

      The lawsuit ≠ Crowdstrike's culpability. The former can lack merit even if the latter is well-established. Odd how many people don't understand this distinction.

  21. MuleD

    CrowdStride did bad.....BUT

    There is no doubt that CrowdStrike has a mess on their hands and that mess is going to be ugly and expensive. But, before we completely crucify them maybe we need to look at some history and how we got here. Channel files exist for a reason, they are updated the way they are for a reason and CrowdStrike got access to write Kernel drivers for a reason. That reason is market demand...Go back in time and look at how some of the antivirus definition files had to be distributed via SMS or some kind of script. The unacceptability of such a time delay between a known signature and an AV update was too great and lead to some of the early virus variants being able to spread like wildfire. The solution..... "Channel Files" they update automagiclly, they meet the need for a quick reaction time that security is demanding, there is not testing delay from the customer because most don't even know they have an option to say no to them and lastly they come from big trusted names in the industry. Certainly they are doing testing before they push them out (sarcasm added for affect). And for years this model worked great, until it didn't. Some readers will be old enough to remember when Symantec did nearly the same thing back in the day and broke a bunch o shit. We the customers are asking nearly impossible things from CrowdStrike. React instantly, never miss a threat and never make a mistake. I get that we pay them well for what we are asking BUT I think back to my first project management training almost 39 years ago when the instructor explained the Triple Constraints of Time, Quality, Money and that you can only have 2 out of the 3 at any given time. In order to get the Time line we are demanding at a price point we are willing to pay CS had to make choices, one of the choices we ALL agreed upon when we allowed them to update Channel files without our own internal testing or even change control approval was that Quality might suffer. From all I have read, this was a human error. A big one and an important one but a human error none the less. If we want perfect we are going to either have to sacrifice speed or money. That's the unfortunate reality of the tripple constraints. No matter how much the Board or the C-Suite does not want to hear this message. There are some facts that even the all great and powerful cannot change. Just my thoughts, feel free to disagree.

  22. gnasher729 Silver badge

    This “canary” request is only one of several items, and not the most important one by far.

    1. Crowdstrike had and probably still has software reading configuration files that crashes when it is given data that it doesn’t like. With adequate code reviews that should never have happened.

    2. A developer created code meant to fix some problem. That developer should have true what happened with his changes and the modified configuration file, which would have revealed the crash. What did the developer actually test?

    3. A script was supposed to be run before deployment that checked the configuration files. This didn’t happen. It is also possible that the configuration files were correct and verifying them wouldn’t have found any problems. So the check should have included actually reading all the configuration files.

    4. If an update like this one got through, not crashing would have been so important, there should be an automatic fallback to the last known working version.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like