back to article CrowdStrike apologizes to Congress for 'perfect storm' that caused global IT outage

CrowdStrike is "deeply sorry" for the "perfect storm of issues" that saw its faulty software update crash millions of Windows machines, leading to the grounding of thousands of planes, passengers stranded at airports, the cancellation of surgeries, and disruption to emergency services hotlines among many more inconveniences. …

  1. Sampler

    "Perfect Storm"

    Of us making boneheaded coding practices combined with little in the way of update testing? That sort of perfect storm that absolutely no one could see coming?

    1. DrkShadow

      Re: "Perfect Storm"

      Oh! And also the immediate rollout of this change to the _entire_world_, *including* the ones who opted for a "stable" release channel? !!

      1. Ken Hagan Gold badge

        Re: "Perfect Storm"

        Twelve times a day.

        1. Anonymous Coward
          Anonymous Coward

          Re: "Perfect Storm"

          I was under the impression that one of the major selling points of the likes of crowdstrike was that it was monitoring behaviour, hence requiring less frequent updates, and not signature based like more generic "old school" AV.

          12 updates a day sounds to me like it is signature based like most AV.

  2. Anonymous Coward
    Anonymous Coward

    Pwn Crowd

    (Falcon "have visibility into everything happening on that operating system.")

    Seriously thinking here about compromising Crowdstrike. It's like the perfect botnet.

    Joking. Maybe not.

    Anonymous, obvious.

    1. DS999 Silver badge

      It is a really attractive target

      Not just Crowdstrike, any AV vendor whose software has kernel access.

      You might think compromising Microsoft would be better, but it would not. Because few targets of interest have Windows updates applied as soon as they become available. By the time the really juicy targets get around to updating, it is highly likely your malfeasance will have been discovered.

      But most computers have AV updates applied immediately, or with a day's lag at most - because they fear leaving themselves open to a fast spreading attack more than they fear the update itself being a problem. Now maybe the Crowdstrike thing changes that equation, but I doubt it does. It will soon pass from memory and everyone will be back to business as usual.

      If you could compromise a major AV vendor and build malware into an update and get it pushed out, by the time it is discovered you have tens of millions of p0wned PCs, including many of the most valuable targets like banks, government, defense/police - whoever it is you are most interested in depending on whether you're a thief, dissident, drug dealer or whatever. Having them p0wned to the kernel level allows you in some cases to leverage EFI to make your "p0wnership" live beyond a complete disk wipe and reinstall.

      1. Anonymous Coward
        Anonymous Coward

        Re: It is a really attractive target

        With love, x64dbg / gdb

        Anonymous, obvious.

      2. Anonymous Coward
        Anonymous Coward

        Re: It is a really attractive target

        My CTO couldn't rush to sign the contract with CS fast enough after the breech. So--I would say that the answer is no.

        What was driving it? Compliance. I had become suspicious before, now I am certain--compliance regimes (HIPAA & friends), just like the old password recommendations (rotating passwords & requiring symbols) actively make things worse.

        Not a good thing.

  3. Anonymous Coward
    Anonymous Coward

    >Argues worse could happen if it loses kernel access

    Cool, so these morons really do insist on going down with their ship.

    This is why I didn't "scoop up" any shares while they're down. I knew from day one they intended to sink to the bottom of the mariana trench with their horrible practices.

    For reference, I compare this "argument" with how antivirus manufacturers tried to bring about a class action suit against MS for making rootkits impossible in Windows 2000 because those same AV manufacturers relied on rootkit techniques to install their own AVs to infect the OS before a virus could. That's inexcusable, and CrowdStrike deserves to rot with the fishes.

  4. Eaten Trifles

    Love the excuses

    You say "If you think about a chessboard trying to move a chess piece to someplace where there's no square. Effectively, this is what happened at the sensor, so when it tried to assess the rule, it was not able to do what the rule was asking it to do, which triggered the issue within the sensor."

    But all I hear is "our software is shit and we don't do proper quality control."

    1. SomeRandom1
      Trollface

      Re: Love the excuses

      - But all I hear is "our software is shit and we don't do proper quality control."

      Microsoft or Crowdstrike?

      1. GregC

        Re: Love the excuses

        Microsoft or Crowdstrike?

        Yes!

    2. Apocalypso - a cheery end to the world

      Re: Love the excuses

      I agree. It's a somewhat bizarre analogy: even Congresspersons will realise that anyone who plays chess and tries to move a piece off the board is a complete moron.

    3. Tom66

      Re: Love the excuses

      What I hear is nobody included the if(ptrToImportantThing == NULL) return; check that anyone who has had bitter experience with C has included.

  5. O'Reg Inalsin

    ... trying to move a chess piece to someplace where there's no square ...

    Sure, it could happen to any grand master. The difference is, for a grand master there would be consequences.

    1. ecarlseen

      Re: ... trying to move a chess piece to someplace where there's no square ...

      This is why chess matches and psychedelic drugs don't mix.

  6. Ken Hagan Gold badge

    No excuse for kernel mode

    These days, most devices are available in USB flavour and/or are implemented by MS. You don't need a third-party kernel driver for either.

    There will always be exceptions but in those cases it is still true that best practice would be a kernel component that did the minimum necessary. Such a component could be submitted to MS and compiled/distributed by the Windows kernel team. Naturally the delusional vendor who claims to neeeeed such a thing would be expected to pay for this privilege.

    I see no reasonable use-case for allowing any code to run in the kernel that has not been vetted by the kernel team. Any remaining kernel vulnerabilities would be obviously Microsoft's fault (and probably rather less likely).

    1. Yet Another Anonymous coward Silver badge

      Re: No excuse for kernel mode

      Didn't Microsoft move the graphics subsystem into the kernel - for performance

      So MSFT would be responsible for testing/certifying, and understanding, Nvidia's entire GPU stack ?

      Sounds like the 737MAX 'commentators' who were proposing that the FAA should be in charge of all aircraft design and manufacture at Boeing

      ps. Of course you could move stuff out of the kernel that didn't need to be there - but that's just silly

  7. MacGuffin

    "Pre"defined

    "Pre" is sometimes the "prefix" for before. Prewar. Prehistory.

    "Predefined rules" might work here because the the rules may have not been defined (before definition).

    No Definition? No rules. Nowhere to go.

    Fail.

    Don't apologize to congress. Apologize to the world.

  8. Ace2 Silver badge

    “Making it worse? How could it be worse? JEHOVAH, JEHOVAH, JEHOVAH!”

    1. Yet Another Anonymous coward Silver badge

      >CrowdStrike's senior VP for counter adversary operations

      Before any possibly destructive test - always mount a scratch VP

  9. gnasher729 Silver badge

    What they shipped to customers just could never work. That was not a bug that you could miss, that bug would have crashed on everybody single machine it was tried without exception. Before testing comes “trying out if it works”. Neither the developer nor their test system did that.

    The bug itself was a combination of a totally broken configuration file reader and a tiny mistake in a configuration file. A configuration file reader should survive _anything_ you throw at it. That is an unforgivable weakness.

    Interestingly, I see nothing that makes this related to the kernel in any way.

  10. Anonymous Coward
    Facepalm

    Copilot, yes.

    If I understand recent marketing correctly, Copilot would’ve definitely solved all of this, for every possible value of “this”.

    By the way, what happened to the Paris icon? She wouldn’t have gone down after unexpected input.

    1. sozz

      Re: Copilot, yes.

      Paris got cancelled on this site - it's not 2015 anymore. Agreed though - I'd also expect greater resilience

  11. Claptrap314 Silver badge

    Malware defense must run in kernel mode

    Two reasons. #1 drivers run in kernel mode. Those drivers continue to be an important source of malware--especially with USB.

    #2 any malware writer out there is going after kernel mode escalation. If they achieve that, they attack the malware defense as their top priority.

    But Microsoft has never properly policed what went in the kernel from a security standpoint, let alone what drivers operate in kernel mode.

    And CS decided to check every "what not to do" box they could find. Yes, they should be kicked out of the kernel. But if MS kicks out AV, that would be a horrible thing for anyone trying to improve their security situation, given that they are running Windows in the first place.

    1. Sok Puppette

      Re: Malware defense must run in kernel mode

      Giant monolithic kernels with completely unconfined drivers are a poor system design. We've known how to do better for a very long time.

      Layer after layer of that kind of bad design is the reason you end up feeling you *need* "malware defense" as a separate category of software.

  12. William K Kelley

    Why is CrowdStrike even necessary?

    Properly architected hardware and operating system software would not require something like CS! Basic problem is that x86 / Windows, PDP-11 / UNIX, and Linux began life as single-user systems with no thought being given to proper isolation and security. They have been grafted on over time, but it's been piecemeal and the gaps continue to be exploited. Systems designed as multi-user systems from the beginning have had fewer of these issues.

    1. Yet Another Anonymous coward Silver badge

      Re: Why is CrowdStrike even necessary?

      >UNIX, and Linux began life as single-user systems with no thought being given to proper isolation and security

      No, if you install bad stuff as a regular user in Unix it is limited to destroying only stuff that you control as a user = your data / pictures / documents / emails / messages

      The system level stuff that can be wiped and re-installed from scratch in 5minutes is perfectly safe.

  13. Sub 20 Pilot

    2024 and this shit is still allowed to happen.

    In my industry, you fuck up, you lose money or get put in jail when a building collapses or does not work.

    Why are the cunts running anything to do with software still allowed to get off free when they fuck up big style and cost other people billions.

    A revolution is needed worldwide where these people are made accountable. Until that happens this shit will go on and on.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like