back to article CrowdStrike CEO summoned to explain epic fail to US Homeland Security committee

The US House Committee on Homeland Security has requested public testimony from CrowdStrike CEO George Kurtz in the wake of the chaos caused by a faulty update. Mark E Green, Chairman of the Committee on Homeland Security, and Andrew R Garbarino, Chairman of the Subcommittee on Cybersecurity and Infrastructure Protection, …

  1. CowHorseFrog Silver badge

    In the new age of bullshit, why do they bother with these stupid parades ?

    Nothing actually changes, like good american tradition of hollywood this is nothing more than two sets of actors, talking bullshit and then its over, and nothing changed, just like all tv series, something happens in the show but all is reset for the next show.

    1. Gene Cash Silver badge

      Dude. Seriously. ALL politics is stupid parades, theatre, and posturing.

      "CrowdStrike had an Endpoint Protection Platform market share of 14.7 percent"

      The problem is that it was a disproportionate share of high visibility installations. It's not run by your average Joe, but large public organizations, usually mandated by insurance. It was concentrated where it would do the most good harm.

      1. Zippy´s Sausage Factory

        I wouldn't be surprised if, in a few months time, the word "had" in that sentence starts to assume the leading role.

        1. Anonymous Coward
          Anonymous Coward

          Did you have McAfee in 2010 when they had a similar issue?

          Did you have Sophos in 2023 when they had a similar issue?

          Did you have Crowdstrike in 2024?

          Bitdefender?

          Norton?

          I challenge you to name one enterprise AV package that hasn't had a similar snafu.

          1. CPU

            Kaspersky

        2. Michael Wojcik Silver badge

          I wouldn't be surprised if Crowdstrike survives this relatively unscathed. There are large costs to switching: financial, cognitive, and regulatory. One of Crowdstrike's main selling points is that it's known to be auditor-friendly, which makes things a lot easier for its customers that have to comply with government and industry regulations.

          It's easy for customers to declare angrily that they're going to switch, but it's also easy for them to subsequently decide it's not worth the trouble.

      2. CowHorseFrog Silver badge

        I have a q....

        who exactly are they trying to impress with this presentation ?

        Most of the masses dont believe... so why bother ?

        1. richardcox13

          Are you aware of the politian's falacy?

          1. CowHorseFrog Silver badge

            And im asking why waste time with this charade ?

    2. Snake Silver badge

      RE: parades, and nothing changes

      You're absolutely correct, and to find the answer to your question all you need to do is Wiki the two politicians who signed the letter of investigation.

      https://en.wikipedia.org/wiki/Mark_Green_(Tennessee_politician)

      https://en.wikipedia.org/wiki/Andrew_Garbarino

      Look at their party affiliation. That's all you need to know. Watch their hands, not what's happening behind the curtain.

      1. DaveLE

        Re: RE: parades, and nothing changes

        I'm loathe to enter low pilitics but R vs D is the nullest debate.

    3. DVG46

      Exactly:

      Q: What went wrong?

      A: We f*kd up

      Q: What action are you going to take?

      A: We’ll try not to f*k up again.

      1. CowHorseFrog Silver badge

        ABout as valuable as Russian or North Korean democracy where everyone pretends the leader makes the sun shine.

        That would also explain America's many problems.

      2. UnknownUnknown

        Although it doesn’t need to be at a Kangaroo Court the ‘we fucked up’’ is still pending some detail about Change Control at Crowdstrike.

  2. Anonymous Coward
    Anonymous Coward

    Patchpocalypse now

    Did I come to the party late? I can't believe the CEO's name is Kurtz!

    "The horror. The horror..."

    1. TimMaher Silver badge
      Mushroom

      Re: The horror.

      “because there’s a conflict in every humancorporate heart between the rational and the irrational, between good and evil. And good does not always triumph”

    2. Jellied Eel Silver badge

      Re: Patchpocalypse now

      Did I come to the party late? I can't believe the CEO's name is Kurtz!

      Yep. Kernel Kurtz is also consistent-

      https://en.wikipedia.org/wiki/McAfee#DAT_5958_update

      On April 21, 2010, beginning at approximately 14:00 UTC, millions of computers worldwide running Windows XP Service Pack 3 were affected by an erroneous virus definition file update by McAfee, resulting in the removal of a Windows system file (svchost.exe) on those machines, causing machines to lose network access and, in some cases, enter a reboot loop.

      ...On a flight, he watched the passenger seated next to him wait 15 minutes for McAfee software to load on his laptop, an incident he later cited as part of his inspiration for founding CrowdStrike. He resigned from McAfee in October 2011.

      Kurtz was McAfee's CTO at the time of their outage. I also wonder if this trait plays a part-

      In March 2020, when discussing company strategy at CrowdStrike, he said that "not one time have I regretted firing someone too fast."

      Which I suspect might include anyone who says things like 'Are you sure that's a good idea?' or 'Maybe we should spend more on QA and QC'

      1. Joe Dietz

        Re: Patchpocalypse now

        Culture is very much at the root here... And McAfee was toxic. There were lessons learned at McAfee with the 5958 incident. Some good, some bad. Ultimately 5958 led to the internal stagnation of the organization and why McAfee just isn't really a relevant player anymore. And yes, he was in the room then as CTO.

      2. Scott 26
        Joke

        Re: Patchpocalypse now

        Update his wikipedia page to suggest he was in a Wuhan open air market, late 2019, and we have the trifecta!

  3. wolfetone Silver badge
    Joke

    Well I hope that after they get the CEO to explain the cock up, they'll continue the pantomime and invite the heads of IT departments affected and ask them where their disaster recovery plans were at.

    1. CowHorseFrog Silver badge

      How can they ask q if they dont understand how software does or doesnt work ?

      Secondly the ceo himself hasnt a clue, so they are both talking about something they dont understand...

      1. williamyf

        In theory, the CEO is briefed and prepared by a multidisciplinary team of underlinggs, including lawyers, pr people and programmers, besides, before being a CEO he was a CTO

        meanwhile, the congresscritters aare briefed and prepared by a multidisciplinary team of aides and subjet matter experts.

        in pactice though.....

        1. CowHorseFrog Silver badge

          > In theory, the CEO is briefed and prepared by a multidisciplinary team of underlinggs, including lawyers, pr people and programmers,

          Its v interesting the order of the advisors you mention. GOing to guess underlings means more managers, again these people dont have a fucking clue. Lawyers also dont have a clue about programming. I dont think i need to give a comment about PR. Lastly its highly doubtful he has asked any programmer, i mean there are 100s of them in crowdstrike, how would he know which to ask ?

          > besides, before being a CEO he was a CTO

          So what if he was ?

          Anyone can call themselves a CTO, you dont need a qualification of any kind, there are university degrees of any kind to be a CTO. Its a bit like how Prince Harry has all them medals, he didnt actually earn them, he just gave them to himself.

    2. Twilight

      They had detailed disaster recovery plans... for their systems. Crowdstrike is a third party vendor and they don't need DR plans to cover that (cf Change Healthcare earlier this year).

      1. CowHorseFrog Silver badge
        1. Jimmy2Cows Silver badge

          Likely some had DR plans, some did not.

          But if your DR plans don't cover what happens when a third-party dependency falls over or fucks your systems, your DR plans aren't worth a damn anyway.

  4. williamyf

    optimum way to make an enpoint security driver

    Last time I checked, when you wrote a driver that:

    a) has to be up in order for the system to boot

    and

    b) it can execute arbitrary code (P-Code) in kernel space

    the driver has to be divided in two parts:

    1) the bulk of the driver as a non essential driver in kernel land

    and

    2) a watchdog rinny driver that has to be up for the system to boot and whose only task is to make sure that the main driver is up and notify otherwise

    sadly the current CEO of cloudstrike was the CTO of mcafee when they had their meltdown, therefore by virtue of top to bottom modeling, most likely crowdstrike programmers do not know and/or do not care.

    1. Gordon 10

      Re: optimum way to make an enpoint security driver

      So much this. But surely the framework for governing this should be developed and run by MS?

      1. CowHorseFrog Silver badge

        Re: optimum way to make an enpoint security driver

        MS made a massive fuckup when they allowed each and every application and driver the ability to write their own files to anywhere on the disk.

        This means theres no way to identify files, isolate, or even make backups...

        The end result is that windows cannot take a snapshot before an update, and then rollback if the update "fails" like it did the other day.

        1. CowHorseFrog Silver badge

          Re: optimum way to make an enpoint security driver

          WOW so many down votes and yet NONE of them can actually refute my statement.

          WHat i described is how all modern popular os operate.

          1. Falmari Silver badge

            Re: optimum way to make an enpoint security driver

            @CowHorseFrog "WHat i described is how all modern popular os operate."

            No that is not how they operate. Not even Windows.

            Programs can only do what their user's level of privilege allows them to do. When the logged on user runs an Application it has the same level of privilege as the user. If the user can't write to an area of the file system neither can the Application.

            But Crowdstrike Falcon runs at a higher privilege level than the logged on user it will be able to write to an area of the file system that non admin users can not. That's because Falcon's user will not be the logged on user Falcon will be using a built in service account from the admin group. A service that uses an admin account requires an admin to install it.

            BTW I have not down voted you

            1. CowHorseFrog Silver badge

              Re: optimum way to make an enpoint security driver

              @CowHorseFrog "WHat i described is how all modern popular os operate."

              Falmari: No that is not how they operate. Not even Windows.

              Cow: Android, IOS, WiNRT does have user/app file isolation.

              Falmari: Programs can only do what their user's level of privilege allows them to do. When the logged on user runs an Application it has the same level of privilege as the user. If the user can't write to an area of the file system neither can the Application.

              cow: And every driver andbasically apps as well in windows can basically write anywhere, thats the problem.

        2. navarac Silver badge

          Re: optimum way to make an enpoint security driver

          I'm just waiting for Microshaft to use this fiasco as an excuse that we really need the much abused feature called Recall.

  5. goblinski

    In the meantime - it's July 27th, and if I had any money - I'd gargle it on Crowdstrike stock today.

    How about we come back to this post in a month, then in six, and see how my virtual money would have been doing by then.

    Here, let's say I just (in my imagination) put ~100k in their stock and bought ~372 shares. Will come back to that one in a month (unless they go belly up before that :-D)

    1. williamyf
      Joke

      Shares are bought in mmultiples of 100, so either get a little more money to reach 300 or get 200 shares only and buy yourself a yaris or somesuch

      1. goblinski

        Blast, my imaginary money just took a hit, but for the sake of numbers - let's say I managed to buy 372 for $100-ish K, which was this morning at $268 a pop at the time when I imaginarily bought them.

    2. Anonymous Coward
      Anonymous Coward

      a GR Yaris please

  6. Pascal Monett Silver badge

    Oh, a summons from Congress

    So, Crowdstrike gonna pull a Zuckerberg, or is the CEO actually going to submit to a public flogging ? 'Cause that's what's waiting for him.

    Not everyone can be a total asshole. I'm guessing Kurtz is going to bend over backwards to try and keep the share price from falling further.

    Time will tell.

    1. CowHorseFrog Silver badge

      Re: Oh, a summons from Congress

      Can you share an example of a public flogging of any ceo in the last 20 years ?

      Did a single one of them lose their job or go to jail ?

      I can name a few that caused hundreds to die because of their policies and they got a public flogging, and a $60M bonus...I wont give you the name o the company im sure you know which one im referring too.

      1. Michael Wojcik Silver badge

        Re: Oh, a summons from Congress

        CEOs in the past 20 years who have gone to prison? Let's see. I'll just do some American ones: Bankman-Fried. Holmes. Kozlowski. Naccio. Israel. Rigas misses the 20-year mark by only a few days, if we go by his conviction; he's in if we go by sentencing.

        Lay only doesn't apply on a technicality: he died between conviction and sentencing.

        Non-CEO officers include Madoff and Hussein.

        There are plenty of other examples outside your arbitrary 20-year limit, such as Ebbers.

        In short, you're an idiot who apparently can't be bothered to do even the smallest bit of research before posting the same tired, ill-informed, uncritical bullshit you repeat over and over and over again.

        1. CowHorseFrog Silver badge

          Re: Oh, a summons from Congress

          MW: CEOs in the past 20 years who have gone to prison? Let's see. I'll just do some American ones: Bankman-Fried. Holmes. Kozlowski. Naccio. Israel. Rigas misses the 20-year mark by only a few days, if we go by his conviction; he's in if we go by sentencing.

          cow: Those people went to jail for committing various crimes, they actively did themselves.

          They did not goto jail for *software fails* that companies under their leadership performed.

          The crowdstrike story is clearly an example of bad stuff happening because of poor software, its not an example of a leader actively lying/stealing/etc.

          BIG DIFFERENCE>

          MW: In short, you're an idiot who apparently can't be bothered to do even the smallest bit of research before posting the same tired, ill-informed, uncritical bullshit you repeat over and over and over again.

          COW:

          No you are an IDIOT because you cant tell the difference between being a clueless CEO for a software company and the crimes that the names you list performed.

  7. Anonymous Coward
    Anonymous Coward

    It's a culture issue

    I found this piece by Ed Zitron quite striking. There appears to be a strong correlation with (largely indiscriminate?) mass lay-offs in the US tech sector and major crises in the months that follow, if you then add a culture of profitability-above-everything-else you have very fertile ground for large scale tech disasters.

    Oh, and there's obviously also this: Spotify CEO Daniel Ek surprised by how much laying off 1,500 employees negatively affected the streaming giant’s operations

    1. CowHorseFrog Silver badge

      Re: It's a culture issue

      Do you really believe anything that the spotify ceo says ?

      Suckers born every minute and all that.

  8. DaveLE

    Did you read our ToS, I'll paraphrase: "our software might break, prepare accordingly"

  9. greenwood-IT

    Basic software common sense

    OK, so they are now saying this was a "definitions/signature" file and not the actual kernel driver itself that caused the problem? This brings on even more questions;

    1) is this critical signature file not signed or checksummed? Both options would have avoided a corrupt file being delivered - which would have avoided this incident. From a security perspective, I'd expect such a critical file to be digitally signed - you wouldn't want bad people modifying its contents now would you?

    2) The contents of the signature file were not corrupt, but just plain wrong - so what happened to input validation? I thought everyone who did anything with security new about bounds and input validation? This is an input file to a critical process, it should not have blue screened just because someone but in too many commas of a date in the wrong format - that is bad programming 101.

    It sounds like a school boy error by CrowdStrike to me - probably a rushed job and now time for "good practice" - possibly because that would have slowed down the release, but also possibly slowed down the application when running. Relying on the "ON ERROR" function in a kernel driver is not a good option.

    1. Webelike

      Re: Basic software common sense

      If the driver failed because of a bad definition file and this wasn't caught during testing and was then pushed out globally there needs to be an independent investigation and complete remediation. Since CrowdStrike is a publicly traded company and a government contractor we need a transparent review of the policies and procedures as well as some form of accountability. The timing, range and scope of the issue plays into the hands of the bad actors. The CEO claims there was no security breach, but his comments are no longer credible -- an outside investigation needs to be initiated.

  10. vcragain

    I was a programmer for more than 30 years & this snafu gave me the shivers, but it brought back my own biggest snafu which made me remember that the biggest problem with coders is that moment when you convince yourself it's 'just a tiny change - won't affect the big picture' and release the 'fix' to production - after all it HAS to get in before xxxxxxxx whatever to make sure we don't have that original problem 'get' us again - so we snuck the change in & BOOM - red face time ! All the change management systems in the world will not protect you if a programmer knows how to get his code 'out there' ! And sometimes that 'fix' is desperately needed before all that change management brouhaha can possibly happen ! Sigh ! Been there !

  11. wibixo2243

    eBPF could be the solution!

    In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.

    For Linux systems, the company behind this outage was already in the process of adopting eBPF, which is immune to such crashes. Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well. These security agents will then be safe and unable to cause a Windows kernel crash.

    https://www.brendangregg.com/blog/2024-07-22/no-more-blue-fridays.html

    https://news.ycombinator.com/item?id=41033579

  12. Conundrum1885

    Well that is unfortunate

    Supposedly, folks are comparing this with SQL Slammer but I have a feeling that the Melissa worm would be about right.

    Someone should graph it, I'd be intrigued.

  13. Skiver

    Nothing will be learned from questioning the CEO because these hearings are performative nonsense.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like