back to article Second NHS IT system confirmed to be affected by CrowdStrike issues

A UK hospital is battling what it is calling a critical incident as the ongoing global IT outage caused by a CrowdStrike update is impacting its Varian system. This is going to turn out to be the biggest cyber incident ever in terms of impact, just a spoiler, as recovery is so difficult Varian Medical Systems is responsible …

  1. KarMann Silver badge
    Trollface

    Completely innocent, I say!

    CrowdStrike said the incident is not malicious in nature – it's not a cyberattack.
    Well, of course they would say that. To suggest otherwise would be to suggest that they're amongst the cyber-criminals, and we couldn't have people thinking like that, could we?

  2. ttlanhil
    Facepalm

    A data file with an invalid format can cause a driver to crash badly enough to take out the entire OS?

    And this file (and those drivers) got pushed out around the world by a big player in the *security* space?

    Fascinating...

    1. Boris the Cockroach Silver badge

      Guess someone missed out some code

      Load "Date.File ;

      Instead of

      Try

      {Load "Data.file; }

      Catch exception

      {PrintLn("This file is bollocks... you're fired"}

      1. TimMaher Silver badge
        Pint

        Re: PrintLn call

        You forgot to close the call with a “)” … so that crashed.

    2. hoola Silver badge

      I really hope CrowdStrike get sued out of existence. They are nothing but snake oil pushing their shonky product that nobody has any control over as some sort of second coming. I evaluated it along with some othe "modern" AV solutions to replace Kaspersky. The conclusion we came to was that it was utter rubbish, just sales spiel for management using all the buzzwords:

      Replacing legacy solutions

      Cloud based

      Lightweight

      Cloud management console.

      Seemless deployment

      Blah blah

      All the usual shite. I hope now that people start to wake up and realise just how tenuous SaaS is.

      1. StewartWhite
        Facepalm

        AI is "Magick" - not so much

        But how can this be possible given that CrowdStrike has been brought to life using AI (see https://www.crowdstrike.com/platform/ and the references to AI all over it)?

        Oh I know, because most "AI" is an expensive big steaming pile of horseshit.

        1. Prst. V.Jeltz Silver badge

          Re: AI is "Magick" - not so much

          AI is the marketeer's golden buzzword

    3. Anonymous Coward
      Anonymous Coward

      > A data file with an invalid format can cause a driver to crash badly enough to take out the entire OS?

      The use of "driver" is - confusing. Unlike, say, a graphics card driver, antimalware has deliberate code that will poke itself into everywhere - if it goes truly apeshit it could amazing amounts of damage! So, yes, it probably could crash the OS.

      Although it is not crashing the OS, it is preventing the OS booting in the first place. Not a subtle difference, as it is almost certainly inserting itself as early into the boot sequence as it can, so probably not that much of the rest of the OS will be operational. Yes, that puts it into a very privileged position (as it would need to be if it can do what it claims).

      That said, the invalid file format is totally inexcusable. Parsers should be robust enough by now to just report back "nope".[1] And at this level you do absolutely everything to validate before you actually load and use a "channel", whatever that really is.

      Someone probably said "we can not parse and validate, that will slow down the boot; it was checksummed in the download and we can totally trust the copy on the server to have been validated"!

      [1] although I have seen morons who believe that a parse error should raise an exception and then you get into all the crap about not catching the exception or letting it pass back up because it is really that layer's responsibility/interest... How the ¢π$¥{¶ is a parse error ever an exceptional situation? Then again, there are lookup tables out there where a "not found" raises an exception! FFS.

  3. Vometia has insomnia. Again. Silver badge

    EMIS was the first system, then; what a surprise. It's kinda hard to tell since it's down half the time anyway, and when it's up it's excruciatingly slow and horrible to use.

    1. Handlebars

      Our incident meeting first thing this morning didn't rule out EMIS just coincidentally falling over at the same time as many other unrelated systems.

  4. glennsills

    Ironic Name

    Hey, it looks like CrowdStrike really is a cloud strike! Get it? Cloud strike. People naming companies should really think things throw.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ironic Name

      More like ClownStrike !

  5. The commentard formerly known as Mister_C Silver badge
    Unhappy

    "The pharmacy could not process electronic prescriptions made after the outage took hold, but those issued before could still be fulfilled"

    Assuming they actually have the items prescribed in stock, which is a whole other chronic failing.

  6. Chris Evans

    Which Windows PCs are effected and should you turn on your PC!

    It's not affected our 4 Windows 10 PCs (mix of home,pro,32 & 64) it might be helpful to know which systems are failing!

    If a Windows PC hasn't yet been turned on today is it safe to turn it on now?

    I suspect old Windows xP, 7 & 8 might not be effected, if so that might help people with old normally unused systems.

    1. Snake Silver badge

      Re: Which Windows PCs are effected and should you turn on your PC!

      It's not a Microsoft / Windows problem, it's a CloudStrike Falcon Sensor product problem. If you aren't using CloudStrike's anti-malware product then your systems are safe.

      1. Chris Evans

        Re: Which Windows PCs are effected and should you turn on your PC!

        I know that but it is only affecting Windows PCs and many users won't know if they are using CloudStrike's anti-malware product. If it would be correct to say this will probably only be effecting organisations that are big enough to have an IT department then stating that will reassure many people!

        1. glennsills@gmail.com

          Re: Which Windows PCs are effected and should you turn on your PC!

          Yeah, the fact that many users don't know if they are using CloudStrike would be a root cause of the entire problem.

        2. Anonymous Coward
          Anonymous Coward

          Re: Which Windows PCs are effected and should you turn on your PC!

          No home user will be using C.S.

          1. Snake Silver badge

            Re: No home user using CrowdStrike

            I would say that would probably be quite true. I'm shocked how common CrowdStrike is on commercial systems! Even our payroll processor just notified us that their systems were affected.

            I'd have to do far more research on this product but maybe it is tied to cloud infrastructure use, the Falcon product is offered and preferred for users who subscribe to a variety of cloud services?

      2. ttlanhil

        Re: Which Windows PCs are effected and should you turn on your PC!

        If you aren't using CloudStrike Falcon, then you're not at risk of this bug hitting your system.

        But we're never really safe :)

    2. katrinab Silver badge
      Alert

      Re: Which Windows PCs are effected and should you turn on your PC!

      Ones with Cloudstrike Anti-Virus.

      If you are using for example McAfee or Norton, then you will have different problems, not this particular one.

  7. Stevie

    Bah!

    Looking forward to the Crowdstrike bod's "Who Me?" writeup.

  8. Anonymous Coward
    Anonymous Coward

    WTF

    Who put these clowns in charge of the circus, reaming is too soft a punishment in my view. In any other industry a mistake of this magnitude would guarantee rolling heads, and multiples thereof.

    1. Anonymous Coward
      Anonymous Coward

      Re: WTF

      > In any other industry a mistake of this magnitude would guarantee rolling heads, and multiples thereof.

      FFS it takes time to sharpen the guillotine and march the prisoner down. What, you want them by Deliveroo? "Heads delivered in 30 minutes or we give you a free ball of wool" Madame le Tricoteuse Anonyme?

      Heads will be rolling, one way or another. In due course. But maybe you won't have the bodies paraded in public.

  9. Dimmer Silver badge

    US courts are down to.

  10. Alan Brown Silver badge

    The surprising part

    Is how much critical infrastructure is running on Windows

    You'd think people would know better by now

    1. Rich 2 Silver badge

      Re: The surprising part

      Sadly, it’s not at all surprising.

      The people responsible for deploying windows in these areas should still be shipped off to some barren lifeless moon somewhere though

  11. Anonymous Coward
    Anonymous Coward

    systmonline seems fine

    I checked my surgery remotely, earlier, and I could still see my records.

    So it's only affecting a subset of GP's, so far.

    1. Chris Miller

      Re: systmonline seems fine

      I can't renew my prescription using the NHS app. No biggie (for me) as I always keep a month's supply in hand, and can hand deliver a request to the pharmacy if the problem persists for weeks.

  12. Doctor Syntax Silver badge

    "they cause the top-level CS driver to crash as they're invalidly formatted,"

    Fail-safe. Defensive programming. All long gone.

    1. Falmari Silver badge

      @Doctor Syntax "Fail-safe. Defensive programming. All long gone."

      Along with Destructive Testing. Test plans need to test more than just expected behaviour.

      1. Disk0
        Mushroom

        If you don't know how to nuke it, is it even in orbit?

  13. Jamie Jones Silver badge

    This whole event is unacceptable

    I'm leaving the examination of CrowdSrike to others.

    My main beef is the fact that this could be possible. People around the world, responsible for important systems farm of responsibility to the lowest bidder just to increase their bonuses.

    It's about time that the whole world took IT infrastructure seriously.

    And it's obvious that this can only be done by legislation. There are checks and balances in most critical jobs. Doctors need to be highly trained, and qualified, so do engineers on great building projects.

    Yet in the technology field, it's the wild west. A simple mistake has caused more problems than a state-sponsored attack has done, but that's just because we have been lucky.

    If a dozy company goes bust because skimping on the IT causes them to lose uptime/data, so be it, but when there are such crippling and dangerous 3rd party effects, something needs to be done.

    1. hoola Silver badge

      Re: This whole event is unacceptable

      Crowd is not cheap. It is the way it is sold. Their sales people go straight in at board level with buzzword bingo.

      This is more to do with reliance on yet another SaaS service over which you have bugger all control.

      That is the way CrowdStrike and similar things work.

      It is how they are sold, lightweight, cloud native, blah blah

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like