Dodgy Kaspersky update borks THOUSANDS of NHS computers A misfiring antivirus update from Kaspersky Lab "effectively levelled several networks" last months including systems at Britain's NHS, The Register has learned. A Kaspersky update released on 25 October falsely classified a Windows system file, specifically tcpip.sys, as malign and quarantined it. The procedure left Windows …
"....to testing updates before they get applied?"
You would think any IT operation of reasonable size would stage definitions a bit, I agree. What I would like to know is why enterprise antivirus deployment tools are happy to roll out buggy deployments to hundreds of machines, borking them one by one without blinking an eye?
Is it so hard so roll out updates to a small number of machines, see if they still work afterwards, then make an intelligent decision on what to do next? Even doctors are smart enough to work like that.
How many companies, let alone NHS trusts do you know that have the budget and resources to employ an AV update bork-checker?
Updates come out for AV at least once a day and sometimes more often. To load this into a test network and try out every bit of software to see if it still works okay for that once in 5 year time that it doesn't is unrealistic.
In an ideal world, yes it would be great but you expect your AV vendor to have the resources to do testing for at least major software (like the WIndows OS) to make sure it is not affected.
@DaLo: "How many companies, let alone NHS trusts do you know that have the budget and resources to employ an AV update bork-checker?"
Maintain the one Windows image that's rolled out across the organizatiion and impliment your own patch cycle. Include a roll-back mechanism for patches that bork the system. Test the updates before adding them to the image. Would take about one-man-days work.
Expecting your vendor to do testing for you is socialism. It's nothing more than entitled sysadmins looking for a hand out. This is something everyone should be doing for themselves. If you want to have someone else do testing then you pay them for it. Specifically. Independently of the definitions.
I can't believe all the crazy socialists around here. This planet sure is going to hell.
On the contrary, this should have been picked up by Kaspersky's internal testing processes. It's not down to the Enterprise sysadmins to test 3rd party products - that's not our job!! If anything, it's easier to test if you work for an SME (5-200 PC's) rather than an Enterprise with 1000+ PC's.
I work for a company that employs over 10,000 people, with over 15,000 workstations and servers and trust me when I say, we are completely reliant on the 3rd party vendors to do their due diligence before releasing updates to their products. We simply don't have the time or manpower to test every update, that's plain ludicrous!
Our job is to design, implement and maintain infrastructure. Not test other people's code.
You (the system administrator) can't test it before it's applied. (Well, I couldn't do that with McAfee a dozen years back.) The definitions go out automatically, because you'd be testing those definitions every day, and the sales staff are opening dodgy attachments right now.
The real question for the various AV firms, and they've all been hit with this, is how did it escape their testing??? Shouldn't this stuff be automatic? Shouldn't the testing come up and say, "hey, this borks a normal installation," and raise a big red flag?
"(Well, I couldn't do that with McAfee a dozen years back.)"
Odd could do that around 8 years ago in McAfee's ePolicy program (2.0 I think it was)?
Several ways to do it but I tended to be lazy so had the IT Department get the first batch as we had a dedicated super agent server, set it up to send out to the other super agents several hours later (If our machine would corrupt jump on server and tell it to revert).
Not the best way but meant no need to manually test anything or use the BETA Dats from McAfee.
Those controls are called "the internal QA process".
Clearly that's lacking over at Kaspersky and several others. However, that said, there's always the internal process that does everything over and over… until someone pushes the wrong button (instead of 'send back', click on 'release').
I used to love Kaspersky, i found that on my home computer that it was the bees knees worked flawlessly. When it came to changing the antivirus at work we decided to go with Kaspersky's Enterprise software. For the first year it worked flawless again, with a fairly easy deployment process. But as soon as the Kaspersky client workstation software found that the licence keys were due to "expire in 7 days". It throw a massive hissy fit and locked up all the computers, blocking all network traffic on each machine.... I then had to go and manually remove and re install each antivirus with new keys... (this took a while) now currently into my second year with Kaspersky 3 weeks ago it decided to lock up all the machines again with these false positives.... I have also been told by a friend that he had the same problem upgrading the versions he was deploying. So to conclude Kaspersky appears to have an inbuilt kill switch that will hold you hostage and the trigger is loose, i will not be using next year...
That sounds like the plot to a sci-fi movie
The AI at Kaspersky's lab was asked the best way to make windows secure.
Computer thought well all these problems are coming from in via the network adapter. I will prevent it working it in the future.
Maybe if given longer it'd think, but they come from the internet I must dismantle the internet.
Longer still and it would realise that people are the problem and get rid of them.
We got of lightly... this time.
This has happened before although incidents haven't been reported as frequently as they used to. The last time I had any AV on my XP machine was when a certain nameless suite deemed WINLOGON.EXE to be malicious and quarantined it. Couldn't un-quarantine it since I was locked out of Windows as a result but was able to manually restore it from Windows install CD - not something a noob would be able to do. I now run naked with no AV or firewall and my Windows 7 system boots in 15 seconds.
I know many of you here do tech support for friends and family like I do. Notice how much a dead weight any AV is against garbage toolbars, search re-directors (CONDUIT et al) and pop-ups? I clean this crap on a regular basis, being paid in dinner/beer. Would hate to do this for a living.
As the news item proves, if you have an AV monoculture, everything's fine until your entire network gets b0rked by the fuck-up.
Perhaps the bloke who suggested this could talk to a health professional about analogues in the medical world (MRSA springs to mind), and see what they think about a single defence vector.
Yeah, on your own son! Using multiple software packages to do a single job is insane. Management would be a nightmare, as the NHS guys are finding out.
So when one of the AV packages b0rks half your computers, and the other half are fine, how are you going to explain that to your staff and managers? Then manage the rollout of the fix without hitting all machines? I could go on but I cant be bothered.
And trying to draw analogues with the health profession is stupid. If they released a vaccine that kills a man, they'd be sacked and possibly jailed. Not only that, how do you compare a biological vaccine to a software algorithm? You really haven't thought about this have you?
Wow.. You think there's only a single antibiotic on the market that everyone uses, because all infections are magically cured by it?
There are loads of antibiotics, and a whole array of drugs used to treat conditions, largely because there are strains of infection, mutation, slight differences in weaknesses between them and so on. If you simply had one drug for one condition, you'd have a reasonably ineffective drug that would get rid of some conditions (screwed if you weren't one of the ones it treated), and it'd very soon be the factor that every infection mutated to overcome (the big target).
Sorry, all this about 'why did the sys admin not test it ..." and all that rubbish ...
The whole point here is that a basic system file, not even something obscure, was borked. In this case, why didn't the AV vendor apparently test it on a windows box before it was unleashed?
Where there's blame ... and all that.
Indeed, this is far from being the first time such a thing has happened, and far from the first vendor to which it has happened as well.
It is clear that there is no basic vendor test for an update against standard Windows files. In this particular case, it is equally clear that the vendor did NOT test it before rolling it out, which smacks very much of lax control procedures.
But given that their LiveDisk was not even able to boot the last time I tried it, I cannot say I am surprised.
Actually, I am surprised. I'm surprised this kind of thing doesn't happen more often.
Presumably you all just allow Windows Update to run on all of your clients and servers without first testing they don't balls up your systems in some obscure way too?
AV cock ups, whilst fewer than they used to be, are anything but unheard of.
Budget? Simple - let your IT team have the releases first. These individuals should have the necessary skills to remediate any damage.
I find it laughable when people say "we don't have the time, budget or resources" - this doesn't have to be any more onerous than you make it.
But downvote away.
I didn't say it _was_ a Windows update but I was making a point that I presume all the people who are too time/budget/whatever excuse constrained to test updates that have, in the past, been known to cause issues occasionally also don't bother to test their Windows updates. Or their application updates.
Yes, for remote working/home working staff it makes sense if you're not using something like an always on VPN/DirectAccess solution to allow their systems to autoupdate and then there are times where you have to weigh up the impact of a zero day exploit / piece of malware etc and the damage it would cause over the potential instability a remediating patch or update would potentially cause if it's installed, but most of the time there is zero excuse for what is effectively laziness.
And yes I agree that the vendor should have caught it but they didn't and the IT staff should have therefore had policies in place that caught it before it became such a huge issue.
Christ - would you just roll out a brand new application suite without any testing?
You really don't work in that kind of environment do you? The way this works in enterprise is that they run very light teams on operations that keep the servers running, monitor and tweak the systems that host hundreds, if not thousands, of applications, check logs for security intrustions, upcoming errors, run diagnostics, herd requested changes through the process and so on.
Where you have a rapidly changing application (such as anti-virus, where if you don't have the absolute to the minute update and you get compromised by lack of it, you can lose your job), there is a balancing of risk very much in favour of trusting the third party. And you have a contract that hurts them if they take down your infrastructure by oversights such as this (and 'insures' you for any damages etc.). Contrary to what you seem to believe, there isn't infinite resource to check absolutely everything all the time, and if you try, it's a long, long queue. The job is managing risk, and in this case, it's the risk of being exposed to a virus infection that may leak your sensitive data to the world (big, big risk, and a fair likelihood), or the major vendor fails to test correctly to the level that it damages your infrastructure.
New application suite and antivirus definition update are two completely different things.
"on the grounds of consistency, easing central support, and economies of scale ... but to my knowledge that never came to fruition"
Says it all about the Public Sector really. They have an unlimited supply of tax-payer funding with no ownership so why would they try and become efficient?
I think I could save the UK £1 Billion pounds on my first day at work if I was in charge of the NHS:
- Tender for a UK-wide public sector mobile phone contract with one supplier, 2 basic handset to choose from
- Tender for a UK-wide contract for all NHS consumables (bandages, syringes, saline, etc)
- No First / Business class travel (all of the public sector)
- No bonuses (all of the public sector)
- Create a standard desktop build for the public sector (yes, there will be exceptions)
- Use open source software
- Secondments from Private Sector to the Public Sector (to change the culture)
- Build a Premier Inn style hotel in London and make MPs / Lords live in it (with suitable security)
Please feel free to add to the list...
My wife's NHS mobile phone is a Vodafone Blackberry Curve - everybody else I know who has a NHS business phone has a Blackberry Curve with Vodafone
Re: UK tendering - given the size of some of the NHS Foundation Trusts - who have their own budgets - these would have to be broken up - that of course will happen for free...
When I worked for the UK MOD I never travelled Business/1st class - I don't know anybody who ever did.
Re: No bonuses - yay - way to go - no incentive to try very hard - just do enough not to get fired...
Re: Standard desktop build - so who would get the Monopoly to provide every single PC to the whole of the Public Sector? HP, Dell, etc? How many machines would that be - how many perfectly good PCs would need to be dumped and replaced right now - who would decide on the minimum spec? You? Apparently (and what would the minimum upgrade time be? Or what if that manufacturer stopped making that PC? And because it's a standard build all those PCs would need rolling out across the whole of the Public Sector at the same time? Or if you phased it in - over how long? How long would the Public Sector have multiple classes of machines to support?)
Re: Open Source Software - brilliant idea - retraining everybody to use Linux instead of Windows, LibreOffice instead M$ Office, etc will happen by magic, be completely free and not disrupt the workings of the NHS system at all (because the manufacturers of Digital X-ray machines, etc will gladly and happily convert all of their software to work on Linux based OSes instead WIndows based ones)
Re: Secondments to/from Public/Private sector already happen - most senior managers across the Public Sector are brought in from successful Private Industries (of course that'll stop when you kill off the bonus system)
Re: MPs/Lords - OK, apart from the fact they're not "Public Sector"...
>> Re: Open Source Software - brilliant idea - retraining everybody to use Linux instead of Windows,
That's the typical F.U.D M$ and their dependents use to justify spending a fortune of tax payer money on software licences every year. If you go from 95 -> XP users need retraining, if you go from XP -> Vista->7->8 (especially 8), users need retraining. Hell, every iteration of M$ Office requires retraining. Retraining is a constant, and therefore it's a blatant lie to suggest it's more costly to move to Linux.
There are many examples of successful migrations to Linux and the subsequent long term savings. The cost of dealing with Anti-virus, Anti-malware, anti-world&dog alone makes Windows incredibly expensive to maintain.
When I hear about incidents like this I just laugh. Windows is disaster waiting to happen. I've seen it so many times, and yet the UK government loves to throw tax payer money at Micro$haft for the privilege of being $hafted. But this isn't anything new. Governments around the world waste vast sums of money on useless things. Procurement is rife with corruption and overpayment.
>>LibreOffice instead M$ Office, etc will happen by magic, be completely free and not disrupt the workings of >>the NHS system at all.
And events like this misbehaving antivirus software don't disrupt the NHS? Heaven forbid they actually get infected with a virus or rootkit. On Linux, the machines wouldn't even need to run an AV. That alone would save a fortune in maintenance and licensing.
*Cough* You work in the NHS in ops or IT service delivery do you? I thought not..
If there's the slightest change that isn't agreed by the Unions (and Linux is a big change for the whole suite of software that's required and available), the Unions can, and will invoke walkouts or strikes if necessary. There's a hell of a lot of negotiation involved.
From XP -> Win7 isn't necessarily a retrain. There's the familiar start button kinda thing, and most icons on the desktop. Win7 to Win 8 is. You'll note, there isn't much windows 8 in industry.
Office XP to Office 2007 was a retraining. 2007 to 2010, not so much. 2010 to Libre Office would be a definite.
Really, after working in big companies on international IT projects, and working inside the NHS in IT ops, running my own show, and consulting with various companies, I hear the kind of rant you're coming out with, and it's almost invariably from people who have never run heterogenous systems, never done major migrations, and never had to be involved in the whole business aspect. The "Well, we can rip everything out over a few weeks, and put it back so the lights look the same, and nobody will notice" plan is something I've seen time and time again. And it leaves a lot of extremely unhappy users, failed services, and project planners kicked out on their derrieres for incompetence.
Oh, and Linux boxes do run antivirus. For a good reason.
Except where it is cheaper than 2nd class, right, because we want to save money? A couple of years ago my County Council employed colleagues and I needed to go to London. County policy was we had to travel by train. We did some shopping around, and found first class return tickets for about £86 each. County policy was no first class travel, so we got told we had to travel second class. At around £150 each. For 10 of us. Common sense not required thank you!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
