Is this the anti-virus equivalent of typing Google into Google?
Sophos users woke up to mayhem on Thursday after the business-focussed antivirus firm released an update that classified itself and any other update utility as a virus. As a result enterprise PCs running the application went haywire, generating false positives reporting SSH/Updater-B malware. Sysadmins were bombarded with …
Thursday 20th September 2012 09:14 GMT Anonymous Coward
Thursday 20th September 2012 09:56 GMT Phil Koenig
Re: Easy to fix, even after quarantine...
This sounds tempting to try, but what does it do exactly? Looks like it just repairs "required registry entries" and shortcuts?
I was getting a "25010" with "NoUpdateInProgress" error while trying to uninstall one of 3 Sophos items in XP control-panel, apparently this has been a bugaboo for Sophos for quite a while now, judging by various online comments and kb articles.
Thursday 20th September 2012 13:26 GMT Anonymous Coward
Thursday 20th September 2012 22:10 GMT Anonymous Coward
Friday 21st September 2012 09:46 GMT Danny 14
Re: @AC 09:14GMT - Easy to fix, even after quarantine...
I sent a new exclusion rule to stop scanning the sophos folders and "reprotected" the clients. This was after repairing the SUM manually.
Worked for all my 350 clients although the AV server took a kicking while they all updated. Network usage took a major spike too.
Friday 21st September 2012 12:05 GMT Anonymous Coward
Thursday 20th September 2012 09:21 GMT geekclick
Easy to resolve...
1. Turn-off 'on-access' scanning in all of your Anti-virus and HIPS policy.
2. Go to the Update Managers in your Enterprise Console, right-click your Update Managers and choose 'Update now'.
3. Wait for the update manager to finish downloading the latest updates (Download status changes to Matches)
4. Edit all of your 'Updating' policies in Enterprise Console. Click on 'Schedule' and change the check for update time to 5 minutes.
5. Wait 8-10 minutes.
10. The number of false-positive Virus/Spyware detection should start falling.
11. Enable the on-access scanner when the number of false-positive detection has fallen significantly.
12. If there are any computers still showing the false-positive alert then they have either not received the latest update or the 'on-access' scanner was still enabled when they tried to update. The above steps can be repeated for just those computers.
Taken from: http://www.neowin.net/news/sophos-releases-update-causes-mayhem-for-corporates
Resolved for all our users this morning, saves having to run a command on every machine....
Thursday 20th September 2012 09:51 GMT Phil Koenig
What a disaster
It took me around 6 hours to get through to Sophos support on the phone.
I didn't even want to bother after reading about everyone else's trouble reaching them, but I had at least one machine which was stuck in a loop - couldn't remove Sophos Endpoint Protection, couldn't re-install it, wasn't working properly, Windows Installer kept trying to re-install the auto-update thing every few minutes, ugh.
I finally got someone on the line 10 minutes after the UK call center started taking calls from the pitifully overloaded Australian call-center. I've been working on this since ~18:00-19:00 PDT, it's now 02:44 9/20 and still waiting to get a response to my diagnostic report sent to them 40 minutes ago.
I was wondering what was going to happen to Sophos after they got bought by some investor group a couple years back. Perhaps this is our answer.
Thursday 20th September 2012 10:00 GMT Anonymous Coward
Not just the sophos program
Problem is if the deafult is chosen where it will delete any known (rather than suspected) virus then you will lose auto-updaters for a lot of software which means you may be at risk for security problems from that software too.
You may also find - like we did that a lot of software stops working completely as it tries to use the updater when it first loads and bombs because the file is missing.
How on earth this got through testing I don't know. Even the most basic of testing should've spotted this.
Thursday 20th September 2012 10:05 GMT Morphius
Re: Not just the sophos program
This is what we have found... so far the list includes Commvault, Flash, Shockwave, Quickbooks, Dell Server Administrator, Java... (although most of those run, just won't update now)
Sophos issuing instructions on their site to fix Sophos is one thing but the damage this false positive has caused by deleting the updaters is a lot worse. Thank god it happened when most of our PCs were offline so only a handful are going to need work, I pity those on US timezones where their computers were all online.
Beer because I think some people are going to need one!
Thursday 20th September 2012 10:33 GMT Anonymous Coward
Re: Not just the sophos program
agreed. Lucky by shear chance I happen to be chaecking my work emails last night at about 2125 just as things were starting. Manage to vpn in and turn off on access scanning and we only had about 50 boxes that were effected by this. As you said pity those in the states that were in the middle of their working day! Still not quite sorted things out as some of the files on our sophos server were moved but the server does seem to be updating ok, I've sorted all the workstations
Thursday 20th September 2012 10:17 GMT Mayhem
Not to mention Small Business Users are currently out in the cold
You can easily fix as a home user, but the SUM/SEC enterprise instructions don't work with the SCC small business version. I've got two sites that have neatly dropped through the cracks at the moment - we can't fix the server, so can't update the endpoints. At least manually clearing the quarantine lists is feasible, if not a pretty option.
Thursday 20th September 2012 10:24 GMT Velv
Shocked and Stunned
I'm just shocked and stunned by the fact this article was published over two hours ago and a fanbois troll fight hasn't broken out in the comments section:
"ha ha ha, look at those muppets who've been hit by an update to software X - we're all fine over here because we use software Y which is immune to such problems"
"ah but software Y is useless, and costs a fortune and you're just showing off how much money you've spent - , my software Z was free and is crowd sourced and open source and used by loads of people"
repeat (until you get more down votes than up votes)
Thursday 20th September 2012 10:37 GMT Anonymous Coward
What's happened to Sophos?
We've used SAV in our business unit for many years. When the renewal came up for for the enterprise AV worldwide, I pushed hard to make the switch to Sophos from TrendMicro.
Since then, we've also taken their SafeGuard product for full disk encryption of the laptop estate.
I'm not sure what's happened at Sophos recently but this is the latest and most public of a long line of snafus which have caused our small team worldwide no end of problems - the most recent being an upgrade to the Enterprise Console which we were advised was just 'a few clicks' and 'no risk' which resulted in both SAV management and SafeGuard management being out of commision for several days whilst we worked through the problems.
The analysis? You shouldn't have the SAV EM and the SafeGuard MC on the same machine. Who new? Certainly not the Platinum Patner consutants who worked with us on both implementations, nor the Sophos Technical Support guy who came along to give us a system heathcheck a couple of months previously.
There are numerous other examples, but I can't be bothered to type them - I'm past the point where this is cathartic - and don't even get me started on the issues around encryption of solid state HDDs with SafeGuard.
In short it appears the QA procedures are completely shot and more worryingly seem to be getting worse. Sure, shit happens and most of the major AV vendors have had similar issues in recent memory, but for our Organisation this is just another example of 'another Sophos problem' in a ridiculously short space of time.
This last cock-up means at the decision whether to renew at the end of the license term has just made itself.
Thursday 20th September 2012 10:50 GMT Anonymous Coward
Not just the US hit badly...
... we have 8000+ workstations and 600+ servers. We're not impressed. As stated ^^^^ how the hell did this get through QA testing?!
Fortunately we had our quarantine configured to disable access rather than delete - we'd be thoroughly stuffed if we had. Interesting to note that Sophos' advice changed through the morning too! I'm glad to report that the latest advice - to stop the on-access scanner via the enterprise console and force an update does seem to be working
Thursday 20th September 2012 11:37 GMT Anonymous Coward
Thursday 20th September 2012 11:45 GMT Zombieman
I think this might be the first time I've heard of an anti-virus/security package classifying itself as malware, though having said that no doubt it has happened before. The more typical rogue AV headline is when operating system files are involved... Bit unfortunate that it targeted the update mechanism... I have this vision of a developer standing like a scolded kid, head down, tracing a partial-circle with the toe of one foot say "uhhm... I did something silly..."
Thursday 20th September 2012 12:08 GMT zaphy42
Thursday 20th September 2012 14:21 GMT DS 1
Got slammed last night, went to bed at 4am.
I was saved, not by planning or judgement, but just luck really. We were conservative in our settings and chose to use deny rather than delete. And seems the two core imlosions were along the line of delete or deny. Deny did not actually delete or move files, so when you disable on access scanning you have a path back to sanity. The delete is nuclear for some people, as the dammed ide really took out all kinds of update processes/programs/settiings.
I was able to pull back from the brink by doing wide sweeps of knobbling on access scanning, updating, and then re-enable - starting with the central console. But not really fun, and only saved by delving to establish our own conservative setting in the on access options in our policies.
Some people are going to need a beer, and I feel for them. Somebody at Sophos or a process at Sophos needs firing/canning, because this got past Qand A and never should have.
How the hell does a process that eats your own product escape past QA. So far not really seeing that being answerd properly. Its all good saying sorry, but what will next week bring. A dislike to Exe files. Maybe DLL files. Or maybe NTuser.dat.
Part of the problem with this is that in this circumstance of unknown calamity - is will the next issue we face leave us in the unforseeable worse off if we all use Deny instead of delete. My crystal ball can't say sadly.
Thursday 20th September 2012 16:14 GMT cyberdemon
Thursday 20th September 2012 19:26 GMT ElSteveo
400+ desktops and my entire serv estate was hit by this, being in a school that has kids constantly bringing infected stuff in we had set our policies to Delete.....after 12 hours of trying to sort this Ive decided to change the decision on that one!! :D
It's more soul destroying to see the long list of other software it appears to have removed as well, as a single person department this is something I REALLY didn't need. A free mug is the least you can do Sophos! :)
Thursday 20th September 2012 21:05 GMT Euripides Pants
Friday 21st September 2012 09:56 GMT Danny 14
mainly because the original update broke the SUM too so even though they released a fix the SUM couldnt download it and sent it to clients. Also the fact clients had their autoupdates quarantined meant the fix wouldnt roll out either.
So unless you have 24hr IT support or live in a timezone that was awake during the debacle then you would wake up to clients going mental.
Friday 21st September 2012 00:09 GMT Anonymous Coward
Friday 21st September 2012 09:54 GMT Danny 14
damn them all to hell
The main pain in the arse for me wasnt the update - only the machines that were on at the time (about 350) go the "bad" update, i'd fixed the SUM by the morning so the majority of machines got the fixed update. No, the pain in the arse was the fact it quarantined a host of other apps too, java, adobe and some bespoke software we run. They will need "manual" care and attention, even using PSTOOLS remotely I will still need some hands on with them. That is not possible in some remote worker cases and they are needing scheduled RDP time.
I have really liked sophos over the years but errors are creeping in slowing (migration to EC 5 was a nightmare for us) I think it is time to look for a different provider come renewal in January.
Any ideas on decent enterprise AV for 1000 machines? Needs an EC style approach to managing the AV rollouts etc.
Friday 21st September 2012 11:52 GMT Psymon
Thank goodness for SCCM
Thankfully, I managed to create a custom task sequence to fix all the clients.
Using file inventory, I managed to create a collection query that listed all the machines containing the agen-xuv.ide.
I then advertised a task sequence that ran:
net stop savservice
It then deleted said file (several caveats for differing install locations, x64 etc.)
net start saveservice
This filtered through and cleaned 6k worth of clients in about 2 hours. I'm just glad I have VPN and RDP on my massively oversized Android phone. I had 90% of the solution in place while I was still on the bus to work.
Our poor email server is another matter - thankfully, not under my care!
Friday 21st September 2012 15:30 GMT Mandoscottie
Looks like Im the luckiest sysadmin (running SAV) on the planet.
Out of 300 installs I had 1 user affected, he was outwith the perimeter so SAV endpoint on his laptop used sophos.com for the update.
Talked him through manually updating when the fix was available and bobs your mothers brother as they say :) Ill give Sophos this one, first real issue ive had in the 6years ive managed their enterprise solution.
no other system was affected onsite.......most odd :o)