back to article Is AV product testing corrupt?

I had a conversation a month or two ago with someone high up in one of the IT security companies. He was bemoaning the fact that his company's AV product had performed poorly in tests run by He was deeply suspicious of the results anyway because his company actually provides its AV engine to another company that had …


This topic is closed for new posts.
  1. Steve

    You're so vague...

    ... (I bet you think this post is abut you, don't you ?)

    In all seriousness, so what ? I thought El Reg had given up on this fool, with vague doom laden FUDmongering.

    Presumably this is in support of the previous series of 'AV is dead, long live the whitelist' FUD pieces ?

    Deeply content free.

  2. Steve Browne

    What the computer industry needs is ...

    ... an OS that works !

    I know we have Linux, I like Linux, I used it for years, fantastic.

    But, most people have Windows which is riddled with problems that M$ cannot or will not address.

    The simple inclusion of a su command, so you don't have to log out and log in as administrator to install software, wouldn't be a bad start. Then we can set ourselves up properly, knowing that NOTHING is going to install itself without us KNOWING what is going on.

    How about read only directories, this will prevent all those useless dlls appearing in system or system32 that are not needed any more, also makes deleting things easier if something has to install into its own directory. Then stop install processes from updating anything other than its own installation directory.

    Get rid of the registry entirely. This has got to have been the worst thing ever done in Windows 95. I thought it a big mistake then and I have seen absolutely nothing to change my mind. Interesting that .NET doesn't use it.

    Make IE an application again.

    Display more information in the process list, so that processes are identifiable.

    Provide a proper "End now" function that cancels the address space, never mind telling something to shut down, just close it, no dialog boxes, no messages, just deleted from memory. When I want to shut down my computer, I mean exactly that, shut down. I don't care if Symantec anti virus doesnt want to go away, it is my machine and it is going to stop, either by itself or with the off switch (the little black on on the back of the case, or the white one on the wall).

    Provide monitors for important services, such as what is sending on the network and where, or what is accessing the disk.

    Add in a facility to make files private. They can be used by the application that created them and nothing else, without explicit permission.

    Reduce the number of executable file types. We have exe, com, pif, bat and probably quite a few more.

    It isn't anti malware we need, it is a properly constructed OS.

  3. t3h

    All we need is... independent review of the reviewers. Problem solved.

  4. voshkin

    Threats Independently Tested Service

    The answer is quite simple...

    1. Install VM

    2. Install XP/Vista on the VM

    3. Clone the VM

    4. Install different AV packages on the VMs

    5. Sysdif the VMs

    On a separate VM without AV (also SysDifed) go to various nefarious websites. Best way to get a virus nowadays is search for “cracks” and “keygens” – any top 20 Google results would infect you 100%. Also, do a search for *anything* on Emule, and save all the <100 KB exe files.

    Download a bunch of “Keygens”, and save the EXE files. Run Sysdif again, and find all the lovely EXEs dropped by browser vulnerabilities. Save those. Restart the VM a couple of times to make sure dropped EXEs download more EXEs, save those too. By now, you would have at least 50 “current” viruses. (To get even more, setup compromised email account, and run everything that comes as attachments, and click on all the links)

    Now that we have a whole bunch of Viruses, and spyware, key loggers, and so on, we can do the actual tests

    6. One by one, run each of the EXEs on a VM with AV product on test.

    7. After each EXE, sysdif the machine (even if AV claimed to have eradicated a threat)

    8. Save the results in xls, plotting a nice graph, that will show... that not a single AV package will protect you from all the threats.

    The problem with the AV industry is that they tend to cater to Joe Blogs who reads the sun online and Jane from accounts, who looks for carrier change as a lion tamer. That’s it. The Viruses that those individuals are likely to come across will be long discovered by the AV companies, and in most cases added to the signature database. They tend not to search for MP3’s nor do they try to download a no cd crack for a game they copied from a friend. But that is where “it’s all at”.

    This AV industry status quo is shattered when you through a teenager in the mix. And bam! Your home computers are well and truly screwed! I mean, which teenager will refuse a file called “best joke ever.exe” from a random contact on MSN?

    To this end, I call for El Reg to setup “Threats Independently Tested Service” I for one, would gladly contribute by submitting all the nasty malware I come across, and believe you me, that is plenty!

  5. Chris Goodchild Silver badge


    Undoubtedly the best influence to decide which AV or any other housekeeping/security product to use is peer recommendation. Your mates have no financial axe to grind when they recommend an AV or Anti-spyware product, whereas almost everybody else has. Even the likes of Which magazine, when recommending domestic products is so busy with the vast selection of goods it tries to test, it uses teams of people who are not really qualified to test the product they receive, it's a question of ecnomics.In the IT industry the testers need to be on some kind of speaking terms with producers of the products to test, nothing to test no business, economics. Much as I dislike rules and regulations there is a need for an international regulatory body with teeth to help raise standards and maintain them, the internet is now a fundamental part of world socio-economic culture and needs something to help maintain standards in a similar way to to the envisaged role of the UN.

  6. Remy Redert


    But then who will review the reviewers of the reviewers? You've got an excellent endless loop going there.

    And Windows is indeed an insecure pile of you know what, but even linux has had its share of problems and bugs. There is no way to get rid of malware entirely, it's going to stay in existance for as long as computers exist.

  7. Anonymous Coward
    Anonymous Coward

    What the AV industry needs...

    is an AV product that un-installs when it's told to. I'm fed up with cleaning the file system and the registry when I want to change product. Most of the time the best way is to re-install the OS.

  8. Anonymous Coward
    Anonymous Coward

    Almost correct..

    "There is no way to get rid of malware entirely, it's going to stay in existance for as long as computers exist."

    Should be..

    "There is no way to get rid of malware entirely, it's going to stay in existence as long as _people_ exist."

    By definition malware is created by a person for a nefarious purpose (and is often installed inadvertently by the person).

    No people, no malware.

    Once the computers work this out we're screwed, of course (search El Reg for ROTM to find out how).

  9. voshkin

    a user who is not an idiot!

    >Steve Browne

    Well, we sort of have one now. Called Vista

    • Su command – got one, called “run as administrator”

    • Display more information in the process list – got tons of free utilities that do this

    • Provide monitors for important services – got this too, built in

    • Provide monitors for important services – got that too. File level security in NTFS

    It’s not bloody Microsoft’s fault that the users are morons! If one elects to download a 100 kilobyte exe/bat/pif/cmd/scr/whatever file purporting to be “die hard 4 movie” then how exactly is that Microsoft’s fault?

    IF you elect to stuff a dead rat down your exhaust pipe, it is NOT your car manufacturer’s fault that they did not cover the exhaust pipe with a mesh!!!

    Nor is it your Microwave Owen manufacturer’s fault that your cat’s eyes, testicles, and cardio-vascular system explode when you decide to dry it.

    Nor is it the fault of your hi fidelity system manufacturer when your system burns when you decide to switch it from 240 to 110 volts, thinking that doubling the supply voltage will double the sound output.

    Need I go on?

  10. Dr. Vesselin Bontchev

    Testing AV is an extremely non-trivial task

    It seems that Mr. Bloor is simply incapable of posting anything that I can't disagree with - even though most of this particular article isn't even his. :-)

    But, basically, yes, AV tests suck. Big time. The problem, however, is not that the testers fiddle with the statistics or that AV companies submit "special" samples. The problem is competence, or more exactly the lack of it.

    Testing an AV product *properly* is an *extremely* difficult job. Although we occasionally see tests that are not so bad (e.g., Virus Bulletin's, VTC Hamburg's, etc.), *none* is what I would call "excellent" and the vast majority of them are terribly bad. I can confidently say that nobody of those currently testing AV products has sufficient competence, resources, time and manpower for the job. Those who have the competence are world-class AV researchers - and they have been snapped up by the AV companies long time ago and, as such, cannot do independent AV product testing due to conflict of interest.

    Describing how to conduct AV tests properly is waaay outside the scope of this simple comment (I recently participated a 2-day workshop on the subject where my speeches covered only a small aspect of the job) but basically you need:

    1) Proper malware collection. This means about half a million currently known different malicious programs and a testing team who is able to analyze every single one of them, figure out which ones are viruses (and which samples are simply non-working crap), replicate them, figure out which samples contain the same virus (possibly polymorphed), classify and order them properly. Wrong shortcuts currently used by incompetent testers: put in the collection anything that a scanner reports as something or use only the set provided by the WildList Organization.

    2) Testers who understand exactly how every single tested product works, what are its components and how to test them properly. And, believe it or not, the different AV products work in vastly different ways. Wrong shortcut currently being used by incompetent testers: just test the on-demand scanner component of the product.

    3) Lots of time, people (competent ones!) and disk space (terabytes), plus helper tools that you develop in-house to facilitate some of the tasks.

    Nobody currently has the capability to do all of the above properly - and I mean NOBODY. That's why the AV tests all suck.

    To Steve Browne: you can already do most of what you want in WinXP+NFTS. SU-ing only for installation, read-only directories, browsers that are applications, monitors, private files, limitation on what is executable. Problem is, in order to do it, you have to be a competent Windows sysadmin (which you obviously aren't). And if you put a Joe Luser in charge of adminisering a Linux box, he will screw it up just as surely as a Windows box. The problem is that Windows is used en-masse, while Linux is used by a few tinkerers who know what they are doing. The mass of people are *not* competent sysadmins. They will screw up *whatever* they are forced to administer, no matter what OS it is running. Make Linux as widespread and as easy-to-use as Windows (*both* factors are essential) and the malware problem will remain the same - except that it will be malware for Linux.

  11. Mark Totton


    This is no different from the behaviour of all special interest groups. Car tire manufacturers pay to ensure their tires score well in "independent tests". About the only reliable source of information is anecdotal reports from people you trust: "I use xxxxx and I haven't had any problems for years".

  12. Richard Thomas

    Aweful Journalism

    This is just a plain made-up article.

    Having worked in the industry for near a decade this just isn't how AV products are tested.

    There in fact IS a standard test set by which all products are tested

    How can El Reg allow someone to publish an article on something they obviously no nothing about?

  13. Dan White

    "My computer hasn't had a malware infection in YEARS..."

    Am I the only one that thought, "How do you know?"?

    Given that he admits his product was in the bottom half of the reviews, could it be that he *has* been infected and his crappy detection software doesn't know it? :-)

  14. Anonymous Coward
    Anonymous Coward

    Trusted source

    The basic premise of this article is that the virus samples are apparently 'distorted' prior to comparitive product testing. Surely if a trusted organisation like Thwarte held a definitive collection of up to date in-the-wild sample viruses, then running procedural tests using vms and these definitive viruse samples should produce a consisent - ie reproduceable - set of results. Anything less than this is little short of corruption & fraud IMHO.

    I'm sure that there would be persuasive corporate 'encouragement' to find a given result, due to the financial implications riding on the outcome - so perhaps the entire process should be undertaken by someone like Thwarte with no contact / announcement to the AV industry apart from to reveal test results periodically.

    I have no connection with Thwarte - it just seemed like a good example of a trusted company. Substitute Thwarte with any other company you considered trusted ;)

  15. Kevin Hughes

    The best of everything

    Doing your own testing and, as Chris already pointed out, peer review are the only two sources I trust. If you believe what you see on the Internet and TV then every company is the "greenest", every product is "product of the year", every AV app "got the highest detection rate in independant tests", every Premiership season is to be "the biggest and most explosive yet", each political party is "listening to what the people want", Daz makes your clothes "whiter than the old Daz" and Vista is "A great OS" :P

  16. Anonymous Coward
    Anonymous Coward

    FUD? Certainly.

    -Fear that AV is not going to protect your data.

    -Uncertainty that vendors really are doing what they need to.

    -And Doubt they will ever get it right the way they are going about it now.

    The best testing I have read about is the Wild List. We don't care that our AV could catch 3k bits of bad code [many of them relegated to the dark archives], but we surely want it to catch the top 10, 15, 20+ most prevalent and active items out there. The statistical probability that I will get infected by something with a very low global footprint is, well also very low. Hmm, wait, what if I'm one of those execs who is extremely targetted... well, AV will probably not save me there anyways.

    For me this is a legitimate article. I Fear it won't cause a significant discussion that causes AV vendors to get their act[s] together; I'm uncertain whether white-lists will solve the problems and I certainly doubt the game against the VXers will be addressed without significant changes in the consumers of these products demanding more from the AV industry...

    Statistics? Hmm, why is it the worst performing vendors seem to be the most popular. Aha! They're not! Or they are... depends how you count these things.

  17. Anonymous Coward
    Anonymous Coward

    Here we go again; what a bloor.

    Monster mighty mouth Bloor; king of the anti-antivirus advocacy is at it again spouting such gems as "As an analogy, if I was looking to build a system to detect cancer, I'd build one that detects every kind of cancer that's out there." Yet another piss poor article built around a piss poor analogy.

    What I say is — Ha... hardy ha ha haar.

    This man must be part of a new age religious sect. I can see him now leaning over his Windix crystal ball as it coughs up the coding from vxers yet to be born or even conceived using its direct MQ-Series XML interface that pumps the new coding into his system to detect every kind of malware.

    I'll tell you what; if I had that ability I'd be queuing up outside my local branch of the U.S. patent office patenting every algorithm that pops out of my Windix.

    Without a doubt signature based Antivirus isn’t the be all and end all but I certainly wouldn’t take it out of my armoury as this man suggested in his article

    I think only a multi-tiered security strategy will save the day.

    So bloor, go research your nads matey: it’s likely to be the only thing you can grasp and truly understand.

  18. Pascal Monett Silver badge

    @Dr. Bontchev

    "if you put a Joe Luser in charge of administering a Linux box, he will screw it up just as surely as a Windows box"

    Now that is a golden nugget of truth if ever I have seen one.

    For me, all AV is wrong from the start anyway. Security should be done by allowing only known processes to run, not by running anything first and checking it's okay second.

    But hey, the industry has to pander to grandma Higgs just as much as has to suit Professor Sprout, PhD. Schizophrenia is never a good thing to have.

  19. lansalot

    re: Threats Independently Tested Service

    This sounds to me like a fantastic idea.

    If such a system gets off the ground and is needing volunteers, then I'd like to extend an open hand towards T.I.T.S. Indeed, I'm sure many of my brethren, sick and fed up of malware, would rather spend time with T.I.T.S than trying to work out which AV is best.

    Perhaps some certification for those T.I.T.S-testers would be in order, I mean it's not like 'we' (see, I'm already in the community-spirit) could let just anyone get into T.I.T.S.

    ISC/SANS has their handlers, and, without question, T.I.T.S should have theirs ! There'd need to be accreditation of some sort - to qualify and be recognised as an *U*n-biased *P*rofessional for example

    To that end, I would suggest the T.I.T.S-U.P accreditation programme, whereby those showing keen-ness and willing would be elevated within their community, their pink noses held-high.

    It's a starting point - any offers? Who likes the idea of T.I.T.S? I'm sure we could get it off the ground, if we all pull together !!

  20. Mark Allen

    AV Programs - or selling black magic

    "My computer hasn't had a malware infection in YEARS and according to the latest "review" in a major magazine, my product was in the bottom half. Hmmm, kind of makes you wonder, doesn't it?"

    This tells me you don't have teenage children. From my experience, it's the "immortal" teenager who thinks (s)he is safe walking into the roughest of websites to download more free music\cracks\pr0n.

    Next are those people with "friends" who constantly email all kinds of junk, "funnies", chain letters, and fake virus warnings. It is these people who then get caught out by infected e-cards.

    I tell my clients that the Internet is like a strange, unkown city. Stay in the recognised shops\websites and life will be trouble free. But the more Googling\Yahooing\searching, then the more a game of Russian Roulette is being played.

    My personal annoyances at AV Reviews is the effective load on the host computer when running. The average home user has an old, underpowered Celery PC with 256MB RAM. They then walk into a highstreet store and pick up Symantec\McAfee products that are now so overloaded with useless "features" that the PC grinds to a halt, and they get in a habit of saying "yes" to every sinlge manic dialog box that pops up.

    So many of these people loose effective use of thier PC, but yet never ever trigger the scanners on anything nasty. So I'll clean these bloated programs away and replace with something smaller like AVG\NOD32\AntiVir\etc and separate spyware scanners. The comedy is that these often immeditaly find nasty infections the "big boys" have been ignoring!!

    Want a good "real world" test? Just install the AV programs on some PCs in an all boys school, then leave them access to the PCs and tell them no one is supervising them..... Soon find out which are the best protected products. :D

  21. Edward Fingleton

    @ Steve Browne

    right-click on an installation file and click "run as" it's effectively su

  22. Andy Silver badge

    My computer hasn't had a malware infection in YEARS either

    ... since I stopped using IE and Outlook, in fact.

    And don't get me started on Symantec. My sister's PC was rendered completely unusable by Norton AV chewing up 100% of the processor, all of the time. And she paid money for it. Just shows that marketing beats good coding every time.


  23. Anonymous Coward
    Anonymous Coward

    Journalism 101

    Check your sources!

    "I had a conversation a month or two ago with someone high up in one of the IT security companies. He was bemoaning the fact that his company's AV product had performed poorly in tests run by He was deeply suspicious of the results anyway because his company actually provides its AV engine to another company that had performed better in the test. He didn't see how that could be, unless a mistake had been made in running the tests."

    The someone may have been feeding you bullcrap by the bushel. Have you checked his story? Are you sure that Secret Company B (which got rated higher) isn't using some sort of secret sauce in addition to the engines provided by Secret Company A (which is doing the bitching)? Something like, oh, I don't know... a second antivirus engine used to double-check on suspicious items? But then, if you had and were, you'd be naming names and pointing fingers, instead of peddling vague rumors and misinformed analogies.

  24. Mike P

    The best AV...

    ... is VirusTotal (, and a good backup software in case of infection.

    I have a PC since 1991, WITHOUT any AV, WITHOUT any firewall, and NO spreading infection so far (on Win95/98 and now W2K). And still, I used to visit sites like How? I know my PC very well, detect all strange behaviours, and in particular avoid running any strange exe. It's only since the last 2 years that I'm much more careful due to explosion of malware, and hence I use VirusTotal much more often, but only on files that I decide to scan. I won't let a stupid AV scanning a file that I know for sure it is clean since it is on my PC for years !

    AV SW DON'T HELP YOU! They simply fuck up your PC, turning your superb ferrari into a 2CV ! AV are actually the only true viruses on earth, and you ought to pay for it??? What a joke!

    I had an infection on the PC on my wife, however protected by commercial AV. An unfortunate visit on a underground cracking site triggered an exploit on IE. Javascript error, with strange title in the windows. Then few sec later, WinXP telling me that PC is at risk because FW was turned off. Then a strange icon on the desktop of a malware detection software that was not installed on the PC before. The app launched automatically, telling me the PC was infected with more > 50 adware, viruses, ... All this in less than 15-30 sec !!!

    I shut off the PC, pull the network plug. Restarted the PC, and run the AV. He detected ~20 virus. By comparing the new files on the PC, I noticed there was at least 50 new files that was created in the last minutes before shut off, and the AV only detected LESS THAN 50%. I copied those files to my own PC, and run a online AV scanner on it. The online scanner detected 90% of the files. Two days later, I rescanned the files, 95% was detected online.

    The PC was completely unrecoverable !!! How would you trust any AV to correctly clean the machine, when YOU CAN SIMPLY BACKUP-RESTORE IT very easily, with 100% confidence on the result ???

    If you understand computer a bit, you don't need AV ! AV simply sucks the power of your PC, continuously scanning and rescanning over and over the same files again and again. I doesn't make sense. Backup-restore and common sense is the best answer. AV are for neophytes and for company PCs only, and yet...

  25. Anonymous Coward
    Anonymous Coward

    Real World vs Simulated

    Ok i am a bit out my technical depth here but i would have thought the soultion would be to get a nice fresh PC then dump as much Malware on in it as you can without killing it then run your AV products ans see who finds, removes or misses the most.

    Or is that to easy?

    I remember cleaning an infection from a friends PC, i just went through all the AV products out there until i found one that cleaned it. Interestingly though it was a free one that did the business and i have stuck with it ever since.

  26. Morely Dotes

    Re: What the AV industry needs...

    "is an AV product that un-installs when it's told to. I'm fed up with cleaning the file system and the registry when I want to change product. Most of the time the best way is to re-install the OS."

    If you'd refrain from installing crapware such as Norton/Symantec/McAfee you'd find the problem doesn't really exist. The only ones that are difficult to uninstall are the ones that also demand an annual "protection" fee. The free AV products are both more effective, and perfectly willing to uninstall if told to do so in the proper way. I'm currently fond of AVG, but have used ClamWIN and Avast Home in the past, with good results.

    And, contrary to Bloor's unsupported allegations, I find that VB100 is a perfectly valid standard, using scientifically-sound and reproducible methods detailed by Virus Bulletin. I question who paid him to publish this press release, thinly disguised as an "informative" article.

  27. Anonymous Coward
    Anonymous Coward


    Oh i just read that post on symatec and i couldn't agree more in fact i have refused to use it, build a system with it, allow it within 10 metres of my home since win95 OSR2.

    On every machine i have ever encountered it on it chews up system resources whilst in my opinion failing to cure some issues that free products did. My main issue with it though is the sheer unadulterated grief of trying to un-install it. I personally think it should be classified as malware.

    Still shocks me that dare ask money for it but not unfortunately that people buy it. After all people will buy anything and i am living proof of that at least.

  28. Rob Crawford

    No change

    This was a problem was was pointed out in the early / mid 1990s

    In magazine tests McA**e always came out tops, but strangely enough all the samples that all the scanners where expected to find where also supplied McA**e

    When samples taken from the wild where used McA**e dropped down the list, and Thunderbyte and fProt suddenly became the most successful.

    The reports where a hyperlinked file that apepared monthly, whos name I unfortunitely cannot remember. As far as I am aware they where the only 'independant' source of antivirus packages around

  29. James

    Malware for Linux? su?

    @Dr. Bontchev

    "Make Linux as widespread and as easy-to-use as Windows (*both* factors are essential) and the malware problem will remain the same - except that it will be malware for Linux."

    While there would certainly be an increase in Linux-targeted malware as its profile grew, you are wrong if you think said malware would have anywhere near the penetration level currently enjoyed by Windows-targeted virus/malware. Go ahead and try it on one of your own Linux systems or pay somebody who knows how to do it. Whatever ... that comment impugns the rest of your wisdom, which is otherwise quite good. You forget, or didn't realize, that Linux has the largest share of the web server market, making it an extremely attractive target. And yet, we haven't seen a successful mass infection like we do several times per year with Microsoft servers. The current Storm comes to mind. If Linux is as insecure as Windows, why isn't there a Storm for Linux? The payoff would be larger.

    And @a few others: "Install as Administrator" is NOT like su. Having set up 25 XP Pro workstations and, more recently, 12 Vista Premium boxes, I can assure you that many programs have a great deal of difficulty running when using "Install as..", and the same programs run just fine when I switched users to the Admin and installed from there. Even then, for one example, WordPerfect 9 will not allow a normal user to run it because of a system call that is restricted to Admin use only. Once installed by an Admin, Limited Account users still need to use "Run as.." to get all of the required permissions despite choosing to allow any user to run the app during the installation.

    su ACTUALLY gives you the full rights of the user you switch to ... and, oh yeah, su can be used to switch to ANY other user, with the proper permissions ... not just the default Admin. "Almost su" is NOT "su".

  30. Phil

    No different to the game industry they

    I remember going to a party around 15 years ago thrown by a very well known owner of a very well known games company (company since brought out, owner still always in the news with his current company)

    Was a great party, hash chilli, most people stoned and I had a great time.

    What was interesting to me was the guests. They consisted of people like me (friends of the programmers), the staff of the company, but the vast majority of the guest were various magazine reviewers etc, all on first name terms with this guy.

    Guess what, his games always got good reviews.

  31. Anonymous Coward
    Anonymous Coward

    I like what he said (quote inside)

    What the AV industry needs...

    "is an AV product that un-installs when it's told to. I'm fed up with cleaning the file system and the registry when I want to change product. Most of the time the best way is to re-install the OS."

    Wow, now that would be amazing. In particular being able to un-install the 'free 30 day trial' that came with your computer cleanly would be nice.

    Its bad enough having to deal with spyware and other foo'y with out having to also fight with your AV or similar product too. I wonder why the reviewers never make any mention of ease of use and things like un-installing.


  32. eris23

    never had a provable virus, but...

    I ran AV programs and multiple anti-spyware programs (on Windows XP). Never had a virus, but had a piece of spyware indistinguishable from one (pop-up, after pop-up, after pop-up -- I had to reboot to Safe Mode to get rid of it).

    No one has the resources to check everything. Any advertising--supported media has a disinclination to negatively review their advertisers.

    I once subscribed to 4 Macintosh magazines. The one I believed had on ads (but cost 5-10 times as much).

    You get what you pay for (sometimes).

  33. A J Stiles

    You get what you deserve

    There is one important criterion for rating security software of any kind, and it's so important that the others alone won't make up for its absence.

    That is: Access to Source Code.

    Without access to the Source Code, you cannot verify that your software "does what it says on the tin" -- and you don't know for sure that it isn't doing things they wouldn't mention on the tin.

    More people need to get vocal about this! If you've paid good money for an application, you should damn well demand access to the Source Code. Would you buy a cake if it didn't have a list of the ingredients, and the breakdown of calories / fat / protein / carbohydrates / fibre on the box? Of course not! So why do you tolerate it when software vendors behave like this?

  34. Steve

    RE FUD? Certainly.

    Which would all be fine, if this were even as analytical as some of the comments that have been posted. As it stands it's not far off just saying "Some bloke down the pub told me that AV software isn't all it's cracked up to be."

    Exactly what we've come to expect from Robin Bore, but still.

  35. Dr. Vesselin Bontchev

    It ain't that folks don't know, it's that folks know so much that just ain't so

    Richard Thomas: By "standard" you're probably referring to the WildList set of samples? I see that some anonymous poster has mentioned it too. Well, folks, the WildList is CRAP. It's an arbitrary set of viruses that bears no similarity with what is actually in-the-wild (ITW). Things that are ITW are not on it. Things that are on it are not ITW. It's crap - mostly because of the utter incompetence of its maintainers. I have an article on this subject, google for it. The only thing such a "standard" provides is reproducibility of the tests. However, the tests based on it are absolutely no measure of how well AV products fare against what is actually out there, infecting people's machines.

    Anonymous: Who the heck is Thwarte? Do you mean Thawte - the PKI guys? Anyway, NOBODY outside the AV industry has sufficient expertise to maintain a well-organized set of virus samples. NOBODY. Not Thawte, not anybody. How do you think this is done? By running a bunch of scanners on the samples and seeing what they repport? The proper way to do it is by analyzing every single sample - which means that you must know a lot about viruses, reverse-engineering, file formats and a whole bunch of other things. And if you already know that, you work in the AV industry, guaranteed. Which means that you're too busy developing your product and don't have time to organize somebody else's test set. And how would Thawte, who have ZERO virus expertise, decide exactly which viruses are ITW?! Ho-ho-ho.

    Pascal Monett: You're falling into the pit of "whitelisting" - and I've debunked this myth here before (see also my paper in the August issue of Virus Bulletin). Who exactly will decide which processes are "known" and "should be allowed to run"?!

    Mike P: Sites like VirusTotal and Jotti cause more harm than good. I'm writing an article for Virus Bulletin on this subject right now. The people running them have absolutely no clue. The scanners they use are often different from what is sold to the customer. If a couple of scanners have a false positive, the whole AV industry is forced to deal with the "but why don't you detect this" syndrome. The samples that the sites provide to the AV people are 99.99% crap and unnecessarily tie their resources to sift through it for the occasional gem (i.e., a genuine, working, new virus). "If you understand computer a bit, you don't need AV !" - very true. Sadly, the remaining 97.24% of the computer users still need one.

    Anonymous: "get a nice fresh PC then dump as much Malware on in it as you can without killing it then run your AV products ans see who finds, removes or misses the most" - problem is, there are half a million known malware programs to choose from. If you're going to use only a subset - how do you decide which ones to choose? If you're going to use every one of them - where would you find the time and other resources to do so? But, yeah, that's, theoretically, the proper way to test AV products - install the full product (not just the scanner!) and keep throwing live malware at it (instead of just scanning static samples safely tucked in a directory and never executed). Fully restore the PC between every two attempts. But it's way too difficult and time-consumming to do it properly.

    Morely Dotes: As I said, Virus Bulletin tests (VB100) are "not bad". Which doesn't mean that they are very good. They still use mainly the WildList test set (thank goodness, not exclusively) and still test only scanners (thank goodness, the on-access scanner too - not just the on-demand one).

    Rob Crawford: You're behind the times. In the early 90s (when Thunderbyte still existed), McAfee's product was indeed total crap. But they later bought S&S International (Dr. Solomon's Anti-Virus ToolKit) and incorporated Alan's excellent scanner in their product. Aquired several of Dr. Alan Solomon's world-class AV researchers, too. That's why McAfee's scanner is nowadays one of the best, as far as virus detection goes.

    James: You aren't paying attention. I said "if Linux becomes as widespread and easy-to-use as Windows (*both* factors are essential)". I cannot "try it on a Linux system" now, because Linux is currently not as easy-to-use as Windows. No easily clickable executable attachments in e-mail. No ActiveX. No Browser Helper Objects. None of the remaining crap that gets so heavily exploited in Windows. Yes, it's because of that crap that Windows is so insecure. But it's because of that crap that it's so easy-to-use and popular, too. Linux won't get as popular as Windows, unless it aquires this stuff too - which means without becoming just as easy to abuse.

    I know perfectly well that Linux is used for Web/ftp/news/email servers a lot. But there it is managed by supposedly competent administrators. Give it in the hands of the average lusers and it will be exploited just as much as Windows.

    "Run As" is *exactly* as su. Your WordPerfect example is irrelevant - it's just a crappy application that does what it shouldn't. There are plenty of those in the *nix world too - applications that need to have the SUID bit set. (Would have been nice to have the equivalent of chroot in Windows - but you can emulate even that with sandboixes and virtual machines.) "Run As" can be used to run as any other user too - not just as Administrator. You can even "Run As" a DOS command prompt or Explorer, in which case anything you launch from there will run as admin too - the equivalent of opening a root shell window.

    Everybody: Face it, folks, it's a free market. If it were possible to make an AV program that would stop all malware - somebody would have made it and we all would be using it by now. If it were possible to produce good AV tests - somebody would have started doing so by now.

  36. Anonymous Coward
    Anonymous Coward

    Keeping the elephants away

    I have heard people say "I use Norton / McAfee / AVG / whatever" and have not had a virus, it's really great". This does not prove it stopped or caught anything, just maybe they did not get attacked since they were not dumb enough to open that unknown file, visit that dodgy website etc.

    My Dad has not had an infection on his computer in my memory, and he uses one of the free-for-home-use products. Does that mean it is really good? If it came bottom of some league table does that make the league table flawed? I currently have no desktop AV, I have previously used a handful of different products (usually for the purposes of being able to review their usability) and basically found none of them particularly useful to me and I got fed up with dialogue boxes and requests for upgrades etc. The majority of the time I kept the live scanning off and just used on-demand to check the system once in a while, part of my routine maintenance along with defragging etc. I have not had an infection on a PC in my home or businesses *ever*.

    In a previous life as a sysadmin I occasionally had users call to say that their AV had caught something so I would trundle along and see what was going on. In every case that was the end of it. It caught it. Move along, there's nothing to see. If I had any doubt, I re-imaged the box.

    Education is better at stopping viruses than most AV software. That said, I heard a support guy tell someone recently that once they had installed their AV software they could check it was all working perfectly and doing a good job by copying the Eicar file. I pointed out that I could probably write a program to detect Eicar (and only Eicar) which would prove nothing else about it's capability as an AV scanner. It tests your AV is alive, but that's about it. It is also a good test for users - see how they react to it, and let them see what their AV does when it sees a virus so they know what to expect.

    What about those elephants?

    I sat next to a guy on the bus and he was tearing off small pieces from his newspaper and dropping them out the window. I aksed what he was doing and he replied "it's to keep away the elephants". I pointed out that there are no elephants in this part of the world. "See - it works", he replied.

    If someone tells you their AV is great because they have no infections, ask how many it actually visibly stopped. If they say "none", tell them about the elephants.

  37. Anonymous Coward
    Anonymous Coward

    RunAs sux, as do most Microsoft applications

    RunAs sux, CPAU gives far more control to serious users:

    I routinely have to run Windows XP Pro as Admin (no thanks to Microsoft and cretin application developers), with an complex password, and have few problems. I check out any unknowns in a VM and run servers in a separate restricted account (not the insecure system account) using CPAU.

    Some things which can help avoid malware issues are:

    * Always have a NAT/firewall router between your computer and your ISP, to prevent port attacks, do not expect security software to reliably block these attacks.

    * Don't believe the hype about Vista security, it isn't much better than XP, just wastes loads more memory and adds more stupid annoyances for users.

    * Run as little closed source software as possible, especially not Microsoft, Symantec or McAfee applications. IE, Outlook and closed source P2P software (e.g. Kazaa) seem to be the main Malware vectors.

    * Lock down IE solid then open it just enough for required IE based (insecure) client applications to run; when not using these applications set the connection/LAN proxy to and an unused port, to block IE based Internet connections.

    * Browse mostly with Firefox (with NoScript and Adblock extensions) and both most flash applets, so that 'adverts' don't infect your PC. Try to avoid sites requiring a plugin (yes I hate flash) because these plugins can bypass browser security and help malware onto your PC.

    * Keep applications up-to-date, exploits can be discovered in both closed source and open source applications, often in libraries, yes even on Linux and OS-X!

    * Be very wary if any software supplier tarts up an applications user interface before fixing poor-design in the core of the application; new code can add many new bugs and make software unusable e.g. Agnitum Firewall Pro 4.0

  38. James

    @Dr. Bontchev

    "You can even "Run As" a DOS command prompt or Explorer, in which case anything you launch from there will run as admin too - the equivalent of opening a root shell window."

    Thanks. Proof that su is not the same as "Run As..." And if you re-read my comment, I was addressing "Install as Administrator", which is even less like su. As you may realize, the above statement illustrates one of the problems with the Windows multi-user environment, and one which is the process invoked by a couple of new exploits for XP/Vista boxes.

    I suggest you grab a copy of Ubuntu and, hell, give Xandros a try if you like the Windows experience. Your not very well-informed about the nature of Linux, and you're surely misguided if you think that user-friendliness comes at the expense of security, just because MS has blundered its way to its present market-dominant position.

    Try a Mac!

  39. Anonymous Coward
    Anonymous Coward

    I haven't had

    any malware in years and I don't have any antivirus at all

    it's quite possible to avoid needing any of this crap and it doesn't

    really even matter what OS your using and the care required

    is minimal so as to whether or not any antivirus I might have randomly

    installed should be considered effective is pointless my personal

    feeling in this matter is don't bother with it because according to the

    tests I have seen in places like isc.sans it's quite likely your antivirus

    is among the vast majority that won't detect a new threat when it's danger is at it's height.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2022