Re: Flawed study?
Excellent points, Alan! (Hi there, BTW. Long time, no see. Yes, I'm still alive.) Here are a few more:
1) If Imperva are selling a security product, then it is highly unethical for them to test (or even comment on the quality of) other people's security products. They are obviously biased. As the following points demonstrate, they are incompetent, as well.
2) They don't seem to distinguish between viruses and malware in general. Most of what they have used in the tests were not viruses but various kinds of Trojans. Trojans don't "spread"; only viruses are able to replicate themselves. It is because of this lack of self-replication that the spread is low and the AV vendors haven't got samples or got around to implementing detection of them. With thousands of new malware variants appearing every day, the AV vendors are forced to concentrate on handling the more widespread threats first.
3) They don't seem to understand how AV works. There are two main kinds of AV solutions - malware-specific ones and generic ones. The malware-specific ones (commonly known as "scanners") is what most people think of when they talk about AV products. As their name suggests, such products detect KNOWN malware - known to their producers, that is. If it is not known to them, they won't detect it. Revealing the "troubling" fact that such products are not very good at detecting unknown malware is like saying that a screwdriver isn't a very efficient tool for nailing nails. It's true, but it is a completely pointless statement and only reveals the incompetence of the person saying it.
The generic AV products (of which there various kinds - heuristic analyzers, behavior blockers, integrity checkers, etc.) try to detect malware not known to them by using some generic knowledge about its structure or behavior (like "if an executable file tries to modify another executable file, this is suspicious" or "if a set of executable files have one and the same code at the end and this code receives control when the file is executed, then they might be infected"). Unfortunately, it is mathematically provable that it is impossible to detect all possible viruses without causing false positives. (The proof is constructive - i.e., if you claim to have an algorithm that does it, the proof shows how to construct a virus for which the algorithm will fail.) In the above examples, the "executable modifying other executables" could be a compiler or a linker, and the files having common executable code at the end might be compressed and executing the decompressor at runtime. So, most AV products of the generic kind try to strike some kind of balance between detection and false positives.
Most AV packages nowadays try to combine products of both kinds. However, VirusTotal uses only the known-malware scanner part of them. Testing it with unknown malware is simply wrong.
Finally, even if Imperva's claim were true (which, I contend, it is not), would you rather use something that gives you a 5% chance of protection or nothing at all?