Bad benchmarks bedevil boffins' infosec efforts

A group of operating systems specialists has said that sloppy benchmarking is harming security efforts by making it hard to assess the likely performance impact of security countermeasures. The researchers from the Netherlands and Australia, decided to take a look at the accuracy of security researchers' systems benchmark. As …

[a_yank_lurker]

Benchmarks Valid?

All the benchmarks I have ever seen are like EPA gas mileage figures; artificial numbers the are optimistic at best. But numbers because of how they are derived have some comparative value at least in a gross sense. EPA mileage figures do not actually account varying load, weather conditions, etc. that affect the real world numbers. Benchmarks have the same problems, what is the proper balance between CPU load, memory load, drive reads, etc. to mimic the mythical average user? The 'crime' is not that benchmarks are skewed but that they researches are misusing them to try predict behavior in the real world when they are only good tell general trends.

[The Count]

Re: Benchmarks Valid?

Can anyone spot the multiple crimes made by this bot against the English language?

[Anonymous Coward]

Is there anyone out there who is good at performance and security?

'As they explain in this paper at arXiv, security papers are littered with so-called “benchmarking crimes”'.

Ah yes, just as the chip optimizing experts have been committing security crimes.

The wonders of specialization...

[John Smith 19]

"Ah yes, just as the chip optimizing experts have been committing security crimes."

But it sure did hit their benchmark targets, didn't it?

I'm curious. How many systems don't start with the assumption (and it is an assumption) that the OS code is absolutely correct, and should therefor have full access to the machine?

Because IRL, over and over again, we've seen that is complete bo***ks.

We know every substantial piece of code will have bugs in it, and they will escape detection whatever level of testing it was given before release (which appears to be f**k all in some cases).

So what OS developers plan for bugs in the code?