Retbleed slugs VM performance by up to 70 percent in kernel 5.19

VMware engineers have tested the Linux kernel's fix for the Retbleed speculative execution bug, and report it can impact compute performance by a whopping 70 percent. In a post to the Linux Kernel Mailing List titled "Performance Regression in Linux Kernel 5.19", VMware performance engineering staffer Manikandan Jagatheesan …

[Anonymous Coward]

VMware

As the article points out, the findings are not surprising and warnings of performance hits caused by mitigations have been made since the beginning, as ElReg readers are aware:

https://www.theregister.com/2022/07/12/amd_intel_retbleed/

The question is why is VMware publishing this now? I assume it's to push their product and I strongly suspect Broadcom had a role in this.

[Anonymous Coward]

Re: VMware

I don't doubt that broadcom influenced it but having actual performance numbers never hurts.

[ssharwood]

Re: VMware

If Broadcom is directing VMware staffers in India to post stuff to the LKML I'd be rather surprised ... but I am not surprised that in the 6 weeks since kernel 5.19 came out VMware has found the time to test and oublish results.

[jake]

Re: VMware

"VMware has found the time to test and oublish results."

They threw 'em into the basement?

[sreynolds]

Re: VMware

I suspect because the mitigations recently made it into the code and they are able to provide some benchmarks.

Broadcoms influence is limited to withholding source code, reducing features and/or services and inflating prices - the typical output of your average marketing department.

[jake]

Well, yes.

Patching broken-by-design hardware with software will, in fact, slow down other software also running on that hardware. Is anyone really surprised?

[VoiceOfTruth]

Re: Well, yes.

Can you design better hardware? Off you go...

[georgezilla]

Re: Well, yes.

" ... Can you design better hardware? ... "

Well no, actually I can't.

But then I'm not in the hardware design business.

They are.

And it seems that neither can they.

Off YOU go ,,,,,,,,

[Twanky]

Faster, better*, cheaper...

We really can't have it all.

*Better in this context includes being resistant to attacks by the bad guys.

[Tom 7]

Re: Faster, better, cheaper, WWW.

"We really can't have it all." Take WWW away and you probably can. Many of the really good advances are fine for things that really need every last drop of grunt which should already be isolated from potentially suspect stuff.

[Twanky]

Re: Faster, better, cheaper, WWW.

Yes, things are faster, better, cheaper than when I started playing with computers in the late 70s. However if what you want is faster, better, cheaper than the current 'normal' then you're going to be disappointed.

'things that really need every last drop of grunt' are usually hugely expensive. If you really need security isolation then it will be expensive and may not be fast. As the article demonstrates, you can make IT cheaper by sharing resources (CPUs, memory, disks) but then you compromise security. When you try to improve the security, you compromise the speed.

Spectre-like attacks show a failure of imagination - a failure to imagine 'what could possibly go wrong?'

[Claptrap314]

Re: Faster, better, cheaper, WWW.

Nope. I was there. EVERYONE knew about the attack path. Many tried--AND FAILED--to realize the attack. And that's where things stood for 20 years.

In the front of the manual of these parts is a full-page warning that the part is not certified for use with government information classified CONFIDENTIAL or higher. If you consider your credit card information to be confidential, you might take than into account before using such a processor to handle credit card information.

[druck]

Just how crippled?

Just how much are x86 systems now crippled by the sum of all the mitigations since Spectre and now Ratbleed?

[sitta_europea]

Re: Just how crippled?

[quote]

Just how much are x86 systems now crippled by the sum of all the mitigations since Spectre and now Ratbleed?

[/quote]

When I 'upgraded' a customer site running Debian on Intel NUC devices (Atom E3815) to the first kernel which had patched some of the early speculative execution vulnerabilities, the performance was at least a thousand times worse. They were clicking something on the screen and then going out for lunch.

To recover I compiled custom kernels without the fixes.

[botfap]

Re: Just how crippled?

You just created shit loads of extra work for yourself. Maybe that was the plan? You will now have to do the same kernel customisation 10-30 times every year, every time a security update is published

Im sure it would have been much easier to boot with "mitigations=off" which would apply now and to all future debian kernel updates

[pip25]

Speculative execution exploits in the wild?

Do we know of instances where these vulnerabilities were successfully exploited by malware to steal sensitive info? The performance hits of the mitigations being as severe as they are, enabling them for use cases such home PCs does not seem like an entirely straightforward question.

[Claptrap314]

Re: Speculative execution exploits in the wild?

That's the hell of these vulnerabilities (not "bugs"). It isn't clear (NOT a security researcher here, but with background in hardware validation) how to attack a home users with these. This looks like a much more serious threat for the cloud providers.

[bombastic bob]

Re: Speculative execution exploits in the wild?

The other thing to consider is whether or not this is needed at all.

In the cases where "sensitive data" is NOT of a concern, why not disable the mitigation?

In the cases where "sensitive data" IS of a concern, put the mitigation(s) in.

Then disclose to customers which is which. Pricing would be related to what the customer wants.

In MY case, my company web site (shared hosting) has public downloads, customer related downloads (modified linux kernel source in one case) and nothing else that would in any way be a compromise if someone leveraged an exploit. On a shared host this might be an issue but not for me.

So if it is a CHOICE for cloud services, maybe not so bad?

[Anonymous Coward]

Re: Speculative execution exploits in the wild?

"In the cases where "sensitive data" IS of a concern, put the mitigation(s) in."

Or, if it's important enough to be that "sensitive", err, run it on POWER or Sparc maybe...

[Anonymous Coward]

Re: Speculative execution exploits in the wild?

Except both Sparc & Power were also susceptible to speculative execution attack vectors. IIRC the only "modern" CPU design which they didn't find these types of issues in was Itanic and no one wanted to avoid this issue that much.

[Nate Amsden]

I was surprised at the hit

Not for this but for the original Spectre/Meltdown performance hits on Linux. For both work and personal I have been actively avoiding the fixes for years whether it be firmware updates or microcode updates(and using "spectre_v2=off nopti" as kernel boot options), and forcing my ESXi hosts to use an older microcode package. I have always felt these are super over hyped scenarios especially if you are running your own infrastructure(I have confidence in what I do managing internet facing infrastructure for the past 25 years). Of course I can't avoid the updates forever, especially since newer systems come with newer firmware at a minimum. Really wish/hope there can be UEFI/BIOS settings to disable these fixes in firmware while maintaining any other fixes that would be useful.

Anyway, for personal use I was given a couple of Lenovo T450s laptops(I think from 2016 time frame) a few months ago and I put Linux Mint 20 on them. They both had 12G of ram and SSDs, but both felt so slow they were practically unusable for me(I didn't use them for anything yet other than just installing Linux). For a while I was suspecting the SSDs, which are Intel, things like software installs were taking a long time, and reviews of these particular Intel SSDs said they were quite slow compared to the competition. So I was ready to buy new SSDs when I realized I hadn't put those "spectre_v2=off nopti" settings in the kernel yet.

I put them in and it was amazing the improvement in performance. Didn't need new SSDs after all. From what I've read the performance overhead for these fixes can vary quite a bit depending on what the CPU is doing, so it's not likely a generic CPU benchmark would register the full effects(or maybe new CPU benchmarks can/do I am not sure).

I really couldn't believe the improvement in performance for just basic desktop tasks it blows my mind. I'm unsure of a good way(in Linux, even though been using Linux on the desktop since 1998) to accurately measure such performance differences(using a stopwatch manually isn't sufficient). I recall back in the 90s on the Windows side I played around a lot with the Ziff Davis benchmarks which were widely used, one of them at least was for desktop apps.

[botfap]

Measuring the difference

Geekbench 5 for Linux is very, very simplified benchmark but it does show you some differences between the mitigations being enabled and disabled and it allows you to perform the same benchmarks across Windows, macOS and Linux. The results on their own are meaningless but when used as a comparative it gives a reasonable, if incomplete indication of performance differences

Phoronix Test Suite is a much more comprehensive benchmark for Linux (and Windows, macOS, Solaris, FreeBSD). While its not perfect, indeed some of the tests are meaningless drivel. It does however contain a lot of real world scenarios and provides a lot more data about individual aspects of performance on many specific workloads. Its well worth a look if you want to understand what aspects of performance change under the mitigations

[Notas Badoff]

twofer

How strange. ElReg has accidentally stuffed the same article twice into the front page, using different URLs.

Retbleed fix slugs Linux VM performance by up to 70 percent

Retbleed fix slugs Linux VM performance by up to 70 percent

I *was* seeing double

[Claptrap314]

THIS IS NOT A BUG

This is NOT a bug.

T.H.I.S. N.O.T. A. B.U.G.

A "bug" is when the published specs are violated. The specs have not been violated.

Consult the front matter for a manual of one of these parts--specifically, the page that says "This product is NOT rated for use with government information classified CONFIDENTIAL or higher."

So...someone buys a plastic shield. They take it into battle with an opponent who has a steel lance. You blame the shield maker?

[IGotOut]

Re: THIS IS NOT A BUG

"So...someone buys a plastic shield. They take it into battle with an opponent who has a steel lance"

Depends. If its 2DPA-1 I'll take my chances