Re: "We could never ever put our customers at risk"
If they used the same mitigation as the other cloud providers, ie, only allowing one customer's VMs per processor, then that should work, regardless of Hyperthreading (or SMT etc.). An attacker could only attack their own VMs.
(Unless they'd somehow gained access to one particular machine belonging to a target company, in which case they could potentially use speculative execution to move laterally to other servers owned by the same company. Of course, there would probably be easier more conventional ways of doing that).
As far as I know, the Azure hypervisor is based on Windows Hyper V, but the majority of VMs are Linux. Either way, any malware trying to make use of speculative execution would probably have to be custom written for a particular cloud platform.
Given that the most likely scenario is an attacker creating a malicious VM in an attempt to steal information out of other VMs on the same CPU, I wouldn't be surprised if the attacking VM was running some variety of linux because that's the OS I've seen most PoC code running on..
