Re: "optimising these drivers for best performance is very important"
And who do you believe will optimize those drivers? Maybe exactly the one who know best the hardware they run for?
This is a key point. I'd like to see the Nvidia drivers open-sourced or, at least, enough that they could talk to an open opencl implementation (people seem to be missing that about the AMD drivers, AMDGPU-PRO is not open source). However, I know people who run a medium sized cluster (i.e. not supercomputer, but a lot of capacity) in a university computer science department, and I don't believe they have the capability to be debugging driver code. That's not to say there aren't a lot of clever people there, but the people employed to run the facility are not driver developers and the academic staff and students are there to solve research problems, not spend months debugging hardware. There are few things less inspiring of confidence than running code that depends entirely on something a PhD student has written, because even if they're very talented most don't care about other people's corner cases.
Open source it and for someone somewhere it will be their topic of interest, and everyone benefits. But the idea that departments that size will stay away from Nvidia because they want to be able to fix their own driver code is based in a fantasy. These people are still buying Nvidia.
As for the "what are you doing using consumer products", we're talking silicon here. If it's consumer it might not be up there in terms of reliability or capacity, but a bug isn't going to respect consumer/professional boundaries when they share the same architecture, see https://www.techarp.com/guides/complete-meltdown-spectre-cpu-list/5/ and look for Xeon. In theory people using GPUs should understand what they can mean for precision (though the days of 16bit only are past), but they should also in theory know about the differences between x87 and SSE calculations... In practice these things are increasingly bought to plug ML algorithms into, and the attitude to precision is that it'll just get taken care of in the algorithm.