Re: Are SPC Benchmarks useful?
Totally reasonable line to take Chris.
The founding premise for the SPC organisation was to provide a level playing field to allow comparisons, judgements, and informed opinion to be made - as you pointed out.
All vendors contributing SPC results should be applauded and recognised for what they are doing. An SPC submission "opens the kimono" to many details regarding a storage systems design and architecture which normally would not be easily available until a customer has purchased, implemented and experienced many months of operations.
A vendor will not submit an SPC result for publishing if they thought they could get a better headline number by tweaking this or that, changing the balance between CPU and Capacity and hosts, etc... it is all a fine balance between cost and ensuring that all resources are utilised efficiently.
This is why I love the SPC-1 Full Disclosure Reports - the detail and nuances disclosed there are amazing and accurate to make an informed opinion and position regarding storage array solutions available on the market.
For example; Why do some vendors go to extreme lengths to implement super-complex host-side file system wide-striping? Answer: because to achieve high performance you need to distribute hot spots, and wide-striping is the best method today to do this. That's why industry leading arrays do the wide-striping inside the array - eliminate the complexity from the customers daily life.
Second example; Why do some vendors configure their array groups, volume pools, whatever they call them - without any sparing? Answer: because if they configured sparing that will reduce (worsen) their capacity utilisation efficiency, increase cost maybe, and potentially decrease performance. I have never met a customer who uses a RAID array and not configured sparing... doh!!
Above are examples - and there are many more - as to why the SPC-1 benchmark is such a good "level playing field" to make informed judgements and decisions (by customers) regarding storage array choices.
Disclosure: I'm a HP Storage Employee.