Re: Are SPC Benchmarks useful?
Totally agree with Rulon, these SSD based SPC-1 numbers aren't about front end, back end port numbers or even the number of SSDs, this is all about controller bandwidth. It's about balance and the key is to size and test up to the controller limits, which is typically the CPU.
Keep in mind the vendors are in full control of the configuration, pricing and the 100% load point and so test up to the controller limits. If they put too few SSD's in the array perhaps they make their solutions buy price look attractive, but they will lower their max IOps and increase their latency (less bragging rights). Conversely if they put too many SSD's in, then they increase their cost exponentially without significantly increasing the performance since the controller CPU's become the bottleneck.
The $ per IOp is the real differentiator of SPC-1 vs most other benchmarks, it allows you to understand how much it actually cost the vendor to hit those numbers. As such it acts as a limiter on the vendor to stop them building crazy configurations, and if they do decide to do that, then they have to disclose both pricing and discount levels in the SPC disclosure. AFAIK no other storage related benchmark has a cost element, hence why you see things like this form EMC http://goo.gl/D03w9 and why those vendors aren't so keen to participate in SPC. Because ultimately the cost and effort required to hit a particular number are completely transparent and that rather undermines marketing..
Now on the face of it non of this sounds very useful to an end user, but it tells you where the systems limits really are. In the case of those sub 150,000 IOps systems, can they really scale performance to support the maximum disks they advertise on the spec sheet, 960 disks for HDS, or worse still scale controller resources to provide virtualization across another back end array, or sacrifice precious CPU resource to compression and raid for IBM ?
Biting the hand that feeds IT
