Are SPC Benchmarks useful?

This topic was created by Chris Mellor 1 .

[Chris Mellor 1]

Are SPC Benchmarks useful?

A commentard ripped into me over the HP SPC-1 benchmark win story - No Dell, no EMC? Well, HP's storage champ then.

Here's what the comment said (between arrow lines):-

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

You're kidding me right?

Chris, have you even read the specs of the arrays you're drawing comparisons against? Do you understand how the SPC benchmarks work and the impact particular type of resources have on the different workload profiles used?

Storage performance and scalability is large dependent on a number of different resource types and the ability to distribute different types of workloads across available resources (utilization rate).

From a workload distribution perspectives in simplistic terms we can think about frontend and backend. For the frontend, the 3PAR 7400 is using more FC ports than both the V7000 and the HUS150. We could get extremely technical on the impact more ports yields from a buffering and queuing perspective, but I think its pretty clear that more ports ultimately means that a host can process more I/Os in parallel.

From a backend workload perspective it more about the disk to offload the workload to. In some workloads profiles its almost always about the disks, with others parts of the I/O chain in-between almost running at line speed. The 7400 again has more disks (SDDs) than both the V7000 and the HUS150, in both cases we are talking double digits more and at 2500IOPS a pop thats 25,000 raw IOPS we can handle without cache! .

When it comes to dealing with the different workload profiles cache (for all but random read workloads) is king, a well designed array's performance scaling profile is based largely on this single resource type all other things being equal. The 7400 has more than 4 times the amount of raw cache than the V7000 and double the cache of the HUS150 When it comes random reads SDD is our saviour and as I said above, we have more SSD spindles in the 7400.

SPC comparisons that are designed in this way prove nothing. Please stop harping that X is better than Y because you've looked at some summaries at the SPC website, its false (performance) economics and just wrong.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

So the SPC-1 benchark result summaries are not a valid way of comparing different vendors' systems.

The SPC web site home page includes this text; "SPC-1 benchmark results provide a source of comparative storage performance information that is objective, relevant, and verifiable. That information will provide value throughout the storage product lifecycle, which includes development of product requirements, product implementation, performance tuning, capacity planning, market positioning, and purchase evaluations. The SPC-1 Benchmark is designed to be vendor/platform independent and are applicable across a broad range of storage configuration and topologies."

The vendors agreed the SPC-1 benchmark and submit systems to it and publish results. So it's valid, I strongly submit, for us hacks to write about them. In other words I disagree with the comment above.

Is that a reasonable line to take?

Chris.

[Rulon]

Re: Are SPC Benchmarks useful?

Hi Chris,

There was nothing wrong with your approach in the article. In fact, I thought you bent over backwards not to raise anyone's hackles. We're all smart, and we're all technical, so it's really easy to poke holes in any benchmark. What makes the SPC benchmark fair is that everyone who participates, whether they admit it or not, does everything they can to get the highest possible number. That's why all those criticisms above don't make sense. If the V7000 or HDS HUS engineers thought they could get a higher number with more cache, host ports, or SSDs, they would have changed their configuration in a heartbeat if they could. The fact is, their number is their number - end of story. The 3PAR 7400 is really, really fast. And the reason why EMC doesn't post a VNX or VMAX SPC result is because the result would hurt their sales - I just had to throw that in.:) Disclosure: I am an HP Storage employee.

[Anonymous Coward]

Re: Are SPC Benchmarks useful?

Totally agree with Rulon, these SSD based SPC-1 numbers aren't about front end, back end port numbers or even the number of SSDs, this is all about controller bandwidth. It's about balance and the key is to size and test up to the controller limits, which is typically the CPU.

Keep in mind the vendors are in full control of the configuration, pricing and the 100% load point and so test up to the controller limits. If they put too few SSD's in the array perhaps they make their solutions buy price look attractive, but they will lower their max IOps and increase their latency (less bragging rights). Conversely if they put too many SSD's in, then they increase their cost exponentially without significantly increasing the performance since the controller CPU's become the bottleneck.

The $ per IOp is the real differentiator of SPC-1 vs most other benchmarks, it allows you to understand how much it actually cost the vendor to hit those numbers. As such it acts as a limiter on the vendor to stop them building crazy configurations, and if they do decide to do that, then they have to disclose both pricing and discount levels in the SPC disclosure. AFAIK no other storage related benchmark has a cost element, hence why you see things like this form EMC http://goo.gl/D03w9 and why those vendors aren't so keen to participate in SPC. Because ultimately the cost and effort required to hit a particular number are completely transparent and that rather undermines marketing..

Now on the face of it non of this sounds very useful to an end user, but it tells you where the systems limits really are. In the case of those sub 150,000 IOps systems, can they really scale performance to support the maximum disks they advertise on the spec sheet, 960 disks for HDS, or worse still scale controller resources to provide virtualization across another back end array, or sacrifice precious CPU resource to compression and raid for IBM ?

[Anonymous Coward]

You're kidding me right?

Are we expected to accept that the comment disparaging the comparison between these three competing platforms really came from someone at HP 3PAR. Surely the losers in this particular benchmark and those vendors who refuse to participate in SPC-1 have the most need to divert attention away from this result.

Really the 7400 didn't just beat the rest of the mid-range systems tested, it pretty much destroyed them, as such it's raised the bar for general purpose mid-range performance at a competitive price. Now if you're not interested in performance then "nothing to see here" buy something cheap with lots of high capacity disk and "move along".

Something faster will likely be along before long, IBM'll likely be strapping multiple V7000's together as we speak.

[PaulHavs]

Re: Are SPC Benchmarks useful?

Totally reasonable line to take Chris.

The founding premise for the SPC organisation was to provide a level playing field to allow comparisons, judgements, and informed opinion to be made - as you pointed out.

All vendors contributing SPC results should be applauded and recognised for what they are doing. An SPC submission "opens the kimono" to many details regarding a storage systems design and architecture which normally would not be easily available until a customer has purchased, implemented and experienced many months of operations.

A vendor will not submit an SPC result for publishing if they thought they could get a better headline number by tweaking this or that, changing the balance between CPU and Capacity and hosts, etc... it is all a fine balance between cost and ensuring that all resources are utilised efficiently.

This is why I love the SPC-1 Full Disclosure Reports - the detail and nuances disclosed there are amazing and accurate to make an informed opinion and position regarding storage array solutions available on the market.

For example; Why do some vendors go to extreme lengths to implement super-complex host-side file system wide-striping? Answer: because to achieve high performance you need to distribute hot spots, and wide-striping is the best method today to do this. That's why industry leading arrays do the wide-striping inside the array - eliminate the complexity from the customers daily life.

Second example; Why do some vendors configure their array groups, volume pools, whatever they call them - without any sparing? Answer: because if they configured sparing that will reduce (worsen) their capacity utilisation efficiency, increase cost maybe, and potentially decrease performance. I have never met a customer who uses a RAID array and not configured sparing... doh!!

Above are examples - and there are many more - as to why the SPC-1 benchmark is such a good "level playing field" to make informed judgements and decisions (by customers) regarding storage array choices.

Disclosure: I'm a HP Storage Employee.

[SPC1]

Re: Are SPC Benchmarks useful?

The SPC-1 benchmark provides a real source for comparing performance of storage systems. It is a single workload that demonstrates storage system performance under typical critical applications such as OLTP and database operations. As Chris mentioned, the process of submitting guarantees that the results are in fact valid and match to an existing product that customer can buy - not a special configuration that is built only for the purpose of achieving a result. It’s a benchmark running on a valid product and includes all the other elements of the product beyond performance, including high availability and Enterprise storage features that customers expect to get beyond the performance. The process of reviewing SPC-1 results guarantee that the results are accurate and real. An SPC benchmark measurement becomes a new SPC benchmark result upon successful completion of the required SPC Audit and SPC submission process. At that point the new result is in “Submitted for Review” status for a minimum of 60 days during which the SPC Peer Review occurs and allows SPC members an opportunity to review the details of the SPC benchmark result and raise any compliance issues resulting from that review. A result that is approved means that it passed this process. Which is what happened, when we, (full disclosure here, I run engineering at Kaminario) Kaminario submitted and won the world record title that Chris referred to in his article. There is no other storage performance benchmark that guarantee this level of validation. Oh, and p.s. we’re still the record holder.

[shen]

I think I need answers, thanks friend for sharing, I think I want to be more an advanced understanding.