Posts by testlabnut

Mangstor, Mellanox flash rig crowned 'fastest in the lab'... for RAID-0

Re: too bad

Kevin @ SR here

We included some screen grabs of the performance level while the tests were running, but the main attraction really is the application performance numbers. We have a lot of historical data on Sysbench if you look through some of our past reviews. We didn't chart anything head to head on this one since it was pretty far from the norm. Closest top number would be the ScaleIO project (which this had more than doubled) but its very dissimilar storage.

On the DRAM size, we ended up narrowing in on 24GB since it stressed the storage the most. The test was around showing MySQL performance across various high-end storage types. At the end of the day, the number worked and its been one we have lots of apples-to-apples comparison points against.

Nutanix digs itself into a hole ... and refuses to drop the shovel

On that things to think about category you are so itching to try and find fault with:

Percona Sysbench: MySQL TPC-C with help from Micron who isn't the fastest in that test. Audited by Percona among others to verify the accuracy and relevance of the test data.

FIO: Help with deployment by Jens (developer) while he worked for Fusion-io. Audited by many others in the space and we share workload parameters used in reviews. Open source workload generator...

MarkLogic NoSQL: Deployed with MarkLogic... no storage vendor attachment

SQL Server TPC-C: Worked with Microsoft Server and SQL Server teams to build and participate, using Benchmark Factory from Dell/Quest. Click the profile of choice you want to run, set the scale and click "run test".

VMmark: Worked with the developers of VMmark at VMware to deploy almost two years ago (well before VSAN or before we even worked with that team). One of the most audited benchmarks out there.

OpenLDAP: Worked work the developer of the software to deploy. No storage vendor attachment

Veeam backup test: Worked with Veeam to deploy. No storage vendor attachment

Its not a bad thing to ask for help. It helps to have an open mind. Before any new test gets added to our site we run the idea past most vendors we work with and industry insiders to check for relevance. When vendors can provide input and offer help, it builds trust in ways many don't seem to understand.

Kevin here - not hiding

Funny you keep reaching there to find some way to think we literally have no idea what we're doing in tests. As Trevor has pointed to, our tests have been audited before. Most tier 1 vendors have replicated our environments specifically to check our results. The synthetic numbers hilariously you liked to bring up are one area of probably the least importance. You can game the system easily to get different numbers to publish. Applications on the other hand don't really mess around.

The part about getting vendors to participate in our lab and offer feedback on tests is to help build trust. Like you say, Brian and myself had zero enterprise or storage experience going into this a number of years ago. Another way we prefer to explain it is we had no biases or legacy baggage to worry about. A lot of vendors like that. Funny how that's worked out. Lots and lots of "firsts" in the industry where previously closed off companies literally ship us any gear because they trust us.

Trust and credibility are literally the only things that keep the lights on in our Cincinnati, OH building.