SPC says up yours to DataCore

The 1,510,090.52 IOPS DataCore Parallel Server single node result in March this year was the fourth highest SPC-1 reading, and has now been withdrawn. After someone, possibly a competitor, went over the results with a microscope they noticed there was no UPS. While its presence or absence wouldn't affect the IOPS result, it …

[Steve Chalmers]

The rule is there for a reason -- but SPC-1 has been broken

The rule which demands that the UPS be there in this particular circumstance is there for a reason -- to ensure that what's being measured as a storage system in fact exhibits durability of writes. Otherwise someone could spin up a stateless AWS instance, run memcached-like software in it knowing that the data would be lost if the instance ever crashed or was reset, and claim a storage benchmark result.

I haven't been in the storage performance game for 14 years, or even in the storage business for 8 years. So take this with a grain of salt. But what we used to say is that the life cycle of a benchmark started with small numbers, grew to medium numbers as people learned to design and tune for the benchmark, and that at some point someone figured out how to "break" the benchmark turning in very large numbers which were no longer meaningful to the real-world customer use the benchmark intends to represent. Once broken, a benchmark needs to retire and/or be replaced.

On the surface, without doing deep study, the result in question tells me SPC-1 has been broken for large storage systems.

@FStevenChalmers

(speaking for self, not for employer, which happens to be HPE)

[asdf]

Re: The rule is there for a reason -- but SPC-1 has been broken

Come on let the author enjoy his Walmart priced system find.

[Paul Crawford]

"UPS costs $1,000"

You are seriously underestimating the cost of a UPS for that sort of system, why it is likely to be MUCH higher, probably around $5,000 which will make an impact of almost 4% to the $/IOPS cost!

[Michael Duke]

Re: "UPS costs $1,000"

Why? Its protecting a single 2U server crammed with PCIe flash. 1500VA will be fine.

[Paul Crawford]

Re: "UPS costs $1,000"

Just. It is a Lenovo X3650 M5 Server and a Dell PowerVault MD1220 Storage Array which have PSU rated for 900W+600W = 1500W but allowing a bit of margin for PF not exactly 1.0 you would really be looking at a 2kVA UPS (even though average power is likely a bit lower). Oddly enough the SPC benchmarks have $/IOPS but not pJ/IOPS or equivalent indication of actual power consumption which in this day is likely to factor in to the overall ownership cost as well.

p.s. I guess you missed the joke icon?

[Paul Crawford]

Re: "UPS costs $1,000"

I guess it should be uW per IOPS (or SI alternative) given that IOPS is I/O per second, and energy per I/O would then be energy/second = power.

[CheesyTheClown]

Why use and array of any type anyway?

I definitely want near-line storage and for that I'm trying to design a new controller with power control so that a full 42U array shouldn't be consuming much more than about 400w at any given time.

What I don't understand is why anyone would want to run centralized storage anymore. It was a terribly failed experiment on so many levels. With a theoretical average of 90,000 IOPs per SATA SSD and an average of 24x drives per server configured as mirror, let's assume a little less than 2 million IOPs per server. Then, distribute that out using RDMA over Ethernet with SMBv3 and Scale-Out-FS. Then consider that the servers themselves will have 4 or 8 40GbE Ethernet ports.

Centralized storage doesn't have the performance to cope with modern data center servers. The goal of data centers is higher density and lower power foot print. I'm in 12U these days what used to us take 6 racks even 3 years back.

DataCore, NetApp, Hitatchi.... they're all selling crap you just don't need anymore. Get 3 good servers for each data center, configure some proper networking equipment (stuff that accelerates DC tasks, I recommend Cisco ACI) and then just use the storage built into the hypervisors. And don't waste money on Enterprise SSD. Just buy a few boxes of Samsung 850's.

As for centralized storage, build a big ass server with lots of spinning 8-10TB disks and run Veeam on it.

I've been testing Windows Storage Space Direct in a small data center on relatively old hardware with relatively wimpy networking. It's still hitting insane IOPS. And that's 4 Windows Server 2016 Hyper-V blades with 6 SSDs per blade on an archaic Dell C6100 cloud system from ebay for $1500 (all blades included). Each machine as 8 cores and 96GB RAM, an Intel x520-DA2, 6xCrucial 250GB SATA SSD (deal of the day).

I will NEVER EVER EVER go back to centralized storage. Unless someone tells me a way I can guarantee network and IOPS scaling for every blade in the data center using centralized storage, it's just a crap solution.

P.S. On VMware, I use centralized... Virtual SAN is a little too 1990's or 1970's depending on how you see it. Their underlying storage architecture is simply not something I would be willing to depend on. When they learn what sharding is, I'll consider otherwise.

[Paul Crawford]

Re: Why use and array of any type anyway?

I can think of a few very good reasons for centralised storage, such as (1) simplifying the task of recognising, protecting and managing your data (snapshots, data replicated to off-site store and/or tape robot), and (2) allowing common data/programs to be updated in one go for everyone who needs access, (3) allowing applications on differing native OS to share data.

However, (1) it will never match local storage for speed on any comparable basis, and (2) you get a degree of redundancy in your company that one central fault won't take down everything.

So really depending on what attribute matters more you you will go for one, the other, or maybe a bit of both (e.g. fast data local, central for share/replication/off-site transfer).

[dikrek]

Re: Why use and array of any type anyway?

Ok Cheesy I'll bite.

In almost every storage-related post (I've been keeping track) you mention these behemoth server-based systems and how fast they are. We get it. You don't like centralized storage and believe server-based is faster. To get this out of the way: properly designed server-based storage can be extremely fast - but not necessarily less expensive.

However, enterprises care about more than just speed. Far more. Otherwise we'd all just use RAM disks.

Out of curiosity, does Windows-based storage do things like live, automated SSD firmware updates regardless of server vendor?

What about things like figuring out how to automatically stage firmware updates for different components including the server BIOS, HBAs, the OS itself, and other components?

Or analytics figuring out what patches are safe to apply vs what bugs other customers with similar setups to yours have hit in the field? (maybe a certain patch will break your specific config).

How about analytics figuring out what exact part of your infrastructure is broken? Based on advanced techniques like pattern matching of certain errors, machine learning and predictive analytics?

Does server-based storage provide comprehensive protection from things like misplaced writes and torn pages? (Hint: checksums alone don't do the job).

In addition, are you running NVMe over fabrics? Because simply a fast Ethernet switch isn't enough to maintain low latencies. Or are you doing SMB3 RDMA? And what if my application won't work on SMB3? Maybe it's an AIX server needing crazy fast speeds?

Are the servers perchance mirrored? (Since you need to be able to lose an entire server without issue). If mirrored, doesn't it follow that 50GB/s writes to the cluster will result in an extra 50GB/s of intra-cluster traffic? Isn't that wasteful? (server-based should really be triple mirrored BTW).

And if erasure coding is employed, how is latency kept in check? Not the best mechanism to protect NVMe drives at scale.

Honestly curious to know the answers, maybe server-based really is that cool these days.

Thx

D (disclaimer: Nimble Storage employee)

[EMC is the Best <3]

Re: Why use and array of any type anyway?

Cheesy you are so so smart! I am looking forward to your game changing storage startup.

[Androgynous Cow Herd]

Oh FFS

Cheesey...it has been possible to do exactly what you request from Central storage for the past 20 years. If your environment requires a certain level of performance from each host, and all of that performance simultaneously, that can be designed...hell, you could do it with a Data General Clariion, so long as you are willing to hang enough media behind it. The problem with those old spindle bound platforms is that business needs change, and it just takes one new LUN placed on the wrong RAID group to Bork it all up.

In these days of high(er) speed storage, stringing hundreds of disks together to get to a specific I/O level is not such a thing anymore, and a lot of arrays are built with a concept of over subscription...most environments do not run every application at 100% simultaneously, so now a storage company will look at the aggregate performance needed for the environment. If over subscription makes you uncomfortable, I would suggest you also avoid thin provisioning and most Ethernet switches.

I don't think either solution is right or wrong...but putting storage inside a server is not exactly a earth shattering new concept. For every advantage you perceive there is a drawback...centralized storage provides for a more robust data protection set and the ability to design workable large scale DR plans. It also frees the server CPU from spending cycles doing advanced calculation for things like deduplication. It makes the run book for recovery from a host failure incident much shorter and simpler.

It all depends on what you are solving for.

Personally, I never, ever make absolute statements.

[NinWalker]

These classes of benchmarks are pure crap and a relic of the 90s. Cheesy makes a point about how modern architecture scale with core count. Yes, that is the fundamental principle of the modern scale out software. And they shard data across nodes and uses message passing to coordinate.

Of course he didn't mention that to coordinate at scale almost all of these classes of applications increase replication factor or relax consistency. One creates hardware inefficiencies while the latter creates application logic headaches for anything except really trivial problems. Acid is a property that is much nicer to reason about for things that require any form of correctness guarantees. Having one global view of the universe across all nodes is a much easier thing for most developers to understand.

For storage though, and especially centralized (and by necessity) multi application storage that interacts with distributed applications there are no discernable patterns that are really cacheable. Spatial and temporal location of data doesn't work for inputs into centralized sister systems that aggregates shards of data from heavily distributed applications. Cache misses becomes the norm and tail latencies dictate true performance. Vendors who claim average latencies as proof points of end user experience but in reality a spiky tail latency and transactional semantics are rapidly becoming the bigger problem. And most apps can't bound response of data to trade off speed for correctness the way many I'd the web properties do.

Which leads me to the benchmark. It's crap. It's built in an era of tech that is rapidly disappearing. Fat database servers whith concentration of io's thanks to btree style data structures to find data when dram caches misses are the underlying assumption in these tests and it's flawed in a world where a storage device is expected to cater to a few apps running in 1,000 physical machines and 10,000 cores. Data access is concurrent, random and heavy in metadata.

So you don't have this environment you say?

You will, either by yourself as you are forced to evolve beyond your legacy Oracle/Microsoft apps or when you land in the cloud and your provider builds this for you. Either way, dram accelerating systems are a thing of the past and at best a detour in history vs the modern constant rate latency system that the hyperscalers are building to cope with concurrent and arbitrary length processing streams.