SolidFire's thin flash density eats more racks than HPE. What gives?

NetApp's SolidFire arrays needs four times more rack space than HPE and EMC for equivalent amounts of flash storage. It has 1U hardware boxes and these appear to be limited in what capacity SSDs can be supported. With SolidFire arrays being scale-out nodes, then low flash density per rack unit means you need more nodes to …

[dikrek]

An inherently inefficient architecture

SolidFire, due to the requirement for mirroring, leaving a full node's worth of capacity free (to withstand a node failure) and not wanting to be over 90% full on the remaining space left, means usable:raw for a 4-way config is closer to 30%.

Assuming the industry standard 5:1 on the 30% means a very low final multiplier (Effective Capacity Ratio).

See here:

http://recoverymonkey.org/2016/05/19/the-importance-of-the-effective-capacity-ratio-in-modern-storage-systems/

This is all fairly basic math. Solidfire even has a sliding calculator on their website and you can verify the ratios easily using that.

See here:

http://www.solidfire.com/platform/hardware

There's the other wrinkle of metadata handling that makes it harder for SolidFire to use large SSDs since there's a RAM ratio that needs to be enforced, similar to XtremIO.

The value of SolidFire isn't dense flash storage. If you want that, look elsewhere.

Thx

D (Nimble employee & ex-NetApp)

[RollTide14]

Re: An inherently inefficient architecture

Thank you D for a non biased, honest response. I'm still shocked that some people (Chris Mellor) can't recognize that there are ALWAYS trade offs. I can have scale out with inline global data efficiencies but I wont get density. I can have density but I wont get things like true QoS policies.

There is no storage platform that is perfect....otherwise they'd own the market

As D said, Any true scale out platform with data services (like inline GLOBAL dedupe) is going to run into issues with large SSD size. If you want to accommodate those SSD sizes then you need crazy amounts of memory and that will price themselves out.

SolidFire Memory - 384GB

Xtremio Memory (40TB brick) - 512GB

[Anonymous Coward]

Re: An inherently inefficient architecture

This is correct. AFAs with 100% inline, global deduplication must maintain a significant amount of metadata in memory, and controller memory dictates the capacity limit. Given almost all storage controllers are based on 2-socket Intel Xeon designs, that dictates the memory capacity, which dictates the physical capacity.

SolidFire, XtremIO, Pure, and Kaminario could support larger SSDs, but fewer of them. It is the total capacity managed by the system architecture which is the limit.

How does NetApp All-Flash FAS and HPE All-Flash 3Par get around this? Neither use a global pool as the deduplication domain (NetApp uses the volume, 3Par uses the Common Provisioning Group). And both keep a subset of the total metadata database in memory, and rely on background processes to make up the difference. Note, there is nothing inherently wrong with this approach, in fact, given the rapidly increasing sizes of SSDs, and falling prices of SSDs, it may prove superior. The global deduplication approach made very good sense when SSDs were small and the price per GB was very expensive.

D blogged on it here:

Architecture has long term scalability implications for All Flash Appliances

http://recoverymonkey.org/2015/11/26/architecture-has-long-term-scalability-implications-for-all-flash-appliances/

I expect "adapted" All-Flash architectures to incorporate more inline and more global deduplication capability to provide better overall effiencies, but I also expect "built from the ground up" AFAs to incorporate more granular, and more deferred deduplication capabilities to allow larger SSDs. Kaminario already allows deduplication to be turned off for workloads which do not benefit from deduplication.

[Mr.Nobody]

A seriously first world datacenter/business problem

While I get that the reg needs articles to keep eyeballs to see ads, this seems a bit of a ridiculous story to bash a company with.

Since most businesses can only now afford AFAs, I doubt many of them are not going to purchase an array because it only has single digit TB SSDs.

When we went from our old spinning disk array 4 years ago to a new shiny array with all sorts of efficiency tools, let alone incredible performance improvement, we went from 2 full rack spaces to 20U of rack space. This four year old array has grown a bit and now has about 3 times the usable space it did initially, and its still not a full rack of space.

When we make the move to AFA,and we get back down to 20U of space for the whole enchilada, we would be thrilled, but its not what's important. The bottom line is what anyone cares about in the business. Have you seen the prices of these 8 and 15TB SSDs? They are astronomical. I highly doubt using 1/4 of the rack space makes up the difference of the enormous cost of going with this mega dense SSDs.

[dikrek]

Re: A seriously first world datacenter/business problem

Correct, it's not so much about the size of the SSDs as what the overall efficiency of the solution is. SolidFire is still not very dense due to the architectural decisions I mentioned in my first comment. A lot of expensive SSD capacity gets wasted in any server-based grid storage approach. ScaleIO, VSAN are similar. Erasure coding is the enemy of low latency so server-based storage typically goes for 2-3 total copies of the data (ideally 3 copies to avoid exposure in the case of node failure but SolidFire doesn't do that kind of protection at all).

There's no perfect platform, it's all about choosing what's best for your business and it's always down to some sort of compromise:

Price, performance, density, capabilities needed, RAS.

Thx

D

[Anonymous Coward]

Why are you picking on SolidFire? What about XtremIO, Pure, and Kaminario?

XtremIO supports a max of 1.6TB drives. Pure supports a max of 1.92TB drives (in a dual-drive enclosure, delivering 3.84TB per slot). Kaminario supports a max of 1.92TB drives.

So what is your point about SolidFire?

Perhaps there is more to it. The adapted HDD/hybrid arrays (NetApp FAS, HPE 3Par, EMC Unity) support large SSDs, but the "built from the ground up" AFAs do not. There is a technical reason. Can you figure it out, Chris?

[JohnMartin]

Re: Why are you picking on SolidFire? What about XtremIO, Pure, and Kaminario?

I already posted about this here http://forums.theregister.co.uk/forum/1/2016/06/07/hpe_adds_hicap_ssd_support_to_storeserv/#c_2888172

But given this is an expansion on the previous post it probably worth repeating

.. because Solidfire keeps all the metadata in memory, its closer to the way XtremeIO works which in its current incarnation gets about 40TB RAW in 6RU for about 150,000 IOPS. With 6 nodes in 6RU using the new Solidfire 19210, you'd get about 114TB RAW and 600,000 IOPS managed by the worlds best QOS implementation and automation framework, and scales to 100's of nodes ... much much higher than the 8 "bricks" (oh how aptly named) you're limited to with XtremeIO ..

Thanks to keeping all the metadata in memory, solidfire can be optimised for infrastructure automation and 100% predictable performance and outstanding QOS. Other architectures are optimised for rack density. Both are good, but many people building cloud infrastructure would choose effectively unlimited elastic scalability and predictable performance over rack density as their primary concern. Every architecture has trade-offs, this one is a good one.

In any case, over the longer term, who says that keeping metadata in memory requires that memory be implemented as DRAM ?

[shaimaskit]

Re: Why are you picking on SolidFire? What about XtremIO, Pure, and Kaminario?

(disclosure: Kaminario employee)

There are some good points made here, mainly the following made by RollTide14:

"There is no storage platform that is perfect....otherwise they'd own the market"

Having said that, I would like to further explain several architectural trade-offs that were brought up in some of the comments.

Managing metadata has become the most important task in AFAs, especially when deduplication is done globally (otherwise you're just inefficient). So if a certain architecture assumes that all the metadata is stored in DRAM (XtremIO, SolidFire) then yes, the amount of DRAM in the storage controllers will dictate how much capacity those controllers can manage and also the data reduction ratio that can be achieved.

In other words: Using larger drivers == more DRAM.

Then you have the approach of not keeping all the metadata in memory - like Kaminario does. That actually enabled Kaminario to be the first (real AFA) vendor to come out with large capacity drives based on 3D TLC last year:

http://www.theregister.co.uk/2015/08/20/kaminario_gets_triple_vision_with_sd_3layer_flash/

That was done with no change to the amount of DRAM in the controllers. Again, metadata management is key to keep the same consistent performance even with larger drives. This architecture approach will allow Kaminario to continue this trend - so stay tuned.

Kaminario is not alone in this approach, also Pure uses it but as you know, they cannot scale.

As for disabling our inline deduplication per volume basis - peforming a deduplication calculation is an action that requires CPU cycles, no way around that. If you decide not to execute these cycles on data for which deduplication would be negligible, such as databases, then more CPU cycles would be available to run customers' workloads faster.

[JohnMartin]

Re: Why are you picking on SolidFire? What about XtremIO, Pure, and Kaminario?

I've been wondering for a while why Pure is so reticent to use larger SSDs given their architecture should at least in theory allow it, makes me wonder if they're really worried about the size of the SSD as a failure domain (which RAID handles or at least should), or if it exposes some fragility in the purity filesystem in some way.

Oh and by the way .. "Real AFA vendor ?" .. I think the whole "it has to be built from the ground up" thing is well and truly dead as a concept, check out Coming Clean: The Lies That Flash Storage Companies Tell with Dave Wright of NetApp/SolidFire

https://www.youtube.com/watch?v=35KNCOYguBU

[Vaughn Stewart]

Re: Why are you picking on SolidFire? What about XtremIO, Pure, and Kaminario?

== Disclosure: Pure Storage Employee ==

Hey John, hate to do this to you mate but next time you "wonder about Pure Storage" you should disclose you work for NetApp.

Transparency is a good thing

-- cheers,

v