Scale-out storage: Proprietary? Commodity? Or both?

There is no doubting that scale-out storage is now thoroughly achievable. Indeed, scale-out is pretty much the norm for object storage, plus it is now the primary option when it comes to huge volumes of file-based NAS, where scale-out means a highly-scalable file system that supports clustering. It is growing in popularity for …

[HighTension]

GlusterFS is *not* a block-capable product. It's file only. IMHO it's also about the slowest for mixed data. And I'd not trust any really valuable data to anything other than RedHat's supported version (RedHat Storage Server).

For filesystems, you've got many more choices, including BeeGFS, MooseFS, RozoFS and more (OSS or semi-open development model) or Exablox, Isilon, Hitachi, HP StoreAll (proprietary).

[Boyan]

Some points from practice:

1) if you need high performance -> block; if you need huge capacity -> object; For example we're yet to see a database that needs 250k IOPS and is 1PB.

2) commodity vs. specialized HW - no matter how much spell is put on specialized HW by vendors delivering it most cases its simply not true that their solution can out-perform a good SDS implementation on standard hardware And no, good SDS does not need a FS underneath. Not the case with Ceph, but there are enough examples.

3) open-source or not - in most cases it's price discussion, however there is nothing is free in life. A open-source project comes DIY - you spend (tens, sometimes hundreds of) thousands in time and salaries. Proprietary comes "out of the box", so there is a trade off between time and money.

Also open-source solutions usually rely on a number of other solutions to provide vital functionality, which adds layers of complexity, inefficiency and risk. For example - unnamed (mentioned in the article) opesource system needs $10k a piece storage nodes and proprietary alternative needs $2k/piece storage nodes. That's 5x difference on money spent for HW. Again - you always pay in one way or another and at current stage (for good or bad) opensource alternatives simply loose on price/performance and price/functionality to proprietary alternatives.

[random_graph]

Hardware independence and other myths

A good survey.

The instant these systems target mixed workloads, they sacrifice a lot of capability. In particular, the need to support latency-sensitive, high-transaction-consistency workloads through a block interface, and POSIX-compliant distributed lock management in a file-interface, designers cross a road that they can't come back from. Ceph is a good example; by targeting the generalized cloud workloads, they've had to invest in optimizations for both transaction latency as well as throughput.

The cost is that these systems become bound by rigid assumptions of underlying hardware and network topology. Maybe you'd be asked to choose from the HP Xeon 2u 12 HDD server or the Dell 2u Xeon 12 HDD server, but true hardware heterogeneity is mostly a myth. Dependencies that that inhibit flexibility include drive failure detection relying on specific BIOS versions, specific local file systems, assumptions about how namespace is balanced, intolerance for performance variation, consistency in SMART APIs, etc. Even the promised 'multi-generational' platform support usually results in bulk migrations akin to a forklift upgrade.

Lots of pundits are predicting continued polarization of the storage landscape. Where transaction performance matters, enterprises continue adoption of AFA and eventually NVMe. And at the other end, unstructured data stores go to Object Stores that are optimized for RESTful semantics, WAN/hybrid topology flexibility, yet can provide high throughput (don't confuse with latency) where needed for workloads like batch analytics.

And to answer your primary question, scale-out has proven to deliver improved TCO at scale, due to the simplified administrative experience and ^improved^ platform lifecycle management. But is scale-out a commodity? of course...IMO there ever was a time that scale-out commanded a price premium. Even in the extreme performance HPTC world where Mr. King plays, storage is mostly purchased by the pound.