Whip out your blades: All-flash Isilon scale-out bruiser coming

EMC has an all-flash, bladed Isilon Nitro project to boost its scale-out file performance to 250,000 IOPS per node and 400+ nodes. Currently Isilon, with its OneFS operating system, supports up to 144 nodes using an InfiniBand internal cluster connection scheme. Cluster capacity can scale from 18TB to over 30PB. There are four …

[Yaron Haviv]

Flash but low on IOPs

Good observations, things in Chad's post don't add up, posted some on that as well

The may have mixed the terms Blade & Node and have few blades per node or something

re the IOPs number they mention 250K IOPs, the S210 model from 2014 did (on paper) ~100K IOPs, so growing only ~2x in ~2 years means they ride Moor law, and cant really make the 10 year old OneFS a Flash optimized storage. same for latency, 1ms latency is better than the previous model, but still much slower than the new Flash and NVM media can offer (1-100us)

wrote about whats needed in to get to the new levels of performance in File & Object: http://sdsblog.com/wanted-a-faster-storage-stack/

we will soon see vendors delivering >1M IOPs and bare-metal latency on file and higher level APIs/features, so not sure i buy Chad's description of it as "Extreme Performance" for a 2017 product.

Yaron

[Yaron Haviv]

Re: Flash but low on IOPs

After number games we had Chad updated his blog post to mention a Node "may" be composed of multiple Blades as i guessed, i bet its 4 blades per node (each 3.75GB/s) this makes the numbers work out

Yaron

[Anonymous Coward]

Re: Flash but low on IOPs

https://www.spec.org/sfs2008/results/res2014q3/sfs2008-20140609-00249.html

The S210 has disclosed results of 18k per-node, not 100k.

[Anonymous Coward]

Tweedle Dumb Stikes Back

As an outsider the Pure - EMC hate match is amusing. Pure come out with claims about their future product line and EMC feels compelled, like Trumper in a Dilbert strip, to claim next year they will have something twice as good. The general BS levels seem to rise every time these two start talking.

[Mark Hahn]

I wonder who buys these damned things. Their price is astronomical, but you'll still need a cluster of them to avoid SPoF. How many companies need those kinds of IOPS and bandwidth? Sure, Amazon would, but they're smart enough to engineer distributed systems that scale and don't cost much. Something like NYSE or Visa/MasterCard? The latter would almost certainly follow the standard path like Amazon and others.

[Chris Mellor 1]

Chad Sakac on blades and nodes

Here's an update to Chad Sakac's blog post; "UPDATE: many people have scratched their heads at this – note that nowhere have I explicitly stated the relationships between blades and nodes (how many blades/node). That’s intentional. Lots of time before GA, and through that time, more will become evident. It’s not uncommon for some details to be left blank (sometimes to keep cards close to one’s chest, sometimes because there’s still variations likely in the plan). With Project Nitro we’re keeping some blade details back. In similar pre-GA statements from EMC and from almost everyone, there are some details kept back."

If a node contains more than one blade then working out the numbers could become easier.

[Chris Mellor 1]

Node = 4 blades

I'm (Chris Mellor) posting this comment for Peter Serocka -

As poster Yaron Haviv pointed out, blades and nodes apparently are mixed up here, with most of the numbers relating to the 4U chassis-nodes, while "400+" relating to blades which will act as the functional nodes of OneFS in the sense how "nodes" are usually operated and counted in OneFS.

NAS Ops rates are least well-defined, so let's have a look at the other figures first, and see what we can figure out from the published information.

8 x 40GbitE frontend + 8 x 40GbitE backend sum up to 16 connectors per "node" and that must be

"chassis" as it would be overkill for any kind of sub-chassis storage "blades" to have 16 connectors on them. Just imaging the cabling...

One the other hand, present OneFS nodes have 2 x Infiniband backend plus 2 x 10GbitE frontend,

both active-active balancing and failover, so a Nitro blade should have at least 2 x 40GbitE

frontend and 2 x 40GbitE backend.

That would mean 4 blades per node.

(Otherwise, 4 x 40GbitE plus 4 x 40GbitE per blade would mean only 2 blades per chassis - barely makes sense, and wouldn't match the performance and capacity figures either, as we'll see below).

So assuming 4 blades per chassis, and that all capital "B"s mean Bytes not bits, and a max cluster size of 400 blades (aka "nodes" in OneFS speak), we might have:

1 Nitro Chassis (4U):

60 x 15 TB = 800 TB

15 GB/s (10 times more throughput ***out of 4U***, compared to one Isilon X410 4U node)

1 Blade:

15 x 15 TB = 225 TB

3.75 GB/s (2.5 times more throughput ***out of a single node*** in traditional sense, compared to X410).

Large Cluster (100 chassis = 400 blades):

100 x 800 TB = 400 x 225 TB = 80 PB (with 100 PB for "400+" blades)

100 x 15 GB/s = 1.5 TB/s (as claimed)

Note that 3.75 GB/s per blade fits nicely with the assumed network connections of dual redundant 40 GbitE for front and backend, respectively.

The picture of the cluster on Chad Sakac's blog site ***shows*** 100 chassis and ***says*** 400+ nodes, another indication that 1 "OneFS node" = 1 Nitro blade, with 4 of them in going in one Nitro chassis.

Fwiw, if we divide a chassis-node's 15GB/s by claimed 250,000 NAS Ops/s that would give

us an *average* NAS operation block size of 60 KB, which kind of makes sense for mixed

workloads (reads/writes at 100+ KB/transfer, and numerous namespace ops with at a few KB/transfer.) Same result when looking a a single blade of course, as dividing both throughput and Ops rates by 4 with cancel out.

With a latency of claimed 1 ms, the NAS queue depth per blade (= OneFS NAS node) would be 1/4 * 250,000/s x 0.001s = 62.5, which also is a reasonable value.

Makes sense?

---------------------

He adds this point: As for the confusion between "nodes" and "nodes":

With the Isilon OneFS software it is very clear what a NODE is: one instance of the OneFS FreeBSD-based operating system, running on a single SMP machine with disks enclosed.

But for the hardware guys "nodes" are those solid pieces of metal that get mounted in racks.

I think with Isilon clusters, one should keep the original OneFS definition of a node, and refer to the new hardware units in a different way. "Brick" hasn't been used with Isilon yet ;-)

Too bad EMC didn't sort out their terminology before making the Nitro pre-announcement, but such confusion arises often with bladed compute clusters, too.