back to article NVMe over Ethernet is the future. And that's how we roll – Tegile

Yesterday, at a TechLive event in London, Tegile said connecting arrays to servers across low-latency NVMe over Ethernet was the way forward. Chief Marketing Officer Narayan Venkat said shared storage arrays had a future and represents a better way of consolidating data than hyper-converged systems. Networked arrays need to …

  1. Anonymous Coward
    Anonymous Coward

    Awesome!

    I would definitely delay any Tegile purchases based on these announcements, as this sounds a lot better than the spindle bound, ZIL pounding Architecture they are currently selling. The new platform sounds like it could even be an exciting OEM option for Osborne Computers.

    1. Anonymous Coward
      Anonymous Coward

      Re: Awesome!

      Actually this doesn't change the ZFS and ZIL requirements inside Tegile. So if you don't like that, you're SOL.

      It's just NVMe to the outside world (which does lower latencies and improves queuing) and NVMe drives for the internal cache.

      Both easy things to do but it takes more than that to drive crazy low latencies. The entire engine of the array needs to be tuned for this.

  2. Anonymous Coward
    Anonymous Coward

    Naive question

    OK, I'm probably missing something obvious, but how does NVMe over Ethernet get rid of network latency? The network is still there, so... it is simply a matter of stripping away all the networking layers that slow down TCP/IP?

    1. illiad

      Re: Naive question

      NVMe means Non Volatile Memory controller... so that mean your Ethernet becomes EVEN MORE crowded!!! so I don't think I'll go there, its bad enough, the web apps failing due to problems at one end or another!!!

    2. Anonymous Coward
      Anonymous Coward

      Re: Naive question

      And I wonder why, if low latency is needed, why "over Ethernet" is winning over, say, Infiniband which is designed for low latency.

      If you say me "Ethernet wins because everybody has an Ethernet switch" I may agree, and probably in most applications Ethernet may be enough, but if you're going to exploit your NVMe disks fully - and you need a SAN - a lower latency transport is desiderable.

    3. Anonymous Coward
      Anonymous Coward

      Re: Naive question

      You are right, NVME over Ethernet itself cannot get rid of latency issue. However NVME over RoCE (a variant NVME over fabrics) has the potential to reduce latency dramatically by avoiding the entire tcpip stack. Several startups on that bandwagon.

      1. P. Lee

        Re: Naive question

        Indeed. Just because you've got Ethernet doesn't mean you need IP.

        We've had ATA over Ethernet for years.

        Ethernet wins on price due to production economies of scale and competition in a commodity market.

  3. Fazal Majid

    Simple rule of thumb

    If your network latency exceeds your storage's latency, the right architecture is direct-attached storage, not networked storage, with the networking happening at the application layer, e.g the database. Very few Ethernet switches deliver the < 0.2ms latency of a typical NVMe SSD (only Arista comes to mind).

    Networked storage is a paradigm whose time is past, and the increasingly frantic efforts of storage vendors to stave off irrelevance by shoehorning SSDs into legacy architectures designed for disk remind me of other Rube Goldberg contraptions like disk arrays that pretend to be tape auto loaders to work around brain-dead backup software.

    1. Preston Munchensonton

      Re: Simple rule of thumb

      If your network latency exceeds your storage's latency, the right architecture is direct-attached storage, not networked storage, with the networking happening at the application layer, e.g the database.

      Networked storage is a paradigm whose time is past

      I share your sentiment about DAS instead of NAS, but you're being dense if you think that NAS doesn't have suitable use cases anymore. It's really intellectually inconsistent to suggest that someone should prefer DAS over NAS if the use case requirements dictate such, but then to say that no one should ever choose NAS. Choice is a good thing, perhaps the best of things, and IT decision makers can make their minds up for themselves. Will it happen wholesale? No idea, but I wouldn't bet that it's a paradigm whose time has past.

  4. Anonymous Coward
    Anonymous Coward

    even better

    here's a crazy idea - if your application actually needs latency that low, maybe put the storage INSIDE the server! I think in the future servers will be built with the ability to put storage devices inside the actual server. I am filing my patent now so no one get any fast ideas about stealing my innovation!

    1. Anonymous Coward
      Anonymous Coward

      Re: even better

      > maybe put the storage INSIDE the server!

      You mean like HPE's "The Machine" although they are talking about access time a lot less that a 1us.

    2. P. Lee

      Re: even better

      It seems for most people the problem is the high cost of administration skills. For most applications, the performance requirements are not a great concern.

      Judging by the success of the cloud, simple point and click interfaces with minimal technical skill required are the way forward. Hyper-fast hardware is just a way to make that happen. If I have to think about how much stuff I need and get some techie work it all out and put physical storage in a server, its going to cost more than expensive hardware, especially if I have to do that for online storage, backup, compute and networking.

      The result is nvme storage pools with multiple 100G links, Epyc CPUs and the cloud.

      Yes, we want the fastest storage possible, but only if its flexible. DAS may have the edge on speed, but its quite a niche use-case.

      It isn't usually about speed for a particular application, its about the ability to carve up the resources we have in an arbitrary fashion to meet changing needs without hiring expensive staff.

  5. Anonymous Coward
    Anonymous Coward

    ... Everything on E

    Sounds like the 90's recipe. Take a bad stage show, wait 6 months, release the 'on ice' version.

    I'm still waiting for Pizza-o-E

  6. Jackzhu
    Childcatcher

    Question too

    NVMe is the future but why not over fiber

    1. Tom Womack

      Re: Question too

      Ethernet can be over copper or over fibre; it's just a way of wrapping up frames.

      "NVMe over Ethernet" I suspect just means that there are 16 bits reserved for queue-number and 16 for position-in-queue in whatever secondary wrapping they're using

  7. This post has been deleted by its author

  8. jsaucedo1961
    Boffin

    Senior Lead Storage Engineer / EMC PP

    I'm thinking something similar in functionality; as INFINIBAND and the TOE/TCP/IP Off-Load Engine cards.

    An off the shelf solution, rather than included in/with Storage Arrays(Used as Back-End movement) i.e. ISILON, PURE-STORAGE, XTREMIO, others...

    The NVMe Protocol could allow users to implement a Private Robust Data Transfer Protocol NVMe and reek the benefits of Ethernet when it reaches 100Gb, +.

    Today, most enterprises have or are implementing Cisco's Nexus 5000/7000 10Gb Ethernet.

    Depending on the Nexus 7000 model chosen, 40Gb and 100Gb interfaces are available in large numbers.

    I think other Vendors as well, will be searching for solutions that get their current NON-40Gb, 100Gb, products into the pool with vendors such as Tegile.

    Just thinking out loud.. :)

    Joe Jr.

  9. Cloud 9

    Latency

    I'm with people who think NVMe needs to be as close to the CPU as possible to get best benefit. I'm willing to listen, but will be surprised if any solution sticking ethernet in the stack actually makes best use of the technology.

  10. mikegrok

    RoCE is under 2 microsecond

    I was looking into this for a zfs zil.

    From what I understand, a typical use case would be direct attached ethernet, i.e. host to host instead of through a switch, that way there are no bottlenecks.

    The ethernet packet does contain an IP packet with a udp packet.

    The protocol keeps the sequence number from TCP, but otherwise uses UDP. It is assumed that the connection is usually reliable, so instead of acknowledging each packet, the server only sends a response if a data packet is missed.

    For a bunch of file servers you would have a top of rack switch. You may also have a pair of top of rack RoCE boxes, with all of the other storage servers in the rack sending them their write transaction log. In case of server outage, the failover server on the san would read from the transaction log. and continue where the first one left off.

    In case of paired servers with primary and failover, a server could mirror an internal memory space for transparent failover.

    client request for foo.txt with write via smb.

    samba mark in memory that file will be in use for write

    queue to nvme (fastest confirmation is 150microsecond to nvme) lock

    send to paired RoCE servers lock event

    wait 2 microseconds

    inform client of lock

    148 microseconds later nvme write confirms.

    historically 2/3 of file server IO events are lock files.

    replacing your lock files with RoCE can effectively triple your effective IOPS.

    If you combine this with a file system that utilizes write combining (zfs), for a single threaded task, this can theoretically speed up your task by 70 times.

    -Michael McMillan

    mikegrok on linkedin

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon