Meh.
While Tachyon is interesting. It can't stand alone.
You will need to persist data for one simple reason.
You can only fit so much data in to memory before you eventually have to deal with swap.
Tachyon Nexus theorises on ultra fast storage
Any in-memory-focussed clustered system has to deal with failure at some point, and learn how to recover from, or tolerate, it. Replication is a common method but it slows down processing, especially in sequences of jobs in a pipeline. However, upstart Tachyon Nexus thinks it has found a way round that problem, and can go a …
-
-
Tuesday 24th March 2015 11:41 GMT Anonymous Coward
Re: Meh.
Indeed. Were this a couple of years ago when Hadoop workloads were almost always IO bound this would be tremendously interesting, but Spark has put the brakes on that little train. Spark takes the same lineage-based in-memory strategy during a job that Tachyon does for storage, but works over HDFS (and half a dozen other file systems), and will happily run on-disk, so the benefits of an in memory filesystem during a process will be minimal.
This was borne out in an excellent paper released this week demonstrating that Spark workloads have pushed the bottleneck to where it belongs: into the CPU [ http://www.eecs.berkeley.edu/~keo/publications/nsdi15-final147.pdf ]. They demonstrated that across the benchmarks used (which are representative of the vast majority of real-world work), eliminating the disk would improve performance by 19% at most.
So yeah, 100x throughput is nice, but in reality it's going to be a very niche application - the only time I can really see this being of benefit is if you've got a large graph of jobs, requiring a write back to disk to share the data between them, but that's a rare use case in my experience and when it does happen isn't so latency-sensitive as to need the improved throughput.
-
Tuesday 24th March 2015 16:56 GMT Ian Michael Gumby
Re: Meh.
The issue is that you would want to use Tachyon and Spark in conjunction with a distributed file system.
MapRFS, IBM's... even Cleversafe's.
Things are getting interesting. And even Tachyon may become moot with RRAM although if its better writtent than Hadoop components... it could still be a winner.
-
-
Biting the hand that feeds IT
