Drive manufacturer kills possibly efficient storage methodology.
Western Digital has taken software acquired with personal storage outfit Upthere in 2017, packed it gently in a wicker basket, and laid it at the door of a GitHub orphanage. It's a far cry from WD's bullish boast that its acquisition had created "a new and better way to keep, find and share what's important and meaningful to …
Not really. They could sell a ton of kit whatever software you claim to use to manage your storage.
What's surprising is that the idea sounds nothing better than the kinds of things I was scrawling on coasters 20 years ago.
Hell, at one point there was a piece of software that you just ran on ordinary Linux/Windows clients - it took up a certain percentage of your disk space, spoke across the network with all the other clients and they worked out a way to manage all the storage such that X percent of the PCs had to be turned on, but that they could all take advantage of all the space storage doing nothing across the hundreds of half-filled client drives in any large organisation.
It was really cool. It worked. It also had enormous downsides (notably network traffic and resiliency and things like backups) and never became popular. It was a bit like DRBD but running on the free sectors of a support filesystem (which it just allocated as it needed them), with every computer a master and slave node simultaneously, and a bit of clever logic to make sure your data is replicated around enough nodes to stay up. Coupled with, in effect, a Samba server over a virtual filesystem that worked by requesting data from given nodes. And an almost DHT / peer-to-peer like network to make sure that any node could advertise the availability of its data and share in sending it to the right places on request at full line speed. And enough resiliency that Fred turning off his PC didn't really make a difference as he was only storing tiny shards of data.
It was, in fact, Samba-over-Bittorrent, really.
Now... THAT was a brilliant idea. What this was sounds like nothing more than standard data replication on the block level.
It's probably best off dead - UpThere only existed since 2016, had $77 Million of funding, 60 employees, and could barely replicate functionality that's present in Windows DFS, for instance.
I worked on such a thing in 1996 or so. The company was Mango, the software was Medley. It worked pretty much like you describe, and it was a *real filesystem* unlike blb. About the only think blb has that it didn't was erasure coding, because that was hardly a thing back then (I personally didn't hear about it for another four years or so). Wouldn't have been hard to graft on, though.
Every year or two there's a new blob store based on the same old ideas, claiming unprecedented levels of scalability, reliability, and convenience. These claims *always* turn out to be exaggerated. Some of the offerings are fine, just not as fantastic as they claim to be, because it turns out this stuff is actually hard. (This is what I do for a living, BTW.) If it hasn't actually been run on a multi-thousand-node cluster, it *won't* run at that scale and probably won't run for very long even at hundreds before it loses data. FWIW, I wrote a bit more yesterday about the *absolutely predictable* issues/omissions I found in five minutes of looking at the blb source.
There's nothing actually wrong with blb. It looks like a decent starting point if you want to build a truly high-scale object store for fun or for internal use. It's just not as All That as they claim.
Biting the hand that feeds IT © 1998–2021