RAID expansion comes to OpenZFS at last ZFS has been around for 16 years and has a solid reputation, but it does have limitations compared to its rivals. One of these is in the process of being lifted: soon it will be possible to add space to an existing ZFS array. As long as you're using FreeBSD, anyway. Filesystems are one of the core differentiators between …
"(It is worth pointing out that according to the legal principle of laches, it's now too late for Big Red to sue, but that still puts off most Linux vendors.)"
To sue? To sue who, about what? Oracle doesn't own the Linux kernel, whose license would allegedly be breached by using OpenZFS as a separate module. They own Solaris, which includes its own, now proprietary licensed ZFS.
Nobody ever claimed that OpenZFS was in breach of the license of the formerly Sun, now Oracle code it's based on, which was published under the CDDL.
The suggestions of potential lawsuit around the use of ZFS have been a giant red herring (pun not intended) for years now.
ZFS is technically, economically and legally mature and has been in use by giants with deep pockets in the enterprise storage space for years. Entire enterprise storage product lines have been based on ZFS. The Lawrence Livermore National Laboratory uses OpenZFS on Linux to store their unfathomably large data sets. If Oracle had the ability to sue, they would have done that years ago (knowing how litigious they are).
The question is, sue about what? The original ZFS was code open sourced before Oracle bought SUN. Oracle have created their own closed source version of ZFS but outside some Oracle shops nobody uses it (and Oracle is rumoured to have stopped working on ZFS all together some time ago).
Considering the forks (first from SUN to the various open source implementations and later the fork from open source into Oracle's closed source version) were such a long time ago, there is not that much original code left. A lot of storage tech, or even entire storage concepts, did not exist when SUN open sourced ZFS. The various ZFS implementations all developed their own support for TRIM, or Sequential Resilvering, or Zstd compression, or Persistent L2ARC, or Native ZFS Encryption, or Fusion Pools, or Allocation Classes, or dRAID long after 2005. That's is why the majority of the code in OpenZFS 2 is from long after the various forks.
Modern OpenZFS contains new code contributions from Nexenta Systems, Delphix, Intel, iXsystems, Datto and a whole bunch of other companies that have voluntarily offered their code when practically all of the non-Oracle ZFS implementations merged to become OpenZFS 2.0. They are obviously not going to sue either.
So, yes, if you only want to use GPL software then OpenZFS might be a problem because it's CDDL licensed instead. But, lawsuits by Oracle are not going to be a problem.
> The suggestions of potential lawsuit around the use of ZFS have been a giant red herring (pun not intended) for years now.
There is no risk from using it, but there is plenty of risk for anybody distributing it, tied to GPL'd code. That's why the distros are wary.
> there is not that much original code left
That doesn't matter at all. They'll remain derivative works even if the last line of original code is replaced.
> lawsuits by Oracle are not going to be a problem.
Oracle isn't likely to file lawsuits, but they've been cagey and refused to make such a commitment. They very well could. The fact that they haven't yet doesn't provide any added safety.
What would be damn interesting is if RedHat included ZFS in RHEL... Oracle have been repackaging RHEL as their own supported distro and redistributing it for years. If RHEL just sneaked ZFS code into their kernel, Oracle would redistribute it, and the GPL would oblige them to allow it to be treated as GPL-compatible.
But that was true. I have here a long, long list of the ways that the Linux cancer infringes on Microsoft's patents. I won't show you or anyone else the list, but trust me, it's long. And a list.
(What's that, Satya? mumblemumblewhisperwhisperwhisper
Oh, OK. Sure. New plan. Gotcha. I'll tell them that instead.)
As I just said, that was never true. Microsoft has always had the greatest respect for Linux, and loves it, and wants to embrace it completely. And extend it. And who knows what next. By the way, if you're using a non-Edge so-called "browser" you might see some corrupted text at the start of this comment. Ignore it.
The Lawrence Livermore National Laboratory uses OpenZFS on Linux to store their unfathomably large data sets. If Oracle had the ability to sue, they would have done that years ago (knowing how litigious they are).
Given the areas of research that LLNL dabbles in, it might just be a tad unwise to sue exactly them.
Sun didn't want the ZFS code in Linux, so they chose the CDDL license. This was their decision, and made sense as Solaris was a Linux competitor.
Nothing has changed (except that Oracle bought the copyright). If Oracle want to allow ZFS in Linux, they can publish a letter that says it is ok, or something. Maybe someone should ask them? (Hint people have been asking Oracle this for many years)
As Oracle have never said that it is ok, I don 't see why armchair lawyers still think that that ZFS in Linux is fine.
I don’t think there’s much Oracle can do, apart from perhaps open sourcing their own version and releasing it under GPL.
It is, however, a bit like the OpenOffice vs. LibreOffice situation. Oracle may have the “original” but nearly all the modern ZFS development has been at the OpenZFS implementation, not at the Oracle implementation (because OpenZFS 2.0 is the merger of various open source and proprietary versions). The Oracle implementation is thus less interesting.
And getting OpenZFS relicensed under GPL is a matter for the OpenZFS developers and companies that have contributed their code to OpenZFS. There’s nothing Oracle can do about that as the majority of the OpenZFS code is outside Oracle’s control.
That one gave me a good chuckle, it's so far-fetched, and yet asserted with such a straight face.
Using their copyright over some bits of their code in Linux, they would sue Canonical for, let me see, breaching the license of Linux by using code that contains huge bits under Oracle copyright, and for which they themselves (as buyers of the previous owner) picked the license that they'd be asserting is in violation of the GPL.
One may dislike Oracle without believing them to be utterly stupid :)
Thank you - was checking the status of this just a couple of weeks ago, as I had to do a migration while missing a physical disk.
Getting it to boot on Linux is still more of chore than it should be, but once you're up and running OpenZFS is just great. I still remember the dance of ext-on-lvm-on-devices and what a surprise it was when I first hit zfs. CLI tools that were coherent, easy to understand and complete. Most unexpected.
BTRFS on the other hand... I didn't go back after I hit a disk full situation that couldn't be recovered by deleting files. I don't want a hobby project running my disks.
Thanks for the heads up on Btrfs, I'm looking at a Raid 5 device that is near its limit right now, though 10% not 3%.
Guessing from what you said, the array crashed went into read only mode. Before choosing Btrfs, we had a test server set up using small 5 x 250GB drives, and pulled drives, to simulate failed disks, implementing a rebuild to get a good understanding of issues and time to rebuild etc.
Even though we did test a raid array to near capacity, the main thing I remember is the transfers stopping before the limit was reached, and you'd lose access to the server over the network, based on notifications that 3% storage remained, then it would come back. But the raid remained healthy, on checking it.
Might check this again with the test server, time constraints, mean you never get to fully test stuff.
In fairness though, with spare capacity, not reaching it's capacity, Btrfs has been rock solid for several years now. It's a server that regularly powers down, and starts up again with WOL (wake on lan).
Original AC here - to be fair this was at least a decade ago, things move on. And for full disclosure I recently hit issues with ZFS going beserk (ZFS and Java both optimistically reserve memory, so you need to enforce some limits on ZFS to stop Java hitting the stops. But set those limits wrong and the cache flush process goes bananas - first time I've seen a three figure load average).
Otherwise, highly recommended. My recent migration involved raidz from 5 x 1TB to raidz on 4 x 2TB, which I had to connect over USB as I didn't have enough bays. Snapshot, zfs send pipe to zfs receive, done. Follow with the usual collection of grub2 incantations and burnt offerings to the bootloader gods, and after about 300 attempts it came up first time...
I have BTRFS on my boot disc and ran into space problems recently. (It's a very small partition for /)
Confused that deleting files (and even snapshots) didn't seem to free up the expected space, I eventually came across
btrfs balanceWorth a look if you haven't needed to delve into the inner workings of BTRFS previously.
M.
I was originally very excited when BTRFS was presented and the first versions came out. Not so much any more since ZFS became more accessible.
Furthermore, I have to bow to Jim Salter's storage expertise and experience and from what I read from him, even if the RAID data loss issues are now supposedly behind us, there are still too many issues with BTRFS for me to even test it as an alternative to OpenZFS. I'm just not that confident that it will ever catch up, especially not since the OpenZFS 2 merger.
I want to like ZFS, but tons of features is inconsistent with code reliability and maintainability. Reliability and visibility into the guts, and programs I can use to manipulate those guts are important to me. "Trust our fully-automagic system" -- with no manual alternative -- is unacceptable to me.
ZFS offers a userland tool to access the low-level parts of your block devices, vdevs, and pools: zdb. It does not hide anything from you and allows you to see all features, fields, and structures of your pools. ZFS isn't "magic", and is actually very well structured, with its use of feature flags and vendor extensions being clearly deliniated both in source code and in practice, as zdb will show you. Its simplicity might seem untrustworthy, but in reality it is actually that straightforward. Furthermore, the userland tools you are provided with are about as "manual" as you can get without redesigning the wheel—really, you could make the same claim about Btrfs, or any other monolithic, all-encompassing filesystem.
...Of course, no matter how ultimiately trivial ZFS's various systems are once you understand them, there is a good bit of jargon when you dig past the usual basic use-cases, with a lot of additional features, extensions, and configuration tweaks whose purpose and use may not be immediately apparent. There is no shying away from the fact that ZFS, being a well-matured project with lots of different tools under its belt, has a degree of complexity, but you also don't have to use or even enable the features you don't want to use on your pool. It is additionally possible to decypher ZFS internals if you have some lower-level knowledge in other filesystems, and the ZFS official documentation (both for end-users and developers) is very thorough. There are also lots of blog posts where even the practically ancient ones give lots of useful insight. Since a lot of people use it, including large corporations, there is no shortage of experts either, of which you might be able to pick the respective brains of if you ever see the chance arrive.
As an example, while the task seemed daunting at first, I managed to recover data from a completely dusted pool (a problem caused by my own negligence, no fault of ZFS—in fact if I were using it more effectively, I would have had proper redundancy that could have rebuilt the pool) by manually accessing the device with zdb. It is a very useful tool and gives you a curated, structured, parseable view of your data in a way that I have never seen from any other filesystem, and while the ease of something like extundelete would certainly be welcomed if anyone would want to make something like that, I ultimately had everything I needed to play part-time data analyst and recover what I needed without too much hassle.
I used this article at the time to get my bearings with zdb. It should still be relevant. Haven't had the need to look anything up since due to a lack of problems, now that I know what I'm doing.
AC "I want to like ZFS" here ... I'll take my lumps (holy moly, time flies!) -- the last time I looked at ZFS, there was no way to manually invoke a scrub. That's been fixed.
I'm still dubious about the featurefulness vs reliability of ZFS, and more dubious about the number of open ZFS bugs which look significant, but have no one assigned to them yet. To be fair, I haven't yet checked the bug tracker for the ext4 subsystem.
All in all, I ought to spin up some VMs and do some user-level ZFS testing.
I would personally recommend you do. Yes, back when ZFS outside of Solaris barely had any functionality out of the gate (no manual scrubs, oh my) and there were multiple forks for different distros, I would have certainly not recommended it unless there was no other solution, it wasn't ready for primetime by any stretch of the imagination. But by this point, since the OpenZFS merger, there is a command for basically anything you want to do with the system and you are generally no longer restricted on what you want to do. I would not hesitate to use it for production, though I would probably hesitate to update it very frequently, as I ran into this issue myself after an update.
I would also say OpenZFS is probably more stable on FreeBSD, though I have not used a BSD in long enough to say that for sure, and they have been making strides to get feature parity with FreeBSD and Linux. (Solaris/Oracle compatibility is another story.) Here on Alpine Linux, that bug I linked is the only issue I've run into using ZFS on root, encrypted volumes, the System Attributes feature, and more, so my own empirical evidence says it's "stable enough" at least for my uses. Performance has also improved on Linux to the point that only the most heavy of FS I/O will show a clear winner to simpler filesystems like ext or XFS.
If you don't need the features ZFS provides, or if you need redline performance, by all means don't use it. But the tools are easy enough to use by now and the features so plentiful that it can probably suit many different use-cases. I for one use bookmarks and snapshots to do incremental backups to an off-site pool, which is not something you can easily do with a more traditional FS setup.
Also, on the topic of bugs, as far as I can see the GitHub tracker is in decent enough shape for such a large project. You should also pay attention to just how much is fixed on the reg, and not just the open issues. I definitely get where you're coming from though, but I myself have learned to expect a little jank in any project that isn't part of some corporate monolith... and even then, that doesn't guarantee quality (coughWindowscough)
Disclaimer: I am probably biased toward ZFS since I use it extensively.
Worth saving people some time here.
The Ars Technica article {annoyingly} leaves the most important thing till the very last paragraph for those running Btrfs on Synology NAS devices.
"In the meantime, if you're going to run btrfs in any configuration in which it manages multiple disks—as opposed to, eg., Synology and Netgear NAS devices, which crucially layer btrfs on top of traditional systems like LVM to avoid these pitfalls—please do so very carefully."
I can hear the collective 'Phew!'.
{Different Anon to OP}
Ah, you don't really know ZFS then?
The RAID aspect is superior in that it has block checksums separate from the storage device, so catching firmware bugs and controller mistakes, and if you do have unrecoverable errors ZFS tells you the actual files involved! Not just some message in syslog about block 123193474 unrecoverable, etc.
Also ZFS has a 'pool' of storage and multiple file systems share it, so you don't have to decide up front how file systems should be partitioned. Yes, you can set usage limits, but in most cases it works just as shared. However, you can then easily have / /var /tmp /home with different mount options and not waste space on one fs that you later realised you needed for another.
Oh, and snapshots! Not mentioned enough but as ZFS is copy-on-write they allow easy and low-cost roll-back to past file contents. Great for recovering from ransomware (provided they don't get root password on the ZFS system, of course...)
Yes, the combination of block device/physical volume management, logical volume management, and filesystem into one is absolutely a boon for ZFS. Being able to see the entire landscape from one vantage point (or two with zfs vs zpool on CLI) is incredible. Compare and contrast to the pain of setting up RAID encryption the "traditional way", and I never want to go back. This is ignoring the great features ZFS offers like mentioned.
MDADM and LVM are robust and flexible, I use them both, but what filesystem do I put on top if I want data integrity checking? There's btrfs in single device mode which seems reliable, but that can't then correct any errors, only return an error code instead of returning corrupt data.
I'm gradually exploring btrfs, but it's turning out to be a massive dissapointment. It's very existence in the linux world is likely dissuading other potential attempts to create a modern linux filesystem. It's probably why there is no ext5.
btrfs is very intolerant of flaky hardware, if it can't write to one of its disks in raid1 mode it seems to fall over easily, and it can be a pig to recover.
The inability to expand or reduce raidz in zfs is why that's a non-starter for me. I still have to wait 6 months or so to get the partial ability to expand raidz, maybe I'll give that a bash soon.
Is zfs still dog slow on nvme devices? Can it read/write to 4 striped nvme drives at anywhere near the 28GB/s R/16GB/s W theoretical limit? If not, it's a pretty crap filesystem for anything more than spinning rust. Modern file systems need things like zero-copy memory transfers, RDMA support, and afaik zfs has none of that.
If you mean Apple File System (APFS) then I’d agree with you. It benefits from leaving a lot of legacy stuff behind (such as working acceptably on HDDs) but compared to ZFS it’s in a very different space.
APFS is great for everyday use on clients (though unfortunately it only has checksums on metadata and not the data itself as it expects SSD protocols to pick up corruption) and I use it on my clients. I’m typing this from an APFS powered device.
For serious storage where safety and reliability are more important than speed and flexibility, however, I very definitely prefer OpenZFS over APFS. Almost all my servers use OpenZFS.
I'd say that ZFS really shines when it comes to large filesystems. Metadata performance remains excellent with over tens of millions of files. You can layer RAM and NVMe caching over slow disks or network block storage. Dedup, compression, and forced asynchronous mode are valuable in some uses. Parallelism can be cranked up for large numbers of high latency devices.
I wouldn't use it for just one or two NVMe devices because the overhead isn't helpful.
I use it on a personal server and I'm impressed that I lost only one file when all the SATA cables from Fry's simultaneously went bad on my RAID. It survived the week it took me to figure out WTF was going on and order new cables.
"You can use OpenZFS on Linux; the problem is building it into the kernel because Sun's Common Development and Distribution Licence is incompatible with Linux's GPL2. [ ... ] Btrfs is not so encumbered. "
Btrfs is released under the GPL, which mean it's MORE encumbered.
If things aren't compatible with the GPL because they won't adhere to it's restrictions, then it's the GPL that needs to change.
Similarly with ZFS, and the GPL zealots criticising it for not licensing under the GPL. If YOUR rules are stopping you from using something, it's YOUR rules that need to change, not the other parties.
"Btrfs is released under the GPL, which mean it's MORE encumbered."
That's a matter of debate. The philosophy behind the GPL is to prevent "backtracking". Once something is released under the GPL, there's supposed to be no way it can be closed again, unlike other types of licenses. If that's considered an encumbrance, then they'd reply it's a necessary evil.
"If things aren't compatible with the GPL because they won't adhere to it's restrictions, then it's the GPL that needs to change."
Or perhaps the software needs to re-license itself to assert its commitment to full open source? Which is more important?
Why would OpenZFS need to be relicensed? It’s currently open source and working just fine. Thanks to its open source nature it’s receiving support and code contributions from dozens of different places, from one-man volunteers to giant enterprises. It’s in use in a whole number of open source products, from Proxmox, Illumos and FreeBSD to Ubuntu, OPNsense, XigmaNAS and TrueNAS.
The only thing that can’t currently happen is it being mainlined in the Linux kernel. But, considering there are plenty of orgs that provide Linux versions with OpenZFS, that doesn’t seem to be much of an obstacle in practice.
"Or perhaps the software needs to re-license itself to assert its commitment to full open source? Which is more important?"
That's exactly the attitude I was referring to in my post. At the risk of repeating myself, It is full open source. You're second question is therefore a strawman at best.
If someone wants the best software to suceed universally, the GPL is restrictive. Companies have been known to roll their own inferior code to the detriment of the community, because software is GPL rather than some less restrictive license.
I love the utopian view you and others have on the software world, where everyone does the right socialist thing, but it doesn't exist.
I'm not kicking the GPLs philosophy, but its naive to think it will work in every situation, and downright arrogant to blame non-GPL licensed software for the GPLs faults which you choose to adhere to.
To put it another way, HOW would the GPL goal be compromised by allowing OPENZFS (and other similarly licensed projects into the Linux kernel?) - it wouldn''t. Those stipulations are purely idiological.. "Do it our way or you can't come in". And then worse, people like you say "they have to change to be more open source."
Of course, ZFS on linux is thriving despite this. It's not killing the GPL, but the GPL is making it more of a hassle. Anyone who then says its openzfs that needs to change because of this is biased beyond rationality.
The CDDL is incompatible with the GPL, and Sun chose that so that RedHat and others could not ship ZFS on Linux to compete with Solaris back in the day.
That you seem to think "I found source code on the Internet, so I can paste it into my project" probably invalidates most thoughts you might have on the subject of software licenses, of course.
So? There are projects that can't use the GPL because the licenses are incompatible. You don't hear people shouting "they must change their GPL license".
The fact you can't see that speaks volumes. Or, to put it another way:
That you seem to think that I think "I found source code on the Internet, so I can paste it into my project" probably invalidates most thoughts you might have on the subject of software licenses, of course.
FTFY
The article is misleading, implying that ZFS pools could not be expanded earlier. They could, if not since "day 1" then at least early in Sun era.
Bigger disks may replace older smaller disks one by one - and when a top-level vdev is all upgraded, new free space can be reaped.
New top-level vdevs can be added.
The innovation of this feature is about changing the amount of disks in a raid-z top-level vdev, which was complicated because logical addressing should remain consistent across your years of snapshots and cloned datasets and terabytes of data, without changing all old metadata trees.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
