A beefy Linux 5.14-rc2 and light at the end of the tunnel for Paragon's NTFS driver

The latest release candidate of the 5.14 Linux kernel is a hefty beast, Linus Torvald remarked yesterday, seemingly impatient over how long it is taking Paragon to send in its long-awaited and much-reviewed NTFS driver. It has been nearly a year since Paragon submitted code for a read-write NTFS driver in the Linux kernel. The …

[Blackjack]

I admit my ignorance

But if the FUSE driver is read and write, works more or less okay and has been tested and used for many years, why not have a version of that in the kernel instead?

[Anonymous Coward]

Re: I admit my ignorance

The NTFS-3G FUSE driver has been used by all Linux variants for over 10 years and it has more advanced features than the Paragon driver like user mapping, ACL, encryption, deduplication, etc. Porting all of those features would be an enourmous amount of work.

[MatthewSt]

Re: I admit my ignorance

I think (not sure) that kernel mode drivers offer better performance as you're logically closer to the hardware. Every time data needs to be transitioned between user mode and kernel mode you lose some speed

[Anonymous Coward]

Re: I admit my ignorance

A FUSE study showed storage is typically the performance bottleneck. Their conclusion was "Modern file systems are complex software that are difficult to develop and maintain, especially in kernel space. We argue that a lot of file system development can be moved to user space in the future (see Apple, QNX, etc). Contrary to popular belief, our experiments demonstrated that for over 40 workloads, the throughput of user-space file systems has an acceptable level (95% of the kernel driver)"

Google "Terra Incognita: On the Practicality of User-Space File Systems - USENIX" for details.

It's pity no performance comparison was done between the kernel and the FUSE driver.

[Malcolm Weir]

Re: I admit my ignorance

Unfortunately, this dates from 2015 when NVME storage was just a glimmer in a few specialist eyes!

The killer is this from slide 9: "Extra memory copying" (with a side order of "Costly context switches"). Fairly commodity "Ice Lake" CPUs have a memory bandwidth of around 230GB/s, which is good, but a PCIe Gen4 x16 will move data at 32GB/s (nearly) and finding storage subsystems that will use most of that is fairly straightforward. But if you have to use twice the memory bandwidth because you're using FUSE your ~30GB/s storage "costs" you ~60GB/s in memory, which is a fair old chunk of available bandwidth that might want to use for, you know, doing stuff.

It's not impossible, of course. But the shift in relative performance between storage and main memory over the past few years makes some of the casual assumptions from back then more questionable.

[Anonymous Coward]

Re: I admit my ignorance

FUSE supports splice which eliminates the extra memory copy. I don't know if ntfs-3g uses it or not. If the performances are the same then it should. Otherwise adding this feature may have been much easier and faster than porting (reinventing?) the kernel driver which apparently took three years according to Paragon copyrights.

Nowadays there are several advanced technologies like DPDK, SR-IOV, netmap, io_ring, RDMA, etc which enable same or close to kernel level performance for user space apps used by Amazon, Apple, Blackberry, IBM, HPE, etc. The benefit is much faster development time, so more efforts can be spent on innovation.

[Malcolm Weir]

Re: I admit my ignorance

All perfectly reasonable comments, but not entirely relevant to a comparison of two existing codebases (NTFS-3G over FUSE vs Paragon in kernel).

The Usenix paper cited seemed to think there were extra memory copies and context switches. Perhaps they were wrong? But assuming, in the absence of evidence, that the authors were NOT wrong, the existence of splice (etc) seems of less than total relevance!

As others have noted, NTFS is an extremely complex beast with, uh, "challenges" mapping it into the Linux space. Yes,*if* you were starting from scratch, there are plenty of technologies that could be used, but you'd need to fully understand NTFS, too. This is not trivial, so we're left with using what we've got.

Fortunately, in my world (at least) WSL2 seems to be reducing the need of Linux NTFS support somewhat.

[Missing Semicolon]

Re: I admit my ignorance

Because the current driver (at least, the way it seems to be integrated into the Ubuntu variants) trashes any NTFS volume you browse to.

The old read-only driver just ignored NTFS permissions, so you could happily browse a mounted Windows system drive with impunity. The FUSE driver gives you the same experience, but it does it by replacing the owner of every file and directory you view with some internally-generated random generated UID. With the result that a mounted Windows system volume rapidly becomes unbootable. I'm amazed more noise isn't made about it.I know you can manually mount the volume read-only, but the default desktop auto-mounter mounts it read-write, with destructive results.

[Blackjack]

Re: I admit my ignorance

That has never happened to me but I use Linux Mint and Puppy Linux Fatdog. Sure sometimes it mounts as read only but a reboot usually fixes it.

[502 bad gateway]

Happy days

Excellent news, the next time I see a user in my favourite Linux forum complaining that they cannot synchronise their NTFS partitions I'll point them at 5.14-RC2.

[martinusher]

I suppose it wouldn't be too much to ask to go the other way?

Microsoft has shown very little inclination to build drivers that know how to work with non-Microsoft filesystems. I suppose its a combination of 'not invented here' and 'still wedded to drive letters' plus their other idiosyncrasies (like using the escape character as a path separator and a lack of soft or hard links) which makes it difficult for them to design mountable filesystems.

Its really corporate petulance (IMHO) -- Cygwin cracked the problem of matching Linux to Windows decades ago.

[Nugry Horace]

Re: I suppose it wouldn't be too much to ask to go the other way?

NTFS has had hardlinks since the beginning, and symlinks since Vista.

[Malcolm Weir]

Re: I suppose it wouldn't be too much to ask to go the other way?

The biggest mess I see with Windows filesystems is the fact that you have two namespaces: the "drive letter" space and then the "filesystem on drive" one.

It's pretty easily solved (e.g. by creating a new space, such as just a ":" with no letter, and then putting all the letters under that, so you'd have :\C\Windows\, or something like that, or modifying the share semantics so that \\localhost\C etc. is automatically and irrevocably shared to just the local host), but it requires MS thinking there's value to this, and, for whatever reason, they don't.

[Malcolm Weir]

Re: I suppose it wouldn't be too much to ask to go the other way?

Well, let me introduce you to Windows Subsystem For Linux 2... aka WSL2.

It's basically a (very) lightweight VM that exports Linux filesystems to Windows. You get what appears to be a network share of the the Linux filesystems. It's not perfect, but it's improving release-by-release.

(Biggest issue I had when I dug into it was that it didn't support removable media very well, which is kinda understandable for situations where you have a USB drive with both NTFS and xfs filesystems on it... which side gets control of the physical USB target? But they seem to be moving towards solving those sorts of problems with e.g. GPUs, so I'm optimistic).

[Sanguma]

I for one

would welcome an NTFS Linux kernel driver that doesn't force me to take my external drives to a MS Windows machine to fix the errors left by attempting to write to them by a Linux driver that isn't fully compatible with the NTFS file system. I've had to do that on several such, and it gets annoying. (Admittedly it would be nice if the external USB drive suppliers sold them for ext4 as well as NTFS ... dreams are free, but that's not what the Dream Police say ... :) )