Re: stunnel, wireguard
There are also situations where NFS should NEVER EVER EVER be run over UDP. I guess you can save stunnel for those scenarios.
Isn't there also a userspace implementation of wireguard? Perhaps you would be happier with that version.
From "man 5 nfs:"
Using NFS over UDP on high-speed links
Using NFS over UDP on high-speed links such as Gigabit can cause silent data corruption.
The problem can be triggered at high loads, and is caused by problems in IP fragment reassembly. NFS read and writes typically transmit UDP packets of 4 Kilobytes or more, which have to be broken up into several fragments in order to be sent over the Ethernet link, which limits packets to 1500 bytes by default. This process happens at the IP network layer and is called fragmentation.
In order to identify fragments that belong together, IP assigns a 16bit IP ID value to each packet; fragments generated from the same UDP packet will have the same IP ID. The receiving system will collect these fragments and combine them to form the original UDP packet. This process is called reassembly. The default timeout for packet reassembly is 30 seconds; if the network stack does not receive all fragments of a given packet within this interval, it assumes the missing fragment(s) got lost and discards those it already received.
The problem this creates over high-speed links is that it is possible to send more than 65536 packets within 30 seconds. In fact, with heavy NFS traffic one can observe that the IP IDs repeat after about 5 seconds.
This has serious effects on reassembly: if one fragment gets lost, another fragment from a different packet but with the same IP ID will arrive within the 30 second timeout, and the network stack will combine these fragments to form a new packet. Most of the time, network layers above IP will detect this mismatched reassembly - in the case of UDP, the UDP checksum, which is a 16 bit checksum over the entire packet payload, will usually not match, and UDP will discard the bad packet.
However, the UDP checksum is 16 bit only, so there is a chance of 1 in 65536 that it will match even if the packet payload is completely random (which very often isn't the case). If that is the case, silent data corruption will occur.
This potential should be taken seriously, at least on Gigabit Ethernet. Network speeds of 100Mbit/s should be considered less problematic, because with most traffic patterns IP ID wrap around will take much longer than 30 seconds.
It is therefore strongly recommended to use NFS over TCP where possible, since TCP does not perform fragmentation.
Jumbo frames are the top-rated workaround.
p.s. Olaf Kirch's overview of NFS on Linux says that TCP was always the default.