Re: Meanwhile, on the super low-end... :)
Cool. :)
The native ethernet support for Mellanox adapters seems to "just work" (after compiling in the driver). It's been pretty much trouble free.
IPoIB mode though... not so much. Kernel panic when switching from Ethernet → Infiniband mode (post boot - still chasing this bug down), and connected mode IPoIB exhibits weird behaviour too at the moment.
I think the Infiniband mode problems are rooted in something to do with the TrueOS base (not pure FreeBSD) that FreeNAS is built upon, as the same drivers on a FreeBSD base are pretty flawless. I'll be digging into this more over the next few days to see if I can figure out WTF is causing the issues, and then get it fixed. :)
If someone's just after Ethernet support (10GbE/40GbE/etc) out of Mellanox adapters though, it seems to be already good enough. Naturally, test the living heck out of it before deploying to production, just to be safe. :D