Re: So the problem seems to be that....
You're absolutely correct, it's not tricky.
If these users trust the source code because they can read it, they can easily confirm that the binary blob is never called unless the user affirmatively opts in.
Surely that's exactly what's intended by an open source license that permits linking to closed source binaries? If you don't trust the binary, you can satisfy yourself that it is never called or if it were to be called you can modify the code. If those are beyond your skill level then you can pay someone appropriately qualified to perform the task.