We use passwordless VNC. Admittedly we only let it listen on 127.0.0.1 for SSH tunnelling.
And to be fair, when we update our server, I am going to ideally try to get it to listen on a UNIX domain socket instead. More recent versions of OpenSSH allow encrypted tunnelling to UNIX socket endpoints.