Re: "If your password is brute-forceable, you shouldn't be using it."
1. I doubt you have to type in the 2^4096 SSH key. Which is what we're talking about here.
2. Even your "million second example" takes the wrong conclusion. Yes, if you really want to break into THAT ONE computer, it's not long. It is, however, very long for a lazy dumbass hacker out for a quick buck with some random poor sod's computer. The former is the problem for professional spies, which, frankly, I don't worry much that I'm a target of. The latter is the threat most everyday passwords aim to protect against.