RE: False positives
>> False positives cause infinitely more harm than false negatives.
That I agree with, and there are many, many ways (some quite creative) that people have come up with for breaking their systems !
>> I rely on email for business (eg. responding to a request for a quote) and I have to be able to assume that if I send an email it either arrives at the recipient or I get a bounce message, not been silently gobbled up by some overzealous antispambot.
Your complaint is with the clueless imbeciles configuring their mail servers. My No 1 rule is "don't accept a message for delivery unless I'm going to deliver it". So I reject messages (ie scan them at receipt time) because once you've accepted a message, you have few options. If you send a bounce message then you are part of the problem because you'll add to the backscatter. If you silently delete it then you've failed rule no1.
Unfortunately, many mail servers aren't easy to configure to do "before acceptance" checking of mail. That means it's harder to do, and so harder for the aforementioned clueless people who think they can administer a server on the basis that they ticked a few boxes and it seems to work.
If everyone applied rule no 1 then you would at least know that your message hadn't been delivered !
And don't get me started on the people who "fix" the backscatter problem by dropping non-delivery notices ...
I don't see the situation changing because SMTP is too entrenched and too few people are prepared to accept that it can't be fixed by further porking about - SPF is **NOT** a fix, it breaks as much as it fixes). As someone has already said, it needs a complete review of the protocols, and designing something that has security and sender verification built in from the start. Unless a good few of the bigger players get involved then it will never happen.
Look up IM2000, seems to tick most of the boxes - but even then it won't deal with the unscrupulous operators who don't care what sort of **** is sent through their service. So we'll still need some sort of blacklist service to deal with them - the difference is that the blacklist can be distinctly more targetted. The biggest factor in it's favour is that it shifts the cost from the recipient to the server where the mail originates - and that should change the economics of sending spam enough to greatly reduce teh problem without any other technical measures.