Time to blacklist blacklists Blacklists have their place for detecting and identifying malicious content and activity, with the whole signature-based malware detection industry effectively being built around the concept that blacklists are reliable mechanisms. The only problem is that they aren't. They certainly are an important element of security …
"One of the greatest of human follies is belief in a panacea."
I have been a security consultant for about 15 years and it still amazes me that people believe in a silver bullet that will cure all security ills. It's ridiculous to place too much reliance on any one technology. Blacklists are just one such example of this. Others include...
"I have a firewall... why do I need anything else?"
"I have AV... why do I need anything else?"
"I have SpywareUberKiller Mega Edition 2008 v99... why do I need anything else?"
and my favorite one...
"I have a Mac... why do I need anything else?"
"or they could have just had AOL mailing list subscribers who find it easier to report as spam than unsubscribe from something they manually subscribed to"
Amen.
You may use double opt in, remove bounces from your list immediately, have huge unsubscribe notices at the top and bottom of your email, and have fewer than 1% of users mark your email as "junk", but if one of those lazy users was an AOL user you better pray. It is all too easy for legitimate users to end up on blacklists thanks to "zero [fault] tolerance" systems.
The author of the article seems to be against using black lists. Yet they are a line of defense which reduces the amount of spam being sent.
Can someone get caught up in the net because they are sharing a netblock with a spammer? Yes, however, if ISPs were responsive to the spam complaints and cleaned up their acts (Read: No PINK Contracts and a fast response to spammer reports), then the net blocks will not stay in the black lists for long.
Spamhaus.org has a policy of removing a netblock from their black list if there were no spam reports for 60 days.
Forcing people to use their ISPs mail servers unless they can show that they can run a secure e-mail server is another good thing.
This should stop a lot of the junk coming from spam-bots and allow more control by the ISP over their networks.
As the previous poster said, there is no silver bullet, but the author is off his rocker by blasting blacklists.
"Moving on to blacklists of known spam-generating IPs and malware-serving sites, we start to see significant problems emerge with this particular approach to protection."
The only "problems" it has is for the well-known spam- and malware-source ISPs, such as the Turkish state ISP, the largest Israeli ISPs, China's State-sponsored malicious attack network (the whole country, as fasr as I can see), Korea, Thailand, the Philippines...
If you are not expecting email or other legitimate traffic from a well-known "problem" IP range, then the best possible way to prevent attacks from that range is to drop their traffic.
But to expect *only* one solution to fix *all* problems is, frankly, stupid.
I still like the idea of the caller ID for e-mail or SPF records. Only registered e-mail servers can be accepted as legit, all others are spam no matter what. Only this would require than any and all legit e-mail servers be registered. That seems like a small task compared to the larger blocking spam task, but yet few e-mail admins bother doing this. Does your e-mail server do this? If not ask why. It DOES make a difference.
The trick is implementing them.
Most AV vendors are trying to add techniques, other than signature files, to their products.
And the same is true of many anti-spam companies.
The problem isn't knowing that creating lists of undesirable files and IP addresses means always playing catch-up, the problem is creating an effective replacement that works well enough to be marketable.
So full points to the authors for observing what has been obvious for a couple of years now. But, sadly, zero points for coming up with a replacement solution.
relying on a whitelist is the same as living in a cleanroom bubble.
Effective enough, but not what you would call a life.
and anything that gets in will screw you over totally and terminally. Not to mention what happens if the people maintaining your clean rooms air/water/food get bought out and offshored to mongolia..
Like a medical clean room, there are applications it is suited for.
No single technique will suit all situations, for one segment of the industry to badmouth another is like wheelchair manufacturers slandering the polio vaccine. (extending the medical analogy far past its expiry date)
"Simple signature based detection" was *never* good enough for reliable protection against malware, because even a trivial change (i.e., the creation of a new variant) could bypass it.
Unfortunately, it is the only kind of protection that the average user can understand, maintain and use. A scanner tells the user "no, you don't have a virus" or "yes, you have a virus, do you want me to remove it?". As opposed to that, a heuristic analyzer says "This file could contain a virus". Well, does it, or does it not? A firewall says "svchost.exe tries to communicate over port 1900". What the heck does that mean? An integrity checker says "File foo.exe has been modified". Did a virus do that - or was it the Windows Update? A behavior blocker says "msvc.exe tries to write to Bar.exe". Is that a virus or a compiler? And so on.
The proper way to protect against malware is to implement defense in depth - by using a combination of all available techniques. Unfortunately, the average user can never do that. Whitelisting is not a panacea. Nothing is. Some techniques like that are applicable in a very small set of restricted environments, with competent security administrators and where security is more important than convenience. The rest of the world will happily keep using scanners.
Nothing has changed in this aspect for the past two decades. Two decades from now, it will be the same - people will keep using scanners, scanners will fail to protect them adequately, and some "experts" will keep pushing their pet alternative panaceas which almost nobody will use.
*golf clap*
Security by ignorance and obscurity.
I am sure my qnx box will NEVER be hacked. Really, I doubt it will. The same goes with my NT 3.51 box in my garage. I will PAY $50,000 to anyone who can hack it. Really. The box has NOT been patched, 2nd release batch, offical MS install cd. Infact, I believe the latest uptime was running at 6 months (power outtage 6 months ago, and I did not have fuel in the generator). AV? Definetaly not, I've only got 350mhz and 128mb ram. Firewall? Again, no go. Modem? No PCI/ISA slots open. NIC? Again, no slots. CDROM? Doesn't work no more. Floppy? YES!
Any takers? You got to be anonymous and can't have physical access... Good luck. Get the files from my c drive, and it's yours...
The problem is that we're not ready to do the Right Thing and create an immune-system for our computers in the image of the biological mechanisms. Usability would plummet - having a semi-fascist entity in charge of my computer would destroy most of the functionality. Flexibility would be out the window. Only pre-approved programs would be allowed, and exactly who are we going to trust with that kind of power? Microsoft? Not bloody likely. IBM? Haw, haw, haw. Apple? Sony?!? *giggle* Our *government*? Oh boy...
So I'll just take my chances with the current sorry state of affairs and resign myself to the fact that the Bad Guys (parasites the lot of them) will be lucky once in a while, and just make sure that there's a limit to how lucky they can get. Meanwhile I'll pay for blacklists and heuristics and firewall etc. and try not to worry - *that* doesn't help one bit.
Cheers,
Morten
I have been using RBLs on our mail server for at least the past two years and have recently taken an even more aggressive approach. Yes, they do work. I now get no more than 5 spam mails a day out of what used to be close to a hundred plus. The ones that get through are those that have not been reported and we report the ones that get through.
Because our business is targeted at a certain client profile, we can afford to block entire countries and whitelisting bona-fide customers. The rejection message carries our phone number so genuinely interested enquirers can always pick up the phone.
Granted it may not work for everyone but it certainly does for us and they are a Godsend as far as I am concerned.
Is this all we have? Blacklists, security updates, patches and constant fear?
Well I don't know about you, but I'm mad as hell. I spend hours every day cleaning up my online forums after spammers, deleting mail, blacklisting bad words and every possible misspellings of them, installing security updates and so on.
Why can't we find, identify and take out these parasites? I'm a peaceful man, but I would like to see these spammers suffer. As long as they feel safe and go unpunished, we cannot really expect any significant decline in their malicious activities. These people cost us time and money, they do a lot of damage, and they seem to get away with it. I don't advocate lynching, but I would support an anti-spammer law with public hanging for first offense. Maybe it wouldn't completely eliminate spam, but it would be fun to watch...
"Yes, they do work. I now get no more than 5 spam mails a day out of what used to be close to a hundred plus."
To determine whether an RBL is working, you'll need to look at more than just the false negative rate (the 5 spams you receive a day). Do you know how many false positives a day are being generated? (mails that get blocked incorrectly).
Taking those two numbers you would be in a better position to tell how your RBL compares to other spam prevention systems.
Why all the articles about useless security measures? Whitelists, blacklists, AV (which is just another blacklist really) - they have a small place in the defence arsenal, but they aren't the way forward. Write about more intelligent stuff like trusted ownership checking or something...
Isn't this just another example of Newton's third law of motion , for every action , there is an equal and opposite reaction!
And in a real world dynamic , just as fast as you close the loopholes , you open another set of unidentified ones!
Don't forget , the old tale of the car thief , as fast as the various car makers upgrade car security , the thieves find both the deliberate side doors , and simultaneously find the means to counter the new improved car security!
So as with everything , a blacklist is little better then a dog continuing to chase it's tail , but never ever quite catching up to it , given the ever dynamic changing and reacting internet!
Or in the words of one song "the times they are a changing!"
In answer to James Henstridge:
My workplace has a spam filtering system based on both content analysis and blacklists. Each day, at midnight, it sends me an email with brief details (From, Date and Subject lines) of each email that it has blocked on my behalf, with a reason.
The last time I had a false positive was at least three years ago, and that was a friend in China whose institution's mail server had been hijacked by spammers. I alerted him to the fact, and he told his sysadmin, who acted quickly to secure the mail server and request a review from the operator of the blacklist. My Chinese friend was able to send me email again within a couple of days, and he was grateful that the temporary blacklisting had highlighted a security flaw.
I have just been informed by an overseas branch office, that as someone is using an IP address for spam, and the first three octets are the same as ours, that mails our branch are sending to a counterparty (via our head office) are being blocked. Its a pain in the neck and the US administrators of the recipients mail domain basically say "tough". Its a bit like me blocking snail mail from Texas to my offices because I receive a rude letter from someone in San Antonio, not very fair on senders from Houston, Dallas etc.
I have spent time trying to explain this in the past to sysadmins at ISPs out there and they just dont care, despite the fact its their clients they are also messing about. Do these idiots not understand that we are not necessarily responsible for our neighbours (in IP address terms) actions?
I use the Spamhaud blacklist because I find it fairly good at blocking spam - but it's only one tool I use AND I reject rather than discard mails that fail the checks so the user knows that their message has been blocked. I agree that it's broken and has numerous faults - but it's less bad than the alternatives which is about all you can say of most of the 'solutions' being sold !
I see at least one comment calls for SPF - well that person obviously doesn't know how email is actually used and how utterly broken SPF is. Broken ? Yes. It's broken in that it's trivially easy for spammers to publish SPF records for their false domains, AND it completely f***s up mailing lists !
Quite frankly, I cannot see any way that SMTP can be fixed - at least not without breaking it to the point that it's no longer worth using at all. Perhaps it's time for something new ?
How about IM2000 (http://www.im2000.org/). Sounds good, works with mailing lists, shifts the cost of storage onto the sender. Will it stop spam ? NO, it won't completely stop it. It will have a big dent as the senders will have to have storage agents available for their messages to be read - using botnets for this won't work in the vast majority of cases. Once you force the spammers to have online servers then you have single points you can blacklist. Getting the whole world (including Microsoft and AOL) to switch to something new ? Good look selling anything that "wasn't invented here" to them !
They'll adapt though, I predict their next attack vector will be to extract the users credentials from their mail client and use those to send mail via the users normal server - but that still leaves mail going via a path that can be monitored and the mail fitered at source.
I get lots of spam, but I won't use spam blockers other than as a marking aid because they always block legitimate content at some point. If you are encouraging people to contact you for business then it's better to take the spam hit than lose a potential lead or sale. From experience, many tentative business enquiries will actually use personal or 'safe' email accounts on first enquiry, because they want to check you are genuine and they don't want their email address or company details to be revealed (they don't want to get spammed). When the reply is appropriate then they respond with their proper details.
But this means that the leads come with all sorts of email addresses including hotmail and yahoo, even from senior personnel in large fully protected companies who could use 'legitimate' addresses, but who want to keep their work email addresses 'clean'.
Consequently the problem with the spam blacklists is that I've seen them blocking lots of these types of ISPs - they blocked BTInternet for about 6 weeks at one point - good job guys, only the main broadband supplier in the UK. A client was trying to send an important urgent document from home at the time and we couldn't work out why it was being blocked. And because it didn't get to me for it to be unblocked or whitelisted we couldn't immediately know what was the problem.
Similarly they will block our outgoing email - for instance auto-responders on websites for people registering with a website, but we won't know about it because the block list acts as a black hole in the middle, and we are not told.
Sure they can be helpful - my noise content is around 97-98% at the moment from 1000+ emails received, but I really don't want to screen out legit senders by accident.
I just switched ISPs.
My old ISP just delegated my netblock to me.
My new ISP still wants to control their reverse IP and delegate to me via a CNAME record.
Sounds OK in theory, but a royal pain because its not something one does everyday and the "instructions" provided by the ISP is vague at best.
So, am I pissed?
Not really. It *is* an inconvenience because I'm not blocked by most large ISPs and can't send mail. But I do have alternatives like hotmail, google and another server to send e-mail that must get out.
I should be mad at my ISP for being a tad lame, and for SORBs for not really giving a damn, but I'm not. Why? Cause Sorbs gets a lot of lame looney responses, and my ISP is being lazy but protective of their systems.
Moi? I'll figure out the problem and get back to normal. And you can bet that I'm adding SORBS to my set of block lists.
BTW, I get maybe 5-10 spam e-mails a day on my servers and I do have stringent criteria. My wife's account at work was getting 10-15 times that on a daily basis, if not more!
RBL's can work, if the controllers are not overzealous.
Anyone involved in any small, medium or large ISP / Web Host will have had, at some point, a bad experience with one of the many RBL's. SORBS and SpamHaus in my experience are the worst offenders. SORBS even want money to be removed, how is that any different to the extortion methods a DoS gang would use?
The problem is they think they are above the law. They think the fight against spam is their god given right and who cares if legitimate mail is rejected, who cares if businesses are disrupted, loosing them money.
One of our clients has wrongly been listed as a ROSKO on SpamHaus and is in the process of seeking legal advice on taking action against SpamHaus, because SpamHaus refuse to listen to any arguments regarding the incorrect listing.
There’s a few precedents to legal action against SpamHaus, but they simply ignore the judgments and the fines imposed on them, as many of these judgements were outside of the UK.
What might eventually tame the Net to an extent is likely to be stronger identity and the ability to handle good and bad reputation information about known identities based on cryptographic keys and domain name ownership (DNSSEC and domain-key signed emails combined with layered use of multiple black and whitelists). Until then using an IP address within a blocked /24 block or a Hotmail or AOL email address is an indication that you need a more reputable ISP, because the bad reputation of your immediate neighbours washes off on you.
This post has been deleted by its author
RBLs are a cure worse than the disease from the user's point of view. From the admin's point of view, they are a magic bullet that reduces the email workload.
As an example, note the previous comment from an ADMIN in favor of RBLs. Only the amount of spam blocked is cited, NOT the amount of LEGIT email that was also blocked in the process, or what hell you have to go through to clear up an erroneous entry. By the time a spam connection makes it to an RBL the spammer is long gone, and it's only you who's address (or you're friend's addresses) that are now blocked that has to wait days or even weeks for the interconnect problem to finally clear up. RBLs are about as ineffective as it gets except for those admins who only measure success as a reduction of email traffic-- RBLs do that, there is no doubt.
As soon as I hear of an ISP who uses an RBL, THEY go on MY blacklist of ISPs to block from consideration of any future patronage...
Simon Hobson: "I reject rather than discard mails that fail the checks so the user knows that their message has been blocked."
That is cretinous.
As *everyone* knows, pammers systematically falsify the sender address, so whoever gets notified, it certainly isn't the sender. Unless it was a false positive. But you wouldn't use a filter that generated false positives, would you?
Bouncing spam is exactly the same as relaying it. Don't do it.
There's two kinds of rejecting mail which need to be distinguished here:
- Accepting all mail first, and then sending a bounce when messages proves undesirable/undeliverable. This is indeed bad, as in the case of spam the bounce will go to a spoofed address.
- Refusing the message during the SMTP dialogue. This way, the receiving server never becomes responsible for delivery of the message, rather the sending server has to send the bounce. Ideally, unwanted messages will never leave their origin this way, rendering spoofed sender addressed ineffective.
So I hope Simon is using the second kind of rejecting.
To Andy, please go back and read what I wrote - I do NOT bounce messages since, as you say, it would be cretinous. Actually that is doing a disservice to cretins to associate them with such assinine behaviour.
As Sebastian correctly points out, if you REJECT the message during the initial SMTP dialog then in the vast majority of cases the message just gets dropped as the smap software the user is infected with doesn't bother with failures. The ONLY situation when such rejections result in backscatter (the sending of bounce messages to innocent third parties) is if the message was relayed through a properly functioning mail server - but that is rare and spam software doesn't normally try and do that because they know that most of their messages would never leave the building if they did.
But, in the case of false positives, the user WILL get a bounce message and know that their mail didn't get through - which I think is very important. I really cannot think of anything more stupid than accepting mail and then silently deleting it, which is of couse what most people do. Lets face it, there's a whole industry selling solutions based on just silently deleting email ! In fact, I'm currently having 'discussions' with my ISP who won't let me NOT use their server as a backup MX*, and it is porgrammed to do just that - apart from the fact that it's broken.
* And I can't move my DNS elsewhere (like home) because if I do then their web hosting turns off for the domain.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
