On Wednesday, November 10, 2004, 5:25:43 AM, Rob McEwen wrote:
1st, if you are converting domains to IPs and then checking these IPs against spamhaus, you may have to make sure your system can whitelist the domains **before** conversion to IP since the IPs can change without notice.
Interesting. I was going to whitelist after detection. Whitelisting first would prevent some processing.
Note that we're not proposing making a list of IP addresses. The output is still mostly a list of domains.
2nd, SpamHaus keeps listing the following: msn.click-url.com, (& variations) (These show up FREQUENTLY in hams, so I'd Whitelist these up front. They seem to go in an out of SpamHaus intermittently.) FOR EXAMPLE: msn.click-url.com = 216.39.69.75 http://www.spamhaus.org/query/bl?ip=216.39.69.75 ...points to... http://www.spamhaus.org/sbl/sbl.lasso?query=SBL20705
click-url.com is already manually whitelisted so it would not be on our version of the lists. We would likely apply the SURBL whitelisting to these lists.
3rd, in fact, SpamHaus is going to list a lot of greymarketers that shouldn't be listed in SURBL (flowgo, euniverse, etc)
That is one area where we disagee with Spamhaus, and we've whitelisted most of those since they appear in legitimate newsletters, etc. However our whitelists of those domains may not be complete.
4th, most of the FPs I find in SpamHaus are XBL listings where the data source for that particular FP was http://cbl.abuseat.org/
CBL catches a LOT of spam... but it also periodically will list the mailserver for respected IPS where that ISP had one user who send out a bunch of spam and then CBL listed the IP address of that server. Unfortunately, this creates a lot of collateral damage. Recently, I experienced this with one of my clients's customer's BellSouth E-mail services. (I don't know the ratio of XBL stuff via CBL versus XBL stuff from other sources. I'd be curious to know this.)
Queries into our DNS servers almost never match domains that resolve into XBL, which makes sense since those are mostly zombies. However a domain list of XBL hits may be a useful early warning of spammers starting to use zombies for hosting, DNS, etc, which fortunately they haven't done much yet. In practical terms the XBL hits are so few now as to be a non-issue.
(Really I just included XBL for completeness; SBL is generally more relevant for URIs, which is why it's what's used by uridnsbl in SapmAssasisn by default. uridnsbl was probably designed with SBL in mind. If we do this, the SBL and XBL lists would be separate.)
Jeff, very likely, (I have a feeling) I've misunderstood your original intended use of SpamHaus? But maybe this information will be helpful anyway? I would definitely recommend NOT using the strategy I've described as an **automatic** way to get listed in SURBL. This would defeat MOST of the hard work we've done to minimize FPs. But, on the other hand, there are many great possibilities here for using this as a tool for evaluating URIs or as a honeypot for queuing URIs for evaluation where the URI wasn't already in SURBL.
Rob McEwen
The reason for looking at this is a way to avoid the DNS resolution on wild URI domains that urbdnsbl does in SA 3. This process is an approximation of what uridnsbl does with sbl. I suspect that uridnsbl gets some false positives similar to what you notice in your own processing. Presumably uridnsbl is scored lower than SURBLs because of the FPs, and a SURBL version of the sbl data should probably also be scored lower than other SURBLs for similar reasons. Our whitelists would tend to reduce the FP rate somewhat, if applied, which seems likely.
I share your concerns about FPs, but since we're doing something very similar to what uridnsbl does but with much less DNS overhead, the same concerns apply to FPs with uridnsbl, it's just that this new way of doing things would be much faster.
We have not turned this data into lists yet, but the reasons for considering it are as I describe: to bypass the very time consuming name resolution that urndnsbl does against domains in wild messages. It's meant to be a potential speedup/replacement for uridnsbl.
We should definitely discuss this more, and I'd like to hear from the SA developers.
Jeff C. -- "If it appears in hams, then don't list it."