Please start testing:
ob.surbl.org
on your low-volume mail servers. Currently it's hosted only on a few name servers, so please don't put it into production on any high volume mail servers yet.
We are particularly interested in hearing your False Positive rates.
If the testing is successful we will ask the rest of our name servers to carry it, document it, announce it, etc.
This list has about 40k domains on it. We still looking for ways to prune it down, for example by expiring domains that no longer resolve. If someone had a reliable master list of expired domains, that could be very helpful.
The data for this list is kindly supplied by Outblaze, and they use it internally for blocking spams. Here is how Yusuf Goolamabbas describes the data sources: __
We gather the domains 4 different ways:
1) spam complaints that have been handled by a human (postmaster/abuse/support people who actually see the spam).
for the next 3, "new" is defined as: Whois reports the domain as newly registered(registered within last 90 days)
2) crawling thru our undeliverable outbound queues. When we get spam from a domain (mailFrom), some bounces are generated. These bounces are handled by a central queue. That central queue is looked at every 60 minutes to see where emails are bound to. If the domain they are bound to is "new" - the domain is blocked.
3) spamtrap body analysis: We have an extensive set of spamtraps. The emails to these accounts are analysed and URLs are extracted. For any domain found in these emails, we check for "new" and if so, its blocked.
4) spam complaint body analysis: Similar to spamcop/yahoo/AOL, we have a feature that allows our users to complain about individual emails "This is spam". All these complaints are analysed and URLs extracted from bodies. If any of these are "new", they are automatically blocked.
We currently do not have a whitelist as we have never needed it. Several things prevent the need for a whitelist: a) humans that are allowed to do the blocks know what they are doing :-) b) machines that are doing the blocks will only block based on "new" c) machines cannot block any domain that was previously blocked and then unblocked by human.
We do not currently remove/expire domains automatically although its under consideration.
Removal procedure is by contacting postmaster at outblaze.com __
We are working with Outblaze to not include some of the domains that are less appropriate for SURBLs, but so far the data looks like it could be quite useful. Please give it a try and let us know what you find.
Thanks,
Jeff C.
Jeff Chan wrote:
Please start testing:
ob.surbl.org
...
This list has about 40k domains on it. We still looking for ways to prune it down, for example by expiring domains that no longer resolve. If someone had a reliable master list of expired domains, that could be very helpful.
I'm very interested in this, as I use my own URLBL plus surlbl. Some time before I was trying to program this, but I had no time to end it.
I'll work on it again, unless there is an efficient solution before.
The problem, to remember, is to quickly check if a bunch of some thousands domains are already valid or not.
Jose-Marcio
Good day, Jeff, all,
On Wed, 16 Jun 2004, Jeff Chan wrote:
This list has about 40k domains on it. We still looking for ways to prune it down, for example by expiring domains that no longer resolve. If someone had a reliable master list of expired domains, that could be very helpful.
I have a list of dead domains on the sa-blacklist processing system. Those are domains that were once on sa-blacklist, but removed because they either expired or were disabled by their registrar. I have a static version of the directory listing and will send that in a separate email to Jeff and Yusuf. Cheers, - Bill
--------------------------------------------------------------------------- "``Threads are like salt. You like salt, I like salt, but we eat a lot more pasta than salt.'' The thread guys are trying to tell you that diet of salt is a good idea. They are wrong, don't listen, eat more pasta and be happy." -- Larry McVoy lm@bitmover.com -------------------------------------------------------------------------- William Stearns (wstearns@pobox.com). Mason, Buildkernel, freedups, p0f, rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org --------------------------------------------------------------------------
On Friday, June 25, 2004, 7:48:27 AM, William Stearns wrote:
On Wed, 16 Jun 2004, Jeff Chan wrote:
This list has about 40k domains on it. We still looking for ways to prune it down, for example by expiring domains that no longer resolve. If someone had a reliable master list of expired domains, that could be very helpful.
I have a list of dead domains on the sa-blacklist processingsystem. Those are domains that were once on sa-blacklist, but removed because they either expired or were disabled by their registrar. I have a static version of the directory listing and will send that in a separate email to Jeff and Yusuf.
I have removed Bill's list of dead domains from all SURBLs. That should reduce all lists in size somewhat. Thanks much for sharing this resource Bill.
Jeff C.
Jeff Chan wrote:
I have removed Bill's list of dead domains from all SURBLs. That should reduce all lists in size somewhat.
Are you sure that this is a good idea ? After all SURBL is a way to identify spam, and stupid spammers spamvertize "dead" domains. Or they create 1 domain per spam run and don't care how fast it's "dead". Rogue registrars like DirectI probably have this procedure on auto-pilot, including their "we'll look into this" delay tactics. Bye, Frank
On Friday, June 25, 2004, 6:17:27 PM, Frank Ellermann wrote:
Jeff Chan wrote:
I have removed Bill's list of dead domains from all SURBLs. That should reduce all lists in size somewhat.
Are you sure that this is a good idea ? After all SURBL is a way to identify spam, and stupid spammers spamvertize "dead" domains. Or they create 1 domain per spam run and don't care how fast it's "dead". Rogue registrars like DirectI probably have this procedure on auto-pilot, including their "we'll look into this" delay tactics.
I agree with your point that if spammers continue to use dead domains in their spams we could continue to block on them, but I think several factors could influence that:
1. Unusable domains do spammers no good. They probably don't have much incentive to keep including them since they would send people to a site that doesn't work.
2. There will be quite a bit of latency in our removing dead domains. The delay will probably let them expire out of use in spams also.
FWIW They're not actually removed; they're suppressed with a whitelist. So we can easily restore them if needed by taking them off the whitelist.
Pruning these lets us reduce the size of our lists quite a bit which is generally a good thing for practical reasons.
Jeff C.
Jeff Chan wrote:
- Unusable domains do spammers no good. They probably
don't have much incentive to keep including them since they would send people to a site that doesn't work.
ACK, and as far as SC is concerned, SC wont't find a report address for "dead" domains, and therefore they should drop from sc.surbl.org automatically. I have to admit that I never checked how the other SURBL zones really work, do they also have some kind of automatical expiration ?
FWIW They're not actually removed; they're suppressed with a whitelist.
Yes, and that's the bad thing, because it's something you have to do manually. Sooner or later spammers will abuse the delay caused by manual whitelists. Maybe you could use the SC input to automatically remove alive-again-domains from the dead list.
Pruning these lets us reduce the size of our lists quite a bit which is generally a good thing for practical reasons.
If you want to prune the list replace all listed *.biz domains by one wildcard *.biz entry (just kidding ;-) Bye, Frank
On Saturday, June 26, 2004, 12:12:58 AM, Frank Ellermann wrote:
Jeff Chan wrote:
- Unusable domains do spammers no good. They probably
don't have much incentive to keep including them since they would send people to a site that doesn't work.
ACK, and as far as SC is concerned, SC wont't find a report address for "dead" domains, and therefore they should drop from sc.surbl.org automatically. I have to admit that I never checked how the other SURBL zones really work, do they also have some kind of automatical expiration ?
Yes, the sc entries currently expire after 4 days of no reports, so they're data-driven. The ph entries have a lifetime also, and ab has a time window of 7 days. However the ob and ws lists did not have automatic expiration, so they could grow indefinitely, which is why it's good that Bill is working on an expiration function based on the domains becoming unregistered.
FWIW They're not actually removed; they're suppressed with a whitelist.
Yes, and that's the bad thing, because it's something you have to do manually. Sooner or later spammers will abuse the delay caused by manual whitelists. Maybe you could use the SC input to automatically remove alive-again-domains from the dead list.
FWIW I stopped removing the dead domains from the lists for other reasons. I may add the removal back again later. The number of entries removed was not too great, even for ob, which is about the same size as ws at about 20k entries.
Jeff C.
Hi!
have this procedure on auto-pilot, including their "we'll look into this" delay tactics.
I agree with your point that if spammers continue to use dead domains in their spams we could continue to block on them, but I think several factors could influence that:
- Unusable domains do spammers no good. They probably
don't have much incentive to keep including them since they would send people to a site that doesn't work.
But if there is backlog on spamruns, and a lot of times you see runs spread over various days, then its still annoying. The spam itself cant do anything, url is dead, but you still get the spam. I would suggest since the zones are somehow small anyway, to keep those and put the 31 day grace period on those. Autoremove them after that but not earlier...
Bye, Raymond.
-
Frank Ellermann -
Jeff Chan -
Jose Marcio Martins da Cruz -
Raymond Dijkxhoorn -
William Stearns