Looks like senderbase.org has a database of the domains and IPs used to send the most mail. Normally that would not be too interesting to us since we care about message body URIs, i.e. content, and not senders or their ISP addresses, but I'm thinking about whitelisting all the legitimate NSPs, ISPs and telcos in their top domains list:
http://www.senderbase.org/search?page=domains
we would exclude the few that appear to be spammers according to spamhaus:
imgmailer.com TAM Network stocksntalk.com iMedia Networks Inc. havagreatday.com
But I'd like to whitelist all the rest which are obviously large ISPs, etc. In essence we're just using it as a list of some of the top ISPs in the world.
Does anyone have any comments on this?
Note that this won't have a major effect on bad guys since spammers would not have much incentive to advertise their ISPs, and we don't "whiten" spams for mentioning non-spam domains anyway. It also does not mean that we're whitelisting the ISP address space, senders, or anything like that, just mail that mentions these large ISP URIs.
Jeff C.
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Looks like senderbase.org has a database of the domains and IPs used to send the most mail. Normally that would not be too interesting to us since we care about message body URIs, i.e. content, and not senders or their ISP addresses, but I'm thinking about whitelisting all the legitimate NSPs, ISPs and telcos in their top domains list:
http://www.senderbase.org/search?page=domains
we would exclude the few that appear to be spammers according to spamhaus:
imgmailer.com TAM Network stocksntalk.com iMedia Networks Inc. havagreatday.com
But I'd like to whitelist all the rest which are obviously large ISPs, etc. In essence we're just using it as a list of some of the top ISPs in the world.
Does anyone have any comments on this?
I like this idea as I believe it would cut down the number of false-positives due to false-listings.
Note that this won't have a major effect on bad guys since spammers would not have much incentive to advertise their ISPs, and we don't "whiten" spams for mentioning non-spam domains anyway. It also does not mean that we're whitelisting the ISP address space, senders, or anything like that, just mail that mentions these large ISP URIs.
Quick question: If I have set "spamcop_uri_limit 25" in my spamcop_uri.cf file, and a spammer sends a message containing 30 URIs, all legit except one, and 10 of the legit URIs are whitelisted by SURBL, would all of the remaining URIs get checked, or still only a random selection of the entire 30 URIs found in the message? Just wondering if the whitelisting will help us to be more accurate in tagging the spammer URI in the message, thus cutting down the possibility of the spammer URI not being one of the random 25 selected for checking against the SURBLs.
I'm curious to know what effect the SURBL whitelisting has as it applies to both SA 2.6x with the SpamCopURI plug-in and SA 3.0 with the URIDNSBL plug-in and the random URI check limit threshold.
Bill
On Wednesday, September 8, 2004, 12:44:34 AM, Bill Landry wrote:
Quick question: If I have set "spamcop_uri_limit 25" in my spamcop_uri.cf file, and a spammer sends a message containing 30 URIs, all legit except one, and 10 of the legit URIs are whitelisted by SURBL, would all of the remaining URIs get checked, or still only a random selection of the entire 30 URIs found in the message?
25 would get checked randomly from the 30 IIRC. 5 would be not checked.
Just wondering if the whitelisting will help us to be more accurate in tagging the spammer URI in the message, thus cutting down the possibility of the spammer URI not being one of the random 25 selected for checking against the SURBLs.
I'm curious to know what effect the SURBL whitelisting has as it applies to both SA 2.6x with the SpamCopURI plug-in and SA 3.0 with the URIDNSBL plug-in and the random URI check limit threshold.
Bill
The SURBL whitelist is an internal exclusion list which currently has no direct effect on SpamAssassin. All it does now is to make sure that these domains do not get added to any SURBLs.
Instead of publishing the whitelist for example as a "do not check" RBL, we may ask that SpamCopURI and urirhsbl and urhrhssub take a hard-coded list of the top N most queried legitimate domains and never query on them. But that doesn't happen yet.
Theo, Should I open an RFE about this for URIDNSBL?
Bill, If you'd like, you can have a similar effect right now by adding the top N domains from:
http://www.surbl.org/dns-queries.whitelist.counts.txt
to the SpamCopURI manual whitelist in the conf file. You'd want to include both basedomain.com and *.basedomain.com . In fact this should be a good improvement for everyone using SpamCopURI to add.
Eric Kolve may be able to provide more details, or someone could just try it. :-)
Jeff C.
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Just wondering if the whitelisting will help us to be more accurate in tagging the spammer URI in the message, thus cutting down the possibility of the spammer URI not being one of the
random
25 selected for checking against the SURBLs.
I'm curious to know what effect the SURBL whitelisting has as it applies
to
both SA 2.6x with the SpamCopURI plug-in and SA 3.0 with the URIDNSBL plug-in and the random URI check limit threshold.
[SNIP]
If you'd like, you can have a similar effect right now by adding the top N domains from:
http://www.surbl.org/dns-queries.whitelist.counts.txt
to the SpamCopURI manual whitelist in the conf file. You'd want to include both basedomain.com and *.basedomain.com . In fact this should be a good improvement for everyone using SpamCopURI to add.
Hmmm, tried this overnight and it cut my SURBL hit rate down to almost nothing. I'm not sure why, unless the whitelist data is not being read into memory and thus the list has to be parsed with each individual e-mail message before the SURBL queries can be done. If that's the case, then it could be possible that the queries are taking to long to start and SA is timing them out based on the network tests timer expiration setting.
Or, could it be that if the SpamCopURI plug-in finds a whitelisted domain, that it skips any further SURBL tests for that message? That could also account for the very low hit rates when using the whitelist entries.
Bill
On Wed, 8 Sep 2004 09:23:02 -0700, Bill Landry billl@pointshare.com wrote:
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Just wondering if the whitelisting will help us to be more accurate in tagging the spammer URI in the message, thus cutting down the possibility of the spammer URI not being one of the
random
25 selected for checking against the SURBLs.
I'm curious to know what effect the SURBL whitelisting has as it applies
to
both SA 2.6x with the SpamCopURI plug-in and SA 3.0 with the URIDNSBL plug-in and the random URI check limit threshold.
[SNIP]
If you'd like, you can have a similar effect right now by adding the top N domains from:
http://www.surbl.org/dns-queries.whitelist.counts.txt
to the SpamCopURI manual whitelist in the conf file. You'd want to include both basedomain.com and *.basedomain.com . In fact this should be a good improvement for everyone using SpamCopURI to add.
Hmmm, tried this overnight and it cut my SURBL hit rate down to almost nothing. I'm not sure why, unless the whitelist data is not being read into memory and thus the list has to be parsed with each individual e-mail message before the SURBL queries can be done. If that's the case, then it could be possible that the queries are taking to long to start and SA is timing them out based on the network tests timer expiration setting.
Or, could it be that if the SpamCopURI plug-in finds a whitelisted domain, that it skips any further SURBL tests for that message? That could also account for the very low hit rates when using the whitelist entries.
Mmmhhh this last thing would be awful... does anyone know this?
FTR, I just checked and you have to add 2 entries to whitelist *.yahoo.com AND yahoo.com...
But I don't want the whole spamcopuri surbl process to stop if it finds an internally whitelisted domain... that'd be plain wrong.
On Wednesday, September 8, 2004, 9:23:02 AM, Bill Landry wrote:
From: "Jeff Chan" jeffc@surbl.org
If you'd like, you can have a similar effect right now by adding the top N domains from:
http://www.surbl.org/dns-queries.whitelist.counts.txt
to the SpamCopURI manual whitelist in the conf file. You'd want to include both basedomain.com and *.basedomain.com . In fact this should be a good improvement for everyone using SpamCopURI to add.
Hmmm, tried this overnight and it cut my SURBL hit rate down to almost nothing. I'm not sure why, unless the whitelist data is not being read into memory and thus the list has to be parsed with each individual e-mail message before the SURBL queries can be done. If that's the case, then it could be possible that the queries are taking to long to start and SA is timing them out based on the network tests timer expiration setting.
Or, could it be that if the SpamCopURI plug-in finds a whitelisted domain, that it skips any further SURBL tests for that message? That could also account for the very low hit rates when using the whitelist entries.
Bill
Wow, I hope that's not how the SpamCopURI local whitelist works!
Maybe someone can read the source code and check, or Eric can let us know.
Jeff C.
On Wed, Sep 08, 2004 at 03:53:18PM -0700, Jeff Chan wrote:
On Wednesday, September 8, 2004, 9:23:02 AM, Bill Landry wrote:
From: "Jeff Chan" jeffc@surbl.org
If you'd like, you can have a similar effect right now by adding the top N domains from:
http://www.surbl.org/dns-queries.whitelist.counts.txt
to the SpamCopURI manual whitelist in the conf file. You'd want to include both basedomain.com and *.basedomain.com . In fact this should be a good improvement for everyone using SpamCopURI to add.
Hmmm, tried this overnight and it cut my SURBL hit rate down to almost nothing. I'm not sure why, unless the whitelist data is not being read into memory and thus the list has to be parsed with each individual e-mail message before the SURBL queries can be done. If that's the case, then it could be possible that the queries are taking to long to start and SA is timing them out based on the network tests timer expiration setting.
Or, could it be that if the SpamCopURI plug-in finds a whitelisted domain, that it skips any further SURBL tests for that message? That could also account for the very low hit rates when using the whitelist entries.
Bill
Wow, I hope that's not how the SpamCopURI local whitelist works!
Maybe someone can read the source code and check, or Eric can let us know.
Jeff C.
I've just done a little test on that.
The SpamCopURI, only skips the whitelisted domains. If you have one whitelisted domain, and some not in a mail, it does the right thing. Doesn't check the whitelisted ones, but check the others. At least in my version (0.20).
Discuss mailing list Discuss@lists.surbl.org http://lists.surbl.org/mailman/listinfo/discuss
On Wednesday, September 8, 2004, 12:44:34 AM, Bill Landry wrote:
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Looks like senderbase.org has a database of the domains and IPs used to send the most mail. Normally that would not be too interesting to us since we care about message body URIs, i.e. content, and not senders or their ISP addresses, but I'm thinking about whitelisting all the legitimate NSPs, ISPs and telcos in their top domains list:
http://www.senderbase.org/search?page=domains
we would exclude the few that appear to be spammers according to spamhaus:
imgmailer.com TAM Network stocksntalk.com iMedia Networks Inc. havagreatday.com
But I'd like to whitelist all the rest which are obviously large ISPs, etc. In essence we're just using it as a list of some of the top ISPs in the world.
Does anyone have any comments on this?
I like this idea as I believe it would cut down the number of false-positives due to false-listings.
Note that this won't have a major effect on bad guys since spammers would not have much incentive to advertise their ISPs, and we don't "whiten" spams for mentioning non-spam domains anyway. It also does not mean that we're whitelisting the ISP address space, senders, or anything like that, just mail that mentions these large ISP URIs.
Does anyone else have comments about whitelisting the ISP and NSP domains mentioned on the senderbase top domains page?
To repeat, this would only be listing the ISP and NSP's own domains. It does not mean whitelisting their customers, their IP space, their name servers, the mail servers, etc.
Jeff C.
Jeff Chan wrote:
On Wednesday, September 8, 2004, 12:44:34 AM, Bill Landry wrote:
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Looks like senderbase.org has a database of the domains and IPs used to send the most mail. Normally that would not be too interesting to us since we care about message body URIs, i.e. content, and not senders or their ISP addresses, but I'm thinking about whitelisting all the legitimate NSPs, ISPs and telcos in their top domains list:
http://www.senderbase.org/search?page=domains
we would exclude the few that appear to be spammers according to spamhaus:
imgmailer.com TAM Network stocksntalk.com iMedia Networks Inc. havagreatday.com
But I'd like to whitelist all the rest which are obviously large ISPs, etc. In essence we're just using it as a list of some of the top ISPs in the world.
Does anyone have any comments on this?
I like this idea as I believe it would cut down the number of false-positives due to false-listings.
Note that this won't have a major effect on bad guys since spammers would not have much incentive to advertise their ISPs, and we don't "whiten" spams for mentioning non-spam domains anyway. It also does not mean that we're whitelisting the ISP address space, senders, or anything like that, just mail that mentions these large ISP URIs.
Does anyone else have comments about whitelisting the ISP and NSP domains mentioned on the senderbase top domains page?
To repeat, this would only be listing the ISP and NSP's own domains. It does not mean whitelisting their customers, their IP space, their name servers, the mail servers, etc.
Jeff C.
I'd exclude: ----------------- dartmail.net bezeqint.net havagreatday.com ohthatsfunny.com prod-infinitum.com.mx imgmailer.com blueyonder.co.uk webhostplus.com hinet.net -------------------
Pls don't ask me to justify. To me they're either black, dark grey or abused or don't care if.....
Alex
On Thursday, September 9, 2004, 4:27:49 AM, Alex Broens wrote:
I'd exclude:
dartmail.net bezeqint.net havagreatday.com ohthatsfunny.com prod-infinitum.com.mx imgmailer.com blueyonder.co.uk webhostplus.com hinet.net
Pls don't ask me to justify. To me they're either black, dark grey or abused or don't care if.....
Alex
I guess you took a different snapshot of those domains than I did. Of the ones you mention, only:
bezeqint.net blueyonder.co.uk hinet.net prod-infinitum.com.mx
Are on my copy of the list and not already excluded as spammers.
http://spamcheck.freeapp.net/whitelists/senderbase-isps
These remaining ones are all large ISPs. They almost certainly have been abused briefly to send spams. But that's not the question. The question is should we allow their own domain names to be blocked when mentioned in messages?
How many spammers include the URI of their ISP in their spams? Is that a useful thing for them to do? Probably not, but even if they did, would we want to block on those ISP domain names?
Jeff C.
Jeff Chan wrote:
On Thursday, September 9, 2004, 4:27:49 AM, Alex Broens wrote:
I'd exclude:
dartmail.net bezeqint.net havagreatday.com ohthatsfunny.com prod-infinitum.com.mx imgmailer.com blueyonder.co.uk webhostplus.com hinet.net
Pls don't ask me to justify. To me they're either black, dark grey or abused or don't care if.....
Alex
I guess you took a different snapshot of those domains than I did. Of the ones you mention, only:
bezeqint.net blueyonder.co.uk hinet.net prod-infinitum.com.mx
Are on my copy of the list and not already excluded as spammers.
http://spamcheck.freeapp.net/whitelists/senderbase-isps
These remaining ones are all large ISPs. They almost certainly have been abused briefly to send spams. But that's not the question. The question is should we allow their own domain names to be blocked when mentioned in messages?
How many spammers include the URI of their ISP in their spams? Is that a useful thing for them to do? Probably not, but even if they did, would we want to block on those ISP domain names?
I don't see them as "abused briefly" personally I seem them as a plague and as 99% free webmailers indirectly or diretly contribute to the SCAM/SPAM/Trash floods.
If "free" web mail would cost $5/annum, Nigeria would have to search for new export products.
if Open Relays, Zombies, Bogons, Spamhauses etc are a source of spam... IMHO the list aboves rates a new category.
Alex
On Friday, September 10, 2004, 2:40:43 AM, Alex Broens wrote:
Jeff Chan wrote:
bezeqint.net blueyonder.co.uk hinet.net prod-infinitum.com.mx
I don't see them as "abused briefly" personally I seem them as a plague and as 99% free webmailers indirectly or diretly contribute to the SCAM/SPAM/Trash floods.
If "free" web mail would cost $5/annum, Nigeria would have to search for new export products.
if Open Relays, Zombies, Bogons, Spamhauses etc are a source of spam... IMHO the list aboves rates a new category.
That's interesting, but it's not the question. The question is do these ISP' own domain names appear as URIs in spams?
Jeff C.
Jeff Chan wrote:
On Friday, September 10, 2004, 2:40:43 AM, Alex Broens wrote:
Jeff Chan wrote:
bezeqint.net blueyonder.co.uk hinet.net prod-infinitum.com.mx
I don't see them as "abused briefly" personally I seem them as a plague and as 99% free webmailers indirectly or diretly contribute to the SCAM/SPAM/Trash floods.
If "free" web mail would cost $5/annum, Nigeria would have to search for new export products.
if Open Relays, Zombies, Bogons, Spamhauses etc are a source of spam... IMHO the list aboves rates a new category.
That's interesting, but it's not the question. The question is do these ISP' own domain names appear as URIs in spams?
if you define 409's and all kinds of beggar-mail as spam/unsolicited/notrequested, yes... though I don't have any msgs to prove it right now...
Alex
Alex
On Friday, September 10, 2004, 3:09:35 AM, Alex Broens wrote:
Jeff Chan wrote:
The question is do these ISP' own domain names appear as URIs in spams?
if you define 409's and all kinds of beggar-mail as spam/unsolicited/notrequested, yes... though I don't have any msgs to prove it right now...
LOL I asked the wrong question too. The real question is should we block all email that contains those ISPs' URIs?
Should everyone who uses webmail to send a message from hinet.net be blocked? I don't think so.
Jeff C.
Jeff Chan wrote:
On Friday, September 10, 2004, 3:09:35 AM, Alex Broens wrote:
Jeff Chan wrote:
The question is do these ISP' own domain names appear as URIs in spams?
if you define 409's and all kinds of beggar-mail as spam/unsolicited/notrequested, yes... though I don't have any msgs to prove it right now...
LOL I asked the wrong question too. The real question is should we block all email that contains those ISPs' URIs?
Jeff, its not your day... asking the wrong person .-)
Should everyone who uses webmail to send a message from hinet.net be blocked? I don't think so.
dunno...... never seen one in english or any language I can identify, have you? :-)
ok. seriously. I guess we shouldn't...
Alex
Jeff Chan wrote:
These remaining ones are all large ISPs.
AFAIK hinet is the TW counterpart of FR wanadoo, DE t-online, or US SpamCast.
would we want to block on those ISP domain names?
No. I'm not sure, but IIRC hinet even answers complaints, with a minor delay of three months. <g> Bye, Frank
OK, we've whitelisted a snapshot of the large ISPs from the senderbase.org top domains list. Some notes:
1. This is a snapshot of the list of the top senders of mail as determined by sendebase.org's techniques.
2. We have excluded a few obvious spammers and included only large ISPs and NSPs.
3. Whitelisting these ISP domains does not mean their customers, name servers, IP blocks or anything other than the ISPs' own domain names will be excluded from SURBLs.
4. None of these domains appeared on any SURBLs before whitelisting.
5. The sorted list can be found at:
http://spamcheck.freeapp.net/whitelists/senderbase-isps.sort
Please let me know if you have any questions or comments.
Jeff C.
Jeff Chan wrote:
Please let me know if you have any questions or comments.
You could probably add t-online.de - the old system used to be *.bei.t-online.de (e.g. frank.ellermann.bei.t-online.de), the free part of the new system uses *.privat.t-online.de IIRC.
The *.t-dialin.net is less important, it's mainly used in host names for dialin users (*.dip.t-dialin.net) by T-Online. Okay, maybe a desperate spammer runs his own server on a dialin IP, but that IP changes at least once per day.
Bye, Frank
On Sunday, September 12, 2004, 11:23:24 AM, Frank Ellermann wrote:
Jeff Chan wrote:
Please let me know if you have any questions or comments.
You could probably add t-online.de - the old system used to be *.bei.t-online.de (e.g. frank.ellermann.bei.t-online.de), the free part of the new system uses *.privat.t-online.de IIRC.
The *.t-dialin.net is less important, it's mainly used in host names for dialin users (*.dip.t-dialin.net) by T-Online. Okay, maybe a desperate spammer runs his own server on a dialin IP, but that IP changes at least once per day.
We've already whitelisted t-online.de as a major ISP, and we could add t-dialin.net, but please remember that we are not a sender IP list. We are not listing any sender IP addresses, only URI domains. Someone sending mail from a t-dialin address probably is not adding a t-dialin.net URI to their mail (unless there is a t-dialin.net webmail address perhaps?), so I don't see how it's relevant to us.
If someone ran their own mail server on a dialup address, that's essentially irrelevant to SURBLs. We are not a list of sending IP addresses or sender domains. We only look at the message body URIs, not senders.
Jeff C.
Jeff Chan wrote:
I don't see how it's relevant to us.
That's exactly what I wanted to say about *.t-dialin.net You asked for comments and your list under point 5...
http://spamcheck.freeapp.net/whitelists/senderbase-isps.sort
...contains t-dialin.net. Probably irrelevant for SURBL.
Bye, Frank
On Monday, September 13, 2004, 6:11:46 AM, Frank Ellermann wrote:
Jeff Chan wrote:
I don't see how it's relevant to us.
That's exactly what I wanted to say about *.t-dialin.net You asked for comments and your list under point 5...
http://spamcheck.freeapp.net/whitelists/senderbase-isps.sort
...contains t-dialin.net. Probably irrelevant for SURBL.
We want to prevent emails that mention legitimate sites from being blocked. That is the reason for whitelisting t-dialin.net.
I agree that t-dialin.net is unlikely to be mentioned in spams, but we want to protect it if it is mentioned in hams.
Klar? :-)
Jeff C.
Jeff Chan wrote:
[t-dialin.net]
Klar? :-)
Yessir. One unnecessary WL entry is no problem. On a similar issue: From time to time I spot an "IB" identified by SC as spamvertized (the last case was eco.de / msn.de / antispam.de).
Then I cancel the corresponding SC reports, and submit the IB to deputies@sc. Do you want a copy in these cases ? Or maybe only if the IBs somehow already made it on sc.surbl.org ?
Bye, Frank
On Wednesday, September 15, 2004, 2:17:13 PM, Frank Ellermann wrote:
One unnecessary WL entry is no problem.
It really doesn't hurt to whitelist the "good guys". Yes, it's probably unnecessary, but the cost is only a few bytes of storage on the whitelist and a couple milliseconds of part of a join -v against the whitelist. Plus we then know they can't turn into FPs later.
On a similar issue: From time to time I spot an "IB" identified by SC as spamvertized (the last case was eco.de / msn.de / antispam.de).
Then I cancel the corresponding SC reports, and submit the IB to deputies@sc. Do you want a copy in these cases ? Or maybe only if the IBs somehow already made it on sc.surbl.org ?
Yes, please send those to me off list, and I'll add them to the whitelist. I just whitelisted:
eco.de msn.de antispam.de
since they're obvious whitehats.
Jeff C.
On Thu, 9 Sep 2004 04:07:40 -0700, Jeff Chan jeffc@surbl.org wrote:
On Wednesday, September 8, 2004, 12:44:34 AM, Bill Landry wrote:
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org
Looks like senderbase.org has a database of the domains and IPs used to send the most mail. Normally that would not be too interesting to us since we care about message body URIs, i.e. content, and not senders or their ISP addresses, but I'm thinking about whitelisting all the legitimate NSPs, ISPs and telcos in their top domains list:
http://www.senderbase.org/search?page=domains
we would exclude the few that appear to be spammers according to spamhaus:
imgmailer.com TAM Network stocksntalk.com iMedia Networks Inc. havagreatday.com
But I'd like to whitelist all the rest which are obviously large ISPs, etc. In essence we're just using it as a list of some of the top ISPs in the world.
Does anyone have any comments on this?
I like this idea as I believe it would cut down the number of false-positives due to false-listings.
Note that this won't have a major effect on bad guys since spammers would not have much incentive to advertise their ISPs, and we don't "whiten" spams for mentioning non-spam domains anyway. It also does not mean that we're whitelisting the ISP address space, senders, or anything like that, just mail that mentions these large ISP URIs.
Does anyone else have comments about whitelisting the ISP and NSP domains mentioned on the senderbase top domains page?
To repeat, this would only be listing the ISP and NSP's own domains. It does not mean whitelisting their customers, their IP space, their name servers, the mail servers, etc.
I'm fine with this... with the provision that if an ISP proves to be just plain black, it can be removed from the WL...
On Thursday, September 9, 2004, 6:25:31 AM, Mariano Absatz wrote:
On Thu, 9 Sep 2004 04:07:40 -0700, Jeff Chan jeffc@surbl.org wrote:
Does anyone else have comments about whitelisting the ISP and NSP domains mentioned on the senderbase top domains page?
I'm fine with this... with the provision that if an ISP proves to be just plain black, it can be removed from the WL...
Thanks. We would definitely list a mostly bad guy ISP, but these particular large ISPs aren't.
Jeff C.
-
Alex Broens -
Bill Landry -
Frank Ellermann -
Jeff Chan -
Leonardo Helman -
Mariano Absatz