Discuss May 2004

discuss@lists.surbl.org

44 participants
91 discussions

RE: [SURBL-Discuss] This ROCKS!
by Chris Santerre 03 May '04

03 May '04

>-----Original Message----- >From: Daniel Quinlan [mailto:quinlan@pathname.com] >Sent: Friday, April 30, 2004 9:13 PM >To: Chris Santerre >Cc: 'SURBL Discussion list'; SpamAssassin-dev(a)incubator.apache.org >Subject: Re: [SURBL-Discuss] This ROCKS! > > >Chris Santerre <csanterre(a)MerchantsOverseas.com> writes: > >> Hey there guys! This was the crazy idea I was discussing >with Doc. I wished >> for a realtime form of DB or flat file to be updated >continuously on rule >> hits. No grepping thru logs or anything. Simply when an >email is sent thru >> SA, whatever rules hit, increase a counter in a db or flat >file for that >> rule. Seperate db or flat file for ham and spam. This gives >live stats on a >> system. No grep'n going on. Just a counter per rule. > >That might give you a good hit rate, but it won't give you an accurate >S/O number. Well if it counts spam and ham hits, then an S/O is but a division away ;) > >> This is to be used on some advanced rule writing we want to >work on. It also >> alows an admin to see what might not be worth keeping >around. Allowing them >> to remove poor performers and increase system speed. > >Sort of like http://www.pathname.com/~corpus/DETAILS.new ? Sort of. I didn't know you guys did that nightly. Very nice. I'm looking for a more localized process that doesn't require a run of anything. Having counters generated means just checking totals. And even if one didn't use the exact same rules as another, they could easily combine totals for the ones they do. However your method is more accurate. > >The corpora have to be sorted by humans to be accurate and runs need to >be synchronized so everyone tests the same rules so runs only happen >once a day, which is fast enough. > >We've been doing this for well over a year and it works great. If only >we had more active developers working on rules... I'm not quite sure how to take that last line. Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin Charles Darwin

2 1

[SURBL-Discuss] Sendmail milter and SURBLs? sbl.spamhaus.org and SpamCopURI or urirhsbl?
by Jeff Chan 03 May '04

03 May '04

Saw a link on the Spamhaus site to a sendmail milter: http://www.five-ten-sg.com/dnsbl.html "This milter will also decode (base64, mime, html entity) and scan for HTTP and HTTPS URLs and bare hostnames in the body of the mail. If any of those host names have A records on the SBL (or a single configurable list), the mail will be rejected unless previously whitelisted." Seems to me that should be just as usable with SURBLs as the more limited spam body domain names in sbl.spamhaus.org. Has anyone tried this milter with SURBLs? Conversely has anyone tried sbl.spamhaus.org as a SURBL in SpamCopURI or urirhsbl? Either could be interesting.... Jeff C. Additionally: Installation and configuration Usage: Note that this has ONLY been tested on Linux, specifically RedHat Linux. In particular, this milter makes no attempt to understand IPv6. Your mileage will vary. You will need at a minimum a C++ compiler with a minimally thread safe STL implementation. The distribution includes a test.cpp program. If it fails this milter won't work. If it passes, this milter might work. Fetch dnsbl.tar.gz and tar xfvz dnsbl.tar.gz bash install.bash

2 2

Spamcop conflict of interest ?
by Simon Byrnand 02 May '04

02 May '04

Hmm....just noticed these comments in a recent slashdot article. Being slashdot it's hard to know whether to take it seriously or not ;-) http://yro.slashdot.org/comments.pl?sid=105978&cid=9022567 Regards, Simon

5 8

Re: better TLD handling
by Jeff Chan 02 May '04

02 May '04

On Friday, April 30, 2004, 7:12:10 PM, Justin Mason wrote: > ... is now in SpamAssassin SVN trunk. it uses quite a lot of data > from the SURBL list of TLDs and their subdomains, btw, thanks! ;) Hehe, I assumed *you* guys had the ccTLD magic bullet. ;-) Will check it out! (I hate to ask but is there any kind of direct URL access into SVN so I don't need to install the entire structure?) An open request to everyone: if you know of or can find any registrar lists of ccTLDs that are not already in our list: http://spamcheck.freeapp.net/two-level-tlds then please forward them to me for addition. I think we're lacking places like Deutschland .de and Italia .it, and probably some smaller ones also. N.B. found at http://www.denic.de/de/domains/index.html and added: moebel.de glueckwunsch.de buecher.de boerse.de kueche.de buero.de fluege.de kuechen.de aerzte.de reisebuero.de Surely there are many more. (Those look strange without Umlauten... :-) This is for country code TLDs which should themselves be whitelisted. Doing so allows us to check the third levels after them. Therefore we want: category.country but not domain.country where an abstract category is like doctors, a city name, etc. where a potential spammer would register at a lower level like spamming.doctor.be or spamming-subdomain.london.uk . We don't want to whitelist spammer.uk so please don't include registered base domains that terminate at the second level. TIA for any help with this, Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

2 2

Problems with uri/_foreign
by Matt Yackley 01 May '04

01 May '04

Hi all, I'm hoping that someone can whack me with a clue-stick here. I'm running into problems getting SURBL up and running. System: RedHat 9 SA 2.63 Amavisd-new SpamCopURI 0.14 I downloaded and ran through the install of the SpamCopURI 0.14 package, put my rules in place, and restarted my services. Here is the error that I receive, each time a messae passes through the system: Failed to run SPAMCOP_URI_RBL SpamAssassin test, skipping: (Can't locate URI/_foreign.pm in @INC (@INC contains: lib /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl) at /usr/lib/perl5/site_perl/5.8.0/URI.pm line 56. ) 1. I re-copied the three .pm files from the source, like the install docs advised. 2. The _foreign.pm file does exist at /usr/lib/perl5/site_perl/5.8.0/uri/_foreign.pm 3. CPAN claims that URI is up to date 4. A quick google search and search of the SURBL archives don't seem to offer much info on this error I'm sure that I'm missing something silly, so any pointers would be great. Thanks, Matt

2 4

Good news/bad news on .us ccTLDs
by Jeff Chan 01 May '04

01 May '04

1. Found a list of .us reserved domains at: http://www.nic.us/registrars/fcfs/dotus_reservedlist_v3.zip 2. The list has 50k domains at 1 MB of text uncompressed, so it will probably be somewhat impractical to use on the client side. 3. It only needs to be processed on the data side to make sure none of the .us ccTLDs gets into the SRUBLs, and that's now in place. 4. There are very many more reserved domains as described at: http://www.nic.us/faqs/index.html#list_reserved_names as: > Where can I see a list of reserved .US names? > See http://www.neustar.us/registrars/. In addition to the > specific names contained in this reserved name file, the > following categories of names are also reserved: > > * All numbers five digits and higher > * All numbers in the format five digits-four digits (zip codes) > * All telephone numbers including toll-free numbers > * Tagged domain namesall labels with hyphens in the third > and fourth character positions (e.g. "bq--kn2n4h4b") 5. It's very unlikely that any of these geographic or reserved domains will become a major source of spam hosting. I don't envision city or country registrars handing subdomains under reserved domains to random spammers. I had a heck of a time trying to get the late, lamented Jon Postel to let the City of Mountain View change theirs from mtview or mtnview to mountainview.... I know things are different now, but I still doubt reserved subdomains are going to be handed out like Mardi Gras candy or anything like that. :-) 6. Note that none of this prevents spammerdomain.us from getting onto the blocklists. That will still happen as it should. Jeff C.

1 0

Re: better TLD handling
by Daniel Quinlan 01 May '04

01 May '04

It's lib/Mail/SpamAssassin/Util/RegistrarBoundaries.pm -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

1 0

[SURBL-Discuss] Re: Don't want to give ham lists for creating whitelists
by Daniel Quinlan 01 May '04

01 May '04

For what it's worth, most of the URLs hitting SURBL in my ham are spam URLs anyway. On Saturday, 140 messages hit BE or WS SURBL out of 14971 total ham in my corpus. Retesting those 140 messages now, only 105 messages hit, so you've probably removed some domains since Friday. Hopefully, because they're not spamming domains. In those 105 messages, there were about 123 hits (40 when looking for unique domain-rule pairs) on 34 different domains as follows: count domain rule 80 xxxxx.xxx URIBL_WS_SURBL 3 xxxxxxxxxx.xxx URIBL_WS_SURBL 2 xxxxxxxxx.xxx URIBL_WS_SURBL 2 xxxxxxxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxx.xxx URIBL_BE_SURBL 1 xxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxx.xxx URIBL_BE_SURBL 1 xxxxxxxxxxxxxxx.xxxx URIBL_WS_SURBL 1 xxxxxxxxxxxxxxx.xxxx URIBL_BE_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxx.xxx URIBL_BE_SURBL 1 xxxxxxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxxxxxxxx.xxx URIBL_BE_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxx.xxx URIBL_WS_SURBL 1 xxxx.xxx URIBL_BE_SURBL 1 xxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxx.xxx URIBL_BE_SURBL 1 xxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 x-xxxxxx.xxx URIBL_BE_SURBL 1 xxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xxxxxxxxxxx.xxx URIBL_WS_SURBL 1 xx-xxxxxx-xxxxxxxxxx.xxx URIBL_WS_SURBL Most of them look like spam domains. -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

2 1

[SURBL-Discuss] Re: Don't want to give ham lists for creating whitelists
by Daniel Quinlan 01 May '04

01 May '04

Jeff Chan <jeffc(a)surbl.org> writes: > Your arguments make perfect sense in terms of SA development > and use, but hopefully my reasons for asking also make some > sense. Not really once you consider than most of the listed URLs in our ham are actually spam domains being discussed in the context of anti-spam efforts. Looking at URLs in the body is never going to have a perfect S/O ratio -- even non-developers discuss spam, forward funny spam, innocently link to interesting content on primarily spamvertized sites, etc. > I guess it's time to stop asking. :-( There are plenty of ways for you to get whitelist URLs without relying on our corpora. -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

2 1

[SURBL-Discuss] Don't want to give ham lists for creating whitelists
by Daniel Quinlan 01 May '04

01 May '04

(was: Want your ham lists for creating whitelists (Was: Re: [SURBL-Discuss] This ROCKS!)) Jeff Chan <jeffc(a)surbl.org> writes: > Is there any way we can get the message body URI domains (or even raw > messages) from some of the ham lists? This is a really really really really really really bad idea. > The intent is not to "cheat" the S/O scores, but to compile a good, > hand checked whitelist of legitimate message body domains. *I don't > really care where they come from*, but we have this nice, untapped > source of them just sitting there, and a glaring need for some > legitimate domains to keep off the blocklists.... I'm very concerned that if people who plan on submitting their corpus results into the SpamAssassin mass-check process for 3.0 will end up with no ham hits whereas your average user will have some. Cheating is not the word I would use, but it would completely completely throw off our GA process by making it look like SURBL cannot ever issue a false positive, screwing over non-developers. Even if you only get results from people who are never going to submitting corpus results, you're going to end up with a very heavy whitelist bias towards technical users, penalizing non-technical users who will have the rule scored too high compared to what it should be. > I'm results oriented. If there are some good hand-checked ham lists, > I see no practical reason why we should not use them to generate > message body domain whitelists. See above. It would be much better if you came up with other methods for generating a whitelist besides using our benchmark data which is comparatively tiny compared to the world of ham. One other note: just because a URL comes out of ham does *not* mean it should be listed in SURBL. I expect spam URLs to be present in some ham, discussion of spam mostly, so there should always be a small false positive rate (and the rules should be appropriately scored). Daniel -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

2 1

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss May 2004