Discuss

discuss@lists.surbl.org

1862 discussions

[SURBL-Discuss] Don't want to give ham lists for creating whitelists
by Daniel Quinlan 01 May '04

01 May '04

(was: Want your ham lists for creating whitelists (Was: Re: [SURBL-Discuss] This ROCKS!)) Jeff Chan <jeffc(a)surbl.org> writes: > Is there any way we can get the message body URI domains (or even raw > messages) from some of the ham lists? This is a really really really really really really bad idea. > The intent is not to "cheat" the S/O scores, but to compile a good, > hand checked whitelist of legitimate message body domains. *I don't > really care where they come from*, but we have this nice, untapped > source of them just sitting there, and a glaring need for some > legitimate domains to keep off the blocklists.... I'm very concerned that if people who plan on submitting their corpus results into the SpamAssassin mass-check process for 3.0 will end up with no ham hits whereas your average user will have some. Cheating is not the word I would use, but it would completely completely throw off our GA process by making it look like SURBL cannot ever issue a false positive, screwing over non-developers. Even if you only get results from people who are never going to submitting corpus results, you're going to end up with a very heavy whitelist bias towards technical users, penalizing non-technical users who will have the rule scored too high compared to what it should be. > I'm results oriented. If there are some good hand-checked ham lists, > I see no practical reason why we should not use them to generate > message body domain whitelists. See above. It would be much better if you came up with other methods for generating a whitelist besides using our benchmark data which is comparatively tiny compared to the world of ham. One other note: just because a URL comes out of ham does *not* mean it should be listed in SURBL. I expect spam URLs to be present in some ham, discussion of spam mostly, so there should always be a small false positive rate (and the rules should be appropriately scored). Daniel -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

2 1

Re: [SURBL-Discuss] This ROCKS!
by Daniel Quinlan 01 May '04

01 May '04

Chris Santerre <csanterre(a)MerchantsOverseas.com> writes: > Hey there guys! This was the crazy idea I was discussing with Doc. I wished > for a realtime form of DB or flat file to be updated continuously on rule > hits. No grepping thru logs or anything. Simply when an email is sent thru > SA, whatever rules hit, increase a counter in a db or flat file for that > rule. Seperate db or flat file for ham and spam. This gives live stats on a > system. No grep'n going on. Just a counter per rule. That might give you a good hit rate, but it won't give you an accurate S/O number. > This is to be used on some advanced rule writing we want to work on. It also > alows an admin to see what might not be worth keeping around. Allowing them > to remove poor performers and increase system speed. Sort of like http://www.pathname.com/~corpus/DETAILS.new ? The corpora have to be sorted by humans to be accurate and runs need to be synchronized so everyone tests the same rules so runs only happen once a day, which is fast enough. We've been doing this for well over a year and it works great. If only we had more active developers working on rules... Daniel -- Daniel Quinlan anti-spam (SpamAssassin), Linux, http://www.pathname.com/~quinlan/ and open source consulting

2 1

This ROCKS!
by Doc Schneider 30 Apr '04

30 Apr '04

Just wanted to say how much this software and spam fighting tool really Rock and rolls them spams into neat little balls and tosses them into never never land. I am on my days off and hope to get some type of stats gathering ditty going. But not sure what stats folks want to see. I had some conversations with Chris "The Big Evil One" and he had some ideas. Am just not sure what direction to pursue. I did join the dev list for SA and sort of got talked into checking the docs for SA 3.0 which I'm slowly working on. Just checking them for proper API coverage in the different modules it is using. So my question to you folks is what sort of statistics would you like to see? Number of rules hit, even if a ham (I always get hungry when talking about ham and spam but I digress) or just total number and the actual rules that are being hit for all spam? And isn't there something already in SA that does these rules and hits? corporra(sic and too lazy to look). Anyway let me know what you all would like and I can whip something up. -- -Doc (The D - Ninja)

3 4

RE: [SURBL-Discuss] This ROCKS!
by Chris Santerre 30 Apr '04

30 Apr '04

>-----Original Message----- >From: jm(a)jmason.org [mailto:jm@jmason.org] >Sent: Friday, April 30, 2004 1:10 PM >To: SURBL Discussion list >Cc: SpamAssassin-dev(a)incubator.apache.org >Subject: Re: [SURBL-Discuss] This ROCKS! > > >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > > >Doc Schneider writes: >>I am on my days off and hope to get some type of stats >gathering ditty >>going. But not sure what stats folks want to see. I had some >>conversations with Chris "The Big Evil One" and he had some ideas. >>Am just not sure what direction to pursue. >>I did join the dev list for SA and sort of got talked into >checking the >>docs for SA 3.0 which I'm slowly working on. Just checking them for >>proper API coverage in the different modules it is using. >> >>So my question to you folks is what sort of statistics would >you like to >>see? Number of rules hit, even if a ham (I always get hungry when >>talking about ham and spam but I digress) or just total >number and the >>actual rules that are being hit for all spam? And isn't there >something >>already in SA that does these rules and hits? corporra(sic >and too lazy >>to look). > >Hi Doc -- cc'ing sa-dev, since it's really an SpamAssassin thing >rather than a SURBL thing ;) > >I'm not sure if you mean measuring rule accuracy in advance to >pick good >scores, or reporting stats after the fact for sysadmins. > >For the first one, read: > > http://wiki.apache.org/spamassassin/MassCheck > http://wiki.apache.org/spamassassin/HitFrequencies > >For the second: > >We recently added some additional stats output to spamd in SpamAssassin >3.0.0. This should improve the accessibility of info about rules >being used during scanning for tools to summarise. > >- --j. Hey there guys! This was the crazy idea I was discussing with Doc. I wished for a realtime form of DB or flat file to be updated continuously on rule hits. No grepping thru logs or anything. Simply when an email is sent thru SA, whatever rules hit, increase a counter in a db or flat file for that rule. Seperate db or flat file for ham and spam. This gives live stats on a system. No grep'n going on. Just a counter per rule. This is to be used on some advanced rule writing we want to work on. It also alows an admin to see what might not be worth keeping around. Allowing them to remove poor performers and increase system speed. HTH Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin Charles Darwin

1 0

query failed
by Grant Baxter 29 Apr '04

29 Apr '04

Just installed spamcop uri .14. I ran spamd in debug mode just to make sure everything is alright. For every email I've seen so far I see a bunch of "query failed yahoo.com.sc.surbl.org; query failed ying.com.sc.surbl.org; query failed usda.gov.sc.surbl.org; query failed yahoogroups.com.sc.surbl.org". However, I received one email that shows a hit on "SPAMCOP_URI_RBL " with a value of 3. Did I do something wrong when I set this up? Or is this normal behavior? TIA, grant

2 1

[SURBL-Discuss] RE: Add "$DATASET dnset @" directive to SURBL rbldnsd zone files?
by Dallas L. Engelken 28 Apr '04

28 Apr '04

> > > >> Can we please add a $DATASET definition to rbldnsd zone > >files for sc,ws, > >> and be? > >> If this gets added, rbldnsd operators would need to change their invocation of rbldnsd from zone:dnset:file to zone:combined:file To run a single zone... Because dnset does not expect to set $DATASET... Or if they want to combine lists into a single zone, use zone:combined:befile,scfile,wsfile how about making a duplicate of each set available with the $DATASET line added... So you have something like this, [root@rhn-1 surbl]# ll total 1040 -rw-r--r-- 1 root root 112368 Apr 28 10:22 be.surbl.org.dataset.rbldnsd -rw-r--r-- 1 root root 112268 Apr 27 06:13 be.surbl.org.rbldnsd -rw-r--r-- 1 root root 13957 Apr 28 10:22 mw.surbl.org.dataset.rbldnsd -rw-r--r-- 1 root root 13857 Apr 24 00:15 mw.surbl.org.rbldnsd -rw-r--r-- 1 root root 185420 Apr 28 10:22 sa.surbl.org.dataset.rbldnsd -rw-r--r-- 1 root root 185320 Apr 27 06:12 sa.surbl.org.rbldnsd -rw-r--r-- 1 root root 9416 Apr 28 10:22 sc.surbl.org.dataset.rbldnsd -rw-r--r-- 1 root root 9316 Apr 27 23:47 sc.surbl.org.rbldnsd -rw-r--r-- 1 root root 185420 Apr 28 10:22 ws.surbl.org.dataset.rbldnsd -rw-r--r-- 1 root root 185320 Apr 27 06:12 ws.surbl.org.rbldnsd That would allow you to combine zones -> uribl.yourdomain.com:combined:be.surbl.org.dataset.rbldnsd,sc.surbl.org. dataset.rbldnsd,ws.surbl.org.dataset.rbldnsd Or still run each individual zone by -> be.yourdomain.com:dnset:be.surbl.org.rbldnsd -> ws.yourdomain.com:dnset:ws.surbl.org.rbldnsd -> sc.yourdomain.com:dnset:sc.surbl.org.rbldnsd And not create any issues, besides having to maintain duplicate zone files. Dallas

1 0

[SURBL-Discuss] RE: Add "$DATASET dnset @" directive to SURBL rbldnsd zone files?
by Chris Santerre 28 Apr '04

28 Apr '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Tuesday, April 27, 2004 10:07 PM >To: SURBL Discuss >Cc: SpamAssassin Users >Subject: RFC: Add "$DATASET dnset @" directive to SURBL rbldnsd zone >files? > > >Dallas Engleken of SARE has suggested that we add a >dataset directive to our rbldnsd zone files: > >> Can we please add a $DATASET definition to rbldnsd zone >files for sc,ws, >> and be? >> >> Ie.. On the 3rd line after $NS and $SOA, add a line labeled, >> >> $DATASET dnset @ >> >> It will not break anything currently set up, but it will >give those of >> use that use the 'combined' type with multi files in rbldnsd >(called via >> uribl.surbl.org:combined:sc,be,ws) to merge ws, sc, and be >together to >> create a single query. > >Does anyone have any comments on this, good, bad or otherwise? >Do other RBLs do it? Is it safe? > >Here's the man page entry: :-) > >> man rbldnsd >> >> combined >> This is a special dataset that stores no data >by itself but acts >> like a container for several other datasets of >any type except >> of combined type itself. The data file >contains an optional >> common section, where various specials are >recognized like $NS, >> $SOA, $TTL (see above), and a series of >sections, each of which >> defines one (nested) dataset and several >subzones of the base >> zone, for which this dataset should be >consulted. New (nested) >> dataset starts with a line >> $DATASET type subzone subzone... >> and all subsequent lines up to the end of >current file or to >> next $DATASET line are interpreted as a part >of dataset of type >> type. Note that combined datasets cannot be >nested. Every subâ >> zone will always be relative to the base zone >name specified on >> command line. If subzone specified as >single character "@", >> dataset will be connected to the base zone itself. >> >> This dataset type aims to simplify subzone >maintenance, in order >> to be able to include several subzones in one >file for easy data >> transfer, atomic operations and to be able >to modify list of >> subzones on remote secondary nameservers. >> >> Note that $NS and $SOA values applies to the >base zone only, >> regardless of the placement in the file. >Unlike the $TTL values >> and $n substitutions, which may be both global >and local for a >> given (subâ)dataset. > >Thumbs up or thumbs down? ;-) > >Jeff C. >-- But we are combining them anyway? Well WS and BE combined. So it would be WS and SC. I see no reason not to try it. --Chris

1 0

Re: second and third level domains - again!
by John Fawcett 28 Apr '04

28 Apr '04

Jeff Chan wrote on Mon Apr 26 03:58:58 CEST 2004 > The underlying principle as I see it is that most > major site will have functional anti-abuse and anti-spam > policies, so either a base domain is good or bad. I > know that seems simplistic, but it's easy and fast to > implement AND it seems to match reality pretty well. There may come a point where a "big" domain starts to appear in spam despite an outward image of "antispam" and the choice which the current infrastructure offers is to block all of it or none of it. Blocking all of it may produce too many FPs whereas blocking a subset may be acceptable. At the moment there haven't been any tough decisions to take on listing or whitelisting. Everything has been very clear cut. It's not guaranteed to stay that way. I don't know how real a risk this is, but I was worried about an infrastructure which effectively ties our hands on this point. (Once surbl is deployed in many different client software, I suppose it will be hard to change anything about the "public interface" to the data....) > That's an interesting idea. Basically you want to > signal redirection to higher domain levels with a > special result for levels that should never get > checked like co.uk. > That might be doable, but it would require extra > logic on the client side as you note. That already > sounds more complex than I like, though I see what > you're getting at. Better to control what goes into > the data (i.e. never let the TLD itself co.uk in), > and make sure the client is following similar rules. What I was proposing is a *change* to client processing logic, but not added *complexity*. rather a simplification. What I mean is that currently the client has to contain - processing logic - data on the ccTLDs (in the long run there will be multiple versions of the ccTLD data implemented in various client versions at any one time. Users will have to upgrade the client software to keep current with domain data). Using an A record in surbl to indicate "this domain is not listed but a subdomain is. Try again" would mean the client just follows a simple processing rule. It doesn't need to know anything about specific domain data. The logic would be something like this. lets assume that "url" is found in an email. level=2 more_info_result = '127.0.0.255' listed_result = '127.0.0.2' do { result = query_dns ( extract_domain (url, level)) level = level +1 }while (result == more_info_result) if (result == listed_result) { score it } > We will always catch bigspammer.co.uk with the current > strategy. I think that some of the ccTLDs have a mixed assignment strategy. This means that they should sometimes be checked at the 2nd level sometime at the 3rd level. The current logic always checks at a single predefined level. The two example I saw were: .fr and .ca Currently we check .ca at the third level, but it is possible to register a second level domain at .ca which we never catch so bigspammer.ca will get through. We check .fr at the second level however there are many "standard" second level domains (like .nom.fr) which means we probably want to be checking these ones at the third level. (Translation: any bigspammer.nom.fr domain is imune to the current strategy unless we want to upset everyone who has a nom.fr domain by listing that). > I know a lot of what I argue for above seems simplistic > when a more complex solution could have more interesting > results, but very often the simpler solutions are better, > especially in terms of resource consumption. You're right to argue for the most simple solution. (In fact there are probably simpler solutions than the one I suggested!). My concern was to avoid inflexibility in the infrastructure and automatic immunity for various classes of domains. John

4 5

access.db -> Bind zone file converter
by David Hooton 28 Apr '04

28 Apr '04

Hi All, We're wanting to move a bunch of our rather huge sendmail access files into bind compliant zone files. Does anyone have a tool that can do this in a slightly more automated way than our current options? We're also looking for a tool to convert a domain list into our own surbl zone (for the moment private, but we may release it later) again any pointers greatly appreciated :-) Regards, David Hooton Senior Partner Platform Networks www.platformnetworks.net ======================================================================== Pain free spam & virus protection by: www.mailsecurity.net.au Forward undetected SPAM to: spam(a)mailsecurity.net.au ========================================================================

4 6

[SURBL-Discuss] RFC: Add "$DATASET dnset @" directive to SURBL rbldnsd zone files?
by Jeff Chan 28 Apr '04

28 Apr '04

Dallas Engleken of SARE has suggested that we add a dataset directive to our rbldnsd zone files: > Can we please add a $DATASET definition to rbldnsd zone files for sc,ws, > and be? > > Ie.. On the 3rd line after $NS and $SOA, add a line labeled, > > $DATASET dnset @ > > It will not break anything currently set up, but it will give those of > use that use the 'combined' type with multi files in rbldnsd (called via > uribl.surbl.org:combined:sc,be,ws) to merge ws, sc, and be together to > create a single query. Does anyone have any comments on this, good, bad or otherwise? Do other RBLs do it? Is it safe? Here's the man page entry: :-) > man rbldnsd > > combined > This is a special dataset that stores no data by itself but acts > like a container for several other datasets of any type except > of combined type itself. The data file contains an optional > common section, where various specials are recognized like $NS, > $SOA, $TTL (see above), and a series of sections, each of which > defines one (nested) dataset and several subzones of the base > zone, for which this dataset should be consulted. New (nested) > dataset starts with a line > $DATASET type subzone subzone... > and all subsequent lines up to the end of current file or to > next $DATASET line are interpreted as a part of dataset of type > type. Note that combined datasets cannot be nested. Every subâ > zone will always be relative to the base zone name specified on > command line. If subzone specified as single character "@", > dataset will be connected to the base zone itself. > > This dataset type aims to simplify subzone maintenance, in order > to be able to include several subzones in one file for easy data > transfer, atomic operations and to be able to modify list of > subzones on remote secondary nameservers. > > Note that $NS and $SOA values applies to the base zone only, > regardless of the placement in the file. Unlike the $TTL values > and $n substitutions, which may be both global and local for a > given (subâ)dataset. Thumbs up or thumbs down? ;-) Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

3 6

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss