Discuss September 2004

discuss@lists.surbl.org

62 participants
229 discussions

Re: [SURBL-Announce] ANNOUNCE: Adding new JP list to multi.surbl.org
by Mariano Absatz 24 Sep '04

24 Sep '04

On Thu, 23 Sep 2004 00:58:42 -0700, Jeff Chan <jeffc(a)surbl.org> wrote: > [Please post follow ups to the SURBL discuss list or to me.] > > One of the distinct data sources currently feeding into > ws.surbl.org includes data from Joe Wein and Raymond Dijkxhoorn > with his colleagues at Prolocation. Raymond and Prolocation > are currently processing more than 300,000 potential spams per > day using Joe's jwSpamSpy server software and combining those > with Joe's own results. In addition to the data processing > software, Joe has an elaborate, thorough, and well-thought-out > set of inclusion criteria which includes age of domain > registration, manual checks, and other factors. The resulting > data are an extensive list of spam URI domains with a very > low false positive rate (hits on legitimate messages). We > are calling this resulting data JP for Joe Wein + Prolocation. > > The bottom line is that JP (called PJ in the table below) has a > significantly lower false positive rate than WS while having > similar spam detection rates, for example as measured against a > large corpora set belonging to Theo Van Dinter of SpamAssassin: > > OVERALL% SPAM% HAM% S/O RANK SCORE NAME > 2424443 2357143 67300 0.972 0.00 0.00 (all messages) > 100.000 97.2241 2.7759 0.972 0.00 0.00 (all messages as %) > 7.595 7.8122 0.0045 0.999 1.00 0.00 URIBL_SC_SURBL > 76.754 78.9448 0.0178 1.000 0.80 0.00 URIBL_OB_SURBL > 77.230 79.4340 0.0208 1.000 0.60 1.00 URIBL_PJ_SURBL > 0.985 1.0126 0.0045 0.996 0.50 0.00 URIBL_AB_SURBL > 82.119 84.4600 0.1367 0.998 0.40 0.00 URIBL_WS_SURBL > 0.021 0.0216 0.0045 0.829 0.00 0.00 URIBL_PH_SURBL > > So we feel the data could usefully be broken out into a > separate list which could safely be scored higher than > WS. We also continue to work on improving the False Positive > rate of WS of course. We propose making JP a separate list > within multi.surbl.org, but *not* a standalone list like > jp.surbl.org, since it's a major effort to set up entirely > new lists and most people should be using multi now. > > The main reason for announcing this change ahead of time > is to allow developers of the many programs (in addition to > SpamAssassin) now using SURBL data to update their code or > configurations to take into account that the result codes in > multi will be changing as a result of adding JP. JP would get > the 64 bitmask, as in: > > 2 = comes from sc.surbl.org > 4 = comes from ws.surbl.org > 8 = comes from phishing list (labelled as [ph] in multi) > 16 = comes from ob.surbl.org > 32 = comes from ab.surbl.org > 64 = comes from jp list > > So a record in SC, WS, and JP would give a value 127.0.0.70. > One with WS, OB, and JP would resolve to 127.0.0.84, etc. > Programs using multi.surbl.org should be updated accordingly. > > Since JP is currently included in WS, there will be 100% > overlap of JP entries in WS so that any record in JP will > also be in WS. In other words about half of the WS records > in multi will increase by 64 due to overlap with JP. But > WS will continue to use the 4 bit, as before. If your > programs are decoding the multi results using the bit > positions, they should need no adjustments to continue to > handle the WS data. > > We hope that 5 days is not too short notice for this kind of > change.... I will try to contact the developers of the various > (non-SA) programs separately to make sure they're aware of the > coming change. Hopefully most of them are on this announcement > list however. > > We were not able to get JP as a separate list in yesterday's > SpamAssassin 3.0.0 full release, but we have gotten it into > SA 3.1 development. > > For now the JP data will continue to be included in WS, > but just before Spam Assassin 3.1 gets released (probably in > 6 months to a year from now), we will remove JP data from WS > to make them separate lists within multi. This means that > SpamAssassin 3.0 and other current users of WS will continue > to to get the benefits of JP under their default shipping > configurations, and that JP can also be used separately by > those who modify their configurations to take advantage of it. > > In summary, we will: > > 1. Add JP to multi.surbl.org on Monday September 27th. > (Note that like PH, JP would not be available as a separate > list, only as part of multi.) > > 2. Keep the JP data in WS for now, so that regular 3.0 users > get the advantages of JP also (as part of WS). > > 3. Ask the SpamAssassin developers to score JP separately in > SA 3.1. > > 4. Remove JP from WS before the final SA 3.1 mass check and > re-scoring is done, to make the two lists more separate > for 3.1 . (Note that the separation is removal of the > specific subset arrangement suggested in #2. If that is > done, there will still be some minor overlap of the records > in WS and JP.) > > 5. Inform people about removing JP from WS before we do it, > so existing WS users can add JP, etc. > > Please post follow up questions or comments to the SURBL discuss > list or to me personally. It looks to me as a sensible way to handle this... I followed your advise about SURBL scoring in a thread a few weeks ago (I think Theo or another ninja also participated), but WS has a somehow low score... I didn't rise it, 'cause I have a setup that is very FP-sensitive (large ISPs), but would love to see high-quality, high-scoring multi sublists... Do you have a current reasonable scoring for jp? (that is, considering that, for now, this score will be added to the ws score, since, until SA3.1, jp will be a subset of ws. Thanx. -- Mariano Absatz - El Baby el (dot) baby (AT) gmail (dot) com el (punto) baby (ARROBA:@) gmail (punto) com

2 1

RE: [SURBL-Discuss] RFC: SURBL inclusion policy
by Chris Santerre 24 Sep '04

24 Sep '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Friday, September 24, 2004 7:40 AM >To: SURBL Discuss >Subject: [SURBL-Discuss] RFC: SURBL inclusion policy > > >In order to assist people hand-classifying spam URI domains and >IPs for inclusion or non-inclusion in SURBLs, I've made a draft >policy document: > > http://www.surbl.org/policy.html > >Please read it and post your comments. Good work. No footnote for 90% spammy, then send to UC. :) --Chris

1 0

SURBL independent daemon, pluggable into amavis
by Yves Junqueira 24 Sep '04

24 Sep '04

Hi, for those that don't use SpamAssassin and would like to make SURBL checks on their mail, I've adapted Devin Carraway's plugin to qpsmtp, making a daemon that is independent, forking itself as needed. I believe it performs well, even on high traffic mail. Its simple usage interface, that was made with amavisd antivirus code in mind, is possibly usable by other services. Whenever a mail comes, one should connect to it and order it to scan the dir or file that message is on. # telnet 127.0.0.1 20098 SCAN /tmp/ Scanning /tmp/ Checking /tmp/filex8QBXY Checking /tmp/ldap18775 OK. OK Connection closed by foreign host. # # echo 'http://123getnow.com' > /tmp/a # telnet 127.0.0.1 20098 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. SCAN /tmp/ Scanning /tmp/ Checking /tmp/a SPAMMEDURL: 123getnow.com in surblgoiano.com.br (SPAMMER 123getnow.com) Action: deny DENY Connection closed by foreign host. # Just check for "DENY" or "OK" and you're done. In this example, "surblgoiano.com.br" is my local RBL zone for bad URIs. It's easy to plug it into amavis: /etc/amavisd.conf: @av_scanners : (...) ['Suri', \&ask_daemon, ["SCAN {}/../email.txt\n", '127.0.0.1:20098'], qr/^OK/, qr/^DENY/, qr/^SPAMMEDURL:.*[(](.+)[)]/], (...) The important part there is: SCAN {}/../email.txt As you can see, there is a problem here. In this case, all messages with a spamvertised URI will be considered a virus, and actions will be taken aproprietely as per amavis config. In this case, I'd recommend using initially a non-deny "$action" setting, for seeing any potential problems in the logs (syslog) before really applying this "dangerous" policy. In any case, it is very recommended too that you quarantine these "infected" messages. This is a perl beginner's work, so please excuse any mistakes. It was made to fit my needs, as I maintain my own suribl zone, which is small, yet made for as little false positives as possible and with a special attention to brazilian domains. I haven't tested it with SURBL.org zones, yet. I am also making a transparent SMTP proxy for using with Postfix in servers with low traffic and/or running a local RBL daemon (rbldnsd). It would prevent losing e-mail. Any potential false positive would be notified by the SMTP reject code after the end of the DATA command block. Do you think it could be useful? You can get this plugin, that I called SURI, at http://i-admin.blogspot.com . I'd appreciate any comments very much. It's my first free software ever relased :-P -- Yves Junqueira nictuku - irc.freenode.net http://www.lynx.com.br

3 3

Whitelist data: Alexa.com top 500
by Joe Wein 24 Sep '04

24 Sep '04

Alexa by Amazon.com has a top 500 list on its site, which it derives from stats collected via its Alexa toolbar plugin. This may be a good source of whitelist data. Any site making that high score has the potential to cause a lot of collateral damage if blacklisted, since these appear to be sites that lots of real-life users *do* to visit regularly, as opposed to sites that advertisers suggest they visit, so they are likely to be mentioned in legitimate personal or business e-mail. Probably sites popular enough to be there have far more to lose than to gain from spamming anyway. I took the HTML from Alexa's five pages which listed 100 sites each, did a bit of text editing and hey presto: here's the list as an attached ASCII file. A quick check against my local blacklist yielded exactly 0 intersections :-) The following entries appeared in suspicious mail or as sender addresses and had been investigated by my filter (WHOIS lookup, etc.), but were not classified as spam domains: 163.net 39.net 888.com 8u8.com chosun.com ctinets.com dreamwiz.com eastday.com enet.com.cn etang.com freeservers.com globo.com km169.net linksynergy.com marktplaats.nl mingpao.com mingpaonews.com mym.net mypcera.com nastydollars.com nate.com naver.com nifty.com no-ip.com opendiary.com rambler.ru sayclub.com trafficmp.com xaonline.com yesky.com About a third of the top 500 sites (160) were already in my local whitelist. I'll probably add the rest to my whitelist too. Anybody here who can bulk-check these against SURBL, in case there are listed sites? Joe -- http://www.joewein.de/sw/jwSpamSpy/

3 2

Proposal for moving forward with JP list
by Jeff Chan 24 Sep '04

24 Sep '04

OK we heard back from Theo that we probably won't be able to get JP into SpamAssassin 3.0, but we should be able to get it into 3.1. I believe the JP data and policies are different enough that it should be a separate list within multi.surbl.org, so I propose that we: 1. Add JP to multi.surbl.org now. 2. Keep the JP data in WS for now, so that regular 3.0 users get the advantages of JP also (as part of WS). 3. Ask SA to put JP into 3.1 for future use, and most significantly, separate scoring. 4. Remove JP from WS before the final 3.1 mass check and re-scoring is done, to make the two lists more separate for 3.1 . (Note that the separation is removal of the specific subset arrangement suggested in #2. If that is done, there will still be some overlap of the records in WS and JP.) 5. Inform people about removing JP from WS before we do it, so existing WS users can add JP, etc. How does this sound? Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

4 8

Web submissions working again.
by Chris Santerre 23 Sep '04

23 Sep '04

THe last few days no web submissions have come in due to a security fix that was needed. It is all back up and running now. Ninja D. informs me we can have submissions go thru a list of email addresses to send to. Meaning we can round robin submissions to other SURBL commiters. Which means instead of me getting a bunch, a few people will get a few of them a day. Any takers on getting a few web submissions? Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com http://www.surbl.org 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin

5 8

RE: [SURBL-Discuss] Web submissions working again.
by Chris Santerre 23 Sep '04

23 Sep '04

>-----Original Message----- >From: Doc Schneider [mailto:maddoc@maddoc.net] >Sent: Thursday, September 23, 2004 3:30 PM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] Web submissions working again. > > > >Raymond Dijkxhoorn wrote: >> Hi! >> >>>> was needed. It is all back up and running now. Ninja D. >informs me we >>>> can have submissions go thru a list of email addresses to send to. >>>> Meaning we can round robin submissions to other SURBL commiters. >>>> Which means instead of me getting a bunch, a few people will get a >>>> few of them a day. Any takers on getting a few web submissions? >> >> >>> Sure send me a few as well. 8*)) >> >> >> Got a few allready, round robin works nice i guess ;) >> > >Of course they're probably going to end up in my spam folder! HAR! > >Maybe Chris can give a hint what address they'll be coming from????? 1600 Pennsylvania Avenue :) --Chris

2 1

Request for Whitelist
by Bitz 23 Sep '04

23 Sep '04

epromos . com listed at ws.surbl.org We have customers who conduct business with these folks, I googled and didn't find any spam related proof against them. Thanks, -b

2 2

JPEG flaw in Windows - URLs in emails
by Matthew Wilson 23 Sep '04

23 Sep '04

Since proof-of-concept code for the JPEG flaw in Windows has been posted online, we can surely expect at least one mass mailing exploit soon. The form will likely take the form of either: 1) A JPEG file embedded in an email message with the exploit code embedded in the embedded image. Theoretically, the exploit pattern should already be known, no matter what the encoding is, so anti-virus companies should theoretically be able to detect this already, if this method is used. 2) Because of the above, the more likely method seems to be the embedding of a URL in the message that either refers to the actual JPEG itself or refers to a webpage that loads the infected JPEG. It seems then that the only tool that could detect worms of this sort would be SURBL. And so on to my question: if I (or anyone else for that matter) submit a domain name that hosts an infected JPEG file, how quickly will the SURBL databases be updated to reflect this infection? Also, what if the exploit is multi-stage, and tries to infect actual http servers with infected JPEGs, and thousands of websites become infected...? Would it then be necessary to create a separate SURBL list for these infected domains, or could they be listed in, say, the phishing list? Thanks, Matthew Wilson

5 4

RE: [SURBL-Discuss] JPEG flaw in Windows - URLs in emails
by Chris Santerre 22 Sep '04

22 Sep '04

>-----Original Message----- >From: Matthew Wilson [mailto:matthew@boomer.com] >Sent: Wednesday, September 22, 2004 11:33 AM >To: SURBL Discussion list >Subject: [SURBL-Discuss] JPEG flaw in Windows - URLs in emails > > >Since proof-of-concept code for the JPEG flaw in Windows has >been posted >online, we can surely expect at least one mass mailing exploit soon. >The form will likely take the form of either: > >1) A JPEG file embedded in an email message with the exploit code >embedded in the embedded image. Theoretically, the exploit pattern >should already be known, no matter what the encoding is, so anti-virus >companies should theoretically be able to detect this already, if this >method is used. > >2) Because of the above, the more likely method seems to be the >embedding of a URL in the message that either refers to the actual JPEG >itself or refers to a webpage that loads the infected JPEG. It seems >then that the only tool that could detect worms of this sort would be >SURBL. > >And so on to my question: if I (or anyone else for that >matter) submit a >domain name that hosts an infected JPEG file, how quickly will >the SURBL >databases be updated to reflect this infection? > >Also, what if the exploit is multi-stage, and tries to infect actual >http servers with infected JPEGs, and thousands of websites become >infected...? Would it then be necessary to create a separate >SURBL list >for these infected domains, or could they be listed in, say, the >phishing list? > >Thanks, >Matthew Wilson > Well here are some random thoughts in no order: 1) I'm not getting web submissions right now. Something is FUBAR there. 2) Perhaps we can work something out with a third party to maintain virus type links like this? I'll email Jeff Off List as I have some other questions there. 3) Ehhh....I think Jeff is going to say something along the lines of "SURBL wasn't intended to be used like that." To which I agree, yet sadly SURBL is the tool to catch these. --Chris

2 1

← Newer
1
...
4
5
6
7
8
9
10
...
23
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss September 2004