Discuss

discuss@lists.surbl.org

1876 discussions

RE: [SURBL-Discuss] mbox parser
by Chris Santerre 19 May '04

19 May '04

>-----Original Message----- >From: Ted Deppner [mailto:tdeppner@surewest.net] >Sent: Tuesday, May 18, 2004 4:53 PM >To: discuss(a)lists.surbl.org >Subject: Re: [SURBL-Discuss] mbox parser > > >On Tue, May 18, 2004 at 10:58:15AM -0400, Chris Santerre wrote: >> For what it is worth, I have abandoned all script >extractions of URLs. It is >> NOT reliable. The human eye is better, and almost as fast. >Simply look at >> the email source and search for every instance of HTTP. >Then, using your >> firing synapse and some of the jedi force powers, decide if >what you are >> looking at is a legit link. Then report it. :-) >> >> Then a team of squirrels on crack will take that submission >and run it >> through numerous test of validity. Then it will be listed. >> >> To go forwards, we had to go backwards. > >Ugh. You'll miss %xx encodings, Mime messages, and so forth. I scan using outlook (don't ask!). So everything is decoded. >No way you >can analyze 45,000 messages in any reasonable amount of time. Thanks to RBLs, and spammers realising they shouldn't send my company email or they get listed to bigevil, my spam numbers are going down. Also I've found that for every domain I add, there are 10 more spam with the same one that can be deleted straight off. So it goes pretty fast, when I get my lazy butt in gear to do it ;) > >We collect bounce messages and analyze those in aggregate. Very simple >and high volume stuff floats to the top. You are correct on that. I did have some great success with the scripts. Mining the domains and working by hand might be ok, but I like to see the whole email. So while time consuming, this is the best method for me to get close to zero FPs. Goes back to the old saying: "Faster, better, and cheaper. Pick only 2." --Chris

1 0

Re: ANNOUNCE: Mail::SpamAsssassin::SpamCopURI 0.16
by Jeff Chan 19 May '04

19 May '04

[moving announcement reply to discussion list] Subject: RE: ANNOUNCE: Mail::SpamAsssassin::SpamCopURI 0.16 Date: Mon, 17 May 2004 13:18:07 +0200 From: <Rikhardur.EGILSSON(a)oecd.org> To: <jeffc(a)surbl.org>, <announce(a)lists.surbl.org> Cc: <spamassassin-users(a)incubator.apache.org> For those of you who have to use a http_proxy, you should add this patch : --- lib/Mail/SpamAssassin/SpamCopURI.pm.orig 2004-05-17 13:13:18.541107920 +0200 +++ lib/Mail/SpamAssassin/SpamCopURI.pm 2004-05-17 13:13:34.878624240 +0200 @@ -45,6 +45,7 @@ # do this since we never want the content $ua->max_size(0); $ua->timeout($LWP_TIMEOUT); + $ua->env_proxy; dbg("redirect resolving: $url"); Or else, it won't work .. - Ríkharður -----Original Message----- From: Jeff Chan [mailto:jeffc@surbl.org] Sent: 15 May, 2004 11:54 AM To: SURBL Announce Cc: SpamAssassin Users Subject: ANNOUNCE: Mail::SpamAsssassin::SpamCopURI 0.16 Eric Kolve has released SpamCopURI version 0.16 which fixes the handling of the few URIs which use numeric IP addresses. http://sourceforge.net/projects/spamcopuri If you're using SpamCopURI, please upgrade to this new version.

1 0

RE: [SURBL-Discuss] mbox parser
by Chris Santerre 18 May '04

18 May '04

For what it is worth, I have abandoned all script extractions of URLs. It is NOT reliable. The human eye is better, and almost as fast. Simply look at the email source and search for every instance of HTTP. Then, using your firing synapse and some of the jedi force powers, decide if what you are looking at is a legit link. Then report it. :-) Then a team of squirrels on crack will take that submission and run it through numerous test of validity. Then it will be listed. To go forwards, we had to go backwards. --Chris >-----Original Message----- >From: David Coulson [mailto:david@davidcoulson.net] >Sent: Monday, May 17, 2004 8:08 PM >To: discuss(a)lists.surbl.org >Subject: [SURBL-Discuss] mbox parser > > >I've got a decent mailbox containing a variety of spam e-mail. >Is there >a nice little Perl script out there which will spit out the URLs so I >can submit them to Bill's list? > >David > >-- >David Coulson email: >d(a)vidcoulson.com >Linux Developer / web: >http://davidcoulson.net/ >Network Engineer phone: >(216) 533-6967 > >_______________________________________________ >Discuss mailing list >Discuss(a)lists.surbl.org >http://lists.surbl.org/mailman/listinfo/discuss >

2 1

mbox parser
by David Coulson 18 May '04

18 May '04

I've got a decent mailbox containing a variety of spam e-mail. Is there a nice little Perl script out there which will spit out the URLs so I can submit them to Bill's list? David -- David Coulson email: d(a)vidcoulson.com Linux Developer / web: http://davidcoulson.net/ Network Engineer phone: (216) 533-6967

5 4

Re: finding RegEx for extracting URIs
by Jeff Chan 18 May '04

18 May '04

Hello Rob, I'm forwarding your message to the SURBL discussion and SpamAssassin developers lists. All, Rob is looking for regular expressions to extract URIs form message bodies. As we know this is a little more complex than it may appear at first, given MIME decoding, weird cases, deliberate obfuscation, etc. Jeff C. __ On Monday, May 17, 2004, 6:01:57 PM, Rob McEwen wrote: > Jeff: > Thanks for the resources you provided. At the least, these will give me a > rough roadmap. The reason that these won't do much more for me than this is > because I program using a different platform, different filtering software, > a different mail server, and in an entirely different programming language > (not perl or php). Some of this stuff does not translate very well and, > frankly, some of it looks like "greek" to me. > I program in Visual Basic.NET using the Microsoft .NET platform. Given that > I'm also decent at C# programming, I think I could have done better if these > examples were written in Java, for example. > Nevertheless, my original question... the search for a really good regular > expression for extracting URIs... is a VERY platform-independent issue. > Finding the best regular expression for this could assist a variety of > people using a variety of programming platforms and programming languages > who might also attempt to write software to work with SURBL. > Therefore, please consider posting this question on your website in the > hopes that someone will provide a solution. I wouldn't be surprised if > someone has already discovered the best Regular Expression for extracting > URIs. Also, I believe that posting the answer on your site will be very > helpful to others in my same predicament. > (Note that it is easy to find Regular Expressions which extract the full URL > where the URL is preceded by an "href". But this is obviously not enough. > I'd like to find one which takes into account the country codes for > determining whether two or three levels are needed and then only extracts > what SURBL actually needs. Also, I fully admit that I'd have this one solved > if Regular Expressions were my specialty. They are not!) > Thanks for your consideration. Feel free to quote some or all of this e-mail > at will. > Rob McEwen > PowerView Systems > rob(a)PowerViewSystems.com > (478) 475-9032 > -----Original Message----- > From: Jeff Chan [mailto:jeffc@surbl.org] > Sent: Monday, May 17, 2004 5:45 PM > To: Rob McEwen > Cc: jeffc(a)surbl.org; webmaster(a)PowerViewSystems.com > Subject: Re: finding RegEx for extracting URIs > On Monday, May 17, 2004, 5:22:50 AM, Rob McEwen wrote: >> I'm trying to find a good regular expression for extracting URIs from the >> raw text of an e-mail message. This would help tremendously for > programming >> a SURBL based filter. Do you know of any such regular expression? The ones >> I've found on the internet so far are not very good, for numerous reasons. > Hi Rob, > I think your best best would be to copy the code from urirhsbl in > SpamAssassin URIDNSBL or SpamCopURI's code: > http://spamassassin.org/full/3.0.x/dist/lib/Mail/SpamAssassin/Plugin/URIDNSB > L.pm > http://sourceforge.net/projects/spamcopuri/ > Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

1 0

[SURBL-Discuss] Re: Heads up: new open redirecters and new spammer trick for urls
by Jeff Chan 18 May '04

18 May '04

On Monday, May 17, 2004, 4:10:51 PM, John Fawcett wrote: > From: "jdow" >> From: "John Hardin" <johnh(a)aproposretail.com> >> >> > On Sun, 2004-05-16 at 01:46, John Fawcett wrote: >> > >> > >> > > In order to obtain the 302 code the browser sees >> > > 2 things are necessary: >> > > 1. Add a / before the * (That is the correct format for >> > > yahoo redirection) >> > > 2. Change the hTtP:\\ to hTtP:// (The mixed case is not a problem) >> > >> > I think fixing all backslashes to forward slashes in the URL before >> > processing by SURBL would deal with both cases. >> > >> > Are (unescaped or unencoded) backslashes even *valid* in URLs? >> >> Um, who cares? If the email programs parse them the way the spammers >> want then we need to catch them parsed the way the spammers want. Of >> course we COULD simply dump emails containing illegitimate back slashes >> if nothing Microsoftish produces legitimate emails with backslashes. >> I do not want to bet on that. >> >> {^_^} >> > In this case there is an additional consideration: SpamCopURI has logic to > get the spammer domain out of a redirection url, by actually doing an http > retrieval and reading the 302 response header. > This retrieval fails if done on the url as written by the spammer. > So the options are: > - use unescaped \ as a spam indicator > - mimic the broken browsers/email clients which are apparently rewriting > malformed urls by mapping \ to /, thus allowing SpamCopURI to > successfully retrieve the spammer domain for testing against surbl. I would vote for the latter: map \ to / (Although treating \ used as a separator as a --clue indicator could also be useful, likewise using / for command line flags. ;-) Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

1 0

[SURBL-Discuss] Heads up: new open redirecters and new spammer trick for urls
by John Fawcett 17 May '04

17 May '04

NEW OPEN REDIRECTER I just noticed a new (for me) yahoo redirecter in spam received eur.rd.yahoo.com On a hunch, I also tried things like: uk.rd.yahoo.com it.rd.yahoo.com de.rd.yahoo.com which are all open redirecters. There are sure to be more of these using other country code prefixes. So for those using SpamCopURI you probably need this in your spamcop_uri.cf: open_redirect_list_spamcop_uri rd.yahoo.com *.rd.yahoo.com I'd recommend Eric to add this to the default SpamCopURI configuration on the next release, along with others like open_redirect_list_spamcop_uri drs.yahoo.com open_redirect_list_spamcop_uri ads.msn.com g.msn.com which aren't currently in the defaults. NEW SPAMMER TRICK FOR URLS Having added the new redirection service, I found that SpamCopURI 0.16 didn't pick up the url shown at the end of this message. The reason is that resolving the URL through the SpamCopURI gives a HTTP/1.1 403 Forbidden. As the response code does not begin with a 3 (= redirection), the URL is assumed to be the final one. The domain which is subjected to lookup in sc.surbl.org is (after normalizing to the register level) is yahoo.com. So this one gets past SpamCopURI. Howver, in a commonly used browser, the url redirects to the spamvertized site without difficulty. I cannot help thinking that this url has been carefully crafted to avoid processing by SpamCopURI but still be acceptable to a browser. (That's a terrifying thought). In order to obtain the 302 code the browser sees 2 things are necessary: 1. Add a / before the * (That is the correct format for yahoo redirection) 2. Change the hTtP:\\ to hTtP:// (The mixed case is not a problem) While the second one is a general case (other redirection services could be abused in the same way by browser loopholes) the first one is a very specific browser loophole that applies only to yahoo redirection. Here's the URL. I didn't even munge it, since it should get past the filters. <a href="http://eur.rd.yahoo.com/electric\croydon\laity\otherworldly\phonetic\e xplicit\mountaineer\integrable\isadore\wangle\zounds\contumacy\embedded\sang uine\arrangeable\duane\malarial\bremsstrahlung\freshmen\windup\spoon\accompa ny\soldier\throb\boil\harrisburg\quartz\throne\giddap\waistcoat\guzzle\whoop \abreast\corral\latrobe\ct\castor\gallup\click\cretinous\alcoa\lysine\wheelc hair\levy\embedded\faint\floodlight\elmer\fiesta\pistachio\pulp\suppress\fle awort\flick\topcoat\brain\prom\bill\knife\serene\*hTtP:\\7Wv2eg82o19X.zbxra1 .com/gp/iNdeX.ASP?id=BW" target="_blank"><b>hit this</b></a> John

3 4

urls not parsed by spamcop itself
by John Fawcett 17 May '04

17 May '04

Up to now I have been looking watching out for types of url which are not parsed by SpamCopURI. Today I have found one which is parsable by SpamCopURI, but which risks not being blocked for a different reason. When I went to Spamcop to report the spam, the URL was not parsed out by spamcop reporting engine. This domain is therefore not going to enter into the data (at least from this spam run) and so won't be blocked. BTW the domain is: agrinho.com. I suppose I should tell spamcop about this...but I thought I'd mention it here to fellow surblers because anything which stops spamvertized domains going into spamcop will have an affect on blocking. In reality we need the parsing mechanisms of clients (like SpamCopURI or URIDNSBL) *AND* of Spamcop itself kept up to date with new obfuscation methods. Alternatively, we need some way of submitting to one of the surbl lists, but that's a big step to take... John

3 2

Autoreponder setup incorrectly
by John Fawcett 16 May '04

16 May '04

I just got this in response to my posting. Shouldn't these messages be going back to the list admin for appropriate action? I had posted the message to two lists, but I *think* its a member of this list that has the autoresponder setup incorrectly - I'll soon find out :-) as a result of this post. Thanks, John ----- Original Message ----- From: "Michael McIsaac" <Michael.McIsaac(a)mpljasper.com> To: <johnml(a)michaweb.net> Sent: Sunday, May 16, 2004 10:48 AM Subject: Re: Heads up: new open redirecters and new spammer trick forurls (I am away from the office) Thank-you for your email, however I will be out of the office until May 24. I will answer your email personally when I return. If you need immediate assistance please email our Accountant at Angela.VanDeBogart(a)mpljasper.com. Thank-you

3 5

[SURBL-Discuss] RFC: Combined SURBL list details, phishing list ready
by Jeff Chan 15 May '04

15 May '04

It's time to return to the question of a combined SURBL list again mainly because David Hooton's anti-phishing list is now ready. The list is too small to be a separate list at a little over a hundred entries, but it will probably grow. So I'd like to make it part of a combined list. 1. We had discussed two strategies for a combined list A records before: I. Separate A records: spammer.com IN A 127.0.0.1 IN A 127.0.0.2 Where the two addresses indicate it being on two lists. II. Bitmasked address: spammer.com IN A 127.0.0.3 where .3 means it's on the two lists corresponding to the .1 and .2 bits, and similarly for other lists in other bit positions. In the first case resolution on spammer.com.combined-list-name-here.surbl.org would give two separate Addresses 127.0.0.1 and 127.0.0.2 and in the second case it would give one result 127.0.0.3. The querying program would need to act accordingly. For the bitmasked case some SA code would need to be written or reused. As mentioned earlier, various other RBLs combine lists using either of these two strategies. Here's an example cited earlier of the bitmask style: http://opm.blitzed.org/info > Using the DNSBL > In opm.blitzed.org, the A record has an IP address of 127.1.0.x > where x is a bitmask of the types of proxy that have been > reported to be running on the host. The values of the bitmask > are as follows: > > WinGate 1 > SOCKS 2 > HTTP CONNECT 4 > Router 8 > HTTP POST 16 So the code using a combined list could be made to detect specific results, i.e., the specific list which triggered a matching A record could be determined, and not just that it matched "all" or any from the original list. On the other hand, the fact that matches from any list occurred may be good enough for some users. Personally I prefer more a detailed explanation that would come from being able to distinguish the source list, but that's a question of the program implementation and not the combined list itself. The combined list itself would always encode the source list, whether the querying program knew or cared how to decode that or not. 2. Another question would be the name of the combined list. Since there would be three or more lists, someone had suggested a name of "all" before. That sounds good to me unless there are other suggestions. 3. I'm assuming TXT records are no longer really feasible in a combined list and that descriptive messages will need to be signalled by the list (127. address) matched. I suppose it would be possible to create custom TXT records for every entry, but a generic TXT (or perhaps none) might be more likely. Is a generic TXT better than none? Even in a BIND file, where it incurs some use of space? 4. TTLs: If an entry has matches on more than one list, should it get a unique TTL? If so, should such a custom TTL on the multiply-matching entry be the longest TTL or the shortest TTL? I lean towards the inheriting the shortest TTL from the matching source list, plus setting a default TTL for the combined zone file to be near the longest. Am I right in thinking that TTLs are largely irrelevant for rbldnsd, since it reloads zone info whenever the files change? In other words, does the rbldnsd cache clear for a given zone when the zone reloads, or do the cached entries with TTLs longer than the last reload interval remain in the cache? (I'm kind of hoping for the simpler, case of rbldnsd clearing whenever reloading.) 5. We will likely want to combine the ws and be lists into a single entry in a combined list, probably using the .1 bit for both of them, since both lists contain the enumerated (non-wildcarded) domains from SA regular expressions. Also, things are moving towards combining the non-wildcarded domains sa-blacklist and BigEvil/MidEvil, so this would somewhat short-circuit that process and future-proof things. Comments? Jeff C.

6 15

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss