Discuss

discuss@lists.surbl.org

1862 discussions

Re: finding RegEx for extracting URIs
by Jeff Chan 18 May '04

18 May '04

Hello Rob, I'm forwarding your message to the SURBL discussion and SpamAssassin developers lists. All, Rob is looking for regular expressions to extract URIs form message bodies. As we know this is a little more complex than it may appear at first, given MIME decoding, weird cases, deliberate obfuscation, etc. Jeff C. __ On Monday, May 17, 2004, 6:01:57 PM, Rob McEwen wrote: > Jeff: > Thanks for the resources you provided. At the least, these will give me a > rough roadmap. The reason that these won't do much more for me than this is > because I program using a different platform, different filtering software, > a different mail server, and in an entirely different programming language > (not perl or php). Some of this stuff does not translate very well and, > frankly, some of it looks like "greek" to me. > I program in Visual Basic.NET using the Microsoft .NET platform. Given that > I'm also decent at C# programming, I think I could have done better if these > examples were written in Java, for example. > Nevertheless, my original question... the search for a really good regular > expression for extracting URIs... is a VERY platform-independent issue. > Finding the best regular expression for this could assist a variety of > people using a variety of programming platforms and programming languages > who might also attempt to write software to work with SURBL. > Therefore, please consider posting this question on your website in the > hopes that someone will provide a solution. I wouldn't be surprised if > someone has already discovered the best Regular Expression for extracting > URIs. Also, I believe that posting the answer on your site will be very > helpful to others in my same predicament. > (Note that it is easy to find Regular Expressions which extract the full URL > where the URL is preceded by an "href". But this is obviously not enough. > I'd like to find one which takes into account the country codes for > determining whether two or three levels are needed and then only extracts > what SURBL actually needs. Also, I fully admit that I'd have this one solved > if Regular Expressions were my specialty. They are not!) > Thanks for your consideration. Feel free to quote some or all of this e-mail > at will. > Rob McEwen > PowerView Systems > rob(a)PowerViewSystems.com > (478) 475-9032 > -----Original Message----- > From: Jeff Chan [mailto:jeffc@surbl.org] > Sent: Monday, May 17, 2004 5:45 PM > To: Rob McEwen > Cc: jeffc(a)surbl.org; webmaster(a)PowerViewSystems.com > Subject: Re: finding RegEx for extracting URIs > On Monday, May 17, 2004, 5:22:50 AM, Rob McEwen wrote: >> I'm trying to find a good regular expression for extracting URIs from the >> raw text of an e-mail message. This would help tremendously for > programming >> a SURBL based filter. Do you know of any such regular expression? The ones >> I've found on the internet so far are not very good, for numerous reasons. > Hi Rob, > I think your best best would be to copy the code from urirhsbl in > SpamAssassin URIDNSBL or SpamCopURI's code: > http://spamassassin.org/full/3.0.x/dist/lib/Mail/SpamAssassin/Plugin/URIDNSB > L.pm > http://sourceforge.net/projects/spamcopuri/ > Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

1 0

[SURBL-Discuss] Re: Heads up: new open redirecters and new spammer trick for urls
by Jeff Chan 18 May '04

18 May '04

On Monday, May 17, 2004, 4:10:51 PM, John Fawcett wrote: > From: "jdow" >> From: "John Hardin" <johnh(a)aproposretail.com> >> >> > On Sun, 2004-05-16 at 01:46, John Fawcett wrote: >> > >> > >> > > In order to obtain the 302 code the browser sees >> > > 2 things are necessary: >> > > 1. Add a / before the * (That is the correct format for >> > > yahoo redirection) >> > > 2. Change the hTtP:\\ to hTtP:// (The mixed case is not a problem) >> > >> > I think fixing all backslashes to forward slashes in the URL before >> > processing by SURBL would deal with both cases. >> > >> > Are (unescaped or unencoded) backslashes even *valid* in URLs? >> >> Um, who cares? If the email programs parse them the way the spammers >> want then we need to catch them parsed the way the spammers want. Of >> course we COULD simply dump emails containing illegitimate back slashes >> if nothing Microsoftish produces legitimate emails with backslashes. >> I do not want to bet on that. >> >> {^_^} >> > In this case there is an additional consideration: SpamCopURI has logic to > get the spammer domain out of a redirection url, by actually doing an http > retrieval and reading the 302 response header. > This retrieval fails if done on the url as written by the spammer. > So the options are: > - use unescaped \ as a spam indicator > - mimic the broken browsers/email clients which are apparently rewriting > malformed urls by mapping \ to /, thus allowing SpamCopURI to > successfully retrieve the spammer domain for testing against surbl. I would vote for the latter: map \ to / (Although treating \ used as a separator as a --clue indicator could also be useful, likewise using / for command line flags. ;-) Jeff C. -- Jeff Chan mailto:jeffc@surbl.org http://www.surbl.org/

1 0

[SURBL-Discuss] Heads up: new open redirecters and new spammer trick for urls
by John Fawcett 17 May '04

17 May '04

NEW OPEN REDIRECTER I just noticed a new (for me) yahoo redirecter in spam received eur.rd.yahoo.com On a hunch, I also tried things like: uk.rd.yahoo.com it.rd.yahoo.com de.rd.yahoo.com which are all open redirecters. There are sure to be more of these using other country code prefixes. So for those using SpamCopURI you probably need this in your spamcop_uri.cf: open_redirect_list_spamcop_uri rd.yahoo.com *.rd.yahoo.com I'd recommend Eric to add this to the default SpamCopURI configuration on the next release, along with others like open_redirect_list_spamcop_uri drs.yahoo.com open_redirect_list_spamcop_uri ads.msn.com g.msn.com which aren't currently in the defaults. NEW SPAMMER TRICK FOR URLS Having added the new redirection service, I found that SpamCopURI 0.16 didn't pick up the url shown at the end of this message. The reason is that resolving the URL through the SpamCopURI gives a HTTP/1.1 403 Forbidden. As the response code does not begin with a 3 (= redirection), the URL is assumed to be the final one. The domain which is subjected to lookup in sc.surbl.org is (after normalizing to the register level) is yahoo.com. So this one gets past SpamCopURI. Howver, in a commonly used browser, the url redirects to the spamvertized site without difficulty. I cannot help thinking that this url has been carefully crafted to avoid processing by SpamCopURI but still be acceptable to a browser. (That's a terrifying thought). In order to obtain the 302 code the browser sees 2 things are necessary: 1. Add a / before the * (That is the correct format for yahoo redirection) 2. Change the hTtP:\\ to hTtP:// (The mixed case is not a problem) While the second one is a general case (other redirection services could be abused in the same way by browser loopholes) the first one is a very specific browser loophole that applies only to yahoo redirection. Here's the URL. I didn't even munge it, since it should get past the filters. <a href="http://eur.rd.yahoo.com/electric\croydon\laity\otherworldly\phonetic\e xplicit\mountaineer\integrable\isadore\wangle\zounds\contumacy\embedded\sang uine\arrangeable\duane\malarial\bremsstrahlung\freshmen\windup\spoon\accompa ny\soldier\throb\boil\harrisburg\quartz\throne\giddap\waistcoat\guzzle\whoop \abreast\corral\latrobe\ct\castor\gallup\click\cretinous\alcoa\lysine\wheelc hair\levy\embedded\faint\floodlight\elmer\fiesta\pistachio\pulp\suppress\fle awort\flick\topcoat\brain\prom\bill\knife\serene\*hTtP:\\7Wv2eg82o19X.zbxra1 .com/gp/iNdeX.ASP?id=BW" target="_blank"><b>hit this</b></a> John

3 4

urls not parsed by spamcop itself
by John Fawcett 17 May '04

17 May '04

Up to now I have been looking watching out for types of url which are not parsed by SpamCopURI. Today I have found one which is parsable by SpamCopURI, but which risks not being blocked for a different reason. When I went to Spamcop to report the spam, the URL was not parsed out by spamcop reporting engine. This domain is therefore not going to enter into the data (at least from this spam run) and so won't be blocked. BTW the domain is: agrinho.com. I suppose I should tell spamcop about this...but I thought I'd mention it here to fellow surblers because anything which stops spamvertized domains going into spamcop will have an affect on blocking. In reality we need the parsing mechanisms of clients (like SpamCopURI or URIDNSBL) *AND* of Spamcop itself kept up to date with new obfuscation methods. Alternatively, we need some way of submitting to one of the surbl lists, but that's a big step to take... John

3 2

Autoreponder setup incorrectly
by John Fawcett 16 May '04

16 May '04

I just got this in response to my posting. Shouldn't these messages be going back to the list admin for appropriate action? I had posted the message to two lists, but I *think* its a member of this list that has the autoresponder setup incorrectly - I'll soon find out :-) as a result of this post. Thanks, John ----- Original Message ----- From: "Michael McIsaac" <Michael.McIsaac(a)mpljasper.com> To: <johnml(a)michaweb.net> Sent: Sunday, May 16, 2004 10:48 AM Subject: Re: Heads up: new open redirecters and new spammer trick forurls (I am away from the office) Thank-you for your email, however I will be out of the office until May 24. I will answer your email personally when I return. If you need immediate assistance please email our Accountant at Angela.VanDeBogart(a)mpljasper.com. Thank-you

3 5

[SURBL-Discuss] RFC: Combined SURBL list details, phishing list ready
by Jeff Chan 15 May '04

15 May '04

It's time to return to the question of a combined SURBL list again mainly because David Hooton's anti-phishing list is now ready. The list is too small to be a separate list at a little over a hundred entries, but it will probably grow. So I'd like to make it part of a combined list. 1. We had discussed two strategies for a combined list A records before: I. Separate A records: spammer.com IN A 127.0.0.1 IN A 127.0.0.2 Where the two addresses indicate it being on two lists. II. Bitmasked address: spammer.com IN A 127.0.0.3 where .3 means it's on the two lists corresponding to the .1 and .2 bits, and similarly for other lists in other bit positions. In the first case resolution on spammer.com.combined-list-name-here.surbl.org would give two separate Addresses 127.0.0.1 and 127.0.0.2 and in the second case it would give one result 127.0.0.3. The querying program would need to act accordingly. For the bitmasked case some SA code would need to be written or reused. As mentioned earlier, various other RBLs combine lists using either of these two strategies. Here's an example cited earlier of the bitmask style: http://opm.blitzed.org/info > Using the DNSBL > In opm.blitzed.org, the A record has an IP address of 127.1.0.x > where x is a bitmask of the types of proxy that have been > reported to be running on the host. The values of the bitmask > are as follows: > > WinGate 1 > SOCKS 2 > HTTP CONNECT 4 > Router 8 > HTTP POST 16 So the code using a combined list could be made to detect specific results, i.e., the specific list which triggered a matching A record could be determined, and not just that it matched "all" or any from the original list. On the other hand, the fact that matches from any list occurred may be good enough for some users. Personally I prefer more a detailed explanation that would come from being able to distinguish the source list, but that's a question of the program implementation and not the combined list itself. The combined list itself would always encode the source list, whether the querying program knew or cared how to decode that or not. 2. Another question would be the name of the combined list. Since there would be three or more lists, someone had suggested a name of "all" before. That sounds good to me unless there are other suggestions. 3. I'm assuming TXT records are no longer really feasible in a combined list and that descriptive messages will need to be signalled by the list (127. address) matched. I suppose it would be possible to create custom TXT records for every entry, but a generic TXT (or perhaps none) might be more likely. Is a generic TXT better than none? Even in a BIND file, where it incurs some use of space? 4. TTLs: If an entry has matches on more than one list, should it get a unique TTL? If so, should such a custom TTL on the multiply-matching entry be the longest TTL or the shortest TTL? I lean towards the inheriting the shortest TTL from the matching source list, plus setting a default TTL for the combined zone file to be near the longest. Am I right in thinking that TTLs are largely irrelevant for rbldnsd, since it reloads zone info whenever the files change? In other words, does the rbldnsd cache clear for a given zone when the zone reloads, or do the cached entries with TTLs longer than the last reload interval remain in the cache? (I'm kind of hoping for the simpler, case of rbldnsd clearing whenever reloading.) 5. We will likely want to combine the ws and be lists into a single entry in a combined list, probably using the .1 bit for both of them, since both lists contain the enumerated (non-wildcarded) domains from SA regular expressions. Also, things are moving towards combining the non-wildcarded domains sa-blacklist and BigEvil/MidEvil, so this would somewhat short-circuit that process and future-proof things. Comments? Jeff C.

6 15

Domain not identified by SpamCopURI
by John Fawcett 15 May '04

15 May '04

I just got the following in a spam message which was not identified by spamcopuri even though the numeric domain it is in sc.surbl.org <a href=3D"http://69.63.161.232/?rid=3D1528"> The debug output shows that the uri is found by the SA modules but that SpamCopURI queries for the wrong hostname. It should be querying 232.161.63.69.sc.surbl.org > debug: uri tests: Done uriRE > debug: checking url: http://69.63.161.232/?rid%1528 > debug: querying for 63.161..sc.surbl.org > debug: Query failed for 63.161..sc.surbl.org It looks like there are two problem is in SpamCopURI (v 0.15) in the routine _spamcop_uri (1) # strip any non alpha characters off of the end # this is to fix a bug where url parsing in core SA # leaves parens and other junk on the URL that URI # parses to the host This causes the domain to become 69.63.161. (including a trailing dot) (2) The domain is stripped back to a two level domain even though it is/should be an ip address. It becomes 63.161. (including a trailing dot) This domain then does not match an ip address format so it is not reversed and the lookup is done for 63.161. added on to .sc.surbl.org, i.e.: 63.161..sc.surbl.org Eric, can you help? John

3 4

RE: [SURBL-Discuss] RFC: Combined SURBL list details, phishing list ready
by Chris Santerre 14 May '04

14 May '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Thursday, May 13, 2004 5:07 PM >To: SURBL Discuss >Subject: Re: [SURBL-Discuss] RFC: Combined SURBL list details, phishing >list ready > > >On Thursday, May 13, 2004, 7:21:35 AM, Chris Santerre wrote: >>>From: Jeff Chan [mailto:jeffc@surbl.org] > >>>Actually I was getting tricky and proposing to collapse ws and be >>>into a single response within a combined list. This was mainly >>>to prevent needing to remove separate be entries later since it will >>>probably be merged into ws eventually. I was proposing short >>>circuiting that process in the combined list. > >> I would say consider BE to be WS as of now. Just work with >WS, because BE is >> definetly going to be pulled in. How we do that on the >backend won't matter >> to the clients. For all intensive purposes, I won't be >updating BE, I will >> be updating WS directly thru the magic of Paul. (He's just >swamped at the >> moment.) > >> So again, consider BE non exhistant for future upgrades. It >will save one >> lookup ;) > >Sounds like you're saying we should not fold be in with ws for a >combined list. Could we fold be in transparently into ws for the >combined list, then remove it later (all invisibly to the users >of the combined list)? Or are you guys already merging them >behind the scenes? Want to try to get all the domains.... :-) > >Jeff C. Yeah we will just transfer. Actually WS is so more up to date, the amount of unique hits keeps getting smaller. Partly because I used to be the only game in town, I felt the preasure to to keep updated. Now I don't thanks to you guys :) So I'm trying to work on a bunch of things and not updating as often. My ninja regex skills were getting rusty ;) --Chris

1 0

Whitelist discussion
by Chris Santerre 14 May '04

14 May '04

So what do we do with candidates for whitelists? Also, what do we do with things that should NOT be whitelisted, but NOT reported as spam? Examples are cacheing services. Which are often used by spammers for images, but legit as well. I think images.exactis.com is one. And of course we have all the akamai.net servers. I'm just kind of thinking out loud. *man I'm hungry right now* :-) Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin

5 5

RE: [SURBL-Discuss] RFC: Combined SURBL list details, phishing list ready
by Chris Santerre 13 May '04

13 May '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Wednesday, May 12, 2004 7:27 PM >To: SURBL Discuss >Subject: Re: [SURBL-Discuss] RFC: Combined SURBL list details, phishing >list ready > > *snip* > >Actually I was getting tricky and proposing to collapse ws and be >into a single response within a combined list. This was mainly >to prevent needing to remove separate be entries later since it will >probably be merged into ws eventually. I was proposing short >circuiting that process in the combined list. > I would say consider BE to be WS as of now. Just work with WS, because BE is definetly going to be pulled in. How we do that on the backend won't matter to the clients. For all intensive purposes, I won't be updating BE, I will be updating WS directly thru the magic of Paul. (He's just swamped at the moment.) So again, consider BE non exhistant for future upgrades. It will save one lookup ;) --Chris

3 2

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss