Discuss April 2004

discuss@lists.surbl.org

60 participants
132 discussions

Re: update on unparsed url types
by Matthew Cline 19 Apr '04

19 Apr '04

On Sunday 18 April 2004 04:55 pm, Theo Van Dinter wrote: > come to think of this: have spammers used things like tinyurl yet? No instances of TinyUrl in any of the some 8,000 spam I'v received in April, but looking in the news.admin.net-abuse.sightings newsgroup (http://groups.google.com/groups?hl=en&lr=lang_en&ie=UTF-8&oe=UTF-8&safe=off…) shows over 300 examples for spam with TinyUrl links. -- Give a man a match, and he'll be warm for a minute, but set him on fire, and he'll be warm for the rest of his life. Advanced SPAM filtering software: http://spamassassin.org

3 2

[long] summary of currently unparsed url types
by John Fawcett 19 Apr '04

19 Apr '04

I'd just like to summarize the current position with regard to url types which are not currently parsed correctly by sa and ask for some help with tests using version 3. Yahoo offers a public redirection service. You can enter a url like this: http://rds.yahoo.com/*http://www.google.com and you get sent to www.google.com. (By the way I'm not sure what the point of this is, because unlike tinyurl.com the yahoo url is longer. However it sure comes in handy to spammers who are trying to get past sa URI rulesets.) Spam which is not picked up correctly by sa uri filters often contains redirection urls, even though the redirected domain is in sc.surbl.org. Jeff Chan has opened a bug against URIDNSBL.pm to ask for support for parsing out the spammer domain from redirected urls. http://bugzilla.spamassassin.org/show_bug.cgi?id=3261 Things are getting more complicated, because spam coming through seems to contain features which avoid it being picked up even by an altered parser which strips off the http://rds.yahoo.com/* part. I wanted to make a summary of current understanding of the url types which break parsing. I've tested these with SpamCopURI and ver 2.63. If someone offers to test (from case 2 onwards) with URIDNSBL and version 3, I'll post suitable test cases. 1.http://rds.yahoo.com/*http://spammer.domain.tld/aaaaaaaaaa (bug 3261) Workaround in PerMsgStatus.pm: $uri =~ s/^http:\/\/(?:drs|rd).yahoo.com\/[^\*]+\*(.*)$/$1/g; 2.http://rds.yahoo.com/*%68ttp://spammer.domain.tld/aaaaaaaa (follow-up to bug 3261 including test case) (the other possible variations on this which I haven't seen as yet can use %NN instead of any or all the 'http' characters in the redirected domain. e.g. http://rds.yahoo.com/*%68%74%74%70://spammer.domain.tld/aaaaaaaa Workaround in PerMsgStatus.pm: $uri =~ s/\%68/h/g; $uri =~ s/\%74/t/g; $uri =~ s/\%70/p/g; 3. http://rd.yahoo.com/winery/college/banbury/*http:/len= derserv.com?partid=3Darlenders The redirect url is formally incorrect (there is a single slash after http) but browsers have no problem with this. The parser cannot handle it. Workaround in PerMsgStatus.pm: $uri =~ s/http:\/([^\/])/http:\/\/$1/g; By the way, this url contains 'quotable printable' characters ('= newline' and '=3d') which are not causing problems to the parser. Neither is the absence of a trailing slash before the ? causing problems in parsing. 4. URLS without http: in front of them. The following seen in a browser reads: "Please copy and paste this link into your browser healthyexchange.biz " <p> P<advisory>l<aboveboard>e<compose>a<geochronology>s<moral>e<palfrey> <rada= r>c<symptomatic>o<yankee>p<conduit>y<souffle> <intake>a<arise>n<eocene>d <= thickish>paste <impact>this <broadloom>link <road>i<dichotomous>n<quinine>= t<scoreboard>o y<eager>o<impact>ur b<archenemy>r<band>o<wallop>wser <b> he= althyexchange.biz</b> Probably not much that can be dones with this. 5. http://http://www.eager-18.com/_7953f10b575a18d044cdec5a40bd4f22//?d=vision Here the double http prevents this being parsed. (OK it wasn't in sc.surbl.org but even if it was it wouldn't have been picked up) Workaround in PerMsgStatus.pm: $uri =~ s/http:\/\/http:\/\//http:\/\//g; John

3 4

Re: [SURBL-Discuss] update on unparsed url types
by John Fawcett 18 Apr '04

18 Apr '04

----- Original Message ----- From: John_Delisle > > > > Wouldn't we want to decode the URIs before the other tests? Good point. The test cases I was working with didn't have a mixture of the characteristics I mentioned. However, putting the HTML::entities decode before the other tests would be sensible, so as to pick up this possibility. John

1 0

[long] summary of currently unparsed url types
by John_Delisle＠ceridian.ca 18 Apr '04

18 Apr '04

John - Can you please post line numbers or a diff for those changes? Thanks, John Delisle Senior Network Analyst Ceridian Canada Ltd Information Systems & Technology Management Dept, Network and Security Team 204-975-5909 ********************************************************************** This e-mail and any files transmitted with it are considered confidential and are intended solely for the use of the individual or entity to whom they are addressed (intended). This communication is subject to agent/client privilege. If you are not the intended recipient (received in error) or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited. If you have received this e-mail in error please notify the sender immediately. **********************************************************************

2 1

Mail-SpamAssassin-SpamCopURI-0.10 Errors
by Doc Schneider 17 Apr '04

17 Apr '04

I'm still getting the following errors for make test using the .10 spamcopuri plugin, any ideas? I get this on multiple servers running either perl 5.6.1 or 5.8.2 and I've gotten all the updates for modules so far that have been listed on this list and still no joy. PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/local/lib/perl5/5.6.1/i686-linux -I/usr/local/lib/perl5/5.6.1 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/bad..............ok t/blacklist........ok t/cache............ok t/dnsrbl...........NOK 1# Failed test (t/dnsrbl.t at line 16) t/dnsrbl...........NOK 2# Failed test (t/dnsrbl.t at line 18) t/dnsrbl...........NOK 5# Failed test (t/dnsrbl.t at line 25) t/dnsrbl...........ok 6/6# Looks like you failed 3 tests of 6. t/dnsrbl...........dubious Test returned status 3 (wstat 768, 0x300) DIED. FAILED tests 1-2, 5 Failed 3/6 tests, 50.00% okay t/good.............ok t/ipaddress........ok t/longhostname.....ok t/mailto...........ok t/merge_urls.......ok t/shorthostname....ok t/spamcop_fetch....ok t/spamcopuri.......ok t/whitelist........ok Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- t/dnsrbl.t 3 768 6 3 50.00% 1-2 5 Failed 1/13 test scripts, 92.31% okay. 3/62 subtests failed, 95.16% okay. make: *** [test_dynamic] Error 11 -- -Doc

4 7

Openrbl.org
by Chris Santerre 17 Apr '04

17 Apr '04

Can someone contact openrbl.org and get them to add SURBL to the query? I use this tool quite often! Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin Charles Darwin

6 10

[SURBL-Discuss] Re: Fwd: Re: Fwd: Re: Announcing SURBL support in SA 2.63 and 3.0 plugins - bug ?
by Jeff Chan 16 Apr '04

16 Apr '04

Simon Byrnand, Eric Kolve and I were having a discussion of what characters are legal in domain names, due to junk showing up around URIs and apparently confusing some of the SpamAssassin URI parsing code. Wanted to share some research and ask if anyone has any other authoritative information on what characters are currently legal for domain names. This is relevant for anyone trying to work with domain names. Also Eric, please share bugs you find in the SA URI parsing code, preferably by opening a bugzilla, especially if you can isolate the module, etc.: http://bugzilla.spamassassin.org/enter_bug.cgi Here's a little research on the subject: The original domain name RFC had names only with letters, numbers and hyphen: http://www.ietf.org/rfc/rfc1035.txt <domain> ::= <subdomain> | " " <subdomain> ::= <label> | <subdomain> "." <label> <label> ::= <letter> [ [ <ldh-str> ] <let-dig> ] <ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str> <let-dig-hyp> ::= <let-dig> | "-" <let-dig> ::= <letter> | <digit> <letter> ::= any one of the 52 alphabetic characters A through Z in upper case and a through z in lower case <digit> ::= any one of the ten digits 0 through 9 Note that while upper and lower case letters are allowed in domain names, no significance is attached to the case. That is, two names with the same spelling but different case are to be treated as if identical. But RFC 2181 leaves things wide open with respect to names: http://www.ietf.org/rfc/rfc2181.txt 11. Name syntax Occasionally it is assumed that the Domain Name System serves only the purpose of mapping Internet host names to data, and mapping Internet addresses to host names. This is not correct, the DNS is a general (if somewhat limited) hierarchical database, and can store almost any kind of data, for almost any purpose. The DNS itself places only one restriction on the particular labels that can be used to identify resource records. That one restriction relates to the length of the label and the full name. The length of any one label is limited to between 1 and 63 octets. A full domain name is limited to 255 octets (including the separators). The zero length full name is defined as representing the root of the DNS tree, and is typically written and displayed as ".". Those restrictions aside, any binary string whatever can be used as the label of any resource record. Similarly, any binary string can serve as the value of any record that includes a domain name as some or all of its value (SOA, NS, MX, PTR, CNAME, and any others that may be added). Implementations of the DNS protocols must not place any restrictions on the labels that can be used. In particular, DNS servers must not refuse to serve a zone because it contains labels that might not be acceptable to some DNS client programs. A DNS server may be configurable to issue warnings when loading, or even to refuse to load, a primary zone containing labels that might be considered questionable, however this should not happen by default. Note however, that the various applications that make use of DNS data can have restrictions imposed on what particular values are acceptable in their environment. For example, that any binary label can have an MX record does not imply that any binary name can be used as the host part of an e-mail address. Clients of the DNS can impose whatever restrictions are appropriate to their circumstances on the values they use as keys for DNS lookup requests, and on the values returned by the DNS. If the client has such restrictions, it is solely responsible for validating the data from the DNS to ensure that it conforms before it makes any use of that data. After scanning the RFC descriptions that were linked from RFC 1035: > RFC 1035 Domain names - implementation and specification. > > Authors: P.V. Mockapetris. > Date: Nov-01-1987 > Formats: txt pdf > Obsoletes: RFC 0973, RFC 0882, RFC 0883 > Updated by: RFC 1101, RFC 1183, RFC 1348, RFC 1876, RFC 1982, > RFC 1995, RFC 1996, RFC 2065, RFC 2136, RFC 2181, RFC 2137, RFC > 2308, RFC 2535, RFC 2845, RFC 3425, RFC 3658 > Also: STD 0013 it appears that these may be the only two authoritative statements on what characters can be in domain names: RFC 1035: letters, numbers, hyphen RFC 2181: implementations should support anything Does anyone have any more info on what characters are legal in domain names? Jeff C.

2 1

Hello to everybody!
by Markus Zingg 15 Apr '04

15 Apr '04

Hi all I found out about the surbl project some days ago and was all excited, but then a bit frustrated not to find contact information. Well, it seems like I did not looked carefull enough but then got the pointer to join this list from Wayne and well, here I am. I don't know the culture of this list yet, but still take the "risk" to introduce myself instead of pasively reading a couple of days first. I hope not to offend people by doing so. My name is Markus Zingg and I live in Switzerland - Europ. English is not my native language, I hope you bear with me if I sound odd or plain wrong :-). I'm developping all kind of software mostly useing 'C' for the last 20+ years. I'm currently working on a project where we have developped an embedded email server. This is a 4x4x1" sized box which apart from being an SMTP, POP3 and WebMail server also contains a spam filter that in short and apart of some other methods works by extracting URI's from e-mails and matching them against a blacklist database. I'm happy to post the URL to a site describing the box in detail if you want me to. The box - which does not have the luxury of having a superfast Pentium processor - must do the filtering in a very efficient fashion and therefore all of the firmware is implemented in 'C' with some parts even being (some risc processor) assembler. In order to parse the e-mails and split them up MIME wise etc. I wrote a special parser which does all what's needed in one single pass (MIME parsing, content transfer decoding, decoding of hex and decimal encoded HTML areas etc. etc.) and parses the textual parts skiping attachements etc. The parser of course takes into acount wether it works on a html text or a plain text part and can't be fooled by the tricks the spammers used so far. It works extremly efficient and hence I thought it might be of intersted to the surbl project. As a side effect it can also detect dangerous attachements by looking at the filename extension and if configured to do so rename them on the fly. I figure though that this later part might does not go together well with SpamAssassin. Since it was written for this embedded hardware there is some effort needed to make it of general use but I partially did that already in order to test the filter more efficiently with thousands of spam samples I collected over the years and of course also spam that currently is coming in. I must admit that I don't know how SpamAssassin works in detail nor do I curently have a Linux based e-mail server setup. I do have however some PC's standing around that I could set up this way and I also used to work with Unix and Linux for several years in the past. Apart from the fact that doing whatever possible to get the spam problem somewhat under control my interest would be to get as much spamvertised domains as possible. I understand that I could read live surbl data already and even add such functionality to the firmware but did not wanted to do this withouth first asking and also without offering help. I also have some aditional ideas on how to get more spamvertised domains and other issues, but I think this posting got a little long already :-) I honestly have no idea if what I can offer is of any interest here. If not, please accept my apologies of having disturbed you. Markus

2 1

RE: [SURBL-Discuss] No install Problems with 0.10
by Paul Barbeau 14 Apr '04

14 Apr '04

Yep > -----Original Message----- > From: Charles Solomon [mailto:csolomon@aldridge-borden.com] > Sent: April 14, 2004 2:54 PM > To: Paul(a)hypernet.ca; discuss(a)lists.surbl.org > Subject: RE: [SURBL-Discuss] No install Problems with 0.10 > > > Do you get a clean bill of health when you --lint?

1 0

RE: [SURBL-Discuss] No install Problems with 0.10
by ITReading ITReading 14 Apr '04

14 Apr '04

Do you get a clean bill of health when you --lint? >>> Paul Barbeau <Paul(a)hypernet.ca> 04/14/2004 12:11:43 PM >>> > Did you remember to copy the .cf file to your spamassassin > rules folder? (I only mention this b/c I tend to forget to > do the most obvious of steps). Yep i put it in both places > > If you run spamassassin -D -t on a test message, do you see > it attempting to perform the queries? Nope it does not look line it

1 0

← Newer
1
...
6
7
8
9
10
11
12
13
14
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss April 2004