[SURBL-Discuss] ccTLDs and multiple queries

Jeff Chan jeffc at surbl.org
Wed Apr 21 16:00:52 CEST 2004


On Wednesday, April 21, 2004, 12:21:16 PM, John Fawcett wrote:
> From: "Eric Kolve"
>> Initially, when I released spamcopuri I decided to pretty much ignore
>> whether the TLD was a country code or not.  This results in about
>> twice as many queries as necessary, but guaranteed you would get
>> hits if the domain was listed.
>>
>> Now that people are pointing this to other RBL's beside just surbl,
>> should we continue to do second and third level queries? Or just
>> the query that we assume to be necessary?  My concern is that not
>> all RBLs will process the domains according to a list such as
>> http://www.bestregistrar.com/help/ccTLD.htm.  I suppose the worst
>> case scenario is we end up getting a miss when we should be getting
>> a hit because one side presumes that say TLD .za has a subdomain 'foo',
>> when the server doesn't.  The server side would expect a second level,
> while
>> the client would do a third level query (this is why I wanted the wildcard
>> records).  I guess this really isn't that great a consequence considering
>> the savings and the fact that this shouldn't occur very often.
>>
>> I will go ahead and make this change if everyone is comfortable with the
>> known risk.
>>
> I think if an rhsbl is listing a second level registry domain
> (like .co.uk) then I think it's up to the list maintainer to implement
> the wild card so that xxxxx.co.uk returns an A record. I wouldn't
> worry about taking into account such an extreme case,
> since I cannot imagine any list wanting to do such widespread
> blocking.

Yes, the two level ccTDLs like co.uk should never get into a
SURBL.  Only registrar-type domains should, like foo.co.uk.

> I believe there should be a mechanism which distinguishes whether
> a second or third level lookup is required based on a static
> lists of domains known to have or not have subdomains.
> If nothing is known then the default should be to check both
> second and third as at present.

Aha, now I think I understand what's being proposed.

Currently SpamCopURI checks all domains at the second
and third level against a given SURBL, regardless of
whether the domain is in a ccTLD or not.

It sounds like Eric is proposing a change, where if a domain is
in the ccTLD list like co.uk, then the client should try
extract and check a three level domain like foo.co.uk.  Otherwise
it should check two levels like foo.com.

Is that right?  If so it may be ok, though our list of ccTLDs is
slightly underspecified (there are some ccTLDs not in it).  Note
that my ccTLD list:

  http://spamcheck.freeapp.net/two-level-tlds

is (derived from but) slightly more complete than the one at
http://www.bestregistrar.com/help/ccTLD.htm ....

Worst case is that we miss a few ccTLDs.  Probably not too big a
deal given that most of the spam domains are .com, .biz, etc.

I believe Eric is also making a finer point that other SURBL data
sources may miss some unexpected geographic domains where foo.za
occurred and only two-level base-ccTLDs like foo.com.za were
expected. Not sure how to handle unusual cases like that.  I
suppose we'll need to relay on the country code authorities to be
somewhat consistent with respect to what levels they will allow
in their ccTLD.

Philosophical point: it's always possible that some spam domains
slip through the cracks, but if that happens often enough and
we spot them, we can always blacklist them manually.  Perfection
may not be possible, but we're certainly greatly increasing the
spam detection rates with this approach overall.

BTW I'm using the ccTLD list to try to ensure that any two level
ccTLDs do *not* get into any SURBLs.

P.S. ***If anyone has more-complete ccTLD lists or any updates or
additions, please share them,*** else spammers may set up shop in
some unknown Outer Mongolian ccTLD, and we may not catch them. :-)

Jeff C.



More information about the Discuss mailing list