At 13:34 22/04/2004, Jeff Chan wrote:

Simon, you're right that DNS caching means this doesn't hurt much in terms of performance. (It may even be faster than trying to store these in SA to prevent duplication.)

For people with a caching DNS server on the same machine or ethernet perhaps, but there are people running SA on Windoze machines over dialup connections, (crazy but true ;) for them it will most definately not be faster :)

Regards, Simon

At 14:39 22/04/2004, Eric Kolve wrote:

I can add something that will cache on a per test basis the results from the queries so the above scenario should be knocked down to just 3 queries instead of 120. I have been a little hesitant to cache misses since I could see where a miss could become a hit later on, but since I would only be caching per test this shouldn't be an issue.

You mean 6 queries ? Assuming you still test both 2nd level and 3rd level possibilities seperately, as now.

And as far as caching goes, it shouldn't be a problem, because you just want to avoid doing a whole string of identical dns lookups - only cache identical lookups, and the caching should only last for one run of SA processing one message...(We assume that no new blacklist records appear in the middle of processing a specific message, or that if they do we don't care ;-)

Regards, Simon

On Wednesday, April 21, 2004, 8:20:40 PM, Eric Kolve wrote:

On Thu, Apr 22, 2004 at 02:55:33PM +1200, Simon Byrnand wrote:

I have pulled the ccTLD code from URIRBL to extract the 'registrar domain' so we would only do one query per host instead of two (2nd and 3rd level).

This should work fine since Jeff is doing the same thing on his side and will cut queries down quite a bit.

Yep. Sounds good!

Jeff C.

At 15:27 22/04/2004, Jeff Chan wrote:

On Wednesday, April 21, 2004, 7:55:33 PM, Simon Byrnand wrote:

...

At 14:39 22/04/2004, Eric Kolve wrote:

...

I can add something that will cache on a per test basis the results from the queries so the above scenario should be knocked down to just 3 queries instead of 120. I have been a little hesitant to cache misses since I could see where a miss could become a hit later on, but since I would only be caching per test this shouldn't be an issue.

...

You mean 6 queries ? Assuming you still test both 2nd level and 3rd level possibilities seperately, as now.

It sounds like Eric is changing SpamCopURI to test on the number of levels depending on ccTLD membership, which is probably fine, therefore it would do 3 queries.

A thought just struck me...

At the moment it is not entirely foolproof to figure out how many layers there are on a given ccTLD domain name, and at surbl.org in your backend processing system you have a list which you have scraped together from various sources, and updated manually etc.

As Eric pointed out in a previous message, trying to do the "right thing" on the client end for stripping the domain name accurately would mean duplicating the knowledge of that list in the client - something which could change from time to time....and if you get it wrong you potentially miss some hits.

Is there any way that you could automatically publish that information in the surbl itself ? Perhaps a client could retrieve a list of current ccTLD designations and whether they are 2nd level or 3rd level etc, and persistently cache that for a few hours to a day, and refer to that during processing.

That way, as errors in the ccTLD designations come to light, or registrars add new ones etc the changes could be automatically propogated down to the clients...

It's not the sort of thing that would need checking every lookup, probably once a day would be fine, and could be handled in a similar way to how the razor client keeps a note of the last time it downloaded the list of discovery servers, when the time limit expires, the surbl client attempts to download the latest list.

Another good thing would be that using this information at the client end would be optional...

I wonder if it would be worth it though, or maybe more trouble than just doing two queries as now ?

Regards, Simon

Hi!

It's not the sort of thing that would need checking every lookup, probably once a day would be fine, and could be handled in a similar way to how the razor client keeps a note of the last time it downloaded the list of discovery servers, when the time limit expires, the surbl client attempts to download the latest list.

Another good thing would be that using this information at the client end would be optional...

I wonder if it would be worth it though, or maybe more trouble than just doing two queries as now ?

Just a idea, perhaps we can, if we give back 127.0.0.8 for example, it could mean lookup again, this is a country multi level. Then we could list them dynamicly. Only the code behind should know how to deal with that. I dont know if thats possible.

Bye, Raymond.

At 15:44 22/04/2004, Eric Kolve wrote:

...

...

And as far as caching goes, it shouldn't be a problem, because you just want to avoid doing a whole string of identical dns lookups - only cache identical lookups, and the caching should only last for one run of SA processing one message...(We assume that no new blacklist records

appear in

...

...

the middle of processing a specific message, or that if they do we don't care ;-)

Yes, it's not clear if "per test" means per message, but it would seem so. That too should be fine. I don't think we should worry too much about the boundary condition of the SURBL changing in the middle of message processing, which seems like it would be uncommon. Per-message caching is already a big help.

'per test' literally means per test. So if you run three eval:check_spamcop_uri_rbl tests, then you will have at most three queries for the same domain.

I would cache per message, but there isn't a good place to cache this data. I have access to PerMsgStatus, but its not a good idea to start shoving stuff into that hash as other code depends on its structure.

Per test is the right way to do it though isn't it ? So thats fine. The cached results of sc.surbl.org should have no bearing on ws.surbl.org...we're just trying to avoid literally identical dns queries, and spammer.com.sc.surbl.org and spammer.com.ws.surbl.org aren't identical.

But we don't want to see spammer.com.sc.surbl.org being queried 20 times for the same message ;-)

Regards, Simon