On Saturday, February 12, 2005, 3:36:11 AM, Alain Alain wrote:
Hi Jeff
On Saturday, February 12, 2005, 2:34:20 AM, Alain Alain wrote:
Generally speaking it may be better to apply this kind of filtering at the server level since there are economies of scale, especially in terms of things like DNS lookups and caching. If we suddenly get 100k more DNS clients, that could tax the name servers somewhat. If those same 100k users were using 100 servers instead, the DNS loading would be quite a bit less. In that sense centralization is desirable.
Mmmm isn't the dns server from the ISP caching the dns requests? I would think it doesn't make a big difference (except when a server is rsync'ing). The difference could be that end users check their e-mail not when arriving on the MTA, but later.
One difference is that the ISP's mail server may see many of the same spams within a short period of time, and the lookups would probably tend to be cached over that time span. Individual users may POP or IMAP their messages at any random time, so the DNS cache hit rate may be lower for them.
This will only the case for spam e-mail, not for domains inside ham e-mail.
But most well-written applications, e.g. SpamAssassin, are already ignoring most ham URIs due to local whitelisting, so it's spam URI domain caching that's the main issue.
I think we're agreeing, but I've never tried to quantify the difference between these. We can propose that there's some difference but how much is unknown. I would suggest a pretty strong cache effect for mail servers however.
But the good news is : The more users, the more caching. So the burden on the nameservers will grow slower.
The SURBL zone files have a minimal 15 minute TTL, so in order for ISP resolver hits to be cached, the queries will need to occur within some 15 minutes, which seems less likely at MUA download time than at MTA processing time. MTAs probably see similar spam over a short period of time whereas MUA clients can download at any later time.
In this case, I don't think your argument applies. For something like caching yahoo domains, or any with "normal" longer TTLs, it probably applies more strongly.
Jeff C. -- "If it appears in hams, then don't list it."
Hi Jeff
On Saturday, February 12, 2005, 2:34:20 AM, Alain Alain wrote:
Generally speaking it may be better to apply this kind of filtering at the server level since there are economies of scale, especially in terms of things like DNS lookups and caching. If we suddenly get 100k more DNS clients, that could tax the name servers somewhat. If those same 100k users were using 100 servers instead, the DNS loading would be quite a bit less. In that sense centralization is desirable.
Mmmm isn't the dns server from the ISP caching the dns requests? I would think it doesn't make a big difference (except when a server is rsync'ing). The difference could be that end users check their e-mail not when arriving on the MTA, but later.
One difference is that the ISP's mail server may see many of the same spams within a short period of time, and the lookups would probably tend to be cached over that time span. Individual users may POP or IMAP their messages at any random time, so the DNS cache hit rate may be lower for them.
This will only the case for spam e-mail, not for domains inside ham e-mail.
But most well-written applications, e.g. SpamAssassin, are already ignoring most ham URIs due to local whitelisting, so it's spam URI domain caching that's the main issue.
I think we're agreeing, but I've never tried to quantify the difference between these. We can propose that there's some difference but how much is unknown. I would suggest a pretty strong cache effect for mail servers however.
But the good news is : The more users, the more caching. So the burden on the nameservers will grow slower.
The SURBL zone files have a minimal 15 minute TTL, so in order for ISP resolver hits to be cached, the queries will need to occur within some 15 minutes, which seems less likely at MUA download time than at MTA processing time. MTAs probably see similar spam over a short period of time whereas MUA clients can download at any later time.
In this case, I don't think your argument applies. For something like caching yahoo domains, or any with "normal" longer TTLs, it probably applies more strongly.
While not having experience with DNS TTL's, wouldn't it be possible to give individual ttl? For example start with 10 minutes and add one 1 minute for the days the entry is inside the list? (Recalculation can be done one's a day.), this is only usefull if older entries would have significantly less FP's. Given the very low FP's already, this is probably difficult to measure. Well just an idea and probably not needed for performance.
Alain
On Sunday, February 13, 2005, 8:56:32 AM, Alain wrote:
But the good news is : The more users, the more caching. So the burden on the nameservers will grow slower.
The SURBL zone files have a minimal 15 minute TTL, so in order for ISP resolver hits to be cached, the queries will need to occur within some 15 minutes, which seems less likely at MUA download time than at MTA processing time. MTAs probably see similar spam over a short period of time whereas MUA clients can download at any later time.
In this case, I don't think your argument applies. For something like caching yahoo domains, or any with "normal" longer TTLs, it probably applies more strongly.
While not having experience with DNS TTL's, wouldn't it be possible to give individual ttl? For example start with 10 minutes and add one 1 minute for the days the entry is inside the list? (Recalculation can be done one's a day.), this is only usefull if older entries would have significantly less FP's. Given the very low FP's already, this is probably difficult to measure. Well just an idea and probably not needed for performance.
We keep the TTLs low to both speed up the addition of new records to the list and speed up the deletion of old records. The 15 minute TTL was chosen because it experimentally maximized speed of these changes and minimized DNS server traffic. (It turns out that some other RBLs also settled on 15 minute TTLs as optimal.)
TTLs for normal domains would be more like 1 day.
Jeff C. -- "If it appears in hams, then don't list it."
-
Alain -
Jeff Chan