On Friday, July 23, 2004, 1:58:26 PM, Mike Atkinson wrote:
On 7/22/2004 at 6:05 PM, Jeff Chan wrote:
Should we decrease the TTL on the rbldnsd version of multi.surbl.org?
version of multi are the default for the entire file which is currently 28800 (8 hours).
Obviously that's quite a bit longer than the 90 minutes now used on ws or the 10 minutes for sc, so some multi/rbldnsd records
I could probably read the RFCs or source code, but does anyone know if DNS implementations cache negative hits to the default TTL. In other words if a new record gets added to a list, do caching name servers (that negatively cached it before, i.e., got queried and said "I don't have that") get the new record immediately or only after some TTL expires.
I think that BIND will cache negative answers for 3 hours from an authoritative name server without regard to any other TTL settings in the zone. It is possible to leave out the SOA record in the RBLDNSD zone and then BIND will see the remote name server as non-authoritative and will not cache the negative response. (Excerpts from man pages below.)
This means that if, for instance, my site is one of the earlier ones to get hit by a spam with a new URL that ends up in the SURBL; we are not going to see it as listed for up to 3 hours. I'm experimenting with 'max-ncache-ttl 3600;' in my named.conf and may drop that lower. 3600 doesn't seem to be a problem for the last several hours.
As far as the TTL for entries; I'm not overly concerned about lingering entries in multi.surbl.org (which is what we are using.) I think that 90-180 minutes would be a good range for that.
======
From 'man named.conf'
max-ncache-ttl
To reduce network traffic and increase performance the server stores negative answers. max-ncache-ttl is used to set a maximum retention time for these answers in the server is seconds. The default max-ncache-ttl is 10800 seconds (3 hours).
From 'man rbldnsd'
It is recommended, but not mandatory to specify SOA record for every zone. If no SOA is given, negative replies will not be cacheable by caching nameservers.
Thanks for the feedback Mike. We definitely want to keep negative caching active in both BIND and rbldnsd since most queries will probably give a negative result (i.e. not match a SURBL). I'm pretty sure we wouldn't want to lose the performance advantage of negative caching, even if it meant new (positive) matches were delayed, since negative answers are probably a majority. However we need to examine in a little more detail how negative caching is actually handled when a record that didn't match before now matches (i.e. is no longer negative).
Three possibilities are:
1. New (positive) entries in a given zone immediately override previous negative cache hits (previously negative for the same query).
2. The negative cache TTL takes precedence.
3. The positive caching TTL take precedence.
To test this, I watched a new domain get into to sc.surbl.org and saw it served up between 10 and 20 minutes after the zone file original modification time (after manually rsyncing the change to a rbldnsd server). sc has TTL and Refresh times of 10 minutes so this seems about right. So does that mean the negative cache time for a new positive match is constrained by the TTL default *zone file TTL* and not the default negative caching TTL? If that were the case we probably do want to lower the TTL on multi and probably the other constituent files like ob and ab, in order to reduce the delay in new entries becoming usable.
Does anyone have more info on this question?
By the way looking at some of the refresh and ttl times of other RBLs:
list.dsbl.org origin = a.list.ns.dsbl.org mail addr = admin.dsbl.org serial = 1080300617 refresh = 600 (10M) retry = 300 (5M) expire = 86400 (1D) minimum ttl = 600 (10M)
sbl.spamhaus.org origin = need.to.know.only mail addr = hostmaster.spamhaus.org serial = 2004032608 refresh = 3600 (1H) retry = 900 (15M) expire = 604800 (1W) minimum ttl = 300 (5M)
dnsbl.njabl.org origin = ns1.njabl.org mail addr = help.njabl.org serial = 1080298387 refresh = 10800 (3H) retry = 1800 (30M) expire = 720000 (1w1d8h) minimum ttl = 900 (15M)
spam.dnsrbl.net origin = ns1.namesystems.net mail addr = dns@namesystems.net serial = 2004031600 refresh = 3600 (1H) retry = 900 (15M) expire = 864000 (1w3d) minimum ttl = 3600 (1H)
relays.ordb.org origin = a.ns.ordb.org mail addr = hostmaster.ordb.org serial = 1080300600 refresh = 600 (10M) retry = 300 (5M) expire = 604800 (1W) minimum ttl = 1800 (30M)
Perhaps our TTLs should be shorter for all of our zones. sc is very short at 10 minutes ttl and refresh. Perhaps the others should be also.
Comments?
Jeff C.