Should we decrease the TTL on the rbldnsd version of multi.surbl.org?
For some background on the issue, rbldnsd does not support setting individual TTLs per record for dnset type zones, so we could not set individual TTLs on the rbldnsd records in multi.surbl.org, so the TTL for all records in the rbldnsd version of multi are the default for the entire file which is currently 28800 (8 hours).
Obviously that's quite a bit longer than the 90 minutes now used on ws or the 10 minutes for sc, so some multi/rbldnsd records may be longer-lived than their BIND versions. This is in conflict with the fact that we can and do set individual TTLs per record in the *BIND* version of multi. So a particular record in BIND multi versus rbldnsd multi may very likely have different TTLs, which is an asymmetry and somewhat inelegant. In other words a server that uses BIND may not give the same TTL for a given entry that an rbldnsd one would.
One solution to this would be to drop the default rbldnsd TTL for multi to something short like 10 minutes to force them all to expire quickly. Does anyone have any thoughts about this? Would this result in a lot more DNS traffic in general?
I could probably read the RFCs or source code, but does anyone know if DNS implementations cache negative hits to the default TTL. In other words if a new record gets added to a list, do caching name servers (that negatively cached it before, i.e., got queried and said "I don't have that") get the new record immediately or only after some TTL expires.
Jeff C.
On 7/22/2004 at 6:05 PM, Jeff Chan wrote:
Should we decrease the TTL on the rbldnsd version of multi.surbl.org?
version of multi are the default for the entire file which is currently 28800 (8 hours).
Obviously that's quite a bit longer than the 90 minutes now used on ws or the 10 minutes for sc, so some multi/rbldnsd records
I could probably read the RFCs or source code, but does anyone know if DNS implementations cache negative hits to the default TTL. In other words if a new record gets added to a list, do caching name servers (that negatively cached it before, i.e., got queried and said "I don't have that") get the new record immediately or only after some TTL expires.
I think that BIND will cache negative answers for 3 hours from an authoritative name server without regard to any other TTL settings in the zone. It is possible to leave out the SOA record in the RBLDNSD zone and then BIND will see the remote name server as non-authoritative and will not cache the negative response. (Excerpts from man pages below.)
This means that if, for instance, my site is one of the earlier ones to get hit by a spam with a new URL that ends up in the SURBL; we are not going to see it as listed for up to 3 hours. I'm experimenting with 'max-ncache-ttl 3600;' in my named.conf and may drop that lower. 3600 doesn't seem to be a problem for the last several hours.
As far as the TTL for entries; I'm not overly concerned about lingering entries in multi.surbl.org (which is what we are using.) I think that 90-180 minutes would be a good range for that.
======
From 'man named.conf'
max-ncache-ttl
To reduce network traffic and increase performance the server stores negative answers. max-ncache-ttl is used to set a maximum retention time for these answers in the server is seconds. The default max-ncache-ttl is 10800 seconds (3 hours).
---
From 'man rbldnsd'
It is recommended, but not mandatory to specify SOA record for every zone. If no SOA is given, negative replies will not be cacheable by caching nameservers.
On Friday, July 23, 2004, 1:58:26 PM, Mike Atkinson wrote:
On 7/22/2004 at 6:05 PM, Jeff Chan wrote:
Should we decrease the TTL on the rbldnsd version of multi.surbl.org?
version of multi are the default for the entire file which is currently 28800 (8 hours).
Obviously that's quite a bit longer than the 90 minutes now used on ws or the 10 minutes for sc, so some multi/rbldnsd records
I could probably read the RFCs or source code, but does anyone know if DNS implementations cache negative hits to the default TTL. In other words if a new record gets added to a list, do caching name servers (that negatively cached it before, i.e., got queried and said "I don't have that") get the new record immediately or only after some TTL expires.
I think that BIND will cache negative answers for 3 hours from an authoritative name server without regard to any other TTL settings in the zone. It is possible to leave out the SOA record in the RBLDNSD zone and then BIND will see the remote name server as non-authoritative and will not cache the negative response. (Excerpts from man pages below.)
This means that if, for instance, my site is one of the earlier ones to get hit by a spam with a new URL that ends up in the SURBL; we are not going to see it as listed for up to 3 hours. I'm experimenting with 'max-ncache-ttl 3600;' in my named.conf and may drop that lower. 3600 doesn't seem to be a problem for the last several hours.
As far as the TTL for entries; I'm not overly concerned about lingering entries in multi.surbl.org (which is what we are using.) I think that 90-180 minutes would be a good range for that.
======
From 'man named.conf'
max-ncache-ttl
To reduce network traffic and increase performance the server stores negative answers. max-ncache-ttl is used to set a maximum retention time for these answers in the server is seconds. The default max-ncache-ttl is 10800 seconds (3 hours).
From 'man rbldnsd'
It is recommended, but not mandatory to specify SOA record for every zone. If no SOA is given, negative replies will not be cacheable by caching nameservers.
Thanks for the feedback Mike. We definitely want to keep negative caching active in both BIND and rbldnsd since most queries will probably give a negative result (i.e. not match a SURBL). I'm pretty sure we wouldn't want to lose the performance advantage of negative caching, even if it meant new (positive) matches were delayed, since negative answers are probably a majority. However we need to examine in a little more detail how negative caching is actually handled when a record that didn't match before now matches (i.e. is no longer negative).
Three possibilities are:
1. New (positive) entries in a given zone immediately override previous negative cache hits (previously negative for the same query).
2. The negative cache TTL takes precedence.
3. The positive caching TTL take precedence.
To test this, I watched a new domain get into to sc.surbl.org and saw it served up between 10 and 20 minutes after the zone file original modification time (after manually rsyncing the change to a rbldnsd server). sc has TTL and Refresh times of 10 minutes so this seems about right. So does that mean the negative cache time for a new positive match is constrained by the TTL default *zone file TTL* and not the default negative caching TTL? If that were the case we probably do want to lower the TTL on multi and probably the other constituent files like ob and ab, in order to reduce the delay in new entries becoming usable.
Does anyone have more info on this question?
By the way looking at some of the refresh and ttl times of other RBLs:
list.dsbl.org origin = a.list.ns.dsbl.org mail addr = admin.dsbl.org serial = 1080300617 refresh = 600 (10M) retry = 300 (5M) expire = 86400 (1D) minimum ttl = 600 (10M)
sbl.spamhaus.org origin = need.to.know.only mail addr = hostmaster.spamhaus.org serial = 2004032608 refresh = 3600 (1H) retry = 900 (15M) expire = 604800 (1W) minimum ttl = 300 (5M)
dnsbl.njabl.org origin = ns1.njabl.org mail addr = help.njabl.org serial = 1080298387 refresh = 10800 (3H) retry = 1800 (30M) expire = 720000 (1w1d8h) minimum ttl = 900 (15M)
spam.dnsrbl.net origin = ns1.namesystems.net mail addr = dns@namesystems.net serial = 2004031600 refresh = 3600 (1H) retry = 900 (15M) expire = 864000 (1w3d) minimum ttl = 3600 (1H)
relays.ordb.org origin = a.ns.ordb.org mail addr = hostmaster.ordb.org serial = 1080300600 refresh = 600 (10M) retry = 300 (5M) expire = 604800 (1W) minimum ttl = 1800 (30M)
Perhaps our TTLs should be shorter for all of our zones. sc is very short at 10 minutes ttl and refresh. Perhaps the others should be also.
Comments?
Jeff C.
-
Jeff Chan -
Mike Atkinson