On Tuesday, August 3, 2004, 11:32:26 AM, Patrik Nilsson wrote:

At 02:50 2004-08-03 -0700, Jeff Chan wrote:

...

I think that would be done by lowering the TTL. The time to live appears to indicated to DNS how quickly new information should be served up. Do any DNS gurus know if that's correct? In other words if we lower the TTLs on our zone files should we expect new entries to be visible sooner.

If you are using version 9 of Bind, the positive TTL is set by the $TTL directive for the zone, unless there is a specific TTL for the RR, in which case that is used.

Right, and since rbldnsd doesn't support per record TTLs, we need to set global defaults for the zone. (Note that many of our public name servers are using BIND 8. We're trying to get everyone switched over to rbldnsd.)

For the BIND versions of multi, we are setting per record TTLs based on the original lists the entries come from.

But in general I'm thinking we probably need to lower all TTLs to get entries active on the lists sooner.

Negative TTLs (for NXDOMAIN/"not found" replies) are set by the "minimum" SOA directive.

This "negative caching time" use of the miminum directive is specific to Bind 9 though. Earlier versions of Bind and most other name servers treat it as a default miminum TTL in general.

For rbldnsd, it might be possible to do something similar using the soa ttl value for negative and the $ttl directive for positive replies, but I am not sure.

That sounds right, and BIND 9 and rbldnsd both seem to agree in that regard. Both allow setting of positive and negative caching times through the overall $TTL directive and the minimum TTL in the SOA respectively.

...

...

Finally, has any progress been made speeding up the refresh times for multi.surbl.org?

It's certainly something that can be done rather easily but I'd like to get some feedback about the impact on our nameservers as a result. Do shorter TTLs mean more DNS traffic? Does it cause positive caches to expire sooner and therefore cause more querying of authoritative name servers?

At least in theory, with Bind 9, negative and positive caching can be set separately.

Try lowering the 8H 'minimum' value for multi.surbl.org while keeping the 8H RR TTL or zone $TTL value.

What we're going to end up doing is tuning the positive and negative caching values to minimize name server load and activation time for new SURBL records. I'll do that once I have my name server changed over to rbldnsd, though I'd welcome the help of anyone else currently running rbldnsd.

(I don't want to change the production TTL values that everyone is using yet. We should test some new values on individual name servers first.)

Here's what Michael Tokarev, author of rbldnsd, suggested:

There are only 2 parameters to tune: positive TTL (for every individual listed record) and negative TTL for all non-existing records. There's no ability to return different (smaller) negative TTL for a record that is about to be listed. The only option you have is to adjust your negative TTL to find a sort of balance between number of queries to your NSes and amount of time NXDOMAIN may be cached (aka "false negative lifetime"). There's no other choice, logically.

By playing with positive TTLs, you're not affecting your "false negative lifetime" (a time between entering an entry into DB and when it becomes "active" after corresponding NXDOMAIN expires from caches) in any way. Only changing negative TTL affects this time, and you're free to change it while it does not hit your NSes very hard. Just play with negative (and positive as well) TTLs for a while (set it to 3h for one day, set it to 1h for another day and so on and watch your nameservers) - this way you will figure out how exactly your NSes affects by the changes. Usually, short TTLs causes very heavy DNS traffic, but after some value, increasing TTL affects the DNS load very slightly, like this:

load ^ | + | + | + | + | + | +-----=---------------> TTL \deadline, approx

Not sure what he means by deadline, but it's probably meant to be some optimal point on the graph that minimizes both values. That point probably needs to be determined through experimentation.

When tuning the TTLs, just choose numbers (by "try and see" method which is easiest to perform) that will generate acceptable load on nameservers while still having acceptable "response time" (false positives and negatives). For a relatively static data (like SBL for example), a TTL of several hours (1..3, maybe even 6) seems to be acceptable. I don't know how it applies to surbl data - that is, how much spam a spammer able to send using new domain while it will be expiring from ob.surbl.org caches.

Anyone who can help with tuning TTLs in this way is welcomed to try it and let us know what it optimal. Michael recognized that our datasets are different from typical RBLs since we mostly have domain names and different querying patterns.

So the results of IP Address RBL TTL tuning may not apply directly to our case.

Jeff C.