Here are some good comments from Dave Funk about the handling/creation of the SURBLs. Please comment on his suggestions, several of which we may want to implement as time permits.
Jeff C. __
On Tue, 20 Apr 2004, Jeff Chan wrote:
On Tuesday, April 20, 2004, 1:20:05 PM, Charles Gregory wrote:
Would it be possible to have 'surbl.org' run a *combined* blacklist, so that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can do it with ONE dns lookup request, instead of two?
Good question, which Matt asks also. Here's my response :-)
[snip..]
Because ws is larger and more stable, the zone files for it gets a six hour TTL compared to 10 minutes for sc. Due to the differences between the time scales, sizes, and data sources of ws and sc, we probably won't be offering a combined ws plus sc list. For example it would be difficult to say what TTL a merged list should get, and you probably would not want a megabyte plus BIND zone file refreshing every 10 minutes. For those using rsynced zone files that would probably not be an issue, but for those using BIND, the DNS traffic quite well could be.
So the quick answer is they'll probably not be combined.
However we probably will offer a combined version of Bill's list and Chris' BigEvil list since they are more similar in character.
A few comments. 1) It is possible to set a TTL in a DNS zone on a per-record basis. (at least with BIND). So you could combine the two zones and have the 'sc' records flagged with a short TTL, and 'ws' with longer. 2) Newer versions of BIND support incremental zone-transfer, and so will just push changes. 3) We also secondary MAPS RBL+ zone, that's a 54Mbyte zone that updates every 3 hours. (IE 18Mbyte/hour). A 1Mbyte x 10 minutes would be only 6Mbytes/hour, chicken feed. ;) 4) Over half the size of those zones is in the TXT records. Just changing 'Message body contains domain in sa-blacklist. See: http://www.stearns.org/sa-blacklist/' to 'Blocked, See: http://www.stearns.org/sa-blacklist/' reduced the 'ws' zone size by 33% 5) It's possible to combine the zones but keep the data logically seperate so people can differentiate and adjust scores/policys accordingly. Check out how MAPS does RBL+, the A record returns an "IP address" that is effectivly a bit-mask flag to indicate which MAPS zone the original hit was from (DUL, RSS, RBL, OPS, etc). Look at how the 'check_rbl' and 'check_rbl_sub' routines are used inside SA to pull apart a single DNS query against RBL+ (at least in SA 2.6*, havn't looked at 3.0 yet ;)
This is not to imply criticism if your response, just some tech info to show alternatives.
Regardless, I would recommend using 5) when you combine Bill's list and Chris' BigEvil so that people can differentiate in case they have score/policy concerns regarding the two. People who just look for the existence of the A record won't notice the difference but people who know and care can utilize the additional info.
Dave
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jeff Chan writes:
Here are some good comments from Dave Funk about the handling/creation of the SURBLs. Please comment on his suggestions, several of which we may want to implement as time permits.
FWIW, we support multi-meaning DNSBL results with TXT records as well as A records; just ensure that the TXT result includes a short string we can match on (e.g. "ws" results contain the string "/sa-blacklist" and "sc" results contain something else similarly well-defined.)
- --j.
Jeff C. __
On Tue, 20 Apr 2004, Jeff Chan wrote:
On Tuesday, April 20, 2004, 1:20:05 PM, Charles Gregory wrote:
Would it be possible to have 'surbl.org' run a *combined* blacklist, so that people who want to check both 'ws.surbl.org' *and* 'sc.surbl.org' can do it with ONE dns lookup request, instead of two?
Good question, which Matt asks also. Here's my response :-)
[snip..]
Because ws is larger and more stable, the zone files for it gets a six hour TTL compared to 10 minutes for sc. Due to the differences between the time scales, sizes, and data sources of ws and sc, we probably won't be offering a combined ws plus sc list. For example it would be difficult to say what TTL a merged list should get, and you probably would not want a megabyte plus BIND zone file refreshing every 10 minutes. For those using rsynced zone files that would probably not be an issue, but for those using BIND, the DNS traffic quite well could be.
So the quick answer is they'll probably not be combined.
However we probably will offer a combined version of Bill's list and Chris' BigEvil list since they are more similar in character.
A few comments.
- It is possible to set a TTL in a DNS zone on a per-record basis.
(at least with BIND). So you could combine the two zones and have the 'sc' records flagged with a short TTL, and 'ws' with longer. 2) Newer versions of BIND support incremental zone-transfer, and so will just push changes. 3) We also secondary MAPS RBL+ zone, that's a 54Mbyte zone that updates every 3 hours. (IE 18Mbyte/hour). A 1Mbyte x 10 minutes would be only 6Mbytes/hour, chicken feed. ;) 4) Over half the size of those zones is in the TXT records. Just changing 'Message body contains domain in sa-blacklist. See: http://www.stearns.org/sa-blacklist/' to 'Blocked, See: http://www.stearns.org/sa-blacklist/' reduced the 'ws' zone size by 33% 5) It's possible to combine the zones but keep the data logically seperate so people can differentiate and adjust scores/policys accordingly. Check out how MAPS does RBL+, the A record returns an "IP address" that is effectivly a bit-mask flag to indicate which MAPS zone the original hit was from (DUL, RSS, RBL, OPS, etc). Look at how the 'check_rbl' and 'check_rbl_sub' routines are used inside SA to pull apart a single DNS query against RBL+ (at least in SA 2.6*, havn't looked at 3.0 yet ;)
This is not to imply criticism if your response, just some tech info to show alternatives.
Regardless, I would recommend using 5) when you combine Bill's list and Chris' BigEvil so that people can differentiate in case they have score/policy concerns regarding the two. People who just look for the existence of the A record won't notice the difference but people who know and care can utilize the additional info.
Dave
Dave Funk University of Iowa <dbfunk (at) engineering.uiowa.edu> College of Engineering 319/335-5751 FAX: 319/384-0549 1256 Seamans Center Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527 #include <std_disclaimer.h> Better is not better, 'standard' is better. B{ -- Jeff Chan mailto:jeffc@surbl.org-nospam http://www.surbl.org/
Discuss mailing list Discuss@lists.surbl.org http://lists.surbl.org/mailman/listinfo/discuss
On Tuesday, April 20, 2004, 5:48:15 PM, Justin Mason wrote:
Jeff Chan writes:
Here are some good comments from Dave Funk about the handling/creation of the SURBLs. Please comment on his suggestions, several of which we may want to implement as time permits.
FWIW, we support multi-meaning DNSBL results with TXT records as well as A records; just ensure that the TXT result includes a short string we can match on (e.g. "ws" results contain the string "/sa-blacklist" and "sc" results contain something else similarly well-defined.)
Sounds great. If we combine lists, we'll make sure they have something unique in the TXT records to key what source they came from.
It may be worth mentioning that SA 3.0 URIBL may have the ability to do a single query on multiple RBLs, if I read the SA-dev messages correctly.
Jeff C.
Good evening, Jeff, all,
On Tue, 20 Apr 2004, Jeff Chan wrote:
Here are some good comments from Dave Funk about the handling/creation of the SURBLs. Please comment on his suggestions, several of which we may want to implement as time permits.
A few comments.
- It is possible to set a TTL in a DNS zone on a per-record basis. (at least with BIND). So you could combine the two zones and have the 'sc' records flagged with a short TTL, and 'ws' with longer.
Agreed, just placed the TTL on the individual record line.
- Newer versions of BIND support incremental zone-transfer, and so will just push changes.
Ah, cool, didn't know about that.
- We also secondary MAPS RBL+ zone, that's a 54Mbyte zone that updates every 3 hours. (IE 18Mbyte/hour). A 1Mbyte x 10 minutes would be only 6Mbytes/hour, chicken feed. ;)
It all comes down to the bandwidth available Jeff at the primary.
- Over half the size of those zones is in the TXT records. Just changing 'Message body contains domain in sa-blacklist. See: http://www.stearns.org/sa-blacklist/' to 'Blocked, See: http://www.stearns.org/sa-blacklist/' reduced the 'ws' zone size by 33%
Works for me! Jeff, feel free to make that change anytime. Would it even make sense to have a single .txt record with the full notice, and have all the rest be cnames to it? It'll be rarely used, so it's hardly a performance problem to have to go back and get the cname data.
- It's possible to combine the zones but keep the data logically seperate so people can differentiate and adjust scores/policys accordingly. Check out how MAPS does RBL+, the A record returns an "IP address" that is effectivly a bit-mask flag to indicate which MAPS zone the original hit was from (DUL, RSS, RBL, OPS, etc). Look at how the 'check_rbl' and 'check_rbl_sub' routines are used inside SA to pull apart a single DNS query against RBL+ (at least in SA 2.6*, havn't looked at 3.0 yet ;)
No experience with this, so no opinion. Thanks for the ideas, Dave. Jeff, enough people have asked for the combined list that I'm game to set up an "all.surbl.org" combined list if you are. It really sounds like the technical concerns are all handleable. We can still keep the sc and ws subdomains for those that think my taste in domains is questionable... :-) Cheers, - Bill
--------------------------------------------------------------------------- "Not only is UNIX dead, it's starting to smell bad." -- Rob Pike (?) (Courtesy of Mike Castle dalgoda@ix.netcom.com) -------------------------------------------------------------------------- William Stearns (wstearns@pobox.com). Mason, Buildkernel, freedups, p0f, rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org --------------------------------------------------------------------------
[FWIW Looks like mailman is *not* stamping a list reply-to as I set in the configs....]
On Tuesday, April 20, 2004, 5:51:18 PM, William Stearns wrote:
- Over half the size of those zones is in the TXT records. Just changing 'Message body contains domain in sa-blacklist. See: http://www.stearns.org/sa-blacklist/' to 'Blocked, See: http://www.stearns.org/sa-blacklist/' reduced the 'ws' zone size by 33%
Works for me! Jeff, feel free to make that change anytime.
Done. Hope the change doesn't break anyone... Hopefully they're using the A record first, and the TXT for a comment, if at all. :-)
(The default/sample SA 2.63 SpamCopURI and 3.0 urirhsbl rules seem to write their own text descriptions based on the A record, so I think we're ok. Other folks could be using the TXT record however. I'll announce the change and hope we catch them.
I may shorten the sc.surbl.org TXT message also....)
- It's possible to combine the zones but keep the data logically seperate so people can differentiate and adjust scores/policys accordingly. Check out how MAPS does RBL+, the A record returns an "IP address" that is effectivly a bit-mask flag to indicate which MAPS zone the original hit was from (DUL, RSS, RBL, OPS, etc). Look at how the 'check_rbl' and 'check_rbl_sub' routines are used inside SA to pull apart a single DNS query against RBL+ (at least in SA 2.6*, havn't looked at 3.0 yet ;)
No experience with this, so no opinion. Thanks for the ideas, Dave. Jeff, enough people have asked forthe combined list that I'm game to set up an "all.surbl.org" combined list if you are. It really sounds like the technical concerns are all handleable. We can still keep the sc and ws subdomains for those that think my taste in domains is questionable... :-)
We could combine into larger lists with:
1. Different TTLs per record (essentially as now, through the default zone file $TTL) 2. Different A records, i.e. 127.0.0.2 for sc, `127.0.0.3 for ws, etc. 3. Different TXT messsages (as now)
Sounds like keeping all the A records at 127.0.0.2, making the different TTLs and keeping the current TXTs indicating the data sources, while combining all the records into larger lists could work. (I.e. #1, #3 but not #2.)
Comments?
Jeff C.
On Tue, 20 Apr 2004, William Stearns wrote:
- We also secondary MAPS RBL+ zone, that's a 54Mbyte zone that updates every 3 hours. (IE 18Mbyte/hour). A 1Mbyte x 10 minutes would be only 6Mbytes/hour, chicken feed. ;)
It all comes down to the bandwidth available Jeff at the primary.
However if you structure your secondaries in a 'calling tree' format, (IE Jeff feeds 2 secondaries, those 2 feed 4 more, etc), then his bandwidth requirements are minimized.
In the secondary named.conf, you list a superior secondary first in the 'masters' record (as well as other alternate sources). Then it will try the various alteratives in the order listed. This reduces Jeff's bandwidth demands and makes the system more robust (alternative paths) at the cost of a bit more complexity and slight delay in update propogation.
Jeff Chan wrote:
However we probably will offer a combined version of Bill's list and Chris' BigEvil list since they are more similar in character.
BigEvil as a URI RBL would be awesome, as it's a really heavy regex right now. What can I do to expedite implimentation of be.surbl.org?
- We also secondary MAPS RBL+ zone, that's a 54Mbyte zone that updates every 3 hours. (IE 18Mbyte/hour). A 1Mbyte x 10 minutes would be only 6Mbytes/hour, chicken feed. ;)
How long does it take BIND9 to load that thing? :-)
David
-
David B Funk -
David Coulson -
Jeff Chan -
jm＠jmason.org -
William Stearns