Raymond Dijkxhoorn said:
In a era where you talk about gigabytes ram? Talking about this only costs more ;) eh eh ...
Excellent point. And I apologize in advance that this has now veered away from SURBL. (originally, I mistakenly thought this file size bloat might have been an SURBL problem... but it turns out there is NOT really any technical problems)
...but while we are "here"...
On my server, I now have a large combination of various rsync subscriptions to various RBLs and DNSBLs that I use in my RBLDNSD. The problem is that my RBLDNSD program now uses a whopping 500MB of RAM. (again, not SURBL's fault... this is a combination of many different lists)
So far, that isn't enough RAM to cause problems... but I noticed some sluggishness on my server today and, it turns out that sometimes, when RBLDNSD is "refreshing" its data, it re-builds an entire extra copy of the data in memory, along side the original copy. And this rebuilding can sometimes take several minutes to complete. Apparently, when done, it kills the old copy and start using the new copy. (again... I don't know why it has to rebuild the whole thing... but, then again, I don't know a whole lot about the inner programming of RBLDNSD)
When this gigabyte of RAM is being used, I actually see a marked performance decline on my 2GB or Ram server. I know that one could argue that I ought to get more ram for my server... but I think instead I'll program a utility to periodically strip all these comments out of the records and point RBLDNSD to the stripped down versions. I'd be curious as to what the RBLDNSD performance and memory usage ends up being!
Also, **getting back to SURBL** ...if you were to offer a "multi" file with comments stripped out, wouldn't this save bandwidth and rsync server resources if folks used that instead of the current offering?
Rob McEwen PowerView Systems rob@PowerViewSystems.com (478) 475-9032
On Wednesday, January 10, 2007, 7:14:08 PM, Rob Systems) wrote:
On my server, I now have a large combination of various rsync subscriptions to various RBLs and DNSBLs that I use in my RBLDNSD. The problem is that my RBLDNSD program now uses a whopping 500MB of RAM.
Many people set up a seprate server for rbldnsd. It can be an old Celeron or Pentium running Linux or BSD.
So far, that isn't enough RAM to cause problems... but I noticed some sluggishness on my server today and, it turns out that sometimes, when RBLDNSD is "refreshing" its data, it re-builds an entire extra copy of the data in memory, along side the original copy. And this rebuilding can sometimes take several minutes to complete. Apparently, when done, it kills the old copy and start using the new copy. (again... I don't know why it has to rebuild the whole thing... but, then again, I don't know a whole lot about the inner programming of RBLDNSD)
Yes, this has been mentioned on the rbldnsd discussion list. It can also be read in the source code.
http://www.corpit.ru/mjt/rbldnsd.html http://www.corpit.ru/mailman/listinfo/rbldnsd
When this gigabyte of RAM is being used, I actually see a marked performance decline on my 2GB or Ram server. I know that one could argue that I ought to get more ram for my server... but I think instead I'll program a utility to periodically strip all these comments out of the records and point RBLDNSD to the stripped down versions. I'd be curious as to what the RBLDNSD performance and memory usage ends up being!
Also, **getting back to SURBL** ...if you were to offer a "multi" file with comments stripped out, wouldn't this save bandwidth and rsync server resources if folks used that instead of the current offering?
You can easily strip it out the text field with sed, awk, cut, perl, grep, etc. Remove the text before you load the zone into rbldnsd, for example as a filter in your rsync cron job (scheduler).
Jeff C. -- Don't harm innocent bystanders.