[SURBL-Discuss] Why is multi's rsync file so large?

Rob McEwen (PowerView Systems) rob at powerviewsystems.com
Thu Jan 11 04:14:08 CET 2007


Raymond Dijkxhoorn said:

> In a era where you talk about gigabytes ram? Talking about this only costs 
> more ;) eh eh ...

Excellent point. And I apologize in advance that this has now veered away from SURBL. (originally, I mistakenly thought this file size bloat might have been an SURBL problem... but it turns out there is NOT really any technical problems)

...but while we are "here"...

On my server, I now have a large combination of various rsync subscriptions to various RBLs and DNSBLs that I use in my RBLDNSD. The problem is that my RBLDNSD program now uses a whopping 500MB of RAM. (again, not SURBL's fault... this is a combination of many different lists)

So far, that isn't enough RAM to cause problems... but I noticed some sluggishness on my server today and, it turns out that sometimes, when RBLDNSD is "refreshing" its data, it re-builds an entire extra copy of the data in memory, along side the original copy. And this rebuilding can sometimes take several minutes to complete. Apparently, when done, it kills the old copy and start using the new copy. (again... I don't know why it has to rebuild the whole thing... but, then again, I don't know a whole lot about the inner programming of RBLDNSD)

When this gigabyte of RAM is being used, I actually see a marked performance decline on my 2GB or Ram server. I know that one could argue that I ought to get more ram for my server... but I think instead I'll program a utility to periodically strip all these comments out of the records and point RBLDNSD to the stripped down versions. I'd be curious as to what the RBLDNSD performance and memory usage ends up being!

Also, **getting back to SURBL** ...if you were to offer a "multi" file with comments stripped out, wouldn't this save bandwidth and rsync server resources if folks used that instead of the current offering?

Rob McEwen
PowerView Systems
rob at PowerViewSystems.com
(478) 475-9032




More information about the Discuss mailing list