[SURBL-Discuss] Why is multi's rsync file so large?

[Jeff Chan]

On Wednesday, January 10, 2007, 7:14:08 PM, Rob Systems) wrote:
> On my server, I now have a large combination of various rsync
> subscriptions to various RBLs and DNSBLs that I use in my
> RBLDNSD. The problem is that my RBLDNSD program now uses a
> whopping 500MB of 
> RAM.

Many people set up a seprate server for rbldnsd.  It can be an
old Celeron or Pentium running Linux or BSD.

> So far, that isn't enough RAM to cause problems... but I
> noticed some sluggishness on my server today and, it turns out
> that sometimes, when RBLDNSD is "refreshing" its data, it
> re-builds an entire   
> extra copy of the data in memory, along side the original copy.
> And this rebuilding can sometimes take several minutes to
> complete. Apparently, when done, it kills the old copy and
> start using the   
> new copy. (again... I don't know why it has to rebuild the
> whole thing... but, then again, I don't know a whole lot about
> the inner programming of RBLDNSD)  

Yes, this has been mentioned on the rbldnsd discussion list.  It
can also be read in the source code.

  http://www.corpit.ru/mjt/rbldnsd.html
  http://www.corpit.ru/mailman/listinfo/rbldnsd

> When this gigabyte of RAM is being used, I actually see a
> marked performance decline on my 2GB or Ram server. I know that
> one could argue that I ought to get more ram for my server...
> but I think   
> instead I'll program a utility to periodically strip all these
> comments out of the records and point RBLDNSD to the stripped
> down versions. I'd be curious as to what the RBLDNSD
> performance and   
> memory usage ends up being!

> Also, **getting back to SURBL** ...if you were to offer a
> "multi" file with comments stripped out, wouldn't this save
> bandwidth and rsync server resources if folks used that instead
> of the current   
> offering?

You can easily strip it out the text field with sed, awk, cut,
perl, grep, etc.   Remove the text before you load the zone into
rbldnsd, for example as a filter in your rsync cron job (scheduler).

Jeff C.
--
Don't harm innocent bystanders.