On Monday, February 13, 2006, 1:38:39 PM, Matthew Eerde wrote:
Jeff Chan wrote:
I'd be a little concerned about the new DNS queries it would generate.
If load on the servers is a concern there are things that can be done:
- Cache lookups for the duration of the session
- Only look up the main document URL, not src= URLs
- Allow a URL whitelist... bookmarked URLs
- Skip lookups for HTTP servers on the local subnet or on My Computer
All of those should be considered. Caching helps a lot and is highly recommended.
The whitelist that SpamAssassin took a snapshot of is a reasonable starting point:
http://spamassassin.apache.org/full/3.1.x/dist/rules/25_uribl.cf
IIRC it covered about half the whitelist hits at the time. IOW not checking those sites reduced by about 50% the checks of mostly whitehat sites; a desirable result in reducing unnecessary nameserver queries.
Many web pages have a w3.org URI as a reference for html, etc. There's no point in checking it many billions of times per day for example.
Cheers,
Jeff C. -- Don't harm innocent bystanders.