In order to explore some conjunctions of existing lists that
might have fewer false positives, we've created some stats
measuring the number of DNS hits against all blocklists that the
individual lists get, along with some permutations of those lists.
For completeness, we've added checking of AB and PH as individual
(not permuted) lists, and the output can be found at:
http://www.surbl.org/permuted-hits.out.txt
[sc][ws][ob][jp] 762 records of 82592 67115 hits of 232463 is 28%
[sc][ws][ob] 857 records of 82592 67325 hits of 232463 is 28%
[sc][ws][jp] 899 records of 82592 70622 hits of 232463 is 30%
[sc][ws] 1066 records of 82592 71725 hits of 232463 is 30%
[sc][ob][jp] 788 records of 82592 69526 hits of 232463 is 29%
[sc][ob] 916 records of 82592 70292 hits of 232463 is 30%
[sc][jp] 934 records of 82592 73050 hits of 232463 is 31%
[sc] 1193 records of 82592 75597 hits of 232463 is 32%
[ws][ob][jp] 16383 records of 82592 144989 hits of 232463 is 62%
[ws][ob] 21793 records of 82592 150159 hits of 232463 is 64%
[ws][jp] 33123 records of 82592 186633 hits of 232463 is 80%
[ws] 58471 records of 82592 209710 hits of 232463 is 90%
[ob][jp] 17145 records of 82592 150595 hits of 232463 is 64%
[ob] 44636 records of 82592 168053 hits of 232463 is 72%
[jp] 34669 records of 82592 196112 hits of 232463 is 84%
[ab] 368 records of 82592 61920 hits of 232463 is 26%
[ph] 996 records of 82592 307 hits of 232463 is 0%
The records columns show the size of the lists or intersections.
The hits columns shows how many DNS queries out of all blocklist
hits those lists or intersections.
This is run nightly around midnight using the script:
http://www.surbl.org/permuted-hits
This gives some measure of the performance of the different lists,
though it likely undercounts rapidly changing data since it's
based on the previous ten days of data. The more quickly
changing lists like AB and SC have higher detection rates in
actual, real-time operation. The stats above also do not take
into account false positives at all, just hits against existing
blocklists (which do have some FPs).
Additionally we've increased the number of days that frequency
data of DNS queries against our whitelist are kept from 10 to 90.
However it will take another 80 days before we get 90 days
accumulated:
http://www.surbl.org/dns-queries.whitelist.counts.txt
I re-wrote the scripts to accommodate the additional data more
efficiently:
http://www.surbl.org/hourly-dnshttp://www.surbl.org/daily-dns (run around midnight)
After things stabilize we will probably change from the current
10,000 DNS queries sampled every 2 hours to 20,000 sampled every
hour. This will make the results represent about a half million
queries per day. The larger sample sizes should make the results
more accurate, but it means absolute numbers from the past won't
be comparable. Percentages and relative rankings should always
be comparable though.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
As of 12 November 2004, we have added data from
fraud.rhs.mailpolice.com into ph, joining our exiting phishing
data from mailsecurity.net.au. This has doubled the size of
our phishing list to about 1000 records. Unlike other SURBLs,
data from this fraud list includes a few deliberate subdomains
as found in URIs. (Because SURBL clients are expected to reduce
subdomains to base domains, an occasional mismatch in domain
levels between data and client should not cause false positives.)
Thanks to Jay Swackhamer of MailPolice for gathering this data
and making it available to us.
At the time of this message, overlap between
fraud.rhs.mailpolice.com and mailsecurity.net.au's phishing list
was 36 records. Overlap between fraud.rhs.mailpolice.com and
all other existing SURBLs, including ph.surbl.org, is 89 records.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
We'd like to welcome and thank the addition of a new public
SURBL name server g3.surbl.org administered by:
Alex Broens of Apexis - Switzerland
Without our public nameservers and the help of their
administrators, SURBLs would not be possible.
Our thanks to all of them!
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
We'd like to welcome and thank the addition of a new public
SURBL name server a3.surbl.org administered by:
Frank Precissi of InReach Internet
Without our public nameservers and the help of their
administrators, SURBLs would not be possible.
Our thanks to all of them!
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Just a reminder that since Qmail-Scanner supports SpamAssassin,
it also supports using SURBLs with the Qmail mail server.
http://qmail-scanner.sourceforge.net/
Also supports anti-virus, etc.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
http://www.surbl.org/quickstart.html
Important Note Regarding SpamAssassin 3.0.1 and later: When
adding any URIDNSBL rules, including SURBL or SBL ones using
urirhsbl, urirhssub or uridnsbl, be sure to set the rule type to
"body". For example:
urirhssub URIBL_JP_SURBL multi.surbl.org. A 64
body URIBL_JP_SURBL eval:check_uridnsbl('URIBL_JP_SURBL')
describe URIBL_JP_SURBL Contains a URL listed in the JP SURBL blocklist
tflags URIBL_JP_SURBL net
score URIBL_JP_SURBL 4.0
This is a change from earlier versions of SpamAssassin, including
3.0.0, where "body" above was previously "header". Here is the
changelog reference:
r54022 | felicity | 2004-10-07 22:21:30 +0000 (Thu, 07 Oct 2004) | 1 line
bug 3734: uridnsbl rules work on body data, not header data, so
change the rule type from header to body
If you had SpamAssassin 3.0.0 SURBL or uridnsbl rules before, you
will need to edit them with this change or they will no longer
work in 3.0.1 or later!
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
The SURBL Quick Start page has been updated to reflect that
SpamAssassin 3 has default support for SURBLs, to take out
mention of the various lists except JP which is not in some
of the default configs yet, and to generally streamline it.
There are other minor updates to the rest of the site also.
http://www.surbl.org/
Comments welcome,
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
We'd like to welcome and thank the addition of three new public
SURBL name servers e3, h3, and l3.surbl.org respectively
administered by:
Wayne Yeung of Mailshell
Kristopher Austin of Oklahoma Christian University
Tan Chao Ching and Josh Duan of Pacific Internet Ltd
Without all of our public nameservers and the help of their
administrators, SURBLs would not be possible.
Our thanks to all of them!
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/