Below are some stats from our incoming mail since midnight 23/04/05. I'm not going to go into too deep an analysis as it's the weekend and I'm too 'tired' to do that ;).
Total nbr of messages with at least one ??_SURBL hit over the last 22 hours was around 1.4 million. The counts below show how many triggered as 'spam' ("result: Y"), and how many didn't trigger ("result: ."). This is based on our Spam Assassin default threshold of 8, but we have many custom rules so spam threshold is really only meaningful to our config - YMMV ;). We don't currently block or tag via SURBL/RBL at the MTA layer - everything goes through SA.
XS popped up reasonably frequently in all SURBL hits - and we had 10456 XS spam hits where it wasn't listed in any other URIBL (unable to say how many false-positives out of that list, but our default tagging threshold is high in SA and FP's are vanishingly small).
Anyway - make of it as you will. SURBL rocks anyway - XS *may* be a useful addition.
Cheers
Paul
On Saturday, April 23, 2005, 3:42:57 PM, Paul Shields wrote:
Below are some stats from our incoming mail since midnight 23/04/05. I'm not going to go into too deep an analysis as it's the weekend and I'm too 'tired' to do that ;).
Total nbr of messages with at least one ??_SURBL hit over the last 22 hours was around 1.4 million. The counts below show how many triggered as 'spam' ("result: Y"), and how many didn't trigger ("result: ."). This is based on our Spam Assassin default threshold of 8, but we have many custom rules so spam threshold is really only meaningful to our config - YMMV ;). We don't currently block or tag via SURBL/RBL at the MTA layer - everything goes through SA.
XS popped up reasonably frequently in all SURBL hits - and we had 10456 XS spam hits where it wasn't listed in any other URIBL (unable to say how many false-positives out of that list, but our default tagging threshold is high in SA and FP's are vanishingly small).
Anyway - make of it as you will. SURBL rocks anyway - XS *may* be a useful addition.
Cheers
Paul
Hi Paul,
Thanks very much for sharing your data. Your results look about as should be expected for the other lists in terms of FPs and spam detection. Summarizing your numbers:
AB: 521886 spam 604 ham WS: 996200 spam 12578 ham JP: 1234602 spam 4376 ham OB: 1139181 spam 36760 ham SC: 751549 spam 1095 ham PH: 383 spam 1 ham
XS: 939134 spam 6283 ham XS unique: 10456 spam 5300 ham
For XS it looks like the Spam to Ham ratio is only about 2:1 which means it has too many FPs, and doesn't hit much unique spam, which is also reasonable given the lack of significant legitimate domain filtering and high inclusion threshold. We will work to improve those much further before we propose adding it to the production data in multi.
In terms of ratios of the current lists, OB is underperforming the others, judging by your data. I'm ccing Suresh at Outblaze so he can see the measurements you got.
All the lists need to hit less ham, and more aggressive checking and whitelisting is probably needed, assuming the data sources don't change their inclusion policies. I hope to address this in future.
Jeff C. -- "If it appears in hams, then don't list it."
----- Original Message ----- From: "Jeff Chan" jeffc@surbl.org <>
Hi Paul,
Thanks very much for sharing your data. Your results look about as should be expected for the other lists in terms of FPs and spam detection. Summarizing your numbers:
AB: 521886 spam 604 ham WS: 996200 spam 12578 ham JP: 1234602 spam 4376 ham OB: 1139181 spam 36760 ham SC: 751549 spam 1095 ham PH: 383 spam 1 ham
XS: 939134 spam 6283 ham XS unique: 10456 spam 5300 ham
For XS it looks like the Spam to Ham ratio is only about 2:1 which means it has too many FPs, and doesn't hit much unique spam, which is also reasonable given the lack of significant legitimate domain filtering and high inclusion threshold. We will work to improve those much further before we propose adding it to the production data in multi.
In terms of ratios of the current lists, OB is underperforming the others, judging by your data. I'm ccing Suresh at Outblaze so he can see the measurements you got.
All the lists need to hit less ham, and more aggressive checking and whitelisting is probably needed, assuming the data sources don't change their inclusion policies. I hope to address this in future.
Jeff C.
Happy to help Jeff.
Don't forget though that we have many custom SA rulesets, and thresholds that can be applied per mailbox, so my ham to spam ratio is not necessarily going to reflect a 'stock' install (this is ham or spam as defined by a variable threshold, not by someone actually identifying the message!).
Paul
On Sunday, April 24, 2005, 12:40:21 AM, Paul Shields wrote:
Don't forget though that we have many custom SA rulesets, and thresholds that can be applied per mailbox, so my ham to spam ratio is not necessarily going to reflect a 'stock' install (this is ham or spam as defined by a variable threshold, not by someone actually identifying the message!).
Paul
Aha, good to know. Still, in aggregate it's interesting data, and the differences between datasets look approximately as expected.
Cheers,
Jeff C. -- "If it appears in hams, then don't list it."
Hi Paul
On 4/24/05, Paul Shields paul.shields-at-blueyonder.co.uk |surbl list| <...> wrote:
Below are some stats from our incoming mail since midnight 23/04/05. I'm not going to go into too deep an analysis as it's the weekend and I'm too 'tired' to do that ;).
Total nbr of messages with at least one ??_SURBL hit over the last 22 hours was around 1.4 million. The counts below show how many triggered as 'spam' ("result: Y"), and how many didn't trigger ("result: ."). This is based on our Spam Assassin default threshold of 8, but we have many custom rules so spam threshold is really only meaningful to our config - YMMV ;). We don't currently block or tag via SURBL/RBL at the MTA layer - everything goes through SA.
Wow, that's very usefull information. Thanks an awefull lot.
I have placed the nr's in a small table (also based on Jeff's table) :
not tagged means a hit, without being marked as spam by SA. It doesn't mean that those e-mails aren't spam, just that those pass the -conservative- filter.
Spam Not tagged %not tagged % spam total AB 521886 604 0,116% 37,28% WS 996200 12578 1,247% 71,16% JP 1234602 4376 0,353% 88,19% OB 1139181 36760 3,126% 81,37% SC 751549 1095 0,145% 53,68% PH 383 1 0,260% 0,03% XS 939134 6283 0,665% 67,08% XS-unique 10456 5300 33,638% 0,75%
Looking at the percentages I do see that JP (and SC) are good predictors for a e-mail being spam and thus very usable if only a few checks can be made (for example if scoring isn't an option).
Comparing XS with WS and OB it's clear that XS is a better predictor than those two lists...
However that doesn't mean they have more FP's, it's possible that WS, OB and XS do catch many spams that pass the other lists and so pass through this SA setup. In this case those lists would be very usefull.
Looking at XS-unique I do wonder how much the other lists catch "unique" and how many of those unique hits to pass the filter, it's possible there's a big overlap between the lists (partly seen at http://www.surbl.org/permuted-hits.out.txt).
It's possible that XS is so much faster than the other links that it's catching very new spam that's still passing the other lists.
Some remarks :
- While the total nr of e-mails is quite high, it's based on just a day, maybe one or two big spamruns are skewing the results.
- It would also very usefull to have info about the unique results of the other lists and combinations of them.
BTW. As written before a large corpus of domains that have occured several times over a periode of time on not-tagged e-mails, could be a very effective first filter to avoid automatic inclusion. The nice thing is that this info doesn't need to be very new, I would even ignore the last week... A domain that's did occur on several seperate days in the past on several not-tagged e-mails is probably hammy. Of course a manual inclusion inside the blacklist should be possible. This info is probably more usefull than the creation day of the domain and probably easier to make (just a lookup i a internal list).
Alain
-
Alain -
Jeff Chan -
Paul Shields