Hi Jeff
I know that not all FP's are reported and there are probably no exact numbers, but it should give a good idea. Or am I wrong?
The FP reports are probably too few overall to be meaningful in terms of differentiating performance between lists. There just aren't that many, maybe a few a day on average.
Yes, but I wasn't thinking on differentiating between the lists, there are other results for. What I was thinking on was the number of FP's that exists on more than one list. This is very usefull information when combining lists. If almost no FP's do occur on more than one list (at the same time) requiring appearance on at least 2 lists would be a very safe one.
Good point. Anecdotally, FPs don't tend to appear on multiple lists very often, at least the FPs we've seen reported. This is unmeasured, just a subjective opinion. If we had some of the list data in combined form as I had proposed then we could test it better. I suppose I could just do it. ;-)
I f the reported one's are very rare, this would probably even more the case for the not reported one's. If there's a FP the chance for being reported will grow if on more than one list.
Mmm the combined lists just have to be available to someone with a big ham corpus, to test it.
Personaly knowing the results for "at least 2" or "at least 3" , would be nice. It also would be nice to know how those combination would result inside : http://www.surbl.org/permuted-hits.out.txt
Alain
Alain:
I like your brainstorming because you just might come up with an idea or two that I haven't thought of before... so don't let me slow you down too much....
However, do note that there are some limitations to the "if on more than one list, probably not an FP" philosophy.
This philosophy works great if the potential FP was due to "stupid human error" from one guy regarding one list. However, there are a number of scenarios where a mass mailing spam campaign may trigger a URI to get listed on multiple SURBL lists, even if that particular URI is found in ham. Of course, we do all that we can to minimize this possibility... and I believe that we are doing a great job and we are continually getting better...
But, again, there are diminishing returns to the "if found in multiple SURBL lists, less change of FP" idea. This is true to an extent, but **not** always true.
Rob McEwen
Hi Rob
Alain:
I like your brainstorming because you just might come up with an idea or two that I haven't thought of before... so don't let me slow you down too much....
Well reading a year list msg's gives some idea's ;-) (Well I did skip most of the reported FP uri's msg's).
However, do note that there are some limitations to the "if on more than one list, probably not an FP" philosophy.
I'm well aware of that. At the moment I'm mostly asking for info about it. It could be that taken "at least 2" or "at least 3" catches a very high % of the spammsg's (% against all lists "OR'ed"), so that this combination is still performing very well. Given the low nr of FP's on the seperate lists, even decreasing this a little bit would give a big boost.
This philosophy works great if the potential FP was due to "stupid human error" from one guy regarding one list. However, there are a number of scenarios where a mass mailing spam campaign may trigger a URI to get listed on multiple SURBL lists, even if that particular URI is found in ham. Of course, we do all that we can to minimize this possibility... and I believe that we are doing a great job and we are continually getting better...
I think you're already doing a very good job weeding out FP's. I'm aware that there could be conditions where a FP goes into more than one list. I'm also confident that those FP's are reported faster and thus solved faster too.
But, again, there are diminishing returns to the "if found in multiple SURBL lists, less change of FP" idea. This is true to an extent, but **not** always true.
Given that it's for me rather easy to implement a "scoring" combination from the different lists and that this is easy to configure. (I suspect most end-users will understand that and that it's easy to "publish" new recommended weigth's.) I think "at least 2" could be the default, without generating FP's (or almost none). The main filtering app that I write a plugin for (spampal) has nice whitelisting, but this needs a few weeks use before being really active.
PS. I hope I was clear, it's getting late here.
Alain
On Saturday, February 12, 2005, 3:54:42 PM, Alain Alain wrote:
Given that it's for me rather easy to implement a "scoring" combination from the different lists and that this is easy to configure. (I suspect most end-users will understand that and that it's easy to "publish" new recommended weigth's.) I think "at least 2" could be the default, without generating FP's (or almost none). The main filtering app that I write a plugin for (spampal) has nice whitelisting, but this needs a few weeks use before being really active.
If you're looking for weights to apply to different lists, you may want to consider starting with the ones SpamAssassin uses.
Regarding your general question of reducing FPs by conjoining list, I agree that could be useful but haven't had a chance to try it yet.
Jeff C. -- "If it appears in hams, then don't list it."
Hi Jeff
Given that it's for me rather easy to implement a "scoring" combination from the different lists and that this is easy to configure. (I suspect most end-users will understand that and that it's easy to "publish" new recommended weigth's.) I think "at least 2" could be the default, without generating FP's (or almost none). The main filtering app that I write a plugin for (spampal) has nice whitelisting, but this needs a few weeks use before being really active.
If you're looking for weights to apply to different lists, you may want to consider starting with the ones SpamAssassin uses.
Thanks, I've looked at them but they are calculated quite complex and depend on other SA options. I did use them as a start point however, as also the numbers from 2005-01-26. I'm curious how ws will do after the JP removal.
Regarding your general question of reducing FPs by conjoining list, I agree that could be useful but haven't had a chance to try it yet.
Well I've made a new beta for the urlbody spampal plugin which uses "at least two" and hope to get some results from testers. Unfortunately there's no "good" summary report...
Alain
On Saturday, February 12, 2005, 3:51:53 AM, Alain Alain wrote:
Hi Jeff
I know that not all FP's are reported and there are probably no exact numbers, but it should give a good idea. Or am I wrong?
The FP reports are probably too few overall to be meaningful in terms of differentiating performance between lists. There just aren't that many, maybe a few a day on average.
Yes, but I wasn't thinking on differentiating between the lists, there are other results for. What I was thinking on was the number of FP's that exists on more than one list. This is very usefull information when combining lists. If almost no FP's do occur on more than one list (at the same time) requiring appearance on at least 2 lists would be a very safe one.
Good point. Anecdotally, FPs don't tend to appear on multiple lists very often, at least the FPs we've seen reported. This is unmeasured, just a subjective opinion. If we had some of the list data in combined form as I had proposed then we could test it better. I suppose I could just do it. ;-)
I f the reported one's are very rare, this would probably even more the case for the not reported one's. If there's a FP the chance for being reported will grow if on more than one list.
Probably the most common SURBL application, SpamAssassin, uses all the lists (though with different weights on each one) so we can assume that they're most often getting checked together, but not hitting FPs together too often.
FPs in general are sporadic, occasional errors that seem to pop up at random on different lists for different reasons. They're in the "noise" of the data and hard to classify. If they were easier to classify, we'd have an easier time eliminating them in a more formal way. We're apparently already operating at the boundaries of what can be classified in this way.
That all said, I appreciate your interest in helping to achieve lower FP rates, a goal which we should all share.
Mmm the combined lists just have to be available to someone with a big ham corpus, to test it.
Personaly knowing the results for "at least 2" or "at least 3" , would be nice. It also would be nice to know how those combination would result inside : http://www.surbl.org/permuted-hits.out.txt
Yes, the SA or SARE folks, for example, could probably write some test rules to see if combinations of SURBL lists would work better. Or I could combine them on the data side and set up some test lists.
Jeff C. -- "If it appears in hams, then don't list it."
-
Alain -
Jeff Chan -
Rob McEwen