[SURBL-Discuss] FP rate?

[Jeff Chan]

On Saturday, February 12, 2005, 3:51:53 AM, Alain Alain wrote:
> Hi Jeff

>> >> > I know that not all FP's are reported and there are
>> >> > probably no exact numbers, but it should give a good idea.  Or am I
>> >> > wrong?
>> >>
>> >> The FP reports are probably too few overall to be meaningful in
>> >> terms of differentiating performance between lists.  There just
>> >> aren't that many, maybe a few a day on average.
>> >>
>> 
>> > Yes, but I wasn't thinking on differentiating between the lists, there
>> > are other results for.  What I was thinking on was the number of FP's
>> > that exists on more than one list.  This is very usefull information
>> > when combining lists.  If almost no FP's do occur on more than one
>> > list (at the same time)  requiring appearance on at least 2 lists
>> > would be a very safe one.
>> 
>> Good point.  Anecdotally, FPs don't tend to appear on multiple
>> lists very often, at least the FPs we've seen reported.  This is
>> unmeasured, just a subjective opinion.  If we had some of the
>> list data in combined form as I had proposed then we could test
>> it better.  I suppose I could just do it.  ;-)

> I f the reported one's are very rare, this would probably even more
> the case for the not reported one's.  If there's a FP the chance for
> being reported will grow if on more than one list.

Probably the most common SURBL application, SpamAssassin, uses
all the lists (though with different weights on each one) so we
can assume that they're most often getting checked together, but
not hitting FPs together too often.

FPs in general are sporadic, occasional errors that seem to pop
up at random on different lists for different reasons.  They're
in the "noise" of the data and hard to classify.  If they were
easier to classify, we'd have an easier time eliminating them
in a more formal way.  We're apparently already operating at the
boundaries of what can be classified in this way.

That all said, I appreciate your interest in helping to achieve
lower FP rates, a goal which we should all share.

> Mmm the combined lists just have to be available to someone with a big
> ham corpus, to test it.

> Personaly knowing the results for "at least 2" or "at least 3" , would
> be nice.  It also would be nice to know how those combination would
> result inside :
> http://www.surbl.org/permuted-hits.out.txt

Yes, the SA or SARE folks, for example, could probably write some
test rules to see if combinations of SURBL lists would work better.
Or I could combine them on the data side and set up some test
lists.

Jeff C.
--
"If it appears in hams, then don't list it."