For the SARE and SURBL folks -
I would love to decrease the delay between when a spammy URL appears in the message stream and when it's submitted to the WS SURBL gods for blacklisting. Here is an idea that may have been discussed before; if so, just ignore it. :)
Currently, the SARE guys have some kind of rotating schedule of who checks the WS submissions, correct?
Wouldn't it be nice if there were some kind of similar volunteer-based rotating schedule that listed who would be responsible for monitoring the raw message stream and submitting unlisted spam domains? This kind of anonymous schedule handled through rulesemporium.com, and if you had some sort of accountability enforcement.
For instance, each submitter could log in with an account, so the validity of their WS SURBL submissions could be tracked by the SARE folks. If they're submitting a bunch of bogus domains or non-spam domains, their account would be disabled.
Each submitter could sign up for a 15-minute slice of time per week (dividing the week into 672 timeslots), and of course more than 1 submitter could take a timeslot, and each submitter could take more than one timeslot.
Each submitter's "record" (domains successfully submitted and blacklisted) would be *anonymously* available to the public through rulesemporium.com, as a form of reputation incentive.
Benefits:
1. The SURBL community would have the confidence that the message stream is being monitored by *someone* at all times.
2. Each individual member of the SURBL community would have a higher incentive to sacrifice some of each's time to submit spammed domains. The higher incentive is the knowledge the the community is (in a sense) depending on them to submit spammed domains within that certain period of time. The additional incentive to report would come from a decent assurance that it is far less likely that someone else is reporting the same domain, and hence it's less likely that any given submission would be a duplicate.
3. WS SURBL's reporting latency would hopefully decrease, because more people would be submitting.
My Motivation: What prompted me to write this was the fact that some of my customers were complaining that lots of spams were slipping through, so I spent about 30 minutes looking at the false negatives, and all of the domains but 1 of the ones I looked at were not yet listed in SURBL. So I submitted 10-15 domains, and thought to myself... I would do this more often and on a regular schedule if only I knew that others were also willing to sacrifice 15-30 minutes out of their week to the same cause......
Thanks for taking the time to read this suggestion.
Matthew Wilson matthew@boomer.com
On Wednesday, February 9, 2005, 6:16:45 PM, Matthew Wilson wrote: [...]
What prompted me to write this was the fact that some of my customers were complaining that lots of spams were slipping through, so I spent about 30 minutes looking at the false negatives, and all of the domains but 1 of the ones I looked at were not yet listed in SURBL. So I submitted 10-15 domains, and thought to myself... I would do this more often and on a regular schedule if only I knew that others were also willing to sacrifice 15-30 minutes out of their week to the same cause......
Hi Matthew, Thanks for your suggestions! I can't speak to the SARE side of things but if you (or your customers) can report these into SpamCop they may get into sc.surbl.org sooner.
Jeff C. -- "If it appears in hams, then don't list it."
Couldn't this be automated using other spam detection techniques? IE spamassassin detects 100% spam, URL not in SURBL. Spamassassin sends the email to a central repository and any URLs are parsed and added to SURBL.
Checksum clearing houses already do this.
John Delisle, CISA Senior Network Analyst, Network and Security Team Information Systems & Technology Management Dept. Ceridian Canada Ltd 600 - 125 Garry St Winnipeg, MB R3C 3P2 204-975-5909
"Matthew Wilson" matthew@boomer.com Sent by: discuss-bounces@lists.surbl.org 02/09/2005 08:16 PM Please respond to SURBL Discussion list discuss@lists.surbl.org
To "SURBL Discussion list" discuss@lists.surbl.org cc
Subject [SURBL-Discuss] Scheduled, Distributed Email Stream Monitoring
For the SARE and SURBL folks -
I would love to decrease the delay between when a spammy URL appears in the message stream and when it's submitted to the WS SURBL gods for blacklisting. Here is an idea that may have been discussed before; if so, just ignore it. :)
Currently, the SARE guys have some kind of rotating schedule of who checks the WS submissions, correct?
Wouldn't it be nice if there were some kind of similar volunteer-based rotating schedule that listed who would be responsible for monitoring the raw message stream and submitting unlisted spam domains? This kind of anonymous schedule handled through rulesemporium.com, and if you had some sort of accountability enforcement.
For instance, each submitter could log in with an account, so the validity of their WS SURBL submissions could be tracked by the SARE folks. If they're submitting a bunch of bogus domains or non-spam domains, their account would be disabled.
Each submitter could sign up for a 15-minute slice of time per week (dividing the week into 672 timeslots), and of course more than 1 submitter could take a timeslot, and each submitter could take more than one timeslot.
Each submitter's "record" (domains successfully submitted and blacklisted) would be *anonymously* available to the public through rulesemporium.com, as a form of reputation incentive.
Benefits:
1. The SURBL community would have the confidence that the message stream is being monitored by *someone* at all times.
2. Each individual member of the SURBL community would have a higher incentive to sacrifice some of each's time to submit spammed domains. The higher incentive is the knowledge the the community is (in a sense) depending on them to submit spammed domains within that certain period of time. The additional incentive to report would come from a decent assurance that it is far less likely that someone else is reporting the same domain, and hence it's less likely that any given submission would be a duplicate.
3. WS SURBL's reporting latency would hopefully decrease, because more people would be submitting.
My Motivation: What prompted me to write this was the fact that some of my customers were complaining that lots of spams were slipping through, so I spent about 30 minutes looking at the false negatives, and all of the domains but 1 of the ones I looked at were not yet listed in SURBL. So I submitted 10-15 domains, and thought to myself... I would do this more often and on a regular schedule if only I knew that others were also willing to sacrifice 15-30 minutes out of their week to the same cause......
Thanks for taking the time to read this suggestion.
Matthew Wilson matthew@boomer.com
_______________________________________________ Discuss mailing list Discuss@lists.surbl.org http://lists.surbl.org/mailman/listinfo/discuss
*********************************************************************************** This communication is intended to be received only by the individual(s) or entity(s) to whom or to which it is addressed, and contains information which is confidential, privileged and subject to copyright. Any unauthorized use, copying, review or disclosure is prohibited. Please notify the sender immediately if you have received this communication in error (by calling collect, if necessary) so that we can arrange for its return at our expense. Thank you in advance for your anticipated assistance and cooperation.
Ce document est destinee aux seuls destinataires designes, personne ou groupe. Cette telecopie contient des renseignements confidentiels et est regie par les lois sur le droit d'auteur. Toute utilisation, copie, revue ou divulgation non autorisee de la presente est formellement interdite. Si vous avez recu cette telecopie et qu'elle ne vous etait pas destinee, veuillez communiquer immediatement avec l'expediteur, a frais vires si necessaire, afin que nous puissions prendre des dispositions a nos frais pour la recuperer. Nous vous remercions a l'avance de votre cooperation diligente. ***********************************************************************************
On Thursday, February 10, 2005, 7:46:23 AM, John Delisle wrote:
Couldn't this be automated using other spam detection techniques? IE spamassassin detects 100% spam, URL not in SURBL. Spamassassin sends the email to a central repository and any URLs are parsed and added to SURBL.
It's very difficult to fully automate spam detection because not everyone agrees on what constitutes a spam. Certainly in some borderline cases, one person's spam may be another person's ham.
As a global list, we need to be very conservative about adding records so as not to create false positives (FPs). For that reason, we seek to add only hosts that are pretty much universally agreed to be spam.
Hopefully it's somewhat clear from the website that we have different sources of data:
http://www.surbl.org/lists.html
JP and OB are based on large spam traps. They're mostly automated but with some specific techniques for keeping out false positives. Outblaze for example only adds domains that are registered within the last 6 months and which are spewing a lot of spam recently. JP has an elaborate system for weeding out FPs before they get onto their list.
SC and AB are based on SpamCop reports. Both have inclusion thresholds so that only the most commonly reported spams get added. SpamCop reports have already been hand checked, though the quality of the checking and reporting varies, so in a sense they're multiply filtered before they get published as SC and AB. (I'm redoing the way the SC data is handled in a way that should be even better if I ever get around to doing it.)
WS is a manual list, meaning most of the entries are added by hand and human checked.
All the lists have FPs, some more than others. FPs are what prevent SURBLs from being used say at the MTA level in a telco, and it would be nice to eliminate FPs entirely. It's bad to have someone's ham marked as spam.
So yes, some parts of data collection can be automated, but quite a bit more engineering and thought needs to go
Jeff C. -- "If it appears in hams, then don't list it."
-
Jeff Chan -
John_Delisle＠ceridian.ca -
Matthew Wilson