Discuss

discuss@lists.surbl.org

1870 discussions

RE: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 12 Oct '04

12 Oct '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Saturday, October 09, 2004 9:06 PM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains >too > > >On Saturday, October 9, 2004, 1:09:43 PM, Patrik Nilsson wrote: > >> Just create a separate "TLDs or treat as TLDSs" zone that >can be checked >> and cached client side. > >> Or even better - give "TLDs or treat as TLDSs" a >distinguished A value in >> existing lists. >> If a lookup returns XXX.XXX.XXX.XXX, it is a "TLD or treat >as TLD" and >> should be further recursed. >> If we think there is a risk that some bad client >implementatins treat any >> returned A record as a hit, use TXT records. > >This is still an interesting idea, but I'd still be somewhat >concerned about putting out a list that looks like a regular >SURBL that it could get misused. > >But perhaps the larger issues is that the hard core spammers >don't seem to use *subdomains of legitimate shared-domain >hosting providers*. They just register their own full domain >names and use those (lots of them). > >If some legitimate hosting provider has an abuse issue, >then it's in their own interest to stop the abuse. > >SURBLs are arguably best suited for cases where the ISP >is spam-friendly and allows spam hosting on custom domains. >The reality is that's a much larger and tougher problem >than shared, common-domain hosting, like a geocities or >tripod. > >The best use of our time is to focus on the biggest spammers >first, and we're not catching all of those yet. > Big surprise...I disagree :-) I think we are already reviewing these domains, so why not just add to catch the smaller ones instead of throwing them away. But I see your vision, and I can bend like the reed ;) But given enough time, and perhaps enough chocolate, I think I can turn you around :-) --Chris

1 0

New data: SURBL DNS queries
by Jeff Chan 10 Oct '04

10 Oct '04

As has already been mentioned, Theo is patching SpamAssassin to locally whitelist some common whitehat URI domains for use in URIBL (which typically uses sbl and SURBL data) . This will prevent DNS queries on the whitehats and probably save some very significant traffic on the SURBL and spamhaus, etc. name servers. In order to get some better whitehat data, we increased the sampling of DNS queries on a name server from 32k (2k every 3 hours for 2 days) to 1.2 million (10k every 2 hours for 10 days). We're only about a third of the way through the initial 10 days, so the stats are still building up, but the current results are at: http://www.surbl.org/dns-queries.whitelist.counts.txt http://www.surbl.org/dns-queries.blocklist.counts.txt (These files have been mentioned before, but they're starting to get a lot more data behind them now.) Something else which was probably suggestion before, but which we *hadn't looked at before* were the DNS queries that *don't match* either our blocklists or whitelists. Those, sorted in order of decreasing frequency are: http://www.surbl.org/dns-queries.unmatched.30thpercentile.txt That's the top 30th percentile of them (about 3.6k records). The full list of unique domains and IPs with frequencies (about 110k records) is at: http://www.surbl.org/dns-queries.unmatched.count.txt Taking a look the top few of these: 333 56.227.117.38 211 internet.e 196 wwwlowmortnow.info 123 beliefnet.com 119 grisoft.com) 107 specialmax.net 99 democrats.org 99 115.14.249.209 96 and 90 c 82 centrport.net 78 charter.net 73 zdnet.com 65 cf.st 63 nuri1.net 62 red-hot1.com 62 imomentum.net 61 justsaywow.com 60 173.213.115.211 59 www 57 www.cool-loanco.kr 57 superduperfun.com 53 healthinsrus.com 51 e-directnet.net 51 agoramail.net 50 tmcs.net 50 latimes.com 50 dw.com.com 50 168.228.186.64 49 iscsimg.com 48 livedaily.com 48 eversave.com 47 1shoppingcart.com 46 srvimg.com 46 realone.com 46 goodnewsdelivery.com 45 rockbridgemedia.com 45 purdue.edu It's clear that a few are errors, probably due to problems in the applications using SURBLs. Yet it's probably useful to not suppress the errors so that the programs can be updated to handle them correctly. (Unfortunately the source URIs generating the errors are not directly available, but they may be identifiable in other ways if anyone would like to look for them.) Minus the errors, I fed this list into Ryan's GetURI to see what it could find. The results are at: http://ry.ca/cgi-bin/geturi.cgi?id=ham-5lCTzHkxan3xE38RKHa0vx Quite a few appear ok to whitelist, like democrats.org, pudue.edu, latimes.com, charter.net, zdnet.com, etc. and I'll probably go ahead and whitelist obvious ones like these, so some of these will probably be off this "unmatched" list and onto the whitelist hits by the time you read this. Nonetheless I recommend we all take a look at this unmatched list periodically, especially the top few dozen, to look for potential domains to whitelist or blacklist. These most frequently appearing domains are probably good candidates for one or the other. Since this is a list of the unknown "wild" domains coming from live, real-world message URIs, it may be another useful and different source of some data. Cheers, Jeff C. -- "If it appears in hams, then don't list it."

1 1

The plus side of FPs.
by Chris Santerre 09 Oct '04

09 Oct '04

We all hate FPs. We know we want to cut them down. But let me just say I have seen a nice plus to them. The whitelist requests for SURBL have been somewhat of an eye opener. There have been more then a few legit mailers who have drastically changed their ways since contacting us for whitelist. It is actually pretty amazing. They will 'fix' or at least try to 'fix' a lot of their messed up subscriptions just to get off of SURBL. Its all been polite and professional as well. We give examples of why they were added and ask for explanations. We also ask for a copy of their antispam/abuse policy. The responses have been outstanding. So we also have the power to educate. Yeah yeah yeah, I know. With great power comes great .....cheesecake. Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com http://www.surbl.org 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin

1 0

They found away around SURBL
by Chris Santerre 09 Oct '04

09 Oct '04

No link whatsoever. NO phone number either. No site mentioned. Nothing :) Maybe they just want to tease the reader :) Too bad this scored 15.1 points without surbl :) --Chris ----- Message-ID: <41665F37.67DCCE6F(a)siba.fi> From: Kent Bryan <kbryan_cd(a)siba.fi> To: Chris Santerre <csanterre(a)MerchantsOverseas.com> Subject: This worked for me, and will for you! Date: Fri, 8 Oct 2004 05:34:47 -0400 X-Mailer: Internet Mail Service (5.5.2653.19) ---------------------------------------------------------------------- - R.ated N.O.1 Pe.nis Enlargeme.nt P.ill on the Mar.ket! - - Gain up to 3+ full inches in length. - Increase your peni.s width by 20.%. - Stop Prema.ture Ejacu.lation. - Produce Strong.er and R.ock Ha.rd Erect.ions ----------------------------------------------------------------------

6 5

Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Frank Ellermann 09 Oct '04

09 Oct '04

Jeff Chan wrote: > Would there be many FNs (missed spams) if we whitelisted > all of these? In other words are these all truly False > Positives? > If not, which ones do you feel are true spammers and why. If WS and SC agree on something (or even more of the SURBLs) they might have a point. I tried to check kit.net (found in your list), and got an "unknown host" error. So maybe that was a FN (= spammer). I also tried to check made.com (on your original DMOZ list, it was listed in SC and AB IIRC), and came to the conclusion that this might be a Korean hoster. The subdomain listed in DMOZ looked pretty ordinary (at least the non-Flash part), no offers to buy something. But that could be misleading, I didn't visit the site with enabled JS. And of course the reason for the SC listing could be another subdomain, not the one listed by DMOZ. But made.com could be a FP (?) Hard to judge. More than one bit in multi is a hint. Bye.

1 0

Revised DMOZ data, got Wikipedia domains too
by Jeff Chan 09 Oct '04

09 Oct '04

Following up on the earlier check of DMOZ domains against SURBL data, I applied some of Quinlan's suggestions and grabbed different revisions of the DMOZ data and joined (intersected) them in order to eliminate changes due to things like editor- removed spammer/abuser domains, etc. This also means that new additions are ignored, but the corpus is so large that the benefits of capturing editor removals is probably more important. Snapshots are the three most recent available from 9/9/04, 9/25/04 and 10/7/04 (file dates) are intersected, resulting in fewer records (only those that are constant across all three snapshots): http://rdf.dmoz.org/rdf/archive/ Here are the line, word and character counts: 2300851 2300851 38065969 dmoz.srt 1169 11690 123909 dmoz-blocklist.summed.txt 1141 11410 120860 dmoz-blocklist.ws 1169 1169 17977 dmoz-blocklist.txt 7394 7394 97011 dmoz-whitelist.txt The above are revised versions of joins on blocklists, with list info, with ws hits, with just domains, and against the whitelist. These are in the whitelists directory, though we are still *not* applying the dmoz domains as whitelists: http://spamcheck.freeapp.net/whitelists/ The previous dmoz (9/25 version only IIRC) and hits are archived as: 2326173 2326173 38494184 dmoz.srt1 1338 13380 141946 dmoz-blocklist1.summed.txt 1173 11730 124298 dmoz-blocklist1.ws 1338 1338 20533 dmoz-blocklist1.txt 7375 7375 96720 dmoz-whitelist1.txt __ I was also able to grab 4 snapshots of the wikipedia, all sections (all languages). Only the two most recent snapshots had comparable numbers of sections so I used only those two. Using only two sets may be ok, since these are much smaller corpora and there's probably more hand-editing of them. http://download.wikimedia.org/ Where dmoz has about 2.3 million domains, wikipedia has about 174k domains: 173828 173828 2633441 wikipedia.srt 188 1880 19631 wikipedia-blocklist.summed.txt 188 188 2713 wikipedia-blocklist.txt 2437 2437 29581 wikipedia-whitelist.txt __ I also took the intersection of the three dmoz snapshots and the two wikipedia snapshots to get a smaller list containing only the ~102k domains found in both wikipedia and dmoz: 101619 101619 1498653 wikipedia-dmoz.srt 116 1160 11928 wikipedia-dmoz-blocklist.summed.txt 116 116 1591 wikipedia-dmoz-blocklist.txt 2223 2223 26854 wikipedia-dmoz-whitelist.txt This intersection of dmoz and wikipedia domains probably represents the best hope for large whitelist additions so far. This is probably imperfect data, but at least it has some checks by human editors and techniques applied to reduce spammer domains, including comparing the snapshots over time and intersecting the two relatively unrelated sources. Can anyone think of any other hand-edited databases, directories, encyclopedias, etc of URIs of hopefully legitimate (non-spammer) domains that are publically available? Please think about it a little, and speak up! While 102k domains isn't nearly as large as the 2.3M in dmoz, it's certainly more than the 12k or so whitelist records we currently have. How does the intersected list look as a potential whitelist? http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz.srt Please also take a look at these blocklist hits (potential FPs) and share what you think: http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz-blocklist.summed.txt Would there be many FNs (missed spams) if we whitelisted all of these? In other words are these all truly False Positives? If not, which ones do you feel are true spammers and why. Jeff C. -- "If it appears in hams, then don't list it."

4 9

RE: Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 09 Oct '04

09 Oct '04

First, great work Jeff. > >While 102k domains isn't nearly as large as the 2.3M in dmoz, >it's certainly more than the 12k or so whitelist records we >currently have. How does the intersected list look as a >potential whitelist? > > http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz.srt > I think this is just plain nuts to whitelist all of these! Why? If we don't try to whitelist the most popular sites, then what the heck it the point? We could whitelist millions of legit domains forever. The popular ones are the most important. Here is one from the above list. Why would listing this help us? http://oigawa-railway.co.jp/ (looks like a real popular site huh!) >Please also take a look at these blocklist hits (potential FPs) >and share what you think: > > http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz-blocklist.summed.txt I picked of few of these that may give us problems, and none of them met our current criteria to list. (sissy-world.com, good grief that had to be a man at one time!) With the ability to now see whitelisted domains in the crossref page, I don't see a problem with whitelisting all these on the list. Because if they do start spamming again, we can see they are whitelisted and remove them. so: -1 for adding all those intersected to WL +1 for whitelisting the blacklist hits. --Chris

3 4

servepath is pissing me off
by Chris Santerre 09 Oct '04

09 Oct '04

they host these idiots and a bunch more! theviralspiral.com I think servepath.com sould be listed. They host obvious spam houses. And they don't host many domains, so to say they didn't know this was a spammer is a complete joke. Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com http://www.surbl.org 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin

3 2

RE: [SURBL-Discuss] RE: Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 08 Oct '04

08 Oct '04

>-----Original Message----- >From: Bill Landry [mailto:billl@pointshare.com] >Sent: Friday, October 08, 2004 3:06 PM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] RE: Revised DMOZ data, got Wikipedia >domains too > > >----- Original Message ----- >From: "Chris Santerre" <csanterre(a)merchantsoverseas.com> > >> >> I think this is just plain nuts to whitelist all of >these! Why? If we >> >> don't try to whitelist the most popular sites, then what >the heck it >> >> the point? We could whitelist millions of legit domains >forever. The >> >> popular ones are the most important. >> > >> >The points: >> > >> > - whitelisting legitimate domains limits the >effectiveness of joe job >> > attacks that result in FPs in various SURBL blacklists >> > - whitelisting could be used as negative points for MAIL FROM if >> > combined with SPF (and more domains is better) >> >> Yeah, but not everyone is using SPF yet. But if they were, sure! >> > >> >In addition: >> > >> > - I would only whitelist those domains (a) subject to editorial >> > removal (b) so long as their domain registration is old >enough and >> > (c) so long as they pass other criteria such as no SBL >listing for >> > NS->A. >> >> Yeah date seems to be key in more and more cases. Granted a >spammer could >> buy one of these older ones, but hasn't happened often enough. >> >> > - I would maintain the automated whitelist separately >from the human >> > edited whitelist and handle it differently. For >example, perhaps >> > automated whitelist entries can only remove a single >blacklist hit >> > (like SpamCop), but to remove two independent blacklist hits, it >> > requires a human decision. >> > >> >> Did you look at the example from the list I gave? It doesn't >even have a >web >> page! Just says testing. I'm all for whitelisting, but popular/useful >> domains only. >> >> >> so: >> >> -1 for adding all those intersected to WL >> >> +1 for whitelisting the blacklist hits. >> > >> >I think there are other options available due to the miracle of >> >programming. ;-) >> >> Well like they say around here, "You can't argue with success." >> >> But taking away spam points based on an autowhitelist still makes me >> nervous. But you might have a few tricks up your sleeve D.Q. >that I don't >> know about yet ;) > >If the whitelist is setup as a SURWL, for example, using it or >not would be >an option to each individual mail administrator. And the >negative weight >applied to reduce the score would be totally dependant on the >discretion of >the SURWL user. > True, and having the option is good. Having the work overhead to do that on top of doing the blacklists is not. And the logic stands if it ain't blacklisted, it must be good ;) --Chris

2 1

RE: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 08 Oct '04

08 Oct '04

>-----Original Message----- >From: Bill Landry [mailto:billl@pointshare.com] >Sent: Friday, October 08, 2004 3:26 PM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains >too > > >----- Original Message ----- >From: "Ryan Thompson" <ryan(a)sasknow.com> > >> >> I wouldn't appreciate it being whitelisted as then if >there's abuse, >> >> and it does get blacklisted, there's no pressure on the >domain holder >> >> to clean up. >> > >> > I'm hoping UC list will help with that. Anyone who wishes to use it >> > can add a minor score to spam for UC hits. This would put at least >> > some sort of preasure for these grey hats to do something. >> >> Ezzactly. Even if it's worth 0.1 points in SA (or >something), the stigma >> of having one's domain listed in some sort of bad "list" >might be enough >> to promote action, provided the accuracy of UC is quite >good. (I.e., we >> definitely still don't want to list everything :-) "Quite >good" in that >> case probably means < 1% FP, but I think we can do even >better than that >> with a few clever tricks. >> >> Personally, I'm hoping UC will be worth more than 0.1 even with a >> somewhat higher FP rate, since there will be little or no >overlap with >> other SURBL tests, but that'll depend on some mass-checks. :-) > >I have been tracking my hits from UC, and although the hit >rate if fairly >low compared to most of the SURBL lists, it has been giving me >good results >and hitting things I would consider spam. And as such, I have the test >currently weighted at 0.5. > >Bill LOL, thats cause UC just ain't that big yet. I keep trying to keep up with all the domains Jeff keeps whitelisting, but he's like some sort of mad robot on overdrive ;) --Chris (Good grief I've been hungry all day!)

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss