- Discuss - Mailman 3

RE: [SURBL-Discuss] Child porn URLs
by Chris Santerre 15 Oct '04

15 Oct '04

>I must agree. The only content criteria we have for SURBLs is >inclusion in spam and exclusion in ham. Aside from the phishing >list (which also happens to be very spammy), all the SURBL lists >contain spam domains. Spam versus ham should remain our only >criteria for inclusion or now. > Well, I think this info should be told to everyone who is an admin. You all know I'm interested in this because I send the info direct to the FBI SA contact I have. The official place to report such sites is www.missingkids.com Also the site mentioned simply redirects to http://218.189.199.18:2180/portal/portal/pbf/ If you have any more info on this site, let me know OFFLIST, and I will submit to the Special Agent. Thanks --Chris

1 0

RE: [ST #592239]:postmaster: Re: [SURBL-Discuss] FP in OB ?
by Chris Santerre 15 Oct '04

15 Oct '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Friday, October 15, 2004 7:58 AM >To: Tony Bea via RT >Cc: SURBL Discuss >Subject: Re: [ST #592239]:postmaster: Re: [SURBL-Discuss] FP in OB ? > > >On Friday, October 15, 2004, 4:26:09 AM, Tony RT wrote: >> Thank you for your suggestions. I will look to implement, >at least a few of >> them, in the near future in hopes of stopping some of the FPs. > >Execellent! I hope they can help. (Referring to >http://www.surbl.org/policy.html ) > >> In the meantime, feel free to let the mailing list know that >> postmaster(a)outblaze.com works and gets reasonably fast >response if they notice >> anything that looks like an FP. > >> Cheers, >> TonyB > >Done. :-) > Without a doubt, those Outblaze guys are aces in my book! Keep up the great work guys! --Chris (I am the queen of spades.)

1 0

FP: vmobile.us
by Joe Wein 15 Oct '04

15 Oct '04

Got email from vmobile.us today. It's listed on [WS] and [JP]. I checked the evidence and it's kind of flakey. The mail in question was mailing by wirelessdealernetwork.com. WDN has 2 NANAS sightings to it, both dated 2004-09-19 and posted by the same guy. Registered on 2004-06-08, NS blacklisted, listed on SURBL [WS], [OB], [JP]. Whether the recipient opted-in or not is not quite clear, but he's telecom-related. It may be spam, it may be not. In any case, vmobile.us was registered on 2004-05-12 and blacklisted by me on 2004-09-14. I shouldn't have blacklisted at that age without further evidence / checks. The NS is not blacklisted, no Google hits for spam other than my own listing. Oops! The domain of the owning company, usa-telecom.net, is two years old. So I think [WS] should probably also unlist vmobile.us, as I will do. My lesson: Just because the mailing list may be spammy and the TLD notorious doesn't mean a domain mentioned deserves listing on SURBL. Joe

2 3

Re: [ST #592239]:postmaster: Re: [SURBL-Discuss] FP in OB ?
by Jeff Chan 15 Oct '04

15 Oct '04

On Friday, October 15, 2004, 4:26:09 AM, Tony RT wrote: > Thank you for your suggestions. I will look to implement, at least a few of > them, in the near future in hopes of stopping some of the FPs. Execellent! I hope they can help. (Referring to http://www.surbl.org/policy.html ) > In the meantime, feel free to let the mailing list know that > postmaster(a)outblaze.com works and gets reasonably fast response if they notice > anything that looks like an FP. > Cheers, > TonyB Done. :-) Jeff C. -- "If it appears in hams, then don't list it."

1 0

Re: [ST #592239]:postmaster: Re: [SURBL-Discuss] FP in OB ?
by Jeff Chan 15 Oct '04

15 Oct '04

[forwarding my reply to Tony at Outblaze with his permission] On Thursday, October 14, 2004, 8:34:41 PM, Tony RT wrote: > [jeffc(a)surbl.org - Thu Oct 14 14:00:25 2004]: >> Thanks Tony. >> >> May I suggest that you consider checking a domain before >> listing it? Just because a few customers consider it spam >> doesn't necessarily mean other customers might not want to >> get it. I ask because there seem to be some legitimate >> sites getting onto your lists which some customers may >> legitimately want to get. For example none of the recent >> FPs have had to do with pills, mortgages, warez, etc. >> >> Another recent example is browsehappy.com run by the Web >> Standards Project: >> >> http://webstandards.org/act/campaign/happy/ >> >> which seem pretty unlikely to be professional or even >> casual spammers, no matter what users may report. Users >> are sometimes wrong, so data should be checked, IMO. >> >> Jeff C. > Jeff, browsehappy.com problem was reported back to us by schampeon and > immediately removed. > The approach we take (if its new and appears in reported spam) does have FPs, I > agree - but we havent been able to find a good way to "check". > We do look (cursory) at all the blocked domains per day and if anything obvious > shows up we do remove them. The problem is that detailed looking by a human is > not really practical given the volume of domains we block by day. > As you know/see, we are very responsive and remove very quickly. > If you have any suggestions on how to improve the process, I'm all ears and > will implement your suggestions as long as it doesnt consume too much human > time (checking 100s of domains 1 by 1 is just not practical). > Cheers, > TB Hi Tony, Thanks indeed for your responsiveness in removing FPs, and addressing the concerns of us and your users. Regarding some checks that can be done on the incoming data, many of the suggestions in our draft policy for manual lists can be automated: http://www.surbl.org/policy.html and some of those may perhaps be useful for your checking of incoming suspected spam domains. What I'd suggest is perhaps using these to score new domains and to flag ones that rise above a certain score. For example, any domain in SBL probably can be blacklisted immediately. Any domain not in SBL probably begins to add to a ham score, though not 100%. If you have access to the headers, and the senders are in xbl.spamuahs.org, then the domain should probably be listed. Any sender IP not in XBL probably should get ham points. Any domain with few or zero NANAS hits may be hammy, Domains in DMOZ, Wikipedia, etc should perhaps get ham points since it's unlikely the human editors of those would add or allow spam domains, etc. Obviously most of the spam domains we get are fully spammy. Perhaps some of these metrics can help flag ones that are less spammy and worthy of a little further checking? Your feedback, comments, questions, etc would be welcomed since we intend to use a policy like this for our own manual list, ws.surbl.org. We may adopt other parts of this for our automated lists also. Cheers, Jeff C. P.S. Do you mind if I publish this response on our SURBL discussion list? -- "If it appears in hams, then don't list it."

1 0

Child porn URLs
by Mariano Absatz 15 Oct '04

15 Oct '04

Hi, I just reported this mail with its 3 URLs to spamcop and ws.surbl... I urge other list mantainers to add them since it contains disgusting child pornography... For what I can understand in http://www.sg.st, although sg.st is not an 'official' 2LD of st, it seems that this is a bulk registry for the domains HK.ST - CN.ST - TW.ST - SG.ST. The official NIC for st (São Tomé & Principe) seems to be http://www.nic.st. As I undertand it, the SG.ST domain should be whitelisted... Regards. Received: from c-24-12-31-157.client.comcast.net (HELO smtp.hotpop.com) (24.12.31.157) by mail.example.com with SMTP; 13 Oct 2004 20:17:24 -0000 Date: Fri, 15 Oct 2004 22:56:09 +0000 From: mangled <mangled(a)example.com> Subject: Hi. To: FILE <mangled(a)example.com> References: <mangled(a)example.com> In-Reply-To: <mangled(a)example.com> Message-ID: <mangled(a)example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT Hey, I've found true BL Hits! Over 12000 users here. http://pbfiles.sg.st and http://ggboys.sg.st Giant Lo collection here: http://ptz-portal.sg.st Best, Michael Berkly. p.s. looking for your quickest reply. -- Mariano Absatz - El Baby el (dot) baby (AT) gmail (dot) com el (punto) baby (ARROBA:@) gmail (punto) com

5 11

WS FP
by Daniel Kleinsinger 15 Oct '04

15 Oct '04

freepichosting . com In a craigslist email. Thanks, Daniel

2 2

FP?: ultimatebizsource.biz
by Joe Wein 15 Oct '04

15 Oct '04

ultimatebizsource.biz claims to be an opt-in list with 55,000 subscribers. It's currently listed on [WS] and [JP] but not my personal list. I know, the name alone sounds like your typical spammer domain, but is it? Today the owner of getyoursiteongoogle.info mailed me, since I had blacklisted their domain. Turns out the evidence was a mailing from ultimatebizsource.biz which was listed on WS and JP. Since the recipient was a third party from the Prolocation feed, I could not easily verify if it was indeed unsolicited, but I checked what I could check quickly. To my surprize the outgoing mailserver's IP is not listed on any of the RBLs and neither are the name servers or the resolved name. There is only 1 NANAS sighting for this almost one year old domain. That posting is an automated one, and probably related to an ahbl.org listing mentioned in the SA-tags added to the evidence mail. I can find no listing for them at ahbl.org now. A web search for the domain name returns a single hit: http://www.google.com/search?q=ultimatebizsource.biz+spam That looks too little for 55,000 mails per day for one year. Joe

2 1

Re: [ST #592239]:postmaster: Re: [SURBL-Discuss] FP in OB ?
by Jeff Chan 14 Oct '04

14 Oct '04

On Thursday, October 14, 2004, 5:20:56 AM, Tony RT wrote: > [jeffc(a)surbl.org - Thu Oct 14 12:10:09 2004]: >> > In this months Microsoft Security update mailout there is a domain >> > mentioned in the credits, persiax.com. Web version includes the same >> > domain... >> Outblaze Postmaster, >> Can you tell us why persiax.com is listed? > Jeff, > If our spamtraps or our users report spam, we scan the body. > If we find domains that are "new" in the body, the domain gets blocked. > persiax.com is now removed. > TonyB Thanks Tony. May I suggest that you consider checking a domain before listing it? Just because a few customers consider it spam doesn't necessarily mean other customers might not want to get it. I ask because there seem to be some legitimate sites getting onto your lists which some customers may legitimately want to get. For example none of the recent FPs have had to do with pills, mortgages, warez, etc. Another recent example is browsehappy.com run by the Web Standards Project: http://webstandards.org/act/campaign/happy/ which seem pretty unlikely to be professional or even casual spammers, no matter what users may report. Users are sometimes wrong, so data should be checked, IMO. Jeff C. -- "If it appears in hams, then don't list it."

1 0

FP in OB ?
by Joseph Burford 14 Oct '04

14 Oct '04

Guys, not sure what the criteria for listing/de-listing in OB is. In this months Microsoft Security update mailout there is a domain mentioned in the credits, persiax.com. Web version includes the same domain... http://www.microsoft.com/technet/security/bulletin/MS04-oct.mspx This is a new domain, about a month old, 0 NANAS, looks like a personal site with 0 content, some guy in Iran who was about to get his 15mins of fame :) Not a whitelist candidate IMHO but am curious why it's in there. Regards, Joseph

2 1

FP on WS
by Joseph Burford 14 Oct '04

14 Oct '04

Hi All, bluedomino.com In my mind a FP on the WS list. Domain age in days: 2168 Associated with CoffeeCup, the age old HTML editing software. Only place I've seen it mentioned is in the normal mailouts from CoffeeCup to people who have downloaded their software or who are on the mailing list. NANAS sightings about 10, some are just reports of customers they host, nothing really recent. I'd be intrigued to know how it got on WS. Regards, Joseph

2 4

Doesn't multi contain everything from WS and others?
by Thomas Johnson 14 Oct '04

14 Oct '04

(Sorry about that - hit send a little too early). Perhaps I'm confused, but shouldn't something listed in WS and JP also be in multi? Here's a domain from an email that made it through our filters: http://y.net.mystuffsupplyMUNGED.com?f4=T12k62 # dig +short @localhost mystuffsupplyMUNGED.com.ws.surbl.org 127.0.0.2 # dig +short mystuffsupplyMUNGED.com.multi.surbl.org <nothing> Am I missing something?

3 4

RE: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 14 Oct '04

14 Oct '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Saturday, October 09, 2004 3:10 AM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains >too > > >On Friday, October 8, 2004, 11:37:03 PM, Alex Broens wrote: >> tripod and geocities have given spammers a home for many years... and >> there's many other freehosters who have become "victims" as well. > >> if not permitted to blacklist, at least treat as 2nd level >tld so we can >> get rid of the spam coming from them > >That's possible, but not something we've done so far. I don't >see shared domain hosting as a major spam destination. Maybe >they are a minor annoyance, but nothing like the pill spammers >hosted in China, Korea, Brazil, etc. > >The difference is that geocities, tripod, angelfire, etc. ought >to have some incentive to get rid of these minor abusers since >they make so little money from them. The pill/mortgage/warez/etc. >spam hosters probably make a lot more money from their spamming >customers, so they have an incentive too keep them. > >In the case where there is some legitimate company like Yahoo >or Lycos (parents of geocities, tripod, etc) to police their >customers and enforce their AUPs, then spam victims should >report them and let them do the enforcing. Since these >companies are mostly legitimate and probably do spend some >resources dealing with abuse, they have an incentive to >reduce abuse since it probably costs them more money than >it gains them. > >One of the main reasons for doing SURBLs was to be able to >do something about the hosting companies who don't have AUPs >against spam or don't enforce them. > >Bottom line is that tripod, etc. are mostly irrelevant compared >to the professional spam gangs who send a lot more spam. SURBLs >are for listing the domains of the bigger fish who have found >spam-friendly hosts, regardless of where those hosts are. > All I can say to that is, it is small amount....for now. I don't see any harm in treating these as second and thrid TLDs. We are already doing all the same work, and seeing the same domains anyway while looking at spams. Why not just add them as 2-3 level? We are doing all the work now, and just throwing the listing away. --Chris

5 8

voicemessage . com listed in WS
by Fred 14 Oct '04

14 Oct '04

I received a virus today which included a link to virt.voicemessage . com This domain was registered 02-DEC-1996, I hand checked the domain, they appear to be a business partner with Avaya. Just because a virus uses this domain name in a attempt to trick the user, is not a reason to list. This domain appears in ham because when my customers receive undetected viruses, I ask them to send them to me and then it's part of mail I need to receive. Does anyone have any spam from this domain or was it listed solely on the fact that it appeared in a virus the committer received? Thanks!

3 3

submit domain?
by Lindsay Snider 13 Oct '04

13 Oct '04

acarmart DOT com What is our policy for adding domains like that above. It appears to be a link builder w/ no real content. And I appologise for not checking Jeff's website of criteria for adding a domain. I wasn't able to find a link so if any one has a bookmark available, I'd greatly appreciate it. -lindsay

2 2

RE: [SURBL-Discuss] submit domain?
by Chris Santerre 13 Oct '04

13 Oct '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Tuesday, October 12, 2004 7:41 PM >To: SURBL Discuss >Subject: Re: [SURBL-Discuss] submit domain? > > >On Tuesday, October 12, 2004, 1:43:26 PM, Chris Santerre wrote: >>>-----Original Message----- >>>From: Lindsay Snider [mailto:lindsay@pa.net] >>>Sent: Tuesday, October 12, 2004 2:43 PM >>>To: discuss(a)lists.surbl.org >>>Subject: [SURBL-Discuss] submit domain? >>> >>> >>>acarmart DOT com >>> >>>What is our policy for adding domains like that above. It >>>appears to be a >>>link builder w/ no real content. And I appologise for not >>>checking Jeff's >>>website of criteria for adding a domain. I wasn't able to >>>find a link so if >>>any one has a bookmark available, I'd greatly appreciate it. > > >> Jeff made us memorize it! > >> http://www.surbl.org/policy.html > >Speaking of which does anyone else think that policy is >good enough to link/publish yet? > +1 Go ahead ;) and post to public. --Chris

1 0

[ot] If you see this domain in spam....
by Chris Santerre 13 Oct '04

13 Oct '04

I need a shower after we removed this domain from SURBL: hypnoticsellingsecrets.com I would like to know if anyone gets a report of spam that has this domain in it. They just make me feel all icky. The more I read their site, the more I wished we didn't remove them. But...I need a spam. Thanks. --Chris

1 0

OT: tcsh filename expansion in file test
by Jeff Chan 13 Oct '04

13 Oct '04

UNIX question not directly related to SURBLs: does anyone know how to make filename expansion (globbing) happen inside a filetest in a tcsh script: unset noglob if (-e *.surbl.org.changed.multi) then ... Didn't work. It's supposed to test the existence of any files *.surbl.org.changed.multi . Using a specific file name like ws.surbl.org.changed.multi works, but I'd like to wildcard it. This has to do with making multi update sooner when the individual lists update, so maybe it's relevant. ;-) Jeff C. -- "If it appears in hams, then don't list it."

2 2

RE: [SURBL-Discuss] submit domain?
by Chris Santerre 13 Oct '04

13 Oct '04

>-----Original Message----- >From: Lindsay Snider [mailto:lindsay@pa.net] >Sent: Tuesday, October 12, 2004 2:43 PM >To: discuss(a)lists.surbl.org >Subject: [SURBL-Discuss] submit domain? > > >acarmart DOT com > >What is our policy for adding domains like that above. It >appears to be a >link builder w/ no real content. And I appologise for not >checking Jeff's >website of criteria for adding a domain. I wasn't able to >find a link so if >any one has a bookmark available, I'd greatly appreciate it. Jeff made us memorize it! http://www.surbl.org/policy.html ;) --Chris

3 2

RE: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Chris Santerre 12 Oct '04

12 Oct '04

>-----Original Message----- >From: Jeff Chan [mailto:jeffc@surbl.org] >Sent: Saturday, October 09, 2004 9:06 PM >To: SURBL Discussion list >Subject: Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains >too > > >On Saturday, October 9, 2004, 1:09:43 PM, Patrik Nilsson wrote: > >> Just create a separate "TLDs or treat as TLDSs" zone that >can be checked >> and cached client side. > >> Or even better - give "TLDs or treat as TLDSs" a >distinguished A value in >> existing lists. >> If a lookup returns XXX.XXX.XXX.XXX, it is a "TLD or treat >as TLD" and >> should be further recursed. >> If we think there is a risk that some bad client >implementatins treat any >> returned A record as a hit, use TXT records. > >This is still an interesting idea, but I'd still be somewhat >concerned about putting out a list that looks like a regular >SURBL that it could get misused. > >But perhaps the larger issues is that the hard core spammers >don't seem to use *subdomains of legitimate shared-domain >hosting providers*. They just register their own full domain >names and use those (lots of them). > >If some legitimate hosting provider has an abuse issue, >then it's in their own interest to stop the abuse. > >SURBLs are arguably best suited for cases where the ISP >is spam-friendly and allows spam hosting on custom domains. >The reality is that's a much larger and tougher problem >than shared, common-domain hosting, like a geocities or >tripod. > >The best use of our time is to focus on the biggest spammers >first, and we're not catching all of those yet. > Big surprise...I disagree :-) I think we are already reviewing these domains, so why not just add to catch the smaller ones instead of throwing them away. But I see your vision, and I can bend like the reed ;) But given enough time, and perhaps enough chocolate, I think I can turn you around :-) --Chris

1 0

New data: SURBL DNS queries
by Jeff Chan 10 Oct '04

10 Oct '04

As has already been mentioned, Theo is patching SpamAssassin to locally whitelist some common whitehat URI domains for use in URIBL (which typically uses sbl and SURBL data) . This will prevent DNS queries on the whitehats and probably save some very significant traffic on the SURBL and spamhaus, etc. name servers. In order to get some better whitehat data, we increased the sampling of DNS queries on a name server from 32k (2k every 3 hours for 2 days) to 1.2 million (10k every 2 hours for 10 days). We're only about a third of the way through the initial 10 days, so the stats are still building up, but the current results are at: http://www.surbl.org/dns-queries.whitelist.counts.txt http://www.surbl.org/dns-queries.blocklist.counts.txt (These files have been mentioned before, but they're starting to get a lot more data behind them now.) Something else which was probably suggestion before, but which we *hadn't looked at before* were the DNS queries that *don't match* either our blocklists or whitelists. Those, sorted in order of decreasing frequency are: http://www.surbl.org/dns-queries.unmatched.30thpercentile.txt That's the top 30th percentile of them (about 3.6k records). The full list of unique domains and IPs with frequencies (about 110k records) is at: http://www.surbl.org/dns-queries.unmatched.count.txt Taking a look the top few of these: 333 56.227.117.38 211 internet.e 196 wwwlowmortnow.info 123 beliefnet.com 119 grisoft.com) 107 specialmax.net 99 democrats.org 99 115.14.249.209 96 and 90 c 82 centrport.net 78 charter.net 73 zdnet.com 65 cf.st 63 nuri1.net 62 red-hot1.com 62 imomentum.net 61 justsaywow.com 60 173.213.115.211 59 www 57 www.cool-loanco.kr 57 superduperfun.com 53 healthinsrus.com 51 e-directnet.net 51 agoramail.net 50 tmcs.net 50 latimes.com 50 dw.com.com 50 168.228.186.64 49 iscsimg.com 48 livedaily.com 48 eversave.com 47 1shoppingcart.com 46 srvimg.com 46 realone.com 46 goodnewsdelivery.com 45 rockbridgemedia.com 45 purdue.edu It's clear that a few are errors, probably due to problems in the applications using SURBLs. Yet it's probably useful to not suppress the errors so that the programs can be updated to handle them correctly. (Unfortunately the source URIs generating the errors are not directly available, but they may be identifiable in other ways if anyone would like to look for them.) Minus the errors, I fed this list into Ryan's GetURI to see what it could find. The results are at: http://ry.ca/cgi-bin/geturi.cgi?id=ham-5lCTzHkxan3xE38RKHa0vx Quite a few appear ok to whitelist, like democrats.org, pudue.edu, latimes.com, charter.net, zdnet.com, etc. and I'll probably go ahead and whitelist obvious ones like these, so some of these will probably be off this "unmatched" list and onto the whitelist hits by the time you read this. Nonetheless I recommend we all take a look at this unmatched list periodically, especially the top few dozen, to look for potential domains to whitelist or blacklist. These most frequently appearing domains are probably good candidates for one or the other. Since this is a list of the unknown "wild" domains coming from live, real-world message URIs, it may be another useful and different source of some data. Cheers, Jeff C. -- "If it appears in hams, then don't list it."

1 1

The plus side of FPs.
by Chris Santerre 09 Oct '04

09 Oct '04

We all hate FPs. We know we want to cut them down. But let me just say I have seen a nice plus to them. The whitelist requests for SURBL have been somewhat of an eye opener. There have been more then a few legit mailers who have drastically changed their ways since contacting us for whitelist. It is actually pretty amazing. They will 'fix' or at least try to 'fix' a lot of their messed up subscriptions just to get off of SURBL. Its all been polite and professional as well. We give examples of why they were added and ask for explanations. We also ask for a copy of their antispam/abuse policy. The responses have been outstanding. So we also have the power to educate. Yeah yeah yeah, I know. With great power comes great .....cheesecake. Chris Santerre System Admin and SARE Ninja http://www.rulesemporium.com http://www.surbl.org 'It is not the strongest of the species that survives, not the most intelligent, but the one most responsive to change.' Charles Darwin

1 0

They found away around SURBL
by Chris Santerre 09 Oct '04

09 Oct '04

No link whatsoever. NO phone number either. No site mentioned. Nothing :) Maybe they just want to tease the reader :) Too bad this scored 15.1 points without surbl :) --Chris ----- Message-ID: <41665F37.67DCCE6F(a)siba.fi> From: Kent Bryan <kbryan_cd(a)siba.fi> To: Chris Santerre <csanterre(a)MerchantsOverseas.com> Subject: This worked for me, and will for you! Date: Fri, 8 Oct 2004 05:34:47 -0400 X-Mailer: Internet Mail Service (5.5.2653.19) ---------------------------------------------------------------------- - R.ated N.O.1 Pe.nis Enlargeme.nt P.ill on the Mar.ket! - - Gain up to 3+ full inches in length. - Increase your peni.s width by 20.%. - Stop Prema.ture Ejacu.lation. - Produce Strong.er and R.ock Ha.rd Erect.ions ----------------------------------------------------------------------

6 5

Re: [SURBL-Discuss] Revised DMOZ data, got Wikipedia domains too
by Frank Ellermann 09 Oct '04

09 Oct '04

Jeff Chan wrote: > Would there be many FNs (missed spams) if we whitelisted > all of these? In other words are these all truly False > Positives? > If not, which ones do you feel are true spammers and why. If WS and SC agree on something (or even more of the SURBLs) they might have a point. I tried to check kit.net (found in your list), and got an "unknown host" error. So maybe that was a FN (= spammer). I also tried to check made.com (on your original DMOZ list, it was listed in SC and AB IIRC), and came to the conclusion that this might be a Korean hoster. The subdomain listed in DMOZ looked pretty ordinary (at least the non-Flash part), no offers to buy something. But that could be misleading, I didn't visit the site with enabled JS. And of course the reason for the SC listing could be another subdomain, not the one listed by DMOZ. But made.com could be a FP (?) Hard to judge. More than one bit in multi is a hint. Bye.

1 0

Revised DMOZ data, got Wikipedia domains too
by Jeff Chan 09 Oct '04

09 Oct '04

Following up on the earlier check of DMOZ domains against SURBL data, I applied some of Quinlan's suggestions and grabbed different revisions of the DMOZ data and joined (intersected) them in order to eliminate changes due to things like editor- removed spammer/abuser domains, etc. This also means that new additions are ignored, but the corpus is so large that the benefits of capturing editor removals is probably more important. Snapshots are the three most recent available from 9/9/04, 9/25/04 and 10/7/04 (file dates) are intersected, resulting in fewer records (only those that are constant across all three snapshots): http://rdf.dmoz.org/rdf/archive/ Here are the line, word and character counts: 2300851 2300851 38065969 dmoz.srt 1169 11690 123909 dmoz-blocklist.summed.txt 1141 11410 120860 dmoz-blocklist.ws 1169 1169 17977 dmoz-blocklist.txt 7394 7394 97011 dmoz-whitelist.txt The above are revised versions of joins on blocklists, with list info, with ws hits, with just domains, and against the whitelist. These are in the whitelists directory, though we are still *not* applying the dmoz domains as whitelists: http://spamcheck.freeapp.net/whitelists/ The previous dmoz (9/25 version only IIRC) and hits are archived as: 2326173 2326173 38494184 dmoz.srt1 1338 13380 141946 dmoz-blocklist1.summed.txt 1173 11730 124298 dmoz-blocklist1.ws 1338 1338 20533 dmoz-blocklist1.txt 7375 7375 96720 dmoz-whitelist1.txt __ I was also able to grab 4 snapshots of the wikipedia, all sections (all languages). Only the two most recent snapshots had comparable numbers of sections so I used only those two. Using only two sets may be ok, since these are much smaller corpora and there's probably more hand-editing of them. http://download.wikimedia.org/ Where dmoz has about 2.3 million domains, wikipedia has about 174k domains: 173828 173828 2633441 wikipedia.srt 188 1880 19631 wikipedia-blocklist.summed.txt 188 188 2713 wikipedia-blocklist.txt 2437 2437 29581 wikipedia-whitelist.txt __ I also took the intersection of the three dmoz snapshots and the two wikipedia snapshots to get a smaller list containing only the ~102k domains found in both wikipedia and dmoz: 101619 101619 1498653 wikipedia-dmoz.srt 116 1160 11928 wikipedia-dmoz-blocklist.summed.txt 116 116 1591 wikipedia-dmoz-blocklist.txt 2223 2223 26854 wikipedia-dmoz-whitelist.txt This intersection of dmoz and wikipedia domains probably represents the best hope for large whitelist additions so far. This is probably imperfect data, but at least it has some checks by human editors and techniques applied to reduce spammer domains, including comparing the snapshots over time and intersecting the two relatively unrelated sources. Can anyone think of any other hand-edited databases, directories, encyclopedias, etc of URIs of hopefully legitimate (non-spammer) domains that are publically available? Please think about it a little, and speak up! While 102k domains isn't nearly as large as the 2.3M in dmoz, it's certainly more than the 12k or so whitelist records we currently have. How does the intersected list look as a potential whitelist? http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz.srt Please also take a look at these blocklist hits (potential FPs) and share what you think: http://spamcheck.freeapp.net/whitelists/wikipedia-dmoz-blocklist.summed.txt Would there be many FNs (missed spams) if we whitelisted all of these? In other words are these all truly False Positives? If not, which ones do you feel are true spammers and why. Jeff C. -- "If it appears in hams, then don't list it."

4 9

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Discuss