[Update, Chris wrote off list that he's put up a quick be.htm page to be prettified later.]
On Wednesday, April 21, 2004, 7:36:08 AM, Chris Santerre wrote:
Sounds good. Can you let me know what kind of TTL I should set?
Well I am now trying to update at least every other day. This way I won't fall behind. But I'm now getting every day. I always test overnight, because too many people rely on the list now. I usually post before noon EST.
OK sounds like an 8 or 12 hour TTL is appropriate then; setting to 8 now.
Any idea how often Paul updates MidEvil?
Basically I'd like to set the lifetime of the zone info to something relevant towards how often you and Paul usually update the lists. Nothing too specific is needed, just a general idea. Like is it daily, twice a day, every other day on average, etc.
Also does this TXT record work for you guys:
"Blocked in BigEvil. See: http://www.rulesemporium.com/"
It was just a generic placeholder. I'd like comments/improvements on it.
How about www.rulesemporium.com/be.htm ? I can make a page just for that error? Otherwise it is fine.
Done. Please set up a page when you get a chance... :-)
- BigEvil wildcards. Not sure how you would handle these.
Something like
evil\d{2,4}spam.com is a general wildcard. Some of those
domains don't even
exhist. Not sure how SURBL will handle that.
Yes, I should have mentioned that I'm simply discarding them. Unfortunately there's no easy way to deal with them. Domains without any patterns in them, which are a majority, come right through. The script is at:
Can we make sure that when you announce this to the public that they know this! :) I can see the flurry of emails now.
Definitely will mention the differences in the announcement and web site!
- What is the quickest way to check a domain against the
other SURBL lists?
Basically I see no reason to duplicate the listings. *gulp* and on a Windowze machine? (Don't ask!)
I wouldn't worry too much about that for now. For now we just want to get an accurate record of everything. We're working on ways to merge things next.
Well ok, but I still want to look others up if I have a domain in question :) Will there be a quick web page to look up a domain? Or do I do an NSLOOKUP using the SURBL?
You can find the domains currently going into the SURBL lists at:
sc: http://spamcheck.freeapp.net/top-sites-domains
ws: http://spamcheck.freeapp.net/sa-blacklist.current.domains.afterwhitelist
be: http://spamcheck.freeapp.net/bigevil.domains.afterwhitelist
But frankly I like the fact that there is some overlap in the lists. In a sense that represents multiple reporting; i.e. a domain in more than one list is more likely a bad guy. I don't think we should lose that coding.
YMMV, but I'd say keep any overlap in BE. It's a feature not a bug.
- Has there been any talk with the sendmail people? It
would be interesting
to actually block at the MTA level based on an evil URL. I
realise the
inherent dangers in this ;)
Yes, there is talk about sendmail milters using SURBLs. I haven't heard of anyone doing one yet, but they're feasible. The limiting factor is the FP rate. FPs must be as close to zero as possible before people will dare to reject spams at the MTA level using SURBLs, other than perhaps for personal servers, etc.
Dangerous, but so very fun!
Hehe! ;-) Messing with spammers is always fun!
Jeff C.
At 09:49 22/04/2004, you wrote:
- BigEvil wildcards. Not sure how you would handle these.
Something like
evil\d{2,4}spam.com is a general wildcard. Some of those
domains don't even
exhist. Not sure how SURBL will handle that.
Yes, I should have mentioned that I'm simply discarding them. Unfortunately there's no easy way to deal with them. Domains without any patterns in them, which are a majority, come right through. The script is at:
Can we make sure that when you announce this to the public that they know this! :) I can see the flurry of emails now.
Right near the top of http://spamcheck.freeapp.net/bigevil.domains.afterwhitelist there is 123-ebiz - is that a mistake or parsing error ?
But frankly I like the fact that there is some overlap in the lists. In a sense that represents multiple reporting; i.e. a domain in more than one list is more likely a bad guy. I don't think we should lose that coding.
YMMV, but I'd say keep any overlap in BE. It's a feature not a bug.
I think so too. What some people suggesting merging are forgetting, is with lists with totally different sources, that whether a given URL is listed in one two or three of the lists IS an extra piece of information, something listed in all three is more likely to be correct than one listed on only one of the lists.
The SA approach of assigning a score to each list based on it's relative merits, and the scores ADDING if they're in multiple lists seems to be a sensible approach to me...
Of course there is nothing to stop you having merged lists available AS WELL for those that are willing to take the risk of one higher scoring merged list...with choice, everyone is happy ;)
By the way, am I jumping the gun here or is be.surbl.org ready to go, or should I wait a bit ? :)
Regards, Simon
On Wednesday, April 21, 2004, 3:16:12 PM, Simon Byrnand wrote:
At 09:49 22/04/2004, you wrote:
- BigEvil wildcards. Not sure how you would handle these.
Something like
evil\d{2,4}spam.com is a general wildcard. Some of those
domains don't even
exhist. Not sure how SURBL will handle that.
Yes, I should have mentioned that I'm simply discarding them. Unfortunately there's no easy way to deal with them. Domains without any patterns in them, which are a majority, come right through. The script is at:
Can we make sure that when you announce this to the public that they know this! :) I can see the flurry of emails now.
Right near the top of http://spamcheck.freeapp.net/bigevil.domains.afterwhitelist there is 123-ebiz - is that a mistake or parsing error ?
Good eye. I think that may be a bug in the original BigEvil.cf rules for Chris to fix since it fell right out of the expand_regex.pl that way: 123-ebiz (i.e. without a TLD). For now I'll stop it from getting into the RBLs with a manual whitelist, though it likely hurts nothing to have it in there.
But frankly I like the fact that there is some overlap in the lists. In a sense that represents multiple reporting; i.e. a domain in more than one list is more likely a bad guy. I don't think we should lose that coding.
YMMV, but I'd say keep any overlap in BE. It's a feature not a bug.
I think so too. What some people suggesting merging are forgetting, is with lists with totally different sources, that whether a given URL is listed in one two or three of the lists IS an extra piece of information, something listed in all three is more likely to be correct than one listed on only one of the lists.
The SA approach of assigning a score to each list based on it's relative merits, and the scores ADDING if they're in multiple lists seems to be a sensible approach to me...
We can merge the lists in a way to preserve the fact that the entries came from multiple lists. That's what the bitmasked single A record versus multiple A record discussion was about.
Of course there is nothing to stop you having merged lists available AS WELL for those that are willing to take the risk of one higher scoring merged list...with choice, everyone is happy ;)
By the way, am I jumping the gun here or is be.surbl.org ready to go, or should I wait a bit ? :)
It's pretty much ready. We got good feedback from Chris Santerre. I need to update the web site and announce it.
Still waiting to hear back from some of the secondary DNS admins....
Jeff C.
On Wed, 21 Apr 2004 14:49:51 -0700, Jeff Chan wrote:
- BigEvil wildcards. Not sure how you would handle these.
Something like
evil\d{2,4}spam.com is a general wildcard. Some of those
domains don't even
exhist. Not sure how SURBL will handle that.
Yes, I should have mentioned that I'm simply discarding them. Unfortunately there's no easy way to deal with them. Domains without any patterns in them, which are a majority, come right through. The script is at:
Can we make sure that when you announce this to the public that they know this! :) I can see the flurry of emails now.
Definitely will mention the differences in the announcement and web site!
Perhaps the ideal would be if the script that converts bigevil to rbl form could also generate a separate "wildevil" cf file containing only the wildcard entries from bigevil, so that people can have the best of both worlds...
John.
On Wednesday, April 21, 2004, 11:25:32 PM, John Wilcock wrote:
On Wed, 21 Apr 2004 14:49:51 -0700, Jeff Chan wrote:
- BigEvil wildcards. Not sure how you would handle these.
Something like
evil\d{2,4}spam.com is a general wildcard. Some of those
domains don't even
exhist. Not sure how SURBL will handle that.
Yes, I should have mentioned that I'm simply discarding them. Unfortunately there's no easy way to deal with them. Domains without any patterns in them, which are a majority, come right through. The script is at:
Can we make sure that when you announce this to the public that they know this! :) I can see the flurry of emails now.
Definitely will mention the differences in the announcement and web site!
Perhaps the ideal would be if the script that converts bigevil to rbl form could also generate a separate "wildevil" cf file containing only the wildcard entries from bigevil, so that people can have the best of both worlds...
That's an interesting idea. I'd need to investigate how reversible the transforms are however. Note that the number of wildcarded base domains discarded is currently 2%.
Jeff C.
Hi!
Perhaps the ideal would be if the script that converts bigevil to rbl form could also generate a separate "wildevil" cf file containing only the wildcard entries from bigevil, so that people can have the best of both worlds...
That's an interesting idea. I'd need to investigate how reversible the transforms are however. Note that the number of wildcarded base domains discarded is currently 2%.
Just give back:
127.0.0.2 SC 127.0.0.3 WS 127.0.0.4 BE 127.0.0.5 WE
Then you can even with a combined BE/WE decide what to do depending on the results, this is also needed for the combined list anyway.
Bye, Raymond.
On Wednesday, April 21, 2004, 11:32:52 PM, Raymond Dijkxhoorn wrote:
Perhaps the ideal would be if the script that converts bigevil to rbl form could also generate a separate "wildevil" cf file containing only the wildcard entries from bigevil, so that people can have the best of both worlds...
That's an interesting idea. I'd need to investigate how reversible the transforms are however. Note that the number of wildcarded base domains discarded is currently 2%.
Just give back:
127.0.0.2 SC 127.0.0.3 WS 127.0.0.4 BE 127.0.0.5 WE
Then you can even with a combined BE/WE decide what to do depending on the results, this is also needed for the combined list anyway.
The problem is that the wildcarded domains coming from BE don't look like resolvable domains that can be put into an RBL. They look like:
bestmedicine?sonline.biz bestmedicine?sonline.com bestmedicine?sonline.net blue-0[15].com boomnetworks2?.com cabledeals?.biz car[bdl]march.biz
...
dfhfksd{1,5}.info drugs?d{0,4}.biz drugs?d{0,4}.com drugs?d{0,4}.info e-?xtremepenetration.biz e-?xtremepenetration.com e-?xtremepenetration.net e?meds?d{1,3}.com ehostzz?z?.biz ehostzz?z?.com ehostzz?z?.net
What I think John is asking for in essence is a version of BigEvil.cf with the non-wildcarded domains removed, i.e. a subset of the original ruleset with *only* the wildcarded domains.
Jeff C.
-
Jeff Chan -
John Wilcock -
Raymond Dijkxhoorn -
Simon Byrnand