On Saturday, April 10, 2004, 10:31:04 PM, Raymond Dijkxhoorn wrote:

...

Just wanted to say hi! And thanks Jeff and Raymond for the new lists.

You've got the honour of doing the first posting :) I see most are subscribed now on the list, who were in the cc fields of the other mails. I invited the rsync mirrors to the other list also, hopefully they will sign up.

Hows things with the BigEvil list Jeff ? Would love to take that out my configs and put back a RBL for it. Would save some RAM on my setups :)

Hi Everyone, Thanks indeed to everyone for their continued support. You guys have made this project a dream.

Regarding BigEvil, I brought up turning it into an RBL with Chris and he's checking with the data sources last I heard. I certainly hope he gets the green light and we can add it.

BTW, Kelsey and I brainstormed last night and I think we have a way to effectively prejudice new domain reports coming in from SpamCop without reference to SBL or to geographic databases like IP::Country::Fast or any other external sources like I had in mind originally.

It's so simple that I might be tempted to call it elegant:

1. Resolve the incoming spam domains from SC into A and perhaps NS records. 2. Keep a persistent tally counting those IPs. (a history) 3. For As or NSes of incoming domains that match many identical or nearby IP tallies (i.e., the new domains use known bad old IPs), drop their inclusion thresholds in some statistically cool and relevant way.

To our thinking, this will automatically and in a self-tuning way catch spam gangs, rogue IPs, rogue blocks, rogue ISPs in any nation, etc. (Manually resolving some of the domains in spams I get seem to show China and a few gangs a lot. I'd dearly like to crush them early and often. Building this refinement into the second version of the sc.surbl.org data engine may very well do that.)

The big advantage is that far fewer reports would be needed for a *new* domain to get added to the list if it has an IP near previously reported domain's IPs. We would expire IPs like domains, but probably with a longer time window for IPs, so that cleaned IPs would eventually come off the tallies.

To clarify, the IPs would not get added to any lists, just get used internally to lower the inclusion threshold for the number of SpamCop reports needed to get added. Inclusion would still be triggered by SpamCop reports, but in a more sensitive way for bad guy IPs.

Seems almost too good to be true. Am I missing something?

I may bounce this idea off SA-dev also.

Jeff C.

Good morning, Jeff, all,

On Sat, 10 Apr 2004, Jeff Chan wrote:

BTW, Kelsey and I brainstormed last night and I think we have a way to effectively prejudice new domain reports coming in from SpamCop without reference to SBL or to geographic databases like IP::Country::Fast or any other external sources like I had in mind originally.

It's so simple that I might be tempted to call it elegant:

1. Resolve the incoming spam domains from SC into A and

perhaps NS records.

NS is a little tougher. I could see us hurting people that left their name service with their registrar or large ISP's that simply host a lot of domains. It would also be possible for a spammer to simply claim that ns1.earthlink.net and ns2.aol.com are their secondary and tertiary when they aren't. It's not possible - or at least tougher - to fake A records.

2. Keep a persistent tally counting those IPs. (a history)
3. For As or NSes of incoming domains that match many identical

or nearby IP tallies (i.e., the new domains use known bad old IPs), drop their inclusion thresholds in some statistically cool and relevant way.

To our thinking, this will automatically and in a self-tuning way catch spam gangs, rogue IPs, rogue blocks, rogue ISPs in any nation, etc. (Manually resolving some of the domains in spams I get seem to show China and a few gangs a lot. I'd dearly like to crush them early and often. Building this refinement into the second version of the sc.surbl.org data engine may very well do that.)

The big advantage is that far fewer reports would be needed for a *new* domain to get added to the list if it has an IP near previously reported domain's IPs. We would expire IPs like domains, but probably with a longer time window for IPs, so that cleaned IPs would eventually come off the tallies.

To clarify, the IPs would not get added to any lists, just get used internally to lower the inclusion threshold for the number of SpamCop reports needed to get added. Inclusion would still be triggered by SpamCop reports, but in a more sensitive way for bad guy IPs.

Seems almost too good to be true. Am I missing something?

The approach made sense to me as well. http://www.stearns.org/sa-blacklist/spamip.current.txt is the report created from the A record harvesting. I'd be glad to provide the raw data collected over the past 5 months. I also have SOA and whois data for the entire sa-blacklist. Cheers, - Bill

--------------------------------------------------------------------------- "There are two kinds of people, those who work and those who want the credit. Try to stay in the first category. The competition is much smaller." -- Mahatma Ghandi -------------------------------------------------------------------------- William Stearns (wstearns@pobox.com). Mason, Buildkernel, freedups, p0f, rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org --------------------------------------------------------------------------

On Sunday, April 11, 2004, 6:52:56 AM, William Stearns wrote:

On Sat, 10 Apr 2004, Jeff Chan wrote:

...

It's so simple that I might be tempted to call it elegant:

1. Resolve the incoming spam domains from SC into A and

perhaps NS records.

    NS is a little tougher.  I could see us hurting people that left 

their name service with their registrar or large ISP's that simply host a lot of domains. It would also be possible for a spammer to simply claim that ns1.earthlink.net and ns2.aol.com are their secondary and tertiary when they aren't. It's not possible - or at least tougher - to fake A records.

Indeed, that's a strong argument for using A records only. I really appreciate your feedback!

...

2. Keep a persistent tally counting those IPs. (a history)
3. For As or NSes of incoming domains that match many identical

or nearby IP tallies (i.e., the new domains use known bad old IPs), drop their inclusion thresholds in some statistically cool and relevant way.

Update: I'm thinking of storing class C sized bins for the tallies. That's very quick and gets "nearness" automatically. (In other words, any IPs in the same /24 could be counted together initially.) How does that sound? Do I lose much by that deliberate imprecision? How much do the spammers move IPs? Would numerical nearness matter/help in detecting them?

    The approach made sense to me as well.  

http://www.stearns.org/sa-blacklist/spamip.current.txt is the report created from the A record harvesting. I'd be glad to provide the raw data collected over the past 5 months. I also have SOA and whois data for the entire sa-blacklist.

Thanks for the reference! Checking the list of IPs will probably answer some of my questions about numbers above, though I kind of want to get not-invented-here :-) in terms of the engine operation so I can prove the merits of using the SC data alone.

That said, I will check out the IPs!

Jeff C.

On Sunday, April 11, 2004, 8:51:10 AM, Raymond Dijkxhoorn wrote:

...

Update: I'm thinking of storing class C sized bins for the tallies. That's very quick and gets "nearness" automatically. (In other words, any IPs in the same /24 could be counted together initially.) How does that sound? Do I lose much by that deliberate imprecision? How much do the spammers move IPs? Would numerical nearness matter/help in detecting them?

I personally like the way DSBL handles this, if a spammer moved we will find out pretty quickly, if you list a /24 you will get in a lot of Ips that have nothing to do with the blocks, most of the time. For example spammers using a open proxy.

Do they use open proxies for their web hosting? Remember this would only be the IP addresses of their URI domains, not their mail sending.

The other thing that cuts down collateral damage is that the IPs must resolve from message body URIs, which won't be too common for FPs among the heaviest spammers. OTOH, like you, I like precision better also and don't like collateral damage either. Maybe full numbers is better than /24s.

Jeff C.

Hi!

What I have been doing is dropping routes to the, more then likely, virused/trojaned boxes. But need a better solution which is less dependant on my seeing it happening and will just kill it.

I know this isn't the best place to ask this but I figured we have all suffered these attacks and though you all might know of something or be using something that will work.

What patterns are you getting in, and what mailer are you using, if its exim it would be most likely simple, most of the times just one regexp. But please post the patterns so we know what it looks like.

Are they open proxy's? Please check some IPs on www.dsbl.org that you are getting in.

Bye, Raymond.

On Sunday, April 11, 2004, 6:52:56 AM, William Stearns wrote:

    The approach made sense to me as well.

http://www.stearns.org/sa-blacklist/spamip.current.txt is the report created from the A record harvesting.

Hi Bill, http://www.stearns.org/sa-blacklist/spamip.current.txt is nicely done and highly relevant to the questions we're pondering about the handling of resolved A records. It seems to indicate that /24-sized tally buckets could be useful. Certainly /24s are resource stingy to implement in many contexts, including ours.

Jeff C.