I run the URL shortener dft.ba. SURBL keeps sending out emails to us saying our shortener is being used for spam links. So far I have been sent around 50 of these messages, out of those messages most of them are for links that not only return a 404 error but were never even created in the first place. The other few all point to URLs which, although are correctly identified as spam manually, when querying the domains using the tool at http://www.surbl.org/surbl-analysis (And subsequently the system we use in our site) return as 'not blacklisted'.
Our application integrates with both SURBL and WebOfTrust to get reputation for URLs and automatically removes all links we detect as spammy. But what are we to do when SURBL is informing us and our ISP of URLs that don't exist or are not even blacklisted in SURBL itself. An example of this is
-- Please remove the abused shortner: http://dft dot ba /-qTY
[etc] --
This URL has never existed, not 'did exist but has now been deleted' because we don't fully delete things from our database just mark them deleted, this URL has never forwarded to anything other than our 404 page.
Another example of the other behaviour is this -- Please remove the abused shortner: http://dft dot ba /-NqD
[etc] --
This URL did exist (but has now manually been deleted), but forwards to the domain 'li.ru', not blacklisted by SURBL. Trying to access the URL by any methods from our server (CURL, WGET etc.) returns a 500 server error so it looks like the site has blocked us from automatically figuring out where the URLs are redirecting to (I guess on an IP based block, it works from other servers). If SURBL isn't going to blacklist sites why are we being alerted that the link is being abused.
Our web host says SURBL often generates "false positives that should be ignored" but I'm trying to avoid our site getting blacklisted/flagged etc.
Any suggestions? -Sam
Sam Rudge wrote:
This URL has never existed, not 'did exist but has now been deleted' because we don't fully delete things from our database just mark them deleted, this URL has never forwarded to anything other than our 404 page.
Another example of the other behaviour is this
Please remove the abused shortner: http://dft dot ba /-NqD
[etc]
This URL did exist (but has now manually been deleted), but forwards to the domain 'li.ru', not blacklisted by SURBL. Trying to access the URL by any methods from our server (CURL, WGET etc.) returns a 500 server error so it looks like the site has blocked us from automatically figuring out where the URLs are redirecting to (I guess on an IP based block, it works from other servers). If SURBL isn't going to blacklist sites why are we being alerted that the link is being abused.
Our web host says SURBL often generates "false positives that should be ignored" but I'm trying to avoid our site getting blacklisted/flagged etc.
Any suggestions?
I also run a URL shortener (Also known as "A dozen lines of PHP to shorten, and thousands of lines of anti-abuse code"), have had some run-ins with blacklists including this one. So far, I have found SURBL is the only one of the bunch that wasn't tedious or outright impossible to deal with or obviously running with no one minding the store. (My favorite was someone I know who works for a URL shortener, telling me they found themselves blacklisted and were given links to their own best practices documents (which they were following)) as a resource to avoid being listed.
What you're probably soon to be told by someone is that you really didn't want this list, and that you wanted to send an email message to whitelist (at) surbl dot org. That said, you'll definitely want to get to the bottom of the non-existent links thing. It's not out of the realm of possibility that these links which were never valid were nevertheless, used in some kind kind of spam. It's also within the realm of possibility that someone's trying to get you listed. The former much more likely than the latter, but you'll want to find out.
http://en.wikipedia.org/wiki/Joe_job
- Ron
On Tue, Aug 2, 2011 at 4:49 PM, Ron Guerin ron@vnetworx.net wrote:
Sam Rudge wrote:
This URL has never existed, not 'did exist but has now been deleted' because we don't fully delete things from our database just mark them deleted, this URL has never forwarded to anything other than our 404 page.
Another example of the other behaviour is this
Please remove the abused shortner: http://dft dot ba /-NqD
[etc]
This URL did exist (but has now manually been deleted), but forwards to the domain 'li.ru', not blacklisted by SURBL. Trying to access the URL by any methods from our server (CURL, WGET etc.) returns a 500 server error so it looks like the site has blocked us from automatically figuring out where the URLs are redirecting to (I guess on an IP based block, it works from other servers). If SURBL isn't going to blacklist sites why are we being alerted that the link is being abused.
Our web host says SURBL often generates "false positives that should be ignored" but I'm trying to avoid our site getting blacklisted/flagged etc.
Any suggestions?
I also run a URL shortener (Also known as "A dozen lines of PHP to shorten, and thousands of lines of anti-abuse code"), have had some run-ins with blacklists including this one. So far, I have found SURBL is the only one of the bunch that wasn't tedious or outright impossible to deal with or obviously running with no one minding the store. (My favorite was someone I know who works for a URL shortener, telling me they found themselves blacklisted and were given links to their own best practices documents (which they were following)) as a resource to avoid being listed.
What you're probably soon to be told by someone is that you really didn't want this list, and that you wanted to send an email message to whitelist (at) surbl dot org. That said, you'll definitely want to get to the bottom of the non-existent links thing. It's not out of the realm of possibility that these links which were never valid were nevertheless, used in some kind kind of spam. It's also within the realm of possibility that someone's trying to get you listed. The former much more likely than the latter, but you'll want to find out.
http://en.wikipedia.org/wiki/Joe_job
- Ron
Perhaps the abusers have code that creates a shortened link but doesn't check that it works, and they spam the shortened link anyway whether it works or not.
SURBL Whitelisters wrote:
Perhaps the abusers have code that creates a shortened link but doesn't check that it works, and they spam the shortened link anyway whether it works or not.
I have found it to be the case, that abuse links are most often never checked to see if they actually work after creation. I don't really track hits to disabled redirections, so I'm not speaking from facts now, but I suspect at least some of the already disabled URLs do in fact get used even though they'd been disabled long before the actual abuse takes place. The kind of attention to detail I see in the abuse (I have one IP address that I blocked years ago that continually tries to submit abuse still) would not lead me to conclude they bother checking the URLs just before use either.
- Ron
On 8/3/2011 3:33 PM, Ron Guerin wrote:
SURBL Whitelisters wrote:
Perhaps the abusers have code that creates a shortened link but doesn't check that it works, and they spam the shortened link anyway whether it works or not.
I have found it to be the case, that abuse links are most often never checked to see if they actually work after creation. I don't really track hits to disabled redirections, so I'm not speaking from facts now, but I suspect at least some of the already disabled URLs do in fact get used even though they'd been disabled long before the actual abuse takes place. The kind of attention to detail I see in the abuse (I have one IP address that I blocked years ago that continually tries to submit abuse still) would not lead me to conclude they bother checking the URLs just before use either.
This makes sense, given the lack of attention to detail spammers put into their craft in general. Perhaps it would be worthwhile if SURBL's (and others') processes included checking pages for 400 error codes before sending (automated?) abuse reports?
On 08/03/11 18:54, Dave Warren wrote:
On 8/3/2011 3:33 PM, Ron Guerin wrote:
SURBL Whitelisters wrote:
Perhaps the abusers have code that creates a shortened link but doesn't check that it works, and they spam the shortened link anyway whether it works or not.
I have found it to be the case, that abuse links are most often never checked to see if they actually work after creation. I don't really track hits to disabled redirections, so I'm not speaking from facts now, but I suspect at least some of the already disabled URLs do in fact get used even though they'd been disabled long before the actual abuse takes place. The kind of attention to detail I see in the abuse (I have one IP address that I blocked years ago that continually tries to submit abuse still) would not lead me to conclude they bother checking the URLs just before use either.
This makes sense, given the lack of attention to detail spammers put into their craft in general. Perhaps it would be worthwhile if SURBL's (and others') processes included checking pages for 400 error codes before sending (automated?) abuse reports?
Not sure if this is material to this discussion or not. This morning, we got some spam with URL's in it that pointed to fake 404 pages, which then tried to download a trojan exe from yet another site.
This was part of the Federal tax payment rejected (yes I am in the US) series that appeared overnight on my mail servers.
BTW, I routinely click on these as my normal workstation is a linux box and 99.9% of these target Windoze boxes.
Lyle Giese LCR Computer Services, Inc.
So from my POV as operator of the link shortener, so long as I keep an eye on things and make sure our protection system is working and reported links return 404's I shouldn't really worry about the SURBL notifications too much? -Sam
On 4 Aug 2011 01:08, "Lyle Giese" lyle@lcrcomputer.net wrote:
On 08/03/11 18:54, Dave Warren wrote:
On 8/3/2011 3:33 PM, Ron Guerin wrote:
SURBL Whitelisters...
Not sure if this is material to this discussion or not. This morning, we got some spam with URL's in it that pointed to fake 404 pages, which then tried to download a trojan exe from yet another site.
This was part of the Federal tax payment rejected (yes I am in the US) series that appeared overnight on my mail servers.
BTW, I routinely click on these as my normal workstation is a linux box and 99.9% of these target Windoze boxes.
Lyle Giese LCR Computer Services, Inc.
_______________________________________________ Discuss mailing list Discuss@lists.surbl.org http:...
Hi!
So from my POV as operator of the link shortener, so long as I keep an eye on things and make sure our protection system is working and reported links return 404's I shouldn't really worry about the SURBL notifications too much?
SURBL does check the links before sending.
A good thing would be to provide a API 'like the big redirectors alleady do' where we can check things on.
BIT.LY for example has a API we use for checking if a link is available or not.
Not sure if this is material to this discussion or not. This morning, we got some spam with URL's in it that pointed to fake 404 pages, which then tried to download a trojan exe from yet another site.
Exactly. A 404 doesnt tell much. In fact. To make this more specific. There are botnets that give a 404 or alike codes if you are outside the US. So they can target specific audience. Those codes cant be really trusted in cases like this.
Remember, if you get reports of abuse, in 99.9% of the times there is something going on. You can ignore them and think you are ok. Most of the times thats whishfull thinking however.
Bye, Raymond.
I meant if the redirector returned a 404, I.E. the short link doesn't exist. And we do provide an API but I assume I would have to set up something special for this. Are there any guidelines to doing this? -Sam
On 4 Aug 2011 08:30, "Raymond Dijkxhoorn" raymond@prolocation.net wrote:
Hi!
So from my POV as operator of the link shortener, so long as I keep an eye on things and make ...
SURBL does check the links before sending.
A good thing would be to provide a API 'like the big redirectors alleady do' where we can check things on.
BIT.LY for example has a API we use for checking if a link is available or not.
Not sure if this is material to this discussion or not. This morning, we got some spam with UR...
Exactly. A 404 doesnt tell much. In fact. To make this more specific. There are botnets that give a 404 or alike codes if you are outside the US. So they can target specific audience. Those codes cant be really trusted in cases like this.
Remember, if you get reports of abuse, in 99.9% of the times there is something going on. You can ignore them and think you are ok. Most of the times thats whishfull thinking however.
Bye, Raymond.
_______________________________________________ Discuss mailing list Discuss@lists.surbl.org
Hi Sam,
I meant if the redirector returned a 404, I.E. the short link doesn't exist. And we do provide an API but I assume I would have to set up something special for this. Are there any guidelines to doing this?
A good thing would be to provide a API 'like the big redirectors alleady do' where we can check things on.
BIT.LY for example has a API we use for checking if a link is available or not.
Not that i know of. But it would be a good startingpoint i guess to just copy functionality of the BIT.LY API. Thats used in multiple places i know so easier to adopt elsewhere.
Bye, Raymond.
On Wed, Aug 3, 2011 at 5:07 PM, Lyle Giese lyle@lcrcomputer.net wrote:
On 08/03/11 18:54, Dave Warren wrote:
This makes sense, given the lack of attention to detail spammers put into their craft in general. Perhaps it would be worthwhile if SURBL's (and others') processes included checking pages for 400 error codes before sending (automated?) abuse reports?
Not sure if this is material to this discussion or not. This morning, we got some spam with URL's in it that pointed to fake 404 pages, which then tried to download a trojan exe from yet another site.
This was part of the Federal tax payment rejected (yes I am in the US) series that appeared overnight on my mail servers.
BTW, I routinely click on these as my normal workstation is a linux box and 99.9% of these target Windoze boxes.
It's probably safer to use a text browser like curl. Be sure to set the user-agent to something workstation like.
I agree, the systems should check the shortener link to see if it returns 30{1,2} or 40x. Unfortunately our web-host thinks the alerts are an issue (they disabled the site for a while until I'd discussed the situation with them). So there is no way to disable/prevent these alerts? I don't see any way we can prevent spammers putting links to our shortener in emails other than stop the links from working, which we're doing. =( -Sam
On 4 Aug 2011 00:55, "Dave Warren" lists@hireahit.com wrote:
On 8/3/2011 3:33 PM, Ron Guerin wrote:
SURBL Whitelisters wrote:
Perhaps the abusers have code ...
This makes sense, given the lack of attention to detail spammers put into their craft in general. Perhaps it would be worthwhile if SURBL's (and others') processes included checking pages for 400 error codes before sending (automated?) abuse reports?
_______________________________________________ Discuss mailing list Discuss@lists.surbl.org http...
-
Dave Warren -
Lyle Giese -
Raymond Dijkxhoorn -
Ron Guerin -
Sam Rudge -
SURBL Whitelisters