I've got some more false positives.. this one appears to be my.yahoo.com !!!
From: "Mark Roberts" maxmedianews@yahoo.com To: "WGRC" news@wgrc.com Sent: Tuesday, December 07, 2004 5:48 AM Subject: [SPAM] 12/7
Spam detection software, running on the system
"spamcobra.swift-networks.com", has
identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or block similar future email. If you have any questions, see the administrator of that system for details.
Content preview: M.Roberts 12/7/04 News [...]
Content analysis details: (5.5 points, 5.0 required)
pts rule name description
----
0.0 HTML_MESSAGE BODY: HTML included in message 0.5 HTML_20_30 BODY: Message is 20% to 30% HTML 5.0 SPAMCOP_URI_RBL URI's domain appears in SpamCop data at
http://www.surbl.org/lists.html
[my.yahoo.com is blacklisted in URI RBL at] [sc.surbl.org]The original message was not completely plain text, and may be unsafe to open with some email clients; in particular, it may contain a virus, or confirm that your address can receive spam. If you wish to view it, it may be safer to save it to a file and open it with an editor.
Subject: 12/7 From: Mark Roberts maxmedianews@yahoo.com Date: Tue, 7 Dec 2004 02:48:31 -0800 (PST) To: WGRC news@wgrc.com
M.Roberts
12/7/04
News
(Northumberland County)---A car and an oil-delivery truck crashed head-on yesterday afternoon in Northumberland County, killing a young Luzerne County woman. Northumberland County Coroner James Kelley identified the victim as 22-year-old Ann Marie Piasecki of Wilkes Barre. Route 487 was closed after the truck and car crashed about 1:30 p-m near the Fleetwood Motor Homes plant near Elysburg. A passenger in Piasecki's car was flown to Geisinger. Police continue to investigate yesterday's fatal Northumberland County Crash.
(Montour County)---Plea negotiations are under way in the case of an Amish farmer accused of sexually assaulting several young girls at or around his farm in northern Montour County. The press enterprise says 47 year old Eli Hostetler, of Anthony Township, was granted a trial continuance at the request of defense attorney. Hostetler faces a host of charges including rape, corruption and child endangerment filed by state police at Milton. Some of the allegations made against him date back more than 11 years.
(State)---When the Electoral College meets next week, 21 of Pennsylvania's staunchest Democrats will cast their electoral votes for defeated presidential nominee John Kerry. The state's electors will gather in the state House chamber December 13th, the day the 538 members of the Electoral College nationally cast ballots confirming President Bush's victory.
(Drugs)---New research suggests that top-selling pain reliever Celebrex does not carry the same heart attack risk as Vioxx, a similar drug pulled from the market in September because of safety concerns. A study by University of Pennsylvania researchers is the first to compare the two arthritis drugs since the recall.
(Columbia County)--- The new owner of the Columbia Mall in Buckhorn is promising more stores and a movie theater. Bayview Financial of Florida will take over the mall next month from Columbia Mall Associates. The current owners say the mall has lost nearly 20 (m) million dollars in value since 1995.
(Luzerne County)--- The city of Wilkes-Barre has agreed to pay 225-thousand dollars for a former bank building in a prominent downtown location. The building that once housed the First National Bank has been vacant since the flood of 1972. City Leaders are looking at the idea of restoring it.
(Northumberland County)---Three people were injured after a crash yesterday morning at route 54 and 44 in the boro of Turbotville, Northumberland County. State Troopers say the crash occurred as 77 year old Annabelle Temple of Milton was attempting to cross 54 when she pulled into the path of 67 year old Sylvia Lawrence of Riverside. Both Lawrence and Temple were transported to Geisinger for treatment. A 2 year old passenger with Temple also suffered minor injuries and was taken to the Evangelical Hospital for treatment after yesterday's crash in Northumberland County.
(Perry County)---State Troopers are looking into a Burglary which occurred sometime yesterday afternoon at the Millerstown Town Moose in Greenwood Township, Perry County. Officers explain, suspects entered the building and removed an undetermined amount of money, before fleeing thru a side door. Anyone with information into that robbery yesterday afternoon at the Millerstown moose should contact authorities.
(Flu)---More flu vaccine is on the way. The government plans today to announce the purchase some of the five million doses available from plants in Canada and Germany. The Food and Drug Administration has signed off on the safety of a portion of the supply after doing inspections and tests.
(Pearl Harbor)---Today marks the 63rd anniversary of the attack on Pearl Harbor. Survivors are returning to the Hawaii site to honor the more than two-thousand soldiers who died in the Japanese surprise attack.
(Bloomsburg)--- Bloomsburg has plans to draw up a budget for next year that would reduce property taxes by almost 6.7 percent, but would hike a tax on in-town workers from $10 to more than $35. Today's press enterprise reports the idea was brought up at a workshop meeting last night. The state Legislature recently passed a law allowing municipalities to raise the old occupational assessment tax from $10 up to $52. Bloomsburg Councilmen entered the night looking at a roughly $309,000 budget deficit for next year.
(Montour County)---Montour County might lay off a few employees and raise taxes next year to balance its 2005 budget. According to the press enterprise, Montour County is allowed to raise property taxes by only 10 percent, which would mean about $20 more in property tax for the typical resident. Commissioners would not reveal any specific figures from the 2005 budget. They said they will release the tentative budget and new tax rates at their meeting on Thursday.
(Dog License)---Don't forget to get lucky or rover his tags….2005 dog licenses go on sale this week. Licenses are required for all dogs more than three months old. If the animal is spayed or neutered, the license is $6. If they are not, the license is $8. In 2004, county treasurers and state agents sold licenses for more than 900,000 dogs.
Do you Yahoo!? Meet the all-new My Yahoo! – Try it today!
yahoo.com and hotmail.com were briefly on sc.surbl.org due a blank line getting onto a whitelist. That problem has since been corrected and all the lists are working normally again.
Here's a little more info:
1. A blank line got into the whitelist due to some unexpected input to a script. That case is now handled at both an input script and an output script. It should be impossible again. (UNIX join gets thwarted by blank lines.)
2. yahoo.com and hotmail.com were on the SpamCop spamvertised site data. People really should not be reporting these sites to SpamCop. Frankly the reporting users are being rather careless in doing so since neither hotmail nor yahoo are reportable spam URI destinations. But that's why we have whitelisting, etc. to clean those up.
With the whitelisting broken, they were getting through on sc.surbl.org. They don't appear on other SURBL lists, because the other data sources would never deliberately list them.
It also means those were reported in sufficient quantity to make it onto sc.surbl.org which currently requires 10 reports, but in future will require many more (in a different style of counting).
Jeff C. -- "If it appears in hams, then don't list it."
Jeff Chan wrote to Matt on Tue, Dec 7th at 10:04 -0800:
yahoo.com and hotmail.com were briefly on sc.surbl.org due a blank line getting onto a whitelist. That problem has since been corrected and all the lists are working normally again.
Whoa! Thanks for the information, Jeff.
To make my searching easier, can you give me a time interval where this occurred? (Or at least a reasonable upper/lower bound?) I'd like to check our logs here to see if anything might have been nuked by this. As long as it was only yahoo.com and hotmail.com, it should just be a matter of grepping the tests for SC hits on hotmail/yahoo during the affected period, right?
Thanks, - Ryan
On Tuesday, December 7, 2004, 3:32:51 PM, Ryan Thompson wrote:
Jeff Chan wrote to Matt on Tue, Dec 7th at 10:04 -0800:
yahoo.com and hotmail.com were briefly on sc.surbl.org due a blank line getting onto a whitelist. That problem has since been corrected and all the lists are working normally again.
Whoa! Thanks for the information, Jeff.
To make my searching easier, can you give me a time interval where this occurred? (Or at least a reasonable upper/lower bound?) I'd like to check our logs here to see if anything might have been nuked by this. As long as it was only yahoo.com and hotmail.com, it should just be a matter of grepping the tests for SC hits on hotmail/yahoo during the affected period, right?
To be honest it was anything on our whitelists that were also on sc or any other lists. It started from about 02:30 pacific time until I fixed it around 05:30. Here are logs for yahoo and hotmail:
top-sites-domains.new.log:2004-12-07 10:28 yahoo.com top-sites-domains.new.log:2004-12-07 13:22 yahoo.com top-sites-domains.new.log:2004-12-07 10:28 hotmail.com top-sites-domains.new.log:2004-12-07 13:22 hotmail.com
These are in GMT; the second time was probably triggered by the whitelist joining correctly again.
Another way to fix this would be to find a sort order that puts blank lines at the end of the file instead of the beginning, but it's a moot point since it now squashes blank lines at multiple places, including the final step of producing the master whitelist, so it won't happen again.
FWIW here is the full list for SC:
http://spamcheck.freeapp.net/top-sites-domains.new.log
on:
2004-12-07 10:28 abbonati.tiscali.it 2004-12-07 10:28 altavista.com 2004-12-07 10:28 borland.com 2004-12-07 10:28 cafe24.com 2004-12-07 10:28 co.kr 2004-12-07 10:28 co.uk 2004-12-07 10:28 com.ar 2004-12-07 10:28 com.br 2004-12-07 10:28 com.ph 2004-12-07 10:28 com.tw 2004-12-07 10:28 ebay.com 2004-12-07 10:28 geocities.com 2004-12-07 10:28 hotmail.com 2004-12-07 10:28 idv.tw 2004-12-07 10:28 ig.com.br 2004-12-07 10:28 net.tw 2004-12-07 10:28 nom.br 2004-12-07 10:28 nullsoft.com 2004-12-07 10:28 nvidia.com 2004-12-07 10:28 opera.com 2004-12-07 10:28 placementusa.com 2004-12-07 10:28 tiscali.it 2004-12-07 10:28 topica.com 2004-12-07 10:28 wamu.com 2004-12-07 10:28 yahoo.com
and off:
2004-12-07 13:22 abbonati.tiscali.it 2004-12-07 13:22 altavista.com 2004-12-07 13:22 borland.com 2004-12-07 13:22 cafe24.com 2004-12-07 13:22 co.kr 2004-12-07 13:22 co.uk 2004-12-07 13:22 com.ar 2004-12-07 13:22 com.br 2004-12-07 13:22 com.ph 2004-12-07 13:22 com.tw 2004-12-07 13:22 ebay.com 2004-12-07 13:22 geocities.com 2004-12-07 13:22 hotmail.com 2004-12-07 13:22 idv.tw 2004-12-07 13:22 ig.com.br 2004-12-07 13:22 net.tw 2004-12-07 13:22 nom.br 2004-12-07 13:22 nullsoft.com 2004-12-07 13:22 nvidia.com 2004-12-07 13:22 opera.com 2004-12-07 13:22 placementusa.com 2004-12-07 13:22 tiscali.it 2004-12-07 13:22 topica.com 2004-12-07 13:22 wamu.com 2004-12-07 13:22 yahoo.com
If I can make time to redo the engine, it will be more uniform in the handling of everything and should be simpler, somewhat faster, and a bit more programatically correct. (Speed is really a non issue since I have things so lean by design that most programs complete within a few seconds.)
Jeff C. -- "If it appears in hams, then don't list it."
Jeff Chan wrote to Ryan Thompson on Tue, Dec 7th at 21:37 -0800:
To be honest it was anything on our whitelists that were also on sc or any other lists. It started from about 02:30 pacific time until I fixed it around 05:30. Here are logs for yahoo and hotmail: [...]
Thanks Jeff. This information (especially timeline and list of domains) will be very helpful. I'll run some reports to see if anything on our system was tripped by this.
(We use the SC URIBL with SpamAssassin. Our tag score is 7.0 and the score for the SC rule is 4.0, so it's unlikely in our case, unless the message also had a high Bayes score or something). Fortunately, we quarantine and never delete, so anything affected by this will still be in storage.
If I find anything interesting, I'll report back.
- Ryan
Ryan Thompson wrote to Jeff Chan and SURBL Discussion list on Tue, Dec 7th...:
If I find anything interesting, I'll report back.
Ok, thankfully, nothing *too* interesting. :-) Out of the few thousand messages we processed during those hours (10:15GMT - 13:22GMT; early in the morning for us), about ten of them had URIBL hits on at least one of the domains in Jeff's last message, and all ten of them were obvious spam. Our other rules seemingly came to the rescue to separate the wheat from the chaff. This is precisely why I like rules from multiple sources/technologies. :-)
I don't have an easy way to tell if anything scored as _nonspam_ hit the whitelisted entries[1], but that's not a concern, because anything under 7.0 points was already delivered normally.
[1] We don't include SA reports with nonspam, of course, so, it's possible to see if the SC_URIBL rule hit, but it isn't possible (without re-checking each message *against a modified SC blocklist with the bad data included*) to see if the URIBL rule hit one of the domains in question. And, anyway, for reasons discussed above, this would be for academic reasons only. :-)
Thanks again, Jeff. This would have taken a lot longer (and resulted in further delayed mail delivery if there were any FPs) without the timeline and domains you provided.
- Ryan
On Tuesday, December 7, 2004, 11:10:25 PM, Ryan Thompson wrote:
Thanks again, Jeff. This would have taken a lot longer (and resulted in further delayed mail delivery if there were any FPs) without the timeline and domains you provided.
Sure. It's an embarrassing problem but won't happen again.
Jeff C. -- "If it appears in hams, then don't list it."
Jeff Chan wrote to Ryan Thompson on Tue, Dec 7th at 23:26 -0800:
On Tuesday, December 7, 2004, 11:10:25 PM, Ryan Thompson wrote:
Thanks again, Jeff. This would have taken a lot longer (and resulted in further delayed mail delivery if there were any FPs) without the timeline and domains you provided.
Sure. It's an embarrassing problem but won't happen again.
Nonsense. Stuff happens to the best of us. It's how you respond to it that makes the difference.
Of course, all of us using SURBL on production systems are heap glad that this doesn't happen very often, but that's a bit different. :-)
- Ryan
Hi!
To be honest it was anything on our whitelists that were also on sc or any other lists. It started from about 02:30 pacific time until I fixed it around 05:30. Here are logs for yahoo and hotmail:
top-sites-domains.new.log:2004-12-07 10:28 yahoo.com top-sites-domains.new.log:2004-12-07 13:22 yahoo.com top-sites-domains.new.log:2004-12-07 10:28 hotmail.com top-sites-domains.new.log:2004-12-07 13:22 hotmail.com
To be clear, this affected the -global- whitelist. Most of the sublists also have seperate whitelists, those were not effected, so a lot was filtered on the submission end anyway.
Bye, Raymond.
On Wednesday, December 8, 2004, 1:33:18 AM, Raymond Dijkxhoorn wrote:
To be honest it was anything on our whitelists that were also on sc or any other lists. It started from about 02:30 pacific time until I fixed it around 05:30. Here are logs for yahoo and hotmail:
top-sites-domains.new.log:2004-12-07 10:28 yahoo.com top-sites-domains.new.log:2004-12-07 13:22 yahoo.com top-sites-domains.new.log:2004-12-07 10:28 hotmail.com top-sites-domains.new.log:2004-12-07 13:22 hotmail.com
To be clear, this affected the -global- whitelist. Most of the sublists also have seperate whitelists, those were not effected, so a lot was filtered on the submission end anyway.
Correct. The only whitelist we don't really have visibility on is whether SpamCop puts an internal one on the spamvertised site domains that go into SC. So in a sense there are multiple safety devices. Other data sources like ws, jp, ob, etc. have their own whitelists.
Jeff C. -- "If it appears in hams, then don't list it."
-
Jeff Chan -
Matt -
Raymond Dijkxhoorn -
Ryan Thompson