Spam Email

A number of proposals that aim to halt the relentless growth of unsolicited commercial email, or spam, are being debated in Congress ...

http://www.acm.org/technews/articles/2003-5/0523f.html#item8

Routing junk email through unwitting third parties, usually home and office Internet users, is the No. 1 distribution method spammers use, and ISPs such as America Online estimate that over 200,000 computers around the world have been exploited in this fashion over the last two years ...

http://www.acm.org/technews/articles/2003-5/0521w.html#item3

Brightmail estimates that spam will account for about 50 percent of all Internet mail sent this year, while Ferris Research reckons that dealing with junk email will cost American businesses $10 billion ...

http://www.acm.org/technews/articles/2003-5/0521w.html#item4

Reverse MX provides a mechanism in DNS for domains to vouch for certain IP addresses; MTAs which subscribe to the ReverseMX philosophy will only accept messages whose sender domains match the published RMX IP addresses. This concept should have been in SMTP and DNS from the very beginning. It's backed by the Anti-Spam Research Group at IETF. Reverse MX allows Hotmail, Yahoo, and other commonly-forged sender domains to protect their names.  Spammers will have to instead forge sender domains which have not set up ReverseMX entries in their DNS. If ReverseMX is widely adopted, only those domains who do not have ReverseMX set up will show up in forged spam sender addresses.  This encourages domain owners to set up RMX because it is costly to handle the resulting bounce messages and misinformed abuse complaints.

Spam blacklists can then become domain-specific.  Right now most blacklists go by IP network and for political reasons will blacklist an entire hosting provider's IP range in an attempt to pressure them to enforce their AUP against a single spamming customer.  I consider the collateral damage unacceptable; blacklists such as SPEWS cause more trouble to nonspammers than to spammers.  With ReverseMX, blacklists will contain two types of domains: known spamming domains, and known "non-RMX-compliant" domains which are the 21st century moral equivalent of the open relay.

http://www.ietf.org/internet-drafts/draft-danisch-dns-rr-smtp-01.txt
http://www.mikerubel.org/computers/rmx_records

The spam stopping ideas mentioned in two recent messages to IP, designated sender and e-postage, have been debated at length in the e-mail community. Unfortunately, each has technical and social problems that make them unworkable.  The existing e-mail system is large, complex, and has operational aspects that are often subtle, and a lot of superficially plausible ideas have already been evaluated and discarded for good reasons.
Reverse MX and other designated sender schemes attempt to prevent people from sending mail from unauthorized hosts, so that. for example, yahoo.com could identify the hosts that are supposed to be sending mail with Yahoo return addresses, and receiving hosts can reject mail purporting to be from Yahoo that originates elsewhere.
A significant technical problem with Reverse MX is that large mail domains like yahoo.com have distributed mail hosts all over the world.  Reverse MXproposes that a mail server make a DNS query to find the addresses of all of the valid sending servers for an incoming message, but all those addresses won't fit in a 512 byte DNS response packet.  (The DNS spec provides for larger packets sent with TCP, but in practice, larger packets are much slower and many DNS implementations don't handle them right.)This problem isn't hard to fix once you realize it's a problem; see Gordon Fecyk's Designated Mailer proposal which is similar but better thought

http://www.ietf.org/internet-drafts/draft-fecyk-dsprotocol-02.txt
out:
The social problem with designated sender is that there are plenty of perfectly legitimate reasons for mail from a domain to originate someplace other than its home network.  Lots of people maintain accounts at Yahoo or other free mail providers, but send mail with their Yahoo address from their home ISP using the ISP's mail server.  Many others use forwarding services such as pobox.com, which would all be unable to function with designated sender, since mail forwarded by such services correctly retains the original sender's address, not the forwarding service's.  And finally, this won't really block any significant amount of spam, since there will always be some domains who out of political principle, malice, or incompetence designate the entire Internet as their valid sender ranges, and spammers can just use those.  Or spammers can register throwaway domains of their own, since burning an $8 domain for a 10 million message spam run isn't much of a deterrent.

The technical problem here is much more serious: nobody has any idea how to build a micropayment scheme that could scale up to the size needed to handle the world's e-mail and work reliably enough to deter spam. Cybercoin systems require that recipients ask the issuing bank if the coin is genuine and hasn't already been spent.  There are probably a hundred billion e-mail delivery attempts per day in the U.S. (Hotmail alone reports about two billion.)  By comparison, there are maybe 100 million credit card transactions a day, so this would require a system that can handle a thousand times the transaction volume of the credit card system. Some designs use statistical validation, only check some fraction of the coins, but in view of the reality that many systems report 80% or more of their mail is spam, you have to validate everything or else let a lot of spam through.  There are other technical problems (how do you clear a ten cent transaction between individuals the U.S. and Indonesia?)  but the transaction volume is the most obvious. Even if through some technical breakthrough we were able to affix e-stamps to every message, we'd turn the e-mail system into something like a phone network where all the numbers start with 1-900.  Since non-commercial mailing lists like IP couldn't exist if they had to pay postage, and most people don't want to charge their friends to write to them, proposals generally have some way to waive postage from known senders or only cash the coin if you don't like the message or otherwise vary the price at the receipient's option.  But this replaces the spam problem with a new world where you have no idea how much postage you'll be paying, and with lots of innovativive e-postage scams, both to avoid paying postage on outgoing mail, and to trick people into sending mail to receipients who want to collect the incoming postage.  Unlike the spam problem, these scams quickly involve large amounts of real money. Think of chain letters saying "write to this address to get coupons for free beer, they won't even cash your stamp."  (Yeah, right.)  Or let's say a virus on your computer sends out a thousand spams.  Who pays the postage?  If the answer isn't "you do", who decides to waive the postage?  How do you tell a user with a real virus from a spammer who deliberately infects his own computer? Doubtless we can come up with a whole set of laws and rules and adjudication procedures, but I don't see any reason to believe that what we'd end up with would be preferable to the admittedly lousy situation we have now.

I'm not arguing that nothing can work so we should throw up our hands, but it's dismaying that the same old unworkable anti-spam approaches keep reappearing over and over, reinvented by people who haven't done the most rudimentary investigation of prior work, invariably foundering on the same problems that came up the last six times that similar proposals failed.

There's plenty of room for innovative thinking, both to try to identify and deter spam, and to pick out the real mail from among the spam and get it to the receipients.  But please, let's stop going in circles, build some prototypes, run some experiments to see how they work, and try to move forward instead.

Here's an unhappy prediction: The explosion of spam-blocking technology could herald the death of much legitimate e-mail. I wrote about patents relating to this technology, known as challenge-response technology, last week. Basically, when your mailbox is protected by a challenge-response system, people who try to contact you will be greeted with a response saying something like "click on this link to deliver this message" or "type in the word you see in the box above." The idea is to block increasingly obnoxious spam bots but still let actual humans get in touch with you. In theory, well-designed challenge-response utilities won't challenge mail from known correspondents or mail that you've actually asked to receive. Unfortunately, many current challenge-response systems are poorly designed, which could wreak havoc on mailing lists and other legitimate communications. This could make e-mail far less useful than it is today.
It's already starting to happen. SpamArrest.com began challenging mailing list messages last year. Recently Mail-block.com and iPermitMail.com followed suit.
When that happens, the operator of the mailing list receives a message--from each subscriber using the poorly designed challenge-response utility--that asks the list operator to respond to the challenge. Replying to a handful of challenges is no big deal, but if many subscribers start using poor challenge-response software, it will pose a serious problem for mailing list operators. Big corporations may be able to afford to hire someone to sit in front of a computer and spend all day proving they're not a spam bot, but nonprofit groups, individuals and smaller companies probably can't.

A system that backers claim will eliminate e-mail spam is about to be deployed by a major Internet service provider, giving a boost to an emerging technology that if widely adopted would change how people communicate online.

Atlanta-based EarthLink Inc., the country's third-largest provider of for-pay e-mail accounts, will roll out test versions of the system for its 5 million subscribers this month.

Known as "challenge-response" technology, the system thwarts the ability of spammers to reach their intended audience with millions of automatically generated e-mails.  When someone sends an e-mail to a challenge-response user, he or she gets an e-mail back asking to verify that the sender is a live person.

http://www.washingtonpost.com/wp-dyn/articles/A22390-2003May6.html

Spam is a growing problem for email users, and many solutions have been proposed, from a postage fee for email to Turing tests to simply not accepting email from people you don't know. Spam filtering is one way to reduce the impact of the problem on the individual user (though it does nothing to reduce the effect of the network traffic generated by spam). In its simplest form, a spam filter is a mechanism for classifying a message as either spam or not spam ...

http://freshmeat.net/articles/view/964




With the exception of the latest spate of DAV trouble, most of the spam that appears to be from hotmail.com isn't actually sent from hotmail.com.  The fraction that is gets reported to abuse@hotmail.com and hotmail.com takes the appropriate action.

I analyzed 6,810,374 unique deliveries over a two-month period whose senders claimed to be from aol.com, hotmail.com, and yahoo.com.  Those deliveries came from 1,885,248 distinct email senders.  I classified those senders using statistical methods into 1,775,660 spammer addresses and 109,588 nonspammer addresses.

Of the 1,775,660 addresses which my classifier decided were more likely to be spammers than not-spammers, 4,188 actually originated from aol, hotmail, or yahoo.  That is a statistically insignificant number and reflects more on the imperfection of my classifier scheme than anything else.  The classifier scheme is described at http://dumbo.pobox.com/spam-sensor/.

Conclusion: aol, hotmail, and yahoo have successfully implemented outbound antispam technology, eg. ways to that only humans sign up for their accounts, and limits on per-account outbound message volume.

| "how will SPF stop someone from registering asdlfkslt12324349584.com,
| sending out a batch of spam...then losing the account and moving to
| asdlfkslt12324349585.com? "

This is a good question, and it is answered at

  http://spf.pobox.com/faq.html#noprevent

I just uploaded that file; if you didn't see it on your first pass through the site, your eyes are not deceiving you :)

The short answer is, even if it's a throwaway domain, we can eventually subpoena the registrar and track down the spammer, then apply legal methods.

Thanks to the greater level of sender accountability, lawsuits may begin against the spammers, and registrars may be subpoenaed for domain owner information.  SPF makes administrative and legal methods possible.

| "how will the non-spammers of videotron.ca react when someone uses a
| videotron.ca account and videotron.ca gets called a spam domain? "

videotron.ca is responsible for monitoring its outbound mail stream and sensing suspicious activity.  most major ISPs do this already.

if the volume of spam decreases, legal and administrative approaches become more effective; right now they are simply swamped.  if there are only 10 spammers in the world, law enforcement can focus on catching each one.  if there are 10,000 spammers, law enforcement throws up its hands and says the problem is too big to tackle.

|
| In short...I wonder... how will this possibly work?
|

If SMTP were proposed today, would you raise the objection that it would make it possible for "spammers" to send "unsolicited bulk email"?  And would you then say " ... I wonder ... how will SMTP possibly work when it allows such a thing as spam"?