The levels of unsolicited bulk email (spam) has increased dramatically over the last number of years.  Today, in the USA, estimates are that roughly three quarters of all the email being transmitted through the Internet is spam.

Internet service providers, business networks and individuals are suffering from a literal tidal wave of spam.  To cope with this problem, network administrators have built a virtual wall around the access points of their networks to the Internet.

Spam and Filters

The purpose: to filter out spam, emails containing malicious files called viruses or trojans.

Unfortunately, spammers continue to find ways to thwart these defenses.  In the meantime, more and more "wanted" email is simply not being delivered; it's caught in the cross-fire in the war between system administrators and spammers.

The situation has become so serious, people started to say email was "broken" with online business suffering significant costs in coping with the onslaught of spam.

The question became how to fix the system.

Today, the underlying presumption with the email system is that all email is to be trusted as being acceptable, unless shown otherwise.

Filters have changed this understanding to a degree, but the problem with filters is these can result in significant levels of "false-positives."

A false-positive is when a wanted message is rejected, or an unwanted message is accepted.  Real-time block lists can help system administrators, but these can result in all users of a particular mail server finding email delivery being blocked.

Between filters and real-time block lists, senders have been seeing upwards of 30% of their solicited email simply not being delivered to consumer networks, with levels as high as 50% of solicited email not being delivered to business networks.

At the same time, network administrators have been confronted with a virtual deluge of spam, along with rising levels of spam complaints and customer upset over undelivered email.

Sender Authentication, Reputation and Certification

Something had to change. The first step was to change how email is treated.  In the future, all email will be considered as "bad," unless shown otherwise.

Why is this necessary?  To allow system administrators to more easily deal with email volumes, reduce false positives, server loads and costs in handling all the unwanted email.

It is estimated that America Online rejects about 1.5 billion pieces of email a day, down from 3 billion pieces of email a day this spring due to changes made in their systems by adopting this methodology.

What do administrators of ISPs, business networks and personal networks want?  A simple way of authenticating the sender.

Presently a number of schemes are being worked on to allow this to happen.  The two which have received the most wide spread coverage in the media are known as Sender Policy Framework (SPF), an open source solution and Microsoft's proposal called the Sender ID Framework, which is the "merger" of SPF and Microsoft's Caller-ID.

The big ISPs are also looking at a number of other proposals, including Compatible Low-overhead Email Authentication and Responsibility (CLEAR), an open source solution, which may ultimately include Yahoo's DomainKeys, another open source solution.

However, sender authentication is only one step in the process.  Administrators of mail systems will also want to know whether the sender has a good reputation.

So, we are now seeing the implementation of various reputation and certification schemes.

A reputation service can provide a history of a particular domains mail volumes, along with reports of bad behavior through the use of various real-time block list services.

A certification service is different.  The certification service provides information concerning the quality of the sender mailing practices, allowing administrators of receiving systems to make decisions based on the sender's rating.

This can allow for white listing of the sender at the receiving mail server, or assigning a certain number of points to messages received from an certified sender, so giving messages received a favorable rating when filtering mail at the receiving mail server.

There is another benefit for both senders and receivers.  Properly run certification services can respond to complaints from receivers, either directly or through various reporting services.  This ensures that receivers know the listings are accurate and current.  It can also facilitate resolution of delivery issues between senders and receivers.

Back to Home Page