Blog:

I have to be honest with you. If you run a web app and it sends email, right now you’re probably blind. You have an app or site, you setup a local SMTP server and start firing messages through it. You don’t know if they are bouncing, making it to the inbox or just failing altogether.

To be fair, it’s not your fault. This is the way most people handle email. To ensure that those important emails get to the inbox, there is a lot more involved. I’m going to explain how to make it happen in two ways: The hard way and the easy way.

Before getting into the details, I should probably tell you about myself. My company, Wildbit, runs Postmark, which is an API based service for delivering email in web apps. While Postmark is new, we’ve been running Newsberry (an email marketing system) for years, giving us the experience to properly handle a large volume of email for many customers. Let’s jump in.

The Hard Way (Brace Yourself)

Email is old and seemingly simple, while at the same time extremely effective. If you send even a moderate amount of email from your app, you can outsource the complexities to a provider like us, or do it on your own. Here’s what you’ll need to know if you want to do it yourself.

Mail Servers and DNS

Obviously to send email, you need a good mail server. Any well known open source email server will work fine. On the commercial side there are some very good providers that offer more specific configuration options for outbound sending. Personally, I highly recommend PowerMTA from Port25.

The next important step is configuring DNS (Ed: Confused by DNS? Read this Think Vitamin article by Chris Lea). Without this, receiving mail servers have no idea who you are. Make sure to give your mail server a dedicated IP address and a domain. Now, this is where most people stop, and also where they get into trouble.

The final step is having a proper PTR (or reverse DNS) record for the dedicated IP address. Most email receivers require that your IP address resolves to the domain you chose for your mail server. Without this, many ISPs will quickly block your messages.

Email Authentication

Once your mail server is up and running and messages are sending, it’s time to think about authentication. No, I am not talking about a username and password. Email authentication was created to help validate and verify senders. The standards we’ll discuss are SPF and DKIM

SPF (Sender Policy Framework)

SPF is pretty simple. You create a record in your DNS and define the IP addresses or domains who are allowed to send for your domain. When an email is sent, the receiving mail server looks this up and decides if it should accept the message.

DKIM (DomainKeys Identified Mail)

DKIM is also managed via DNS, but is very different. With DKIM you designate the various servers that you send email from (email marketing, transactional, departments, etc) and give each one a private and public key pair.

The public key is added to DNS and the private key is configured on your mail servers. When you send email, each message is “signed” with the key and used by receiving mail servers to identify and verify your domain.

Reputation & Monitoring

If you send even a moderate volume of email, you can be sure that ISPs are building a reputation on you. They watch email coming from your IP or domain, then track things like spam complaints, bounces, unsubscribes and volume. From there, they can determine a reputation and decide how to treat your messages (inbox, reject, spam).

ISPs are now building reputations on the from domain, and further, basing that reputation on things like opens and clicks (engagement). Remember that DKIM stuff? By signing your messages, it helps email receivers identify your sending domain and build a good (or bad) reputation that you can hold on to.

While ISPs don’t publish their reputation data, there are two well known public resources: Senderscore and Senderbase. You can use these for free to track your own IP reputation, but keep in mind they are not definitive and should be used for reference only. The best way to test delivery results is through something like Mailbox Monitor from Return Path, which will track the inbox, missing and spam rates of a seed list to major ISPs worldwide.

Throttling (or Being Polite)

The amount of spam ISPs need to deal with is both overwhelming and expensive. As a reputable sender, it is important to remain polite and follow their rules. Each ISP has rules about how you should send email to their servers. This includes the number of connections you can open, messages per connection, what to do if you receive certain responses.

Remember I mentioned commercial mail servers? This is where they really shine. PowerMTA, for example, has some great options to not only control these settings, but to change them based on the responses from the ISP. For example, during the holidays many email servers get overloaded and start rejecting connections. Instead of overloading them more and reconnecting, the mail server can backoff a bit to reduce the load.

Whitelists and Feedback Loops

Heard enough already? I’m almost done so you can read the easy way. The final important aspect of sending email is getting listed (no, not blacklisted). A growing number of ISPs having something called a Feedback loop. In short, it is a way to get notified when someone clicks “This is spam” in their email client.

Most people don’t even know about it, but it’s just as important as having an unsubscribe option. By tracking spam complaints, you can easily remove these people from your list and avoid damaging your reputation. This is also a good metric to understand how relevant your email is to subscribers. Something to keep in mind, AOL (and others) consider 1 complaint for every 1,000 messages a target rate for bulk senders.

Whitelists still exist, but only with ISPs such as Yahoo, AOL, Verizon, and United Online. If you send a high volume of email, you may want to look into commercial accreditation services like SenderScore Certified and ISIPP.

Okay, even I am bored at this point. Almost all of the stuff I just told you can be accomplished with hardly any effort, so let’s jump to the easy way.

Doing it the Easy Way

An easy way actually exists, and all you have to do is use a service that already has all of this in place and follow their instructions. When it comes to sending transactional email from your web app, there are a few services popping up lately, but I’ll just explain how we tackle the above in Postmark.

Mail Server and DNS

When you create an account, you can add a “virtual server” for each project or site. In the background, all of the magic for configuring the mail server and setting DNS is done automatically. All you have to do is drop some code into your app (we have many libraries) or point to our SMTP server to start sending.

Email Authentication

Before you can start sending, Postmark requires that you create a “sender signature” for each FROM email. This confirms that you own the email address, but it also informs us of the domain you use for sending. By knowing the domain, it is then easy to create a DKIM pair and explain the proper settings to add DKIM and SPF to your DNS. It turns a complex process into something easy and straight forward. Even if you choose not to implement DKIM for your own domain, Postmark will still sign emails with our own domain, allowing you to “borrow” our reputation.

Reputation and Monitoring

By using an Email Service Provider (ESP), you are also using their reputation. It’s extremely important to make sure that the ESP handles abuse before it happens and keeps their customers under control. By maintaining a good reputation across a high volume of email, you then benefit from the reputation for your own messages.

With Postmark, since we sign outgoing messages (DKIM), you can either benefit from building your own reputation or piggy back on our well maintained reputation.

When it comes to tools, we use ReturnPath and ISIPP to monitor our reputation, inbox rates, and blacklists. If any issues arise, they are known immediately so they can be resolved. Of course, maintaining a good reputation is all about stopping things “before” they happen. This is why we have our own algorithms in place to alert if a customer is misbehaving. This could range from high spam complaints to content issues or even anomalies in volume.

Throttling

By using PowerMTA, we can quickly analyze and adapt to what the ISPs want when it comes to sending practices. These rules can change based on reputation as well, so they go hand in hand. For instance, if we add a new IP address to our servers it is basically unknown to the ISPs. We can easily “warm up” this new IP address slowly over time until it has built enough of a reputation to send a higher volume.

Whitelists and FBLs

Out of everything mentioned, this is the easiest part. For each IP or sending domain we just make sure we have feedback loops in place to monitor your spam complaints and maintain any existing whitelists. An elegant reporting interface is available for customers to view spam complaints if they occur.

Wait, There’s More!

There are some features that a hosted provider can offer that a regular mail server might not provide. The most important is bounce handling. By using Postmark, all bounce types are automatically tracked and if appropriate, deactivated. For instance, if an email no longer exists you should stop sending to it.

This is not only polite to the ISP, but also helps you recover and report these problems to your customers or members. You can also categorize (tag) and track messages that were sent, along with a full archive of the content. These features are great for troubleshooting and monitoring.

Summing it Up

If you are still with me, the short answer to Email Delivery for Web Apps is just like anything else: Focus on what you do best and give the rest to the experts. By using a hosted service for email delivery you can have more confidence that your emails are delivered to the inbox.

Further reading and resources:

• Postmark
• Sender Policy Framework
• DKIM and DKIM Core
• MAAWG Senders Best Communications Practices
• ISP Information on Word to the Wise
• Return Path
• ISIPP
• Port25

Questions or Feedback?

If you have any general questions about email, deliverability, or want to discuss, leave a comment below or email me at cn@wildbit.com.