Introduction

When setting up a web site or application under your own domain, your hosting provider may also offer you the option of configuring your own mail server. Although there are many robust open source solutions such as Dovecot, hosting your own mail is often not the best option for many deployments. Because of the relatively complicated way that DNS records, spam filters, and webmail interfaces are implemented, maintaining your own mail server is becoming less popular, and less widely supported by hosting providers. Most people will get more value out of using a hosted mail service. This guide will cover many of the reasons that you may not want to run your own mail server, and offer a few alternatives.

Mail Servers Are Complex

A typical mail server consists of many software components that provide a specific function. Each component must be configured and tuned to work nicely together and provide a fully-functioning mail server. Because they have so many moving parts, mail servers can become complex and difficult to set up.
Here is a list of required components in a mail server:

Mail Transfer Agent
Mail Delivery Agent
IMAP and/or POP3 Server

In addition to those, you will probably want to add these components:

Spam Filter
Antivirus
Webmail

While some software packages include the functionality of multiple components, the choice of each component is often left up to you. In addition to the software components, mail servers need a domain name, the appropriate DNS records, and an SSL certificate.
Let’s take a look at each component in more detail.

Mail Transfer Agent

A Mail Transfer Agent (MTA), which handles Simple Mail Transfer Protocol (SMTP) traffic, has two responsibilities:

To send mail from your users to an external MTA (another mail server)
To receive mail from an external MTA

Examples of MTA software include Postfix, Exim, and Sendmail.

Note: As a general rule, even if you are committed to not running a full mail server, an MTA is still relatively straightforward to deploy on its own in order to send alerts or notifications from your software. This is sometimes challenging because some hosting providers (including DigitalOcean) will automatically block the default outgoing mail port, 25, in order to avoid being used for spam. To avoid this, you can use a third-party SMTP server. You can also review How To Install and Setup Postfix.

Mail Delivery Agent

A Mail Delivery Agent (MDA), which is sometimes referred to as the Local Delivery Agent (LDA), retrieves mail from a MTA and places it in the appropriate mail user’s mailbox.
There are a variety of mailbox formats, such as mbox and Maildir. Each MDA supports specific mailbox formats. The choice of mailbox format determines how the messages are actually stored on the mail server which, in turn, affects disk usage and mailbox access performance, as well as import/export compatibility.
Examples of MDA software include Postfix and Dovecot.

IMAP and/or POP3 Server

IMAP and POP3 are protocols that are used by mail clients — software that is used to read email, for mail retrieval.
IMAP is the more complex protocol that allows, among other things, multiple clients to connect to an individual mailbox simultaneously. The email messages are copied to the client, and the original message is left on the mail server.
POP3 is simpler, and moves email messages to the mail client’s computer, typically the user’s local computer, by default.
Examples of software that provides IMAP and/or POP3 server functionality include Courier, Dovecot, and Zimbra.

Spam Filter

The purpose of a spam filter is to reduce the amount of incoming spam, or junk mail, that reaches user’s mailboxes. Spam filters accomplish this by applying spam detection rules — which consider a variety of factors such as the server that sent the message, the message content, and so forth — to incoming mail. If a message’s “spam level” reaches a certain threshold, it is marked and treated as spam.
Spam filters can also be applied to outgoing mail. This can be useful if a user’s mail account is compromised, to reduce the amount of spam that can be sent using your mail server.
SpamAssassin is a popular open source spam filter.

Antivirus

Antivirus is used to detect viruses, trojans, malware, and other threats in incoming and outgoing mail. ClamAV is a popular open source antivirus engine.

Webmail

Many users expect their email service to provide webmail access. Webmail, in the context of running a mail server, is a mail client that can be accessed by users via a web browser. Gmail is probably the best-known example of this. The webmail component, which requires a web server such as Nginx or Apache, can run on the mail server itself.
Examples of software that provide webmail functionality include Roundcube and Citadel.

Downsides to Running a Mail Server

Although having to maintain a stack of four or five different software components in order to provide basic functionality is not ideal, it may not seem so much worse than other deployments in that regard. This, however, does not take into account the significant “trust” issues of running your own mail server.
In many ways, mail server stacks represent a collision between the tools and values of the early internet — self-hosting open source software using well-defined standards and interoperable protocols — and the reality of the modern internet — a few centralized, trusted authorities. More than web servers, database servers, or other cloud software, they have to handle an enormous amount of untrustworthy input, and the trust standards of commercial mail servers are very high as a result. Because mail servers are constantly handling attachments of potentially harmful files, and constantly filtering spam and spam addresses, it can be quite challenging to run a server that actually keeps up with the expectations of modern webmail providers. Many of them will not hesitate to block traffic from a temporarily compromised sender, especially if it is a small, self-hosted operation.

Staying Off Blacklists

It is not trivial to keep your server off of the various blacklists, also known as DNSBL, blocklists, or blackhole lists. These lists contain the IP addresses of mail servers that were reported to send spam or junk mail (or for having improperly configured DNS records). Many mail servers subscribe to one or more of these blacklists, and filter incoming messages based on whether the mail server that sent the messages is on the list(s). If your mail server gets listed, your outgoing messages may be filtered and discarded before they reach their intended recipients.
When deploying a web server, it is fairly common to experience occasional outages from DNS misconfiguration. There is a substantial ecosystem of CDNs and load balancers whose primary purpose is to prevent these minor outages from otherwise impacting your infrastructure. When it comes to mail servers, however, a minor misconfiguration can make it hard to — literally — restore trust.
If your mail server gets blacklisted, it is often possible to get it unlisted (or removed from the blacklist). You will need to determine the reason for being blacklisted, and resolve the issue. After this, you will need to look up the blacklist removal process for the particular list that your mail server is on, and follow it.

Hosted Mail Services

Hosted mail services fall into two broad categories. The first category is comprised of personal webmail providers. These service providers are widely known for their free service tiers, and usually provide paid options for hosting a custom email domain, supporting multiple users of a shared business account, and so on. They usually provide their own webmail interfaces and dedicated mobile apps.

Google Apps
Zoho
FastMail
Microsoft Office365

A second category is mail delivery services. These providers are not necessarily in the personal email business, but instead provide API access for any software that needs to send mail in bulk, such as password change notifications or advertising campaigns. Usually, these services include dedicated mail server credentials, the relevant trust and filtering features, and a web dashboard to monitor your mail volume and any related issues. They are typically priced by usage.

Mailgun
SendGrid
SparkPost

This list is not exhaustive, but should provide an overview of the service landscape.

Conclusion

Although email is a fundamental internet technology, many cloud providers are reluctant to support self-hosted mail servers because of their inherent challenges. We generally recommend using an external provider to handle email for your cloud.
If you are determined to run your own mail server, you can see a comprehensive example in How To Configure a Mail Server Using Postfix, Dovecot, MySQL, and SpamAssasin.