How to Set Up Selfhosted Email Server

2023-11-25 ⏳10.0 min(4.0k words)

Discussion on HN https://news.ycombinator.com/item?id=38419148

I have used email service with my own domain for several years. Gandi.net were chosen as my domain registrar, not only because its low fees, but also its free mailbox service. However, since the middle of this year, Gandi.net has announced their price increment and would cancel all free mailbox service in the end of November. Although it is a controversial decision1, I have no choice but to find an alternative mail service as soon as possible. As I also own some always free instance of Oracle Cloud Computer, why not set up my own selfhosted mail service? So do I, and in this blog, I will share how to set up one selfhosted mail server securely and freely, yet without any fee.

Before start, we need do learn some theories.

How Email Works

While sending/receiving Email is easy, running a complete mail service is very hard. One reason of this hardness is because Email is one federal system, every message need to be delivered by one server to another server. People can not deliver their message directly to the recipient’s server in now days. All these mail server use the Simple Mail Transfer Protocol (SMTP) to communicate.

When the recipient accept the mail, it will be saved on the server. The user need to use another protocol like POP3 or IMAP to connect to the server to check their messages. In some big mail service provider, the SMTP service and POP3/IMAP service are distributed on different server, which makes the system further complicated.

Overview

So the big picture of email delivery is like this:

            mx.example.com              mx.example.net  
           ┌────┐     ┌───┐     SMTP    ┌───┐   ┌────┐  
           │IMAP│◄────│MX1│◄───────────►│MX2│──►│IMAP│  
           └─┬──┘     └───┘             └───┘   └──┬─┘  
             │          ▲                 ▲        │           Server
-------------│----------│-----------------│--------│---------------------
             │IMAP      │                 │        │IMAP       Local
             ▼          │                 │        ▼
       ┌───────────┐    │                 │    ┌─────────┐
   ┌──►│Thunderbird│────┘ SMTP Submission └────┤AppleMail│─────┐
   │   └───────────┘                           └─────────┘     │
   │                                                           ▼
 Alice                                                        Bob

Let’s dive into the process for Alice sending mail from alice@example.com to bob@example.net.

First, Alice open its mail client Thunderbird, as known as Mail User Agent (MUA). And then, she composite its mail content. When finish, she click the send button. And the MUA try to connect to the Mail Transfer Agent (MTA), MX1, using the SMTP protocol and submit her email to it. In this procedure, the MTA need to authorize the MUA by letting MUA offer the username and password. After the authorization, the MUA use the SMTP protocol to submit the mail.

MX Record

After mail submission finish, the MTA starts to try deliver the mail to Bob’s MTA. The MX1 first lookup the MX DNS record of the domain example.net, which is Bob’s email address’ domain. And it will get a list of host name with weight. MX1 will try to connect to host according their weight in ascending order. So MX1 connect to MX2 using the TCP port of 25. MX1 tells the MX2 that it want to deliver one mail from alice@example.com to bob@example.net. As MX2 is the MTA of example.net, it knows bob@example.net is a legal mail account, so it needs to receive this mail.

Here is the MX records of gmail.com:

drill mx gmail.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 64117
;; flags: qr rd ra ; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; gmail.com.   IN      MX

;; ANSWER SECTION:
gmail.com.      3389    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
gmail.com.      3389    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.      3389    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.      3389    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.      3389    IN      MX      5 gmail-smtp-in.l.google.com.

;; ...

PTR Record

However, in order to prevent spam email, MX1 has to do some check to assure MX1 is the legal MTA of mail domain example.com. There are several task have to be done.

When MX1 establish the TCP link with MX2, it’s first message looks like

EHLO mx.example.com

And this EHLO message tells MX2 the sender’s hostname is mx.example.com. MX2 may lookup the PTR record according to the server’s IP address, and check whether the response match the sender’s hostname reported by EHLO. This is why we need to set the right PTR record.

We can use the drill to query the PTR record:

drill -x 8.8.4.4
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 3912
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; 4.4.8.8.in-addr.arpa.        IN      PTR

;; ANSWER SECTION:
4.4.8.8.in-addr.arpa.   58104   IN      PTR     dns.google.

;; ...

The response says the address 8.8.4.4 is associated to the domain dns.google.

If the sender’s PTR does not point to its hostname, it may be spam sender.

However, as far as I known, modern MTA tends to not do this reverse DNS lookup, because it will cause too much burden to the zone of in-addr.arpa and ip6.arpa, and this will make the mail delivery very slow. But it is still recommend to set the PTR record to fulfill MTA which still do such check.

SPF Record

If the recipient MTA does not check PTR record, how could they detect the spam sender? The answer is the SPF record. The modern MTA will lookup another TXT record with specific structure of SPF2. And here is on simple SPF record,

drill TXT google.com|grep spf
google.com.     1319    IN      TXT     "v=spf1 include:_spf.google.com ~all"

The prefix v=spf1 means it is a SPF record. The include:_spf.google.com means all the SPF rule should lookup from the SPF record of domain _spf.google.com. And the finally ~all means if the sender cannot fulfill the rule of SPF, the email delivered should be move to trash.

Let’s dig the SPF of _spf.google.com:

"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"

It’s still include another three SPF list. So we again lookup up the SPF records of the first two domain:

"v=spf1 ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16 ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19 ~all"
"v=spf1 ip6:2001:4860:4000::/36 ip6:2404:6800:4000::/36 ip6:2607:f8b0:4000::/36 ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all"

The SPF record of _netblocks.google.com specifies several IPv4 address block, and _netblocks2 specifies several IPv6 address block. All addresses belong to these blocks are allowed to send mail from the domain of google.com. Mail from IP address not listed in SPF records will be trashed.

The SPF is more lightweight than PTR, because there is no need to contact the owner of the IP address to modify the PTR records. And the SPF solution is more efficient than PTR. However, it is not the final solution. Because there still are methods to hijack IP address to send spam email. For example, some bad ISP can utilize the BGP protocol to hijack IP address not own by themselves. So some ultra solution is needed. And it is the DomainKeys Identified Mail Signatures (DKIM).

DKIM Record

The main idea of DKIM3 is to public one signing public key using a special DKIM record. Every mail sending from one MTA will be attached the digital signature with the corresponding private key, and the recipient MTA can then lookup the sender’s public signing key to very the signature. Only the DKIM signature check passed can the mail be accepted.

Here is an example of DKIM record:

drill txt default._domainkey.example.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 50686
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; default._domainkey.lvht.net. IN      TXT

;; ANSWER SECTION:
default._domainkey.example.org.    300     IN      TXT     "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAA..."

And here is an example of DKIM signature:

DKIM-Signature: a=rsa-sha256; bh=iQ/2DHA4xYuzNkuty4mmtgXpGyW0lOv+tDU9bZ5y9zA=; c=relaxed/relaxed; d=example.com; h=Subject:Subject:Sender:To:To:Cc:From:From:Date:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Reply-To:In-Reply-To:Message-Id:Message-Id:References:Autocrypt:Openpgp; i=@example.org; s=default; t=1698495966; v=1; x=1698927966; b=fsRwrjAXpqNbPhB...

How to calculate this signature is out of the scope of this article. But we should notice the s=default and d=example.com which will be used to fetch the DKIM record. In this example, the recipient MTA will lookup the DKIM of default._domainkey.example.com, and then verify the signature.

By the DKIM, the Mail Service Provider will totally remove the dependence on the low level ISP, and will assure no mail could be send out of their own infrastructures.

DMARC

Although the recipient MTA could handle the email according the SPF or DKIM in themselves, there is still mechanism for the sender to change the receivers’ behavior. And this is Domain-based Message Authentication, Reporting, and Conformance (DMARC)4.

The DMARC is a little complicated, and I am not going to dig the detail about it. The main idea is utilizing DNS to publish a special DMARC TXT record like

_dmarc.example.org. TXT "v=DMARC1; p=quarantine; ruf=mailto:postmaster@example.org"

The label of TXT record is _dmarc. The v=DMARC1 is fixed currently. The p= means the policy. If the SPF or DKIM check fails, the recipient side should take action according to the DMARC policy. Three values are defined:

The ruf=mailto:postmaster@example.org requires the receiver report the failure information to the mail of postmaster@example.org. So that the sender operator will find the problem as soon as possible.

When Bob want to check his mailbox, he will open it MUA. And then, the MUA will connect its IMAP server to check the new message. If there are new mails, the MUA will display them in its UI and show to Bob.

STARTTLS

This is the total transfer procedure. Even though it is very complicated, the plain text design of SMTP/IMAP makes the system worse. In the early date of Internet, all the data are transferred in plain text, which means they can be sniffed by the ISP. So the original Email system has huge security risk. In order to fix this problem, TLS or STARTTLS5 have been introduced, which make the system even more complicated.

The SMTP and SMTP submission both used the plaintext TCP on port 25. The plain POP3 protocol uses 110 port on TCP, and IMAP uses 143 port on TCP. We people want to introduce TLS to protect the mail traffic, they simply assign new dedicate ports for the TLS version. So they let the TLS version of POP3 use 995 port of TCP, and the TLS version of IMAP use 993. For SMTP with TLS, the port of 465 is assigned.

The solution works but with only one problem, there are only 1024 ports reserved for well known services. If all protocols need one additional port for the TLS version connection, the well known ports will be exhausted very soon. So another solution, STARTTLS, has been introduced. In STARTTLS, when the client side build new TCP link, the server side will indicate whether it support TLS. If it does, the client will try to upgrade the current plaintext TCP link to TLS session. By this design, all legacy plaintext protocols gain the capability on the some port.

MTA-STS

The STARTTLS also has one crucial problem, the Man-in-the-middle attack (MITM). The if the ISP want to hijack the mail traffic, they can intercept the server’s STARTTLS response, and let the client only use the plaintext transportation. In order to address this issue, the SMTP MTA Strict Transport Security (MTA-STS)6 has been designed.

The main idea of MTA-STS is using a special TXT record to indicate the domain of recipient has a STARTTLS ploice, which is published by a well known HTTPS endpoint.

The DNS label is _mta-sts. If we want public MTA-STS we need add the following TXT record to DNS:

_mta-sts.example.com. IN TXT "v=STSv1; id=20160831085700Z;"

The v=STSv1 is fixed value. And the value of id= is the version of policy, by which the sender can determine if it is need to refresh the policy. Once the sender gain the MTA-STS record, it will fetch the real policy from the following URL:

https://mta-sts.example.com/.well-known/mta-sts.txt

And its content looks like:

version: STSv1
mode: enforce
max_age: 604800
mx: mx1.example.org
mx: mx2.example.org

The version: STSv1 is fixed currently. The mode: has several value, but we should set it to enforce which enforce all sending MTA use the STARTTLS unconditionally. The max_age: indicates the max time the sender should cache the policy. And the mx key indicates the MTA hosts allowed to receive email for the destination domain, which will override the MX record from DNS.

TLSRPT

Similar to DMARC, there is failure report mechanism for MTA-STS as well. It is the TLSRPT7. The MTA-STS is used for the sender deliver email to the receiver. If there are some MITM attack, the receiver cannot detect it. So the sender can publish the TLSRPT record to require the sender to report the STARTTLS usage.

Here is an example of TLSRPT:

_smtp._tls.example.org. TXT "v=TLSRPTv1;rua=mailto:postmaster@example.org"

If the sender support TLSRPT, it will send the statistics of success or failure of MTA-STS to the receiver’s mailbox. So the receiver can detect the problem as soon as possible.

Finally, we have finished all the details about how the email works. It’s time to do the real configurations.

Practice

Static IP Address

First of all, we need at least one computer with static IP address. In my case, I have registered the always free tier Oracle account. However, it’s very hard to create the ARM instance, which has 4 CPU and 24G memory, because the low stock. In order to gain my appropriate instance, I have upgraded my account to the Pay As You Go (PAYG) Plan. Even though I am not the always free account, I still be able to use all the source granted to the always free tier account without any fees. But after my upgrade, I got my ARM instance.

The next important task is to obtain one static IP address. If you just create your Cloud Computer and choose that you need public IP address, you will be assigned one ephemeral one, which will be withdrawn after the termination of instance of computer. In order to obtain a static IP address, we need to make the Reserved public IPv4 addresses. It’s said that we reserve up to 6 IPv6 address without any cost. For the IPv6, all we need to do is to check the “Assign an Oracle allocated IPv6 /56 prefix” option when create the Virtual Cloud Network (VPC), and we will get on /56 IPv6 address block, which is also static and freely.

And now, it is time to make our journey.

DNS PTR Record

Only the owner of the IP address is able to change this record. So we need to create support request to Oracle to set this PTR record. This is another reason we need to upgrade to none-always-free plan, because the free plan account can not make the support request.

Before to submit the request, you need to choose one name for your mail server. It’s recommend to use some name like mx1.example.org because the mx means mail exchange.

Here is my request detail

Please help me setup PTR records for the following addresses:

- Address: XX.XX.XX.XX
- OCID: ocid1.publicip.oc1.us-sanjose-1.xxx
- Point to: mx1.example.org

- Address: XXXX:XXXX:XXX:XXX::
- OCID: ocid1.ipv6.oc1.us-sanjose-1.xxx
- Point to: mx1.example.org

Thanks :)

It will take some days to finish, be patient.

Open 25 Port

According to the Oracle’s document8, tenancies made after June 23, 2021 are by default not allowed to send e-mail via outbound TCP port 25 to the internet. Tenancies made prior to June 23, 2021 are unaffected. If you require the ability to send email from your tenancy, open a service limits request to request an exemption.

Again, we need create one support request to do this task. Here is my ticket:

I need to send verification emails to registered users of my website.
Please help me unblock 25 port and set rDNS (PTR) pointed to mx1.example.org
for VPS of address XX.XX.XX.XX and XXXX:XXXX:XXXX:XXXX::

The OCID:
ocid1.instance.oc1.us-sanjose-1.xxx

It takes almost one week to finish. So be patient again.

TLS Certificate

Both STARTTLS and HTTPS need SSL Certificate. We can use acme.sh9 to gain free certificates. Before we run the acme.sh, we need first setup the A/AAAA record for the domain mx1.example.org. I choose this domain as the hostname of my MTA. Both IPv4 and IPv6 should be supported. And then create a CNAME record for the domain _mta-sts.example.org to mx1.example.org. So there is no need to setup the A/AAAA repeatedly.

Now we need install the acme.sh as the root:

curl https://get.acme.sh | sh -s email=hi@example.com

In order to offer the MTA-STS policy and issue certificate, we need to install one HTTP server. I choose the Nginx. And my operating system is Ubuntu, so

aptitude install nginx

After the installation of Nginx, we can use the webroot mode to issue certificate. I prefer to issue different certificate for different domain, so I do the following steps:

# for STARTTL
acme.sh --issue -w /var/www/html -d mx1.example.org
# for MTA-STS 
acme.sh --issue -w /var/www/html -d mta-sts.example.org

The certificates are stored in the location of /root/.acme.sh/{mx1,mta-sts}.example.org.

And then we add crontab to auto re-issue these certificates:

5 21 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null && /usr/sbin/nginx -s reload

Choose Mail Server

As said before, the email system is far more complicated. Different parts has different implements. For example, Postfix/OpenSMTPD/Exim have implemented the SMTP protocol for MTA; Cyrus IMAP/Dovecot have implemented the POP3/IMAP; Courier has implemented all the SMTP/POP3/IMAP, even with a web client. If you want to support, OpenDKIM is your friend. If you need a web interface, you may be interested in Roundcube or Squirrelmail.

There is even a Mail-in-a-Box project10, and its components include:

Besides all this component, I also need to run them on the ARM platform. Do we really need to known all the chaos? Absolutely Not.

Here I recommend the Maddy Mail Server. It is developed by the Golang. It can send messages via SMTP (works as MTA), accept messages via SMTP (works as MX) and store messages while providing access to them via IMAP. In addition to that it implements auxiliary protocols that are mandatory to keep email reasonably secure (DKIM, SPF, DMARC, DANE, MTA-STS).

It replaces Postfix, Dovecot, OpenDKIM, OpenSPF, OpenDMARC and more with one daemon with uniform configuration and minimal maintenance cost.

Lightweight and secure, as well as support the ARM platform, so I choose to use the Maddy Mail Server as my selfhosted mail server.

Install Maddy

First install the C toolchain and Make:

apt install build-essential

Then install the latest Go toolchain (I use the arm64):

wget https://go.dev/dl/go1.21.4.linux-arm64.tar.gz
tar xf "go1.21.4.linux-arm64.tar.gz"
export GOROOT="$PWD/go"
export PATH="$PWD/go/bin:$PATH"

Clone the source code:

git clone https://github.com/foxcpp/maddy.git
cd maddy

Select the appropriate version to build

git checkout v0.7.0      # a specific release
git checkout master      # next bugfix release
git checkout dev         # next feature release

The v0.7.0 has some bugs for STARTTLS, I currently use the master branch.

Finally, build and install:

# enable cgo
export CGO_ENABLED=1
./build.sh --static

In the build directory, you will see the following files:

build
├── maddy
├── maddy.conf
└── systemd
    ├── maddy.service
    └── maddy@.service

Copy systemd/*.service to /etc/systemd/system and copy maddy.conf to /etc/maddy/, and copy maddy to /usr/local/bin.

Configure Maddy

Add the maddy user to run the maddy server process:

useradd -mrU -s /sbin/nologin -d /var/lib/maddy -c "maddy mail server" maddy

Reload systemd service

systemctl daemon-reload

Set hostname and mail domain of the maddy.conf:

; used in the EHLO request
$(hostname) = mx1.example.org
$(primary_domain) = example.org
; if you have multiple domains
$(local_domains) = $(primary_domain) example.com

Setting up the TLS certificate of the maddy.conf:

tls file /root/.acme.sh/$(hostname)_ecc/fullchain.cer /root/.acme.sh/$(hostname)_ecc/$(hostname).key

maddy reloads TLS certificates from disk once in a minute so it will notice renewal.

First run maddy server

systemctl start maddy

maddy will initiate the /var/lib/maddy/ directory and generated the DKIM key.

Now its time to add all the DNS records need.

Add MX record, to let mx1.example.org receive email:

example.org.  MX 10 mx1.example.org.

Add SPF record to allow the servers in MX to send mail.

example.org. TXT "v=spf1 mx ~all"

Add DKIM record. This key is generated by maddy automatically in the path of /var/lib/maddy/dkim_keys/example.org_default.dns.

default._domainkey.example.org. TXT "v=DKIM1; v=DKIM1; k=rsa; p=MIIB..."

Add DMARC record to set the policy and reporting mail.

_dmarc.example.org. TXT "v=DMARC1;p=quarantine;ruf=mailto:hi@example.org"

Mark domain as MTA-STS compatible and request reports about failures.

_mta-sts.example.org.   TXT "v=STSv1; id=1"
_smtp._tls.example.org. TXT "v=TLSRPTv1;rua=mailto:hi@example.org"

Finally, save MTA-STS policy file in the path /var/www/html/.well-known/mta-sts.txt with the following content.

version: STSv1
mode: enforce
max_age: 604800
mx: mx1.example.org

The nginx web site configure should look like:

server {
    listen 443 ssl;
    listen [::]:443 ssl; # support ipv6, please

    ssl_certificate /root/.acme.sh/mta-sts.example.org_ecc/fullchain.cer;
    ssl_certificate_key /root/.acme.sh/mta-sts.example.org_ecc/mta-sts.example.org.key;

    index index.html;
    root /var/www/html;

    server_name mta-sts.example.org;

    location / {
        try_files $uri $uri/ =404;
    }
}

Reload nginx and we can get the above mta-sts.txt from the URL

https://mta-sts.example.org/.well-known/mta-sts.txt

Create your own mail account.

maddy creds create hi@example.org

The maddy creds will require you to input the password of account.

Finally enable the IMAP access:

maddy imap-acct create hi@example.org

Both the account of SMTP submission and IMAP is hi@example.org, not hi.

In the default table.chain, only the mail sending to the exist account will be accept. If you want to let the hi@example.org receive all the sending to none-exist address, you can append the following line to the table.chain local_rewrites block.

optional_step regexp "(.+)@(.+)" "hi@$2"

And the full chain will look like:

table.chain local_rewrites {
    optional_step regexp "(.+)\+(.+)@(.+)" "$1@$3"
    optional_step static {
        entry postmaster postmaster@$(primary_domain)
    }
    optional_step file /etc/maddy/aliases
    optional_step regexp "(.+)@(.+)" "hi@$2"
}

The first rule will forward email sending to foo+git@example.org to foo@example.org. The second make sure the mail recipient to postmaster (without domain) will be send to postmaster@example.org. The third rule will find forwarding rule from the file /let/maddy/aliases. And the final rule will forward all mail with unknown recipient to hi@example.org.

Finally, start the maddy service.

systemctl start maddy

maddy will listen to the following ports:

Port Usage Security
25 SMTP MTA PLAIN/TLS
465 SMTP Submission TLS
587 SMTP Submission STARTTLS
993 IMAP TLS
143 IMAP STARTTLS

Please configure the firewall to allow traffic for these ports.

If there are no problem, you need configure you mail client and test sending and receiving. I recommend use the mail-test.com to check if there is any problem for your configuration. If there is no obvious problem, you should send several test mail to some big player like Gmail or Outlook to check if they treat your mail as spam.

Conclusion

The long article finally finished. I have briefly introduced how the email works and why it is so complicated. And I recommend to use the maddy mail system to build selfhosted mail server. It is lightweight, and built with security in mind. It support SMTP and IMAP. It is still new, and still miss features like web interface or CalDAV/CardDAV. However, it is a Golang project, which means it is far more easy to maintain than projects written in C. I will try to participate into the maddy project if possible. Feel free to comment for any issue about selfhosted email system. Good luck.

  1. https://news.ycombinator.com/item?id=36321783↩︎

  2. https://datatracker.ietf.org/doc/html/rfc7208↩︎

  3. https://datatracker.ietf.org/doc/html/rfc6376↩︎

  4. https://datatracker.ietf.org/doc/html/rfc7489↩︎

  5. https://datatracker.ietf.org/doc/html/rfc3207↩︎

  6. https://datatracker.ietf.org/doc/html/rfc8461↩︎

  7. https://datatracker.ietf.org/doc/html/rfc8460↩︎

  8. https://docs.oracle.com/en-us/iaas/releasenotes/changes/f7e95770-9844-43db-916c-6ccbaf2cfe24/↩︎

  9. https://github.com/acmesh-official/acme.sh↩︎

  10. https://github.com/mail-in-a-box/mailinabox↩︎