Setting up a mailserver from scratch (with docker)

I’ve decided to put this down for two reasons:

1. I may need to come back to it

2. It took me way too long to figure this out, even after reading documentation. Maybe someone who stumbles upon this could benefit

Let’s get started with our technology:

• Docker

• DigitalOcean droplet running Ubuntu

• DigitalOcean DNS management

• Treafik for networking

• Domain name purchased from Namecheap, pointing to DigitalOcean nameservers

• docker-mailserver docker image

1. Bare system setup — starting with Docker Install

In order to follow this setup, make sure you have docker installed (below is from the linked website):

1. Set up Docker's apt repository.

   # Add Docker's official GPG key:
   sudo apt-get update
   sudo apt-get install ca-certificates curl
   sudo install -m 0755 -d /etc/apt/keyrings
   sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   sudo chmod a+r /etc/apt/keyrings/docker.asc
   
   Add the repository to Apt sources:
   echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "${UBUNTUCODENAME:-$VERSIONCODENAME}") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   sudo apt-get update

2. Install the Docker packages.

    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

3. Ensure your /etc/hostname and /etc/hosts file displays whatever your mailserver is going to be as the hostname, this is crucial for mail services like Gmail to accept incoming emails. All incoming emails will be denied by Gmail if you do not update this. (This is referred to as the PTR record by google, and DigitalOcean handles it by using your hosts file and hostname file)

2. Docker Compose setup

Here is an example for a docker compose that will work, it’s easily transferrable to a docker swarm config.

version: "3.8"
services:
  traefik:
    image: traefik:latest
    command:
      - "--entrypoints.http.address=:80"
      - "--entrypoints.https.address=:443"
      - "--entrypoints.smtp.address=:25"
      - "--entrypoints.smtps.address=:465"
      - "--entrypoints.submission.address=:587"
      - "--entrypoints.imaps.address=:993"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.file.directory=/traefik_config/"
      - "--providers.docker.exposedbydefault=false"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=http"
      - "--certificatesresolvers.myresolver.acme.email=me@arrontaylor.me"
      - "--certificatesresolvers.myresolver.acme.storage=/traefik_config/acme.json"
ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "25:25"
      - "465:465"
      - "587:587"
      - "993:993"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefikconfig/:/traefikconfig/"
mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    hostname: mail.arrontaylor.me
    environment:
      - ENABLE_SSL=1
      - SSL_TYPE=letsencrypt
      - OVERRIDE_HOSTNAME=mail.arrontaylor.me
      - ENABLE_IMAP=1
      - ENABLE_POP3=0
      - ENABLE_SASLAUTHD=1
      - PERMIT_DOCKER=connected-networks
      - ENABLE_FAIL2BAN=1
    volumes:
      - ./docker-data/dms/mail-data/:/var/mail/
      - ./docker-data/dms/mail-state/:/var/mail-state/
      - ./docker-data/dms/mail-logs/:/var/log/mail/
      - ./docker-data/dms/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - ./traefik_config/live:/etc/letsencrypt/live
    cap_add:
      - NET_ADMIN # For Fail2Ban to work
    labels:
      - "traefik.enable=true"
      
      - "traefik.tcp.routers.smtp.rule=HostSNI(*)"
      - "traefik.tcp.routers.smtp.entrypoints=smtp"
      - "traefik.tcp.routers.smtp.service=mailserver-smtp"
      - "traefik.tcp.services.mailserver-smtp.loadbalancer.server.port=25"
- "traefik.tcp.routers.smtps.rule=HostSNI(*)"
      - "traefik.tcp.routers.smtps.entrypoints=smtps"
      - "traefik.tcp.routers.smtps.service=mailserver-smtps"
      - "traefik.tcp.services.mailserver-smtps.loadbalancer.server.port=465"
- "traefik.tcp.routers.submission.rule=HostSNI(*)"
      - "traefik.tcp.routers.submission.entrypoints=submission"
      - "traefik.tcp.routers.submission.service=mailserver-submission"
      - "traefik.tcp.services.mailserver-submission.loadbalancer.server.port=587"
- "traefik.tcp.routers.imaps.rule=HostSNI(*)"
      - "traefik.tcp.routers.imaps.entrypoints=imaps"
      - "traefik.tcp.routers.imaps.service=mailserver-imaps"
      - "traefik.tcp.services.mailserver-imaps.loadbalancer.server.port=993"

^^ What’s important above is the HostSNI, traefik.tcp.services setup and ensuring you have all of the correct entrypoints and ports setup in the traefik service. Failure to miss any one of these will result in ports not matching and data not getting transferred

3. Setting up the mailserver

• Once you’ve gotten this far, you’ll need to start the services and then setup your first email address. You can do this with the following steps

1. start the services

   docker compose up -d

2. Add a user

   docker exec -it mailserver setup email add admin@example.com password123

3. Set DKIM keys (necessary for actually verifying your identity when sending emails, mail services like gmail will not receive email from you if you cannot verify your idenity with dkim keys — FYI you’ll need your public key, which we’ll come back to later)

   docker exec -it mailserver config dkim

4. Move the keys to the right place and make sure they are recognized within your /etc/opendkim/KeyTable file

   cp /tmp/docker-mailserver/config/opendkim/ /etc/opendkim

   echo "mail.domainkey.arrontaylor.me arrontaylor.me:mail:/etc/opendkim/keys/arrontaylor.me/mail.private" >> /etc/opendkim/KeyTable                             

   Keep in mind that the generated dkim public key will tall you exactly how this domain “mail.domainkey….” should appear, that file will be found at /etc/opendkim/keys/mail.txt

5. Setup authentication within the mailserver app with postfix, and restart dovecot and postfix (enables logging into the mail account you setup before as admin@example.com

   docker exec -it mailserver bash -c "echo '
   service auth {
     unix_listener /var/spool/postfix/private/auth {
       mode = 0666
       user = postfix
       group = postfix
     }
   }
   ' >> /etc/dovecot/conf.d/10-master.conf" 
   docker exec -it mailserver bash -c "echo 'smtpdsasltype = dovecot' >> /etc/postfix/main.cf"
   docker exec -it mailserver bash -c "echo 'smtpdsaslpath = private/auth' >> /etc/postfix/main.cf"
   
   docker exec -it mailserver chown postfix:postfix /var/spool/postfix/private/auth
   docker exec -it mailserver chmod 777 /var/spool/postfix/private/auth
   docker exec -it mailserver bash -c "echo '[mail.arrontaylor.me]:587 me@arrontaylor.me:password' > /etc/postfix/sasl_passwd"
   docker exec -it mailserver postconf -e "smtpdsaslauth_enable = yes"
   docker exec -it mailserver postmap /etc/postfix/sasl_passwd
   docker exec -it mailserver postfix reload
   docker exec -it mailserver supervisorctl restart dovecot
   docker exec -it mailserver postfix reload
   docker compose restart mailserver

6. 🎉🎉🎉 Mailserver is setup with an email address, DKIM keys, and an authenticated user setup with postfix 🎉🎉🎉

4. DNS setup

1. MX Record

   Hostname: @ (arrontaylor.me)
   value: mail.arrontaylor.me
   priority: 10

2. TXT Record

   Hostname: _dmarc.mail.arrontaylor.me 
   value: v=DMARC1; p=reject

3. TXT Record

   Hostname: mail.arrontaylor.me
   value: v=spf1 a mx ip4:YOURSERVERIP ~all

4. TXT Record

   Hostname: mail._domainkey.arrontaylor.me
   value: v=DKIM1; h=sha256; k=rsa; p=YOURPUBLICKEY

   You can find the public key inside the mailserver /etc/opendkim/keys/mail.txt -- you’ll want to copy the value you see as p= (you may see the key split up into multiple lines, so you’ll have to remove some quotation marks and the extra space)

5. Run the setup, start receiving emails and sending emails

• Connect to an email provider by adding your new mail server as an imap mail service. Use your newly created email address, password and mail server domain (my example is mail.arrontaylor.me)

• Register your domain with Google so your outbound emails don’t get automatically marked as spam. You do this by adding an additional TXT record to your DNS settings

Helpful commands

Make sure dovecot is running

docker exec -it mailserver doveadm status service auth

Restart dovecot

docker exec -it mailserver service dovecot restart

Makesure authentication is working with postfix

docker exec -it mailserver postconf -n | grep smtpdsaslauthenable