Mail Server Basics: Protocols, Configurations & Security

TAKEAWAYS:

• Email relies on three core protocols: SMTP (sending), POP3/IMAP (receiving), and DNS (domain resolution) to route messages globally.
• Mail servers (e.g., Postfix, Exchange) act as intermediaries, storing, forwarding, and filtering emails using MTA (Mail Transfer Agent) and MDA (Mail Delivery Agent) roles.
• Security layers like TLS encryption, SPF/DKIM/DMARC authentication, and firewalls prevent spoofing, phishing, and data leaks.
• Configuration files (e.g., /etc/postfix/main.cf) define server behavior, including relay rules, spam filters, and virtual domains.
• Troubleshooting involves checking logs (/var/log/mail.log), verifying DNS records (MX, SPF), and testing connectivity with tools like telnet and swaks.
• Real-world impact: Banks use DMARC to block fraudulent transaction emails, while eSewa’s SMTP server routes payment confirmations via TLS to user inboxes.

Core Email Protocols: How Messages Travel

Email delivery is a handshake between protocols, each with a distinct role. Visualize the flow:

sequenceDiagram
    participant UserA as Alice (Sender)
    participant ClientA as Mail Client (e.g., Thunderbird)
    participant MTA_A as Alice's MTA (e.g., Postfix)
    participant DNS as DNS Server
    participant MTA_B as Bob's MTA (e.g., Gmail)
    participant UserB as Bob (Receiver)
    ClientA->>MTA_A: SMTP: HELO, MAIL FROM, RCPT TO
    MTA_A->>DNS: MX Query: "Where is mail.example.com?"
    DNS-->>MTA_A: Returns MX record (e.g., mail.example.com)
    MTA_A->>MTA_B: SMTP: DATA (email content)
    MTA_B->>UserB: Delivers via POP3/IMAP

Key Protocols:

Worked Example: Sending an eSewa Payment Alert

1. Alice sends a payment via eSewa’s app → SMTP (Port 25) sends raw email to eSewa’s MTA (Postfix).
2. DNS resolves recipient@ncell.com → returns MX record pointing to Ncell’s mail server (mail.ncell.com).
3. eSewa’s MTA relays the email to Ncell’s MTA via SMTP handshake (HELO, MAIL FROM, RCPT TO).
4. Ncell’s MTA stores the email → Bob fetches it via IMAP (Port 143) on his phone.

Mail Server Types and Components

Servers act as post offices for emails, handling storage, routing, and security. Compare two popular types:

classDiagram
    class MailServer {
        <<abstract>>
        +process_incoming()
        +route_outgoing()
        +apply_security()
    }
    class Postfix {
        +open-source
        +uses `/etc/postfix/main.cf`
        +supports virtual domains
    }
    class Exchange {
        +proprietary (Microsoft)
        +includes calendar/contacts
        +supports Active Directory
    }
    MailServer <|-- Postfix
    MailServer <|-- Exchange

Components of a Mail Server:

• MTA (Mail Transfer Agent): Handles sending/receiving (e.g., Postfix, Sendmail).
• MDA (Mail Delivery Agent): Delivers emails to user mailboxes (e.g., Dovecot for IMAP).
• LDA (Local Delivery Agent): Stores emails locally (e.g., /var/mail/).
• Spam Filter: Uses tools like SpamAssassin or ClamAV to block threats.

Configuring a Mail Server (Postfix Example)

Postfix is configured via /etc/postfix/main.cf. Key directives:

# /etc/postfix/main.cf snippet
myhostname = mail.example.com
mydestination = $myhostname, localhost
relayhost = [smtp.gmail.com]:587  # For outbound relay via Gmail
smtpd_tls_cert_file = /etc/ssl/certs/mail.example.com.crt
smtpd_tls_key_file = /etc/ssl/private/mail.example.com.key

Steps to Set Up Postfix:

1. Install Postfix:

   sudo apt install postfix
   

2. Configure /etc/postfix/main.cf (see above).
3. Restart Postfix:

   sudo systemctl restart postfix
   

4. Test SMTP connectivity:

   telnet localhost 25
   

   Expected output:

   220 mail.example.com ESMTP Postfix
   EHLO example.com
   250-mail.example.com
   

Worked Example: Configuring NEPSE’s Email Server NEPSE (Nepal Stock Exchange) uses Postfix to send dividend alerts to shareholders:

• Virtual Domain: dividends.nepse.com routes to a separate mailbox.
• Relay Rule: All outbound emails must pass through smtp.nepse.gov.np (port 587 with TLS).
• SPF Record: v=spf1 include:_spf.nepse.gov.np ~all prevents spoofing.

Security: Protecting Email from Spoofing and Attacks

Email is a primary attack vector for phishing and data leaks. Security layers include:

stateDiagram-v2
    [*] --> SecureEmail
    SecureEmail --> TLS: Encryption in transit
    SecureEmail --> SPF: Verify sender domain
    SecureEmail --> DKIM: Sign email content
    SecureEmail --> DMARC: Policy for failures
    SecureEmail --> Firewall: Block malicious IPs
    SecureEmail --> [*]

Key Security Measures:

1. TLS Encryption:

   • Ensures emails are encrypted during transit (e.g., smtpd_tls_security_level = may in Postfix).
   • Real Picture: IMAGE: TLS handshake diagram | How SMTP uses TLS to encrypt data between MTAs

2. SPF (Sender Policy Framework):

   • Publishes a DNS record listing authorized sending IPs.
   • Example for eSewa.com:

     v=spf1 ip4:192.0.2.1 ip4:203.0.113.5 ~all
     

   • Worked Example: If a phisher sends from: support@eSewa.com but their IP isn’t in SPF, receivers flag it as spam.

3. DKIM (DomainKeys Identified Mail):

   • Adds a digital signature to emails to prove authenticity.
   • Configuration Snippet (Postfix):

     milter_protocol = 2
     milter_default_action = accept
     smtpd_milters = inet:localhost:8891
     

4. DMARC (Domain-based Message Authentication):

   • Tells receivers what to do if SPF/DKIM fail (e.g., p=reject).
   • Example record:

     v=DMARC1; p=reject; rua=mailto:admin@eSewa.com
     

Troubleshooting Common Issues

Issue 1: Emails Stuck in Queue

• Cause: DNS resolution failure or relay misconfiguration.
• Fix:

  sudo postqueue -f  # Force retry
  sudo postfix check  # Validate config
  

• Log Check:

  tail -f /var/log/mail.log | grep "postfix/smtp"
  

Issue 2: SPF/DKIM Failures

• Symptom: Emails marked as spam despite correct From: header.
• Debug:

  sudo apt install opendkim opendkim-tools
  opendkim-testmessage -d example.com < email.eml
  

Issue 3: Port 25 Blocked by ISP

• Workaround: Use a smarthost (e.g., Gmail’s SMTP relay):

  relayhost = [smtp.gmail.com]:587
  smtp_sasl_auth_enable = yes
  smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
  smtp_sasl_security_options = noanonymous
  

In the Real World

1. eSewa’s Transaction Alerts:

   • Uses Postfix + DKIM to sign payment confirmations, ensuring users receive legitimate alerts.
   • SPF Record: v=spf1 include:_spf.eSewa.com.np ~all blocks spoofed "payment failed" emails.

2. Ncell’s Bulk SMS-to-Email Gateway:

   • Relays SMS messages to emails via SMTP (Port 25) using a custom MTA script.
   • Security: TLS + IP whitelisting to prevent SMS flooding.

3. Google Workspace (Gmail for Business):

   • Combines Exchange-like features with Postfix-like MTA handling.
   • DMARC Policy: p=reject ensures only verified senders (e.g., ncell.com) deliver emails to employees.

4. Nepal Rastra Bank’s Fraud Alerts:

   • Uses DMARC + BIMI (Brand Indicators for Message Identification) to display the bank’s logo in verified emails, reducing phishing.

Exam Tip

1. Protocol Questions:

   • Expect SMTP command traces (e.g., "What happens when Alice sends an email to Bob?").
   • Key Commands to Memorize: HELO, MAIL FROM, RCPT TO, DATA, QUIT.

2. Configuration Files:

   • Postfix: Know /etc/postfix/main.cf directives (myhostname, relayhost, smtpd_tls_*).
   • Dovecot (IMAP): Focus on /etc/dovecot/dovecot.conf (e.g., protocols = imap pop3).

3. Security Shortcuts:

   • SPF/DKIM/DMARC: Be able to read and write these records from a given scenario.
   • TLS: Know the difference between may, encrypt, and secure in Postfix.

4. Troubleshooting:

   • Logs: /var/log/mail.log (Postfix), /var/log/mail.err (general).
   • Tools: swaks, telnet, dig MX example.com.

5. Real-World Scenarios:

   • Case Study: "How would you configure a mail server for a bank to prevent fraudulent transaction emails?"
     • Answer: Use DMARC with p=reject, DKIM signing, and TLS encryption on port 465/587.

Summary Table: Mail Server Protocols and Tools

Final Checklist for Exam Readiness

• Draw the SMTP handshake sequence diagram.
• Write SPF/DKIM/DMARC records for a given domain (e.g., pathao.com).
• Configure Postfix to relay via Gmail (with TLS).
• Explain how Ncell’s SMS-to-email gateway uses SMTP.
• Troubleshoot a stuck email in the Postfix queue.