IPsec: Architecture, Modes, Protocols & Real-World Use

Core Concepts: What is IPsec?

IPsec (Internet Protocol Security) is a suite of protocols that secures IPv4/IPv6 communications by:

1. Authenticating senders (preventing spoofing).
2. Encrypting payloads (confidentiality).
3. Ensuring integrity (detecting tampering).
4. Providing anti-replay protection (stopping replay attacks).

It operates at the network layer (Layer 3), unlike TLS (Layer 4). IPsec is mandatory in IPv6 but optional in IPv4.

Why IPsec?

• VPNs: Secures remote access (e.g., Ncell employees connecting to corporate servers).
• IoT: Protects smart devices (e.g., Daraz’s warehouse sensors).
• E-commerce: Encrypts payment data (e.g., Khalti’s transactions).

IPsec Architecture: AH vs. ESP

IPsec uses two main protocols:

1. Authentication Header (AH)

   • Provides integrity and authentication (via HMAC-SHA).
   • Does not encrypt payloads (only protects headers).
   • Vulnerable to replay attacks (unless anti-replay is enabled).

2. Encapsulating Security Payload (ESP)

   • Provides confidentiality (via AES, 3DES) and integrity (via HMAC).
   • Can encrypt both headers and payloads (in tunnel mode).
   • Preferred for most use cases (e.g., eSewa’s encrypted transactions).

graph LR
    A["IPsec Suite"] --> B["Authentication Header (AH)"]
    A --> C["Encapsulating Security Payload (ESP)"]
    B --> D["Integrity & Authentication\n(No encryption)"]
    C --> E["Confidentiality + Integrity\n(Encryption + HMAC)"]
    C --> F["Tunnel Mode\n(Full packet encryption)"]
    C --> G["Transport Mode\n(Payload-only encryption)"]

IPsec Modes: Transport vs. Tunnel

Transport Mode

• Original IP header is unchanged (except for ESP/AH insertion).
• Used for host-to-host security (e.g., secure email between two PCs).
• Limitation: Cannot secure routing headers (e.g., in NAT traversal).

Tunnel Mode

• Original IP packet is fully encapsulated in a new IP header.
• Used for VPNs (e.g., Pathao’s driver-to-server communication).
• Advantage: Hides internal network topology.

+---------------------+       +---------------------+
| Original IP Header  | ----> | New IP Header (ESP) |
+---------------------+       +---------------------+
|      Payload        |       | ESP Header          |
+---------------------+       +---------------------+
                          | ESP Trailer         |
                          +---------------------+
                          | ESP Authentication   |
                          +---------------------+

Shows original packet wrapped in a new IP header with ESP fields. (Image: Mpk1024, CC BY-SA 4.0, via Wikimedia Commons)

Internet Key Exchange (IKEv2): How Keys Are Established

IPsec uses symmetric keys (AES, 3DES) for encryption. But how are these keys securely exchanged? → Internet Key Exchange (IKEv2) does this in two phases:

Phase 1: Security Association (SA) Setup

1. Initial Exchange (IKE_SA_INIT)

   • Peers exchange nonces (random numbers) and DH public keys.
   • Agrees on cipher suite (e.g., AES-256-GCM).
   • Creates IKE SA (for IKE traffic itself).

2. Authentication

   • Uses X.509 certificates or pre-shared keys (PSK).
   • Example: Ncell’s VPN uses PSK for site-to-site tunnels.

Phase 2: Child SA for IPsec

1. Quick Mode
   • Derives IPsec keys (for AH/ESP).
   • Negotiates lifetime (e.g., 1 hour).
   • Example: eSewa’s payment gateway refreshes keys every 30 mins.

sequenceDiagram
    participant A as Initiator (e.g., Ncell User)
    participant B as Responder (e.g., Ncell Server)
    A->>B: IKE_SA_INIT (Nonce, DH, Cert)
    B->>A: IKE_SA_INIT (Nonce, DH, Cert)
    A->>B: IKE_AUTH (ID, Signature)
    B->>A: IKE_AUTH (ID, Signature)
    Note over A,B: IKE SA Established
    A->>B: Quick Mode (Key Material)
    B->>A: Quick Mode (Key Material)
    Note over A,B: IPsec SA Ready (ESP/AH Keys)

Real-World Applications

1. Ncell’s VPN Tunnels (Tunnel Mode + IKEv2)

• Problem: Ncell needs to securely route voice/data between towers.
• Solution:
  • Tunnel mode IPsec encrypts entire packets between gateways.
  • IKEv2 Phase 1 uses PSK for authentication.
  • ESP with AES-128 ensures confidentiality.
• Result: Eavesdroppers see only ciphertext.

2. Daraz’s Order Routing (Transport Mode + AH)

• Problem: Daraz’s warehouse sensors send inventory updates.
• Solution:
  • Transport mode ESP encrypts only payloads (sensor data).
  • AH ensures no tampering with IP headers.
• Result: Only authorized Daraz servers can read/update inventory.

3. eSewa’s Payment Security (ESP + IKEv2)

• Problem: Payment data must be confidential and untampered.
• Solution:
  • ESP with AES-256 encrypts transaction details.
  • IKEv2 Phase 2 refreshes keys every 30 mins.
• Result: Even if intercepted, data is unreadable.

Security Services Provided by IPsec

Advantages and Limitations

Advantages

✅ Transparency: Works at Layer 3 (no app changes needed). ✅ Flexibility: Supports AH, ESP, or both. ✅ Standardized: Widely supported (IPv6, VPNs, IoT).

Limitations

❌ Performance Overhead: Encryption/decryption slows traffic. ❌ Complexity: IKEv2 handshakes add latency. ❌ No built-in key management: Relies on IKE or manual config.

Exam Tip: What to Focus On

1. AH vs. ESP

   • AH: Integrity only (headers + payload).
   • ESP: Confidentiality + integrity (payload only in transport mode).
   • AH+ESP: Full security (rarely used due to complexity).

2. Transport vs. Tunnel Mode

   • Transport: End-to-end (e.g., user ↔ server).
   • Tunnel: Gateway-to-gateway (e.g., VPNs).
   • Exam trick: Draw the packet diagrams!

3. IKEv2 Phases

   • Phase 1: IKE SA (authentication, DH).
   • Phase 2: IPsec SA (key derivation).
   • Memorize: Quick Mode = IPsec keys.

4. Real-World Mapping

   • Ncell: Tunnel mode + IKEv2.
   • eSewa: ESP + IKEv2.
   • Daraz: Transport mode + AH.

5. Common Pitfalls

   • ❌ Confusing AH (integrity) with ESP (confidentiality).
   • ❌ Forgetting tunnel mode adds a new IP header.
   • ❌ Mixing up IKEv1 (old) vs. IKEv2 (modern).

Worked Example: Securing a Daraz Order

Scenario: A customer orders from Daraz. The order must reach the warehouse securely.

1. Step 1: IKEv2 Phase 1

   • Daraz’s server and warehouse gateway perform DH key exchange + PSK authentication.
   • Result: Secure IKE SA.

2. Step 2: IKEv2 Phase 2 (Quick Mode)

   • Derives ESP keys (AES-128).
   • Sets SA lifetime = 1 hour.

3. Step 3: ESP in Transport Mode

   • Order data is encrypted with AES.
   • Sequence numbers prevent replay attacks.

4. Step 4: Delivery

   • Warehouse decrypts the order using the same ESP keys.

Visual Trace:

sequenceDiagram
    participant C as Customer
    participant D as Daraz Server
    participant W as Warehouse
    C->>D: Order Request (HTTP)
    D->>W: Encrypted Order (ESP Transport Mode)
    W->>D: ACK (ESP)
    Note over W,D: AES-128 decryption

Comparison Table: IPsec vs. TLS

Key Takeaways for Exams

1. IPsec = AH + ESP + IKEv2

   • AH = Integrity, ESP = Confidentiality.
   • IKEv2 = Key exchange (Phase 1 + 2).

2. Modes Matter

   • Transport = End-to-end (e.g., user ↔ server).
   • Tunnel = Gateway-to-gateway (e.g., VPNs).

3. Real-World = Your Friend

   • Ncell = Tunnel mode.
   • eSewa = ESP + IKEv2.
   • Daraz = Transport mode + AH.

4. Draw It Out

   • Packet formats (AH/ESP headers).
   • IKEv2 sequence diagrams.
   • Mode differences (transport vs. tunnel).

5. Avoid Common Mistakes

   • Don’t say AH encrypts data (it doesn’t).
   • Don’t forget tunnel mode adds a new IP header.
   • Don’t mix up IKEv1 and IKEv2 (exams love this!).