IPsec & TLS: Protocols Securing Data in Transit

TAKEAWAYS:

• IPsec operates at the network layer (Layer 3) using AH (Authentication Header) and ESP (Encapsulating Security Payload) to secure IP packets end-to-end.
• TLS (Transport Layer Security) works at Layer 4/5 (transport/application) and replaces SSL, using symmetric encryption (AES) for bulk data and asymmetric (RSA/ECC) for key exchange.
• IPsec modes: Transport (secures payload only) vs. Tunnel (secures entire packet, used in VPNs).
• TLS handshake: ClientHello → ServerHello → Certificate exchange → Key derivation → Symmetric session setup.
• Real-world use: Ncell’s VPN uses IPsec for secure remote access; eSewa’s HTTPS uses TLS for payment data protection.
• Exam focus: Compare IPsec vs. TLS layers, trace a TLS handshake, and explain how ESP provides confidentiality + integrity.

Core Concepts: IPsec and TLS

1. IPsec (Internet Protocol Security)

IPsec is a suite of protocols designed to secure IPv4/IPv6 communications by authenticating and encrypting packets. It operates at the network layer (Layer 3) and is widely used in VPNs, remote access, and site-to-site encryption.

Key Components of IPsec

IPsec consists of two primary protocols:

• Authentication Header (AH): Provides data integrity and authentication (via HMAC) but no confidentiality.
• Encapsulating Security Payload (ESP): Provides confidentiality (encryption), integrity, and optional authentication. ESP is more commonly used than AH.

IPsec Modes

IPsec operates in two modes, each with distinct use cases:

• Transport Mode:

  • Secures only the payload (upper-layer data).
  • Original IP header remains unencrypted.
  • Used for host-to-host communication (e.g., secure email between servers).
  • Example: Secure communication between a bank’s database server and a client application.

• Tunnel Mode:

  • Secures the entire IP packet (original header + payload).
  • A new IP header is added for routing.
  • Used in VPNs (e.g., Ncell’s secure remote access for employees).
  • Example: A user connecting to a corporate network via a VPN uses tunnel mode to encrypt all traffic between their device and the VPN gateway.

IPsec Security Associations (SA)

• An SA is a unilateral relationship between two entities (e.g., a client and server) that defines:
  • Security parameters (algorithm, keys, mode).
  • Lifetime of the SA (keys must be refreshed periodically).
• Two SAs are required for full security:
  1. Outbound SA: For data sent from the initiator.
  2. Inbound SA: For data received by the responder.

IPsec Packet Format (ESP)

• ESP Header:
  • Security Parameters Index (SPI): Identifies the SA.
  • Sequence Number: Prevents replay attacks.
  • Payload Data: Encrypted data (e.g., using AES).
• ESP Trailer:
  • Padding: Ensures block cipher alignment.
  • Pad Length: Indicates padding size.
  • Next Header: Identifies the original protocol (e.g., TCP, UDP).
  • Integrity Check Value (ICV): HMAC for integrity.

IPsec Protocols

1. Internet Key Exchange (IKE):

   • Used to negotiate and establish SAs.
   • Operates in two phases:
     • Phase 1: Establishes a secure channel (ISAKMP).
     • Phase 2: Negotiates IPsec SAs for data transfer.
   • Example: When you connect to a VPN, IKE negotiates the encryption keys before data is sent.

2. Internet Security Association and Key Management Protocol (ISAKMP):

   • Framework for key exchange and SA management.

Advantages and Disadvantages of IPsec

Worked Example: IPsec in a VPN

Scenario: A company in Kathmandu allows remote employees to access internal resources securely using a VPN.

1. Employee’s device initiates an IKE handshake with the VPN gateway.
2. Phase 1 establishes a secure channel (e.g., using AES-256 and SHA-256).
3. Phase 2 negotiates an ESP SA for data transfer.
4. All traffic between the employee and the company network is encrypted in tunnel mode.
5. The original IP header is hidden, and a new header is added for routing.

Visual:

2. TLS (Transport Layer Security)

TLS is the successor to SSL and is used to secure web traffic (HTTPS), email (SMTPS), and other application-layer protocols. It operates at Layer 4 (Transport) or Layer 5 (Application) and is application-aware (unlike IPsec).

TLS Architecture

TLS provides:

• Server authentication (via digital certificates).
• Client authentication (optional, e.g., client certificates).
• Encrypted communication (symmetric encryption for bulk data).
• Data integrity (via HMAC).

TLS Handshake Process

The TLS handshake establishes a secure session between a client and server. Here’s a step-by-step trace:

sequenceDiagram
    participant Client
    participant Server
    Client->>Server: ClientHello (supported cipher suites, TLS version)
    Server->>Client: ServerHello (selected cipher suite, TLS version)
    Server->>Client: Certificate (server’s public key + CA signature)
    Server->>Client: ServerKeyExchange (if needed, e.g., DH/ECDHE)
    Server->>Client: ServerHelloDone
    Client->>Server: ClientKeyExchange (pre-master secret, encrypted with server’s public key)
    Client->>Server: ChangeCipherSpec
    Client->>Server: Finished (MAC of all handshake messages)
    Server->>Client: ChangeCipherSpec
    Server->>Client: Finished (MAC of all handshake messages)
    Note over Client,Server: Symmetric session keys derived and used for encryption

Key Steps:

1. ClientHello: Client sends supported cipher suites (e.g., TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) and TLS version.
2. ServerHello: Server selects a cipher suite and sends its digital certificate (containing its public key).
3. Key Exchange:
   • If using RSA, the client encrypts a pre-master secret with the server’s public key.
   • If using Ephemeral Diffie-Hellman (DHE/ECDHE), the client and server compute a shared secret without transmitting private keys.
4. Finished Messages: Both sides send a MAC of all handshake messages to verify integrity.
5. Session Establishment: Symmetric keys (e.g., AES-256) are derived for encryption.

TLS Record Protocol

After the handshake, TLS uses the TLS Record Protocol to:

• Fragment data into records.
• Compress (optional) data.
• Encrypt data using the symmetric key.
• Add MAC for integrity.

TLS Cipher Suites

A cipher suite defines:

• Key exchange algorithm (RSA, DHE, ECDHE).
• Symmetric encryption (AES, ChaCha20).
• MAC algorithm (SHA-256, SHA-384).
• Example: TLS_AES_256_GCM_SHA384 uses AES-256-GCM for encryption and SHA-384 for MAC.

TLS vs. IPsec: Key Differences

Worked Example: TLS in eSewa Payments

Scenario: A user in Nepal makes a payment via eSewa’s website.

1. The user’s browser sends a ClientHello to https://esewa.com.np.
2. The server responds with its TLS certificate (issued by a trusted CA like Sectigo).
3. The browser verifies the certificate and performs a key exchange (e.g., ECDHE).
4. A symmetric session key (e.g., AES-256) is established.
5. All subsequent data (login credentials, payment details) is encrypted and sent over HTTPS.

Visual:

![TLS handshake diagram](/media/ad0b818caee100dd836c.png "Step-by-step TLS handshake between a client and eSewa’s server (Image: Fleshgrinder and The People from The Tango! Desktop Project., Public domain, via Wikimedia Commons)")

## In the real world

1. Ncell’s VPN Service:

   • Uses IPsec in tunnel mode to secure remote access for employees.
   • How: IKE negotiates SAs, and ESP encrypts all traffic between the user’s device and Ncell’s network.

2. eSewa’s HTTPS (TLS):

   • Secures payment data (card numbers, UPI IDs) during transactions.
   • How: TLS 1.3 with ECDHE key exchange and AES-256-GCM ensures confidentiality and integrity.

3. Daraz’s Order Processing:

   • Uses TLS for secure API calls between the Daraz website and its backend servers.
   • How: JSON payloads (e.g., order details) are encrypted during transit to prevent MITM attacks.

4. Nepal Rastra Bank’s SWIFT Communications:

   • Uses IPsec for site-to-site encryption between banks to secure SWIFT messages.
   • How: Tunnel mode IPsec encrypts entire SWIFT packets between bank networks.

## Exam Tip

1. Compare IPsec and TLS:

   • Layer: IPsec (Layer 3), TLS (Layer 4/5).
   • Use Case: IPsec for VPNs/networks, TLS for apps/web.
   • Key Exchange: IPsec uses IKE, TLS uses RSA/DHE/ECDHE.

2. Trace a TLS Handshake:

   • Memorize the 6-step process (ClientHello → ServerHello → Certificate → KeyExchange → Finished).
   • Know the difference between RSA and ECDHE key exchange.

3. IPsec Modes:

   • Transport mode = payload only (end-to-end).
   • Tunnel mode = entire packet (VPNs).

4. Packet Formats:

   • Draw ESP header/trailer and label fields (SPI, Sequence Number, ICV).
   • For TLS, sketch the record protocol layers (fragmentation → compression → encryption → MAC).

5. Real-World Applications:

   • Link IPsec to VPNs (Ncell, corporate networks).
   • Link TLS to HTTPS (eSewa, Daraz, banks).

6. Common Pitfalls:

   • AH vs. ESP: AH provides integrity/authentication only; ESP provides confidentiality + integrity.
   • TLS 1.2 vs. 1.3: 1.3 removes RSA key exchange and session resumption is simplified.
   • IPsec SA: Always remember two SAs (inbound and outbound) are needed.

Visual Summary: