Replication & High Availability: Strategies, Models & Failover

TAKEAWAYS:

• Replication copies data across nodes to survive failures, but introduces consistency vs. availability trade-offs (CAP theorem).
• Synchronous replication guarantees consistency but slows writes; asynchronous improves speed but risks data loss.
• High availability clusters use active-passive or active-active setups, with failover automating recovery from node crashes.
• Quorum-based voting (e.g., Raft, Paxos) ensures consensus in distributed systems like Google Spanner or Ncell’s backend.
• Disaster recovery separates backup (point-in-time recovery) from HA (zero downtime), using tools like Oracle Data Guard or PostgreSQL streaming replication.
• Real-world cost: Pathao’s ride-matching system uses eventual consistency (asynchronous replication) to handle 100K+ concurrent users, while banks like NMB use synchronous replication for fraud-proof transactions.

1. Why Replication and High Availability?

Databases fail due to:

• Hardware failures (disk crashes, server overheating).
• Software bugs (corrupted transactions, deadlocks).
• Human errors (accidental DROP TABLE).
• Network partitions (split-brain scenarios).

Goal: Ensure 99.999% uptime (five 9s) with minimal data loss.

IMAGE: server rack with redundant power supplies | High availability requires hardware redundancy (N+1, 2N, etc.).

2. Types of Replication

Replication copies data from a primary (master) node to one or more secondary (replica) nodes. The key difference lies in when and how the replicas are updated.

A. Synchronous Replication

• How it works: The primary waits for all replicas to acknowledge a write before confirming success.
• Consistency: Strong (all nodes see the same data).
• Performance: Slow (latency adds up).
• Use case: Financial systems (e.g., NMB Bank’s core banking system) where $100K fraud is unacceptable.

sequenceDiagram
    participant Client
    participant Primary
    participant Replica1
    participant Replica2
    Client->>Primary: WRITE (txn=123)
    Primary->>Replica1: Sync Write
    Primary->>Replica2: Sync Write
    Replica1-->>Primary: ACK
    Replica2-->>Primary: ACK
    Primary-->>Client: SUCCESS

B. Asynchronous Replication

• How it works: The primary sends writes to replicas in the background (e.g., via logs or triggers).
• Consistency: Eventual (replicas lag behind).
• Performance: Fast (no wait for ACKs).
• Use case: Pathao’s ride-matching (users see updated driver locations with slight delay).

sequenceDiagram
    participant Client
    participant Primary
    participant Replica1
    Client->>Primary: WRITE (txn=123)
    Primary-->>Client: SUCCESS (immediate)
    Primary->>Replica1: Async Write (later)

C. Semi-Synchronous Replication

• Hybrid approach: Primary waits for at least one replica to ACK before confirming.
• Trade-off: Balances speed and safety (used in PostgreSQL’s synchronous_commit=remote_apply).

3. High Availability Architectures

HA ensures the database remains available even if some nodes fail. Common setups:

A. Active-Passive (Master-Slave)

• Primary (Active): Handles all reads/writes.
• Secondary (Passive): Standby; takes over if primary fails.
• Failover: Manual or automatic (e.g., Oracle Data Guard).
• Pros: Simple, low cost.
• Cons: Underutilized replicas; downtime during failover.

stateDiagram-v2
  [*] --> Active
  Active --> Failover: Primary Crash
  Failover --> Passive: Promote
  Passive --> Active: New Primary
  state Active {
    [*] --> Active : Normal Operation
  }
  state Passive {
    [*] --> Passive : Standby
  }
  Failover --> [*] : Manual/Automatic

B. Active-Active (Multi-Master)

• All nodes can accept writes.
• Conflict resolution: Timestamps, vector clocks, or application logic.
• Use case: Google Spanner (globally distributed databases).

C. Leader-Follower (Raft/Paxos)

• Leader: Handles all writes; followers replicate.
• Election: If leader fails, followers vote for a new leader.
• Use case: Consul (HashiCorp), etcd (Kubernetes).

sequenceDiagram
    participant Leader
    participant Follower1
    participant Follower2
    Leader->>Follower1: Append Log
    Leader->>Follower2: Append Log
    Follower1-->>Leader: ACK
    Follower2-->>Leader: ACK
    Leader->>Follower1: New Term Election

4. Failover Mechanisms

Automating recovery from failures:

A. Automatic Failover

• Tools: Oracle RAC, PostgreSQL Patroni, MySQL InnoDB Cluster.
• Steps:
  1. Detect primary failure (heartbeat timeout).
  2. Promote a replica.
  3. Reconfigure clients to new primary.

B. Manual Failover

• Steps:
  1. Admin detects failure.
  2. Runs ALTER SYSTEM SWITCHOVER (Oracle) or pg_ctl promote (PostgreSQL).
• Use case: Critical systems where zero data loss is required (e.g., NTC’s billing database).

C. Split-Brain Prevention

• Problem: Two nodes think they’re primary (e.g., network partition).
• Solutions:
  • Quorum-based voting (e.g., Raft requires majority).
  • STONITH (Shoot The Other Node In The Head): Kill conflicting nodes (used in Pacemaker/Corosync).

5. Disaster Recovery vs. High Availability

Worked Example: Daraz’s Order Queue

• Scenario: Daraz’s primary database crashes during Black Friday.
• HA Setup: Active-active replication across Kathmandu and Pokhara.
• Failover: Within 2 seconds, traffic routes to Pokhara node.
• DR Plan: If both data centers fail, restore from hourly snapshots (RPO=1 hour).

6. Real-World Applications in Nepal

A. Ncell’s 4G Network

• Problem: Millions of users; single-point failures cause outages.
• Solution:
  • Synchronous replication for billing records (fraud prevention).
  • Asynchronous replication for call logs (tolerates slight delays).
• HA: Geo-redundant data centers in Kathmandu and Biratnagar.

B. eSewa’s Payment System

• Challenge: High concurrency during Dashain/Tihar.
• Solution:
  • Active-passive for transaction logs.
  • Read replicas for user queries.
• Failover: Automatic switch if primary node crashes (RTO < 5 sec).

C. NEPSE’s Stock Trading Platform

• Requirement: Zero tolerance for stale data (fraud risk).
• Solution:
  • Synchronous replication across multiple servers.
  • Quorum-based writes (3/5 nodes must agree).

7. Performance Trade-offs

Example: Kathmandu Traffic Routes

• Primary: Real-time GPS data (synchronous to avoid accidents).
• Replica: Historical data for analytics (asynchronous, tolerates lag).

8. Tools and Technologies

Exam Tip

1. Define key terms:

   • Replication lag: Delay between primary and replica (critical for async).
   • Quorum: Minimum nodes required for consensus (e.g., 3/5 in Raft).
   • RTO/RPO: Always relate to real systems (e.g., "Ncell’s RTO is 2 sec").

2. Compare setups:

   • Draw a mermaid diagram for active-passive vs. active-active.
   • Explain CAP theorem with an example (e.g., "Pathao chooses A and P over C").

3. Worked problems:

   • Given a scenario (e.g., "A bank wants 99.99% uptime"), calculate:
     • Required replication factor (e.g., 3 nodes for 2 failures).
     • Trade-off (e.g., "Synchronous replication adds 50ms latency").

4. Common pitfalls:

   • Split-brain: Always mention quorum or STONITH.
   • Data loss: Async replication can lose writes (e.g., "If primary crashes before replicating, 10 transactions are lost").

5. Short-answer hooks:

   • "Why does Google Spanner use TrueTime?" → Bounds clock skew for consistency.
   • "How does PostgreSQL’s synchronous_commit=off improve performance?" → No wait for replicas.

Final Visual Summary: