Replication & High Availability: Methods, Failover, and Cloud Sync

TAKEAWAYS:

• Replication copies data across nodes to prevent loss, with synchronous (strong consistency) and asynchronous (speed) trade-offs.
• High availability (HA) uses failover clusters, load balancing, and geo-redundancy to minimize downtime (e.g., Ncell’s 99.99% uptime).
• Multi-master replication allows simultaneous writes (e.g., WhatsApp’s global message sync), while single-master prioritizes consistency (e.g., bank transactions).
• Cloud databases (AWS RDS, Google Spanner) automate HA via automatic failover and read replicas, reducing manual intervention.
• Conflict resolution in distributed systems uses timestamps, vector clocks, or application logic (e.g., Daraz’s inventory updates).
• Disaster recovery pairs replication with backup strategies (e.g., NTC’s daily snapshots + real-time logs).

1. Why Replication and High Availability?

Databases fail due to hardware crashes, network outages, or human errors. Replication copies data across multiple servers to survive failures, while high availability (HA) ensures the system remains operational with minimal downtime. Together, they form the backbone of mission-critical systems like:

• eSewa: Replicates transaction logs across data centers to prevent fraud during online payments.
• Ncell: Uses HA clusters to route calls even if a tower fails.
• Google Cloud Spanner: Syncs data globally with millisecond latency for apps like YouTube’s recommendation engine.

Key Definitions

2. Types of Replication

Replication methods differ in consistency, performance, and use case. Visualize the trade-offs:

mindmap
  root((Replication Types))
    Synchronous
      "Strong consistency (ACID)"
      "Slower writes (waits for all replicas)"
      Example: Bank transactions
    Asynchronous
      "Eventual consistency"
      "Faster writes (fire-and-forget)"
      Example: Social media feeds
    Multi-Master
      "Writes to any replica"
      "Conflict resolution needed"
      Example: Global e-commerce (Daraz)
    Single-Master
      "One write node, multiple reads"
      "Simpler but bottleneck"
      Example: NEPSE stock data

Worked Example: eSewa’s Payment Sync

1. User initiates payment → Primary database (Kathmandu) records the transaction.
2. Synchronous replication → Secondary (Pokhara) mirrors the transaction before acknowledging success.
3. Asynchronous log shipping → Tertiary (Biratnagar) gets updates every 5 seconds (for recovery).
4. Failover: If Kathmandu’s server crashes, Pokhara takes over with <1s downtime.

Why this mix?

• Synchronous ensures no double-spending (critical for money).
• Asynchronous reduces load on the primary server.

3. High Availability Architectures

HA relies on redundancy and automatic recovery. Compare two common setups:

Active-Passive Failover (Step-by-Step)

sequenceDiagram
    participant User
    participant PrimaryDB
    participant StandbyDB
    participant LoadBalancer

    User->>LoadBalancer: Request (e.g., login)
    LoadBalancer->>PrimaryDB: Forward request
    PrimaryDB-->>User: Response
    loop Every 10ms
        PrimaryDB->>StandbyDB: Heartbeat (sync)
    end
    PrimaryDB-->>LoadBalancer: "I'm alive"
    StandbyDB-->>LoadBalancer: "Standby mode"
    PrimaryDB->>PrimaryDB: CRASH!
    LoadBalancer->>StandbyDB: Promote to primary
    StandbyDB->>User: Take over (transparent)

Real Picture: Failover Cluster Hardware

4. Conflict Resolution in Distributed Systems

When multiple masters write simultaneously (e.g., two users editing the same Daraz product inventory), conflicts arise. Solutions:

Worked Example: Daraz’s Inventory Sync

• Conflict: User A adds 5 units to "iPhone 15" in Kathmandu; User B adds 3 units in Pokhara.
• Solution: Daraz’s database uses application logic:

  -- Instead of overwriting, sum the changes:
  UPDATE products
  SET stock = stock + 5 WHERE product_id = 123; -- Kathmandu
  UPDATE products
  SET stock = stock + 3 WHERE product_id = 123; -- Pokhara
  

• Result: Final stock = 8 (no data loss).

5. Cloud-Based High Availability

Cloud providers abstract HA complexity. Compare on-premise vs. cloud:

Cloud HA in Action: Google Cloud Spanner

• Global replication: Data is stored in multiple regions (e.g., US, Europe, Asia) with strong consistency.
• SQL + Spanner-specific features:

  -- Create a globally distributed table:
  CREATE TABLE Users (
    UserID STRING(36) NOT NULL,
    Name STRING(100),
    Email STRING(100),
  ) PRIMARY KEY (UserID)
  INTERLEAVE IN PARENT Orders ON DELETE CASCADE;
  

• Use Case: YouTube’s user profiles must load instantly worldwide.

6. Backup and Replication: Complementary Strategies

Replication protects against current data loss; backups protect against corruption or accidental deletion. Combine them:

Worked Example: NTC’s Disaster Recovery Plan

1. Primary data center (Kathmandu) replicates to secondary (Lalitpur) synchronously.
2. Tertiary site (Pokhara) gets asynchronous snapshots every 6 hours.
3. Daily full backups + hourly transaction logs stored in AWS.
4. RTO (Recovery Time Objective): 15 minutes (failover to Lalitpur).
5. RPO (Recovery Point Objective): 5 minutes (logs ensure no more than 5 mins of data loss).

7. Real-World Applications

Case Study 1: eSewa’s Payment System

• Problem: Fraudsters exploit single-point failures during Diwali sales.
• Solution:
  • Multi-region replication: Transactions sync to Kathmandu, Pokhara, and Biratnagar.
  • Synchronous writes: No money is credited until all replicas confirm.
  • Automatic failover: If Kathmandu’s server crashes, Pokhara takes over in <500ms.
• Result: 99.999% uptime during peak seasons.

Case Study 2: WhatsApp’s Global Sync

• Problem: 2 billion users sending 65 billion messages/day need low latency.
• Solution:
  • Asynchronous replication: Messages sync to nearby servers (e.g., India → Mumbai, Nepal → Kathmandu).
  • Conflict resolution: Vector clocks resolve "edit conflicts" (e.g., two users editing the same group chat).
  • Read replicas: 90% of queries hit local replicas (reducing cloud load).
• Result: 99.9% message delivery rate.

Case Study 3: NEPSE’s Stock Data

• Problem: Stock exchanges need strong consistency (no stale prices).
• Solution:
  • Single-master replication: Only the primary server (Kathmandu) accepts trades.
  • Synchronous logs: All replicas (Pokhara, Biratnagar) mirror trades before acknowledging.
  • No multi-master: Prevents "double-selling" of shares.
• Result: Zero data inconsistencies during market crashes.

8. Performance vs. Consistency Trade-offs

Choose replication based on your SLA (Service Level Agreement):

Visual Trade-off:

pie
    title Replication Trade-offs
    "Consistency" : 30
    "Availability" : 40
    "Partition Tolerance" : 30
    "Latency" : 10

9. Exam Tip: How This Unit Is Tested

1. Definitions: Know the difference between replication, HA, and failover.

   • Example Question: "Explain why synchronous replication is used in banks but not in YouTube."
   • Answer: Banks need ACID compliance (no partial transactions), while YouTube prioritizes speed over consistency.

2. Diagrams: Draw and label:

   • A master-slave replication setup (show arrows for sync direction).
   • A failover sequence diagram (include LoadBalancer, Primary, Standby).
   • A conflict resolution table (e.g., Last-Write-Wins vs. Vector Clocks).

3. Worked Examples: Be ready to:

   • Calculate RTO/RPO for a given scenario (e.g., "If backups run every 4 hours, what’s the max data loss?").
   • Design a replication strategy for a Nepali e-commerce site (consider Kathmandu-Pokhara latency).

4. Cloud vs. On-Premise: Compare cost, scalability, and uptime for:

   • NTC’s local HA vs. AWS Multi-AZ.
   • eSewa’s synchronous replication vs. WhatsApp’s async.

5. Short-Answer Tips:

   • Advantages of multi-master: Scalability, global writes.
   • Disadvantages of async replication: Stale reads, data loss risk.
   • HA tools: Heartbeat (Linux), AWS RDS Multi-AZ, Google Cloud Spanner.

Final Visual Summary:

flowchart TD
    A["Database Failure"] --> B["Replication"]
    B --> C["Synchronous\n(Strong Consistency)"]
    B --> D["Asynchronous\n(Eventual Consistency)"]
    C --> E["Single-Master\n(eSewa)"]
    D --> F["Multi-Master\n(WhatsApp)"]
    E --> G["Failover\n(Active-Passive)"]
    F --> H["Conflict Resolution\n(Vector Clocks)"]
    G --> I["High Availability\n(99.99% Uptime)"]
    H --> I
    I --> J["Cloud HA\n(AWS RDS, Spanner)"]