Cloud Databases: Models, Services, Security & Migration

Key Concepts and Cloud Database Models

1. What Are Cloud Databases?

Cloud databases are managed database services hosted on remote servers (cloud infrastructure) instead of on-premises hardware. They offer scalability, elasticity, and pay-as-you-go pricing, eliminating the need for physical maintenance.

Why Use Cloud Databases?

• Cost Efficiency: No upfront hardware costs; pay for usage.
• Scalability: Auto-scaling handles traffic spikes (e.g., Daraz’s Black Friday sales).
• High Availability: Multi-region replication ensures uptime (e.g., Ncell’s customer data).
• Disaster Recovery: Built-in backups and failover (e.g., eSewa’s transaction logs).

2. Cloud Database Deployment Models

Cloud databases are categorized by service models and deployment models (NIST framework). The key distinction lies in who manages what:

Deployment Models (Physical Location Control):

mindmap
  root((Deployment Models))
    Public Cloud
      Example: AWS, Azure, Google Cloud
      Pros: Low cost, scalable
      Cons: Less control, security risks
    Private Cloud
      Example: On-premises VMware, OpenStack
      Pros: Full control, compliance (e.g., bank core systems)
      Cons: High cost, maintenance overhead
    Hybrid Cloud
      Example: Ncell’s CRM (public cloud) + billing (private cloud)
      Pros: Balance of control and scalability
      Cons: Complex integration
    Multi-Cloud
      Example: eSewa (AWS + Azure for redundancy)
      Pros: Avoid vendor lock-in
      Cons: Management complexity

In the Real World

1. eSewa’s Transaction Logs

   • Idea Used: Multi-cloud DBaaS (AWS Aurora + Azure SQL) for high availability.
   • How: eSewa replicates transaction data across AWS (primary) and Azure (backup) to prevent downtime during peak hours (e.g., Dashain festivals). Uses read replicas to distribute load.

2. Daraz’s Inventory Database

   • Idea Used: Serverless PaaS (AWS DynamoDB) for auto-scaling.
   • How: DynamoDB handles millions of product queries per second during sales events (e.g., 6.6.6) without manual scaling. Uses partition keys (product_id) and secondary indexes (category, price_range) for fast lookups.

3. Ncell’s Customer Data

   • Idea Used: Hybrid Cloud (Private Cloud for PII + Public Cloud for Analytics).
   • How: Sensitive customer data (phone numbers, billing) is stored in a private cloud (compliant with Nepal’s Telecom Regulatory Authority rules), while public cloud (Google BigQuery) is used for marketing analytics (e.g., predicting churn).

3. Cloud Database Services Comparison

Not all cloud databases are equal. Here’s how leading providers compare:

Worked Example: NEPSE’s Trading Database

• Requirement: Handle 10,000+ trades/sec during market hours with zero downtime.
• Solution:
  • Primary DB: AWS Aurora PostgreSQL (multi-AZ deployment for failover).
  • Read Replicas: 3 replicas in different regions (Kathmandu, Delhi, Singapore).
  • Caching: Amazon ElastiCache (Redis) for frequent queries (e.g., stock prices).
  • Backup: Automated snapshots + cross-region replication to Azure.

sequenceDiagram
    participant User
    participant NEPSE_App
    participant Aurora_Primary
    participant Aurora_Replica
    participant Redis_Cache

    User->>NEPSE_App: Request stock price (e.g., NEPSE:123)
    NEPSE_App->>Redis_Cache: Check cache
    alt Cache hit
        Redis_Cache-->>NEPSE_App: Return price
    else Cache miss
        NEPSE_App->>Aurora_Primary: Query DB
        Aurora_Primary-->>NEPSE_App: Return price
        NEPSE_App->>Redis_Cache: Update cache
    end

4. Security and Compliance in Cloud Databases

Cloud databases introduce new attack vectors (e.g., misconfigured IAM roles, data leaks in shared tenancy). Key risks and mitigations:

A. Shared Responsibility Model

B. Compliance Requirements

Example: Khalti’s PCI-DSS Compliance

• Challenge: Store credit card tokens securely while processing payments.
• Solution:
  • Encryption: AES-256 for data at rest (AWS KMS).
  • Tokenization: Replace card numbers with tokens (stored in AWS Secrets Manager).
  • Network: PCI-compliant VPC with private subnets and no public internet access to DB.

5. Database Migration to the Cloud

Migrating on-premises databases to the cloud involves 6 key phases:

A. Assessment Tools

B. Migration Strategies

Worked Example: NTC’s Network Log Migration

• Challenge: Migrate 10TB of historical network logs from on-prem SQL Server to Google BigQuery with zero downtime.
• Solution:
  1. Initial Load: Use AWS DMS to replicate data to a staging Aurora DB.
  2. Cutover: Switch read queries to Aurora while writes go to BigQuery via Change Data Capture (CDC).
  3. Validation: Compare record counts between old and new systems.

6. Cost Optimization in Cloud Databases

Cloud databases can become expensive if not optimized. Key levers:

A. Pricing Models

B. Cost-Saving Techniques

• Right-Sizing: Use AWS Compute Optimizer to adjust instance types.
• Auto-Scaling: Enable horizontal scaling for read replicas (e.g., Daraz’s product catalog).
• Storage Tiering: Move cold data to cheaper storage classes (e.g., S3 Glacier for backups).
• Multi-Region vs. Single-Region: Single-region is cheaper but less resilient.

Example: Daraz’s Cost Optimization

• Before: Used m5.2xlarge instances (fixed cost) for product DB.
• After: Switched to Aurora Serverless + auto-scaling read replicas.
  • Savings: 60% reduction in costs during off-peak hours.
  • Performance: Handled Black Friday traffic (5x normal load) without manual scaling.

7. Performance Tuning for Cloud Databases

Cloud databases require different tuning than on-premises systems due to shared resources and auto-scaling.

A. Common Bottlenecks

B. Query Optimization

• Use Read Replicas: Offload read-heavy queries (e.g., Daraz’s product searches).
• Indexing: Cloud databases support global secondary indexes (e.g., DynamoDB).
• Partitioning: Avoid hot partitions (e.g., Pathao’s ride_id as partition key).
• Materialized Views: Pre-compute aggregations (e.g., Ncell’s monthly billing reports).

Example: eSewa’s Transaction Query Optimization

• Problem: Slow queries during Dashain (high transaction volume).
• Solution:
  • Added composite index on (user_id, transaction_date).
  • Used Aurora read replicas in Mumbai and Singapore for global users.
  • Result: Query time reduced from 500ms → 10ms.

Exam Tip

What to Expect in TU/PU Exams

1. Short Questions (2-5 marks):

   • Define DBaaS, multi-cloud, or shared responsibility model.
   • Compare AWS RDS vs. Google Cloud SQL (focus on pricing, compliance).
   • Explain one migration strategy (e.g., lift-and-shift vs. refactoring).

2. Long Questions (10-15 marks):

   • Case Study: Given a scenario (e.g., "Ncell wants to migrate its CRM to the cloud"), outline:
     • Deployment model (hybrid).
     • Security controls (encryption, IAM, PCI-DSS).
     • Cost optimization (reserved instances for core DB, serverless for analytics).
   • Diagram-Based: Draw a sequence diagram for a cloud DB transaction (e.g., Khalti payment flow) or a mindmap of cloud database services.

3. Practical (NEB-style):

   • SQL Query: Write a query to optimize a cloud DB (e.g., add an index for a slow join).
   • Configuration: Describe how to enable auto-scaling in AWS RDS or set up a read replica in Azure SQL.

Key Formulas to Remember

• Cloud Cost Calculation:

  Total Cost = (Instance Hours × Price/hr) + (Storage × Price/GB) + (Data Transfer × Price/GB)
  

• Replica Lag Calculation (for high availability):

  Replica Lag (sec) = (Primary DB Write Speed) / (Network Bandwidth)
  

  Example: If primary DB writes at 1000 transactions/sec and network bandwidth is 100 Mbps, lag = 1000 / (100 × 10^6 bits/sec) ≈ 0.01 sec.

Common Pitfalls

• Ignoring Data Transfer Costs: Moving 1TB of data between regions can cost $100+ (AWS pricing).
• Over-Provisioning: Using m5.4xlarge when m5.xlarge suffices.
• Neglecting Compliance: Storing PCI-DSS data in a public cloud without tokenization.