NoSQL & Emerging DBs: Models, Use Cases & Tech Trends

TAKEAWAYS:

• NoSQL databases sacrifice strict consistency for scalability and flexibility, using models like document (MongoDB), key-value (Redis), column-family (Cassandra), and graph (Neo4j) to handle unstructured data.
• Emerging systems (spatial, temporal, multimedia) extend traditional databases with specialized indexing (e.g., geospatial queries for Pathao ride requests) and data types (e.g., video streams for YouTube).
• CAP theorem forces trade-offs: NoSQL systems prioritize Availability (e.g., Daraz’s distributed inventory) or Partition tolerance (e.g., Ncell’s CDN) over strong consistency.
• Schema-less design in NoSQL enables rapid iteration (e.g., eSewa’s dynamic user profiles) but requires application-level validation.
• Blockchain databases (e.g., Hyperledger) use cryptographic hashing for immutable ledgers, ideal for supply chain tracking (e.g., Daraz’s vendor authentication).
• Hybrid architectures (e.g., Google’s Spanner) merge SQL and NoSQL to balance transactional integrity with horizontal scaling.

---

1. Why NoSQL? The CAP Theorem and Trade-offs

NoSQL databases emerged to address limitations of traditional relational databases (RDBMS) like MySQL or PostgreSQL:

• Scalability bottlenecks: RDBMS struggle with horizontal scaling (adding more servers) due to ACID constraints.
• Schema rigidity: RDBMS require predefined schemas, slowing down agile development.
• Data variety: Modern apps (e.g., social media, IoT) deal with unstructured data (JSON, XML, blobs).

The CAP Theorem: Choose Two

NoSQL systems prioritize two out of three properties in distributed environments:

• CP (Consistency + Partition Tolerance): Sacrifices availability (e.g., ZooKeeper for configuration management).
• AP (Availability + Partition Tolerance): Sacrifices consistency (e.g., DynamoDB, Cassandra).
• CA (Consistency + Availability): Not partition-tolerant (e.g., traditional RDBMS in a single node).

Real World:

• eSewa uses AP for payment processing: users see "Payment Failed" (unavailable) only if the system is down, but never inconsistent data.
• Ncell’s CDN (Content Delivery Network) uses AP: videos buffer briefly (unavailable) during network partitions but never show corrupted data (consistent).

---

2. NoSQL Data Models: Which One to Pick?

NoSQL databases categorize data differently. Here’s how they compare:

Worked Example: Daraz’s Order Processing

Daraz uses a hybrid model:

• Key-Value (Redis): Stores session tokens for fast login.
• Document (MongoDB): Stores product catalogs (JSON with nested reviews, images).
• Graph (Neo4j): Tracks supplier relationships to prevent fraud.

Trace:

1. User adds items to cart → Redis stores session ID.
2. Checkout → MongoDB validates inventory and user data.
3. Fraud check → Neo4j queries: "Is this supplier linked to past scams?"

---

---

3. Emerging Database Systems

Beyond NoSQL, specialized databases handle niche needs:

A. Spatial Databases

Store geographic data with indexing for location-based queries.

• Example: Pathao’s ride-matching algorithm uses spatial joins to find the nearest driver.
• Features:
  • Geohashing: Encodes latitude/longitude into strings (e.g., u4pruydqqvj for Kathmandu).
  • R-tree indexes: Speed up queries like "Find all restaurants within 500m of my location."

Worked Example: Kathmandu Traffic Routes

• Query: "Show all roads with congestion > 0.8 near Thamel." Uses a spatial index to avoid scanning all 10,000 roads.

B. Temporal Databases

Track data changes over time (e.g., stock prices, medical records).

• Example: NEPSE (Nepal Stock Exchange) uses temporal databases to audit price changes.
• Features:
  • System-versioned tables: Store valid_from/valid_to timestamps.
  • Query: "Show NEPSE’s closing price for NABIL on 2023-05-15."

C. Multimedia Databases

Store unstructured data (images, videos, audio) with metadata.

• Example: YouTube uses Google’s BigQuery to index video metadata (tags, views, likes).
• Features:
  • Content-based retrieval: Find similar images using color histograms or deep learning embeddings.
  • Example Query: "Find all videos with >1M views tagged #KathmanduDurbarSquare."

D. Blockchain Databases

Immutable ledgers for decentralized trust.

• Example: Hyperledger Fabric powers Nepal’s digital land records (pilot in Chitwan).
• Features:
  • Smart contracts: Automate transactions (e.g., "Transfer land title if payment is confirmed").
  • Consensus algorithms: Proof-of-Authority (PoA) for enterprise use (vs. Bitcoin’s PoW).

---

4. NoSQL vs. SQL: When to Use Which?

Key Insight:

• Use SQL for complex queries (e.g., "Show top 10 customers by lifetime value").
• Use NoSQL for high-speed writes (e.g., Ncell’s call logs) or unstructured data (e.g., Khalti’s transaction notes).

---

5. Designing a NoSQL Database: Rules of Thumb

Rule 1: Denormalize for Performance

NoSQL thrives on redundancy. Example:

// Bad (normalized)
"user": { "id": 1, "name": "Ramesh" }
"order": { "id": 101, "user_id": 1, "items": [...] }

// Good (denormalized)
"order": {
  "id": 101,
  "user": { "id": 1, "name": "Ramesh" }, // Redundant but faster!
  "items": [...]
}

Rule 2: Use Composite Keys

Avoid single-column lookups in wide datasets. Example for Daraz’s inventory:

• Composite key: (order_id, product_sku) instead of just order_id.

Rule 3: Shard Strategically

Split data by access patterns. Example for Ncell’s CDN:

• Shard by region: asia.south.ncell.videos
• Shard by content type: asia.south.ncell.images

---

6. Real-World Case Study: eSewa’s Payment System

eSewa processes 10,000+ transactions/minute using a multi-model database:

1. Key-Value (Redis):
   • Stores session tokens for OAuth 2.0 authentication.
   • Example: {session_id: "abc123", user_id: 456, expires: 3600}.
2. Document (MongoDB):
   • Stores transaction metadata (JSON):

     {
       "_id": "txn_789",
       "user": { "id": 456, "name": "Sita" },
       "amount": 500,
       "status": "completed",
       "metadata": { "phone": "98XXXXXXXX", "notes": "Electricity bill" }
     }
     

3. Graph (Neo4j):
   • Detects fraud rings by analyzing relationships:

     MATCH (u:User)-[:TRANSFERRED_TO]->(fraudster:User)
     WHERE fraudster.fraud_score > 0.9
     RETURN u.name
     

Why This Works:

• Redis handles low-latency reads (e.g., "Is this session valid?").
• MongoDB enables flexible queries (e.g., "Find all transactions for user 456 in 2023").
• Neo4j ensures scalable fraud detection.

---

In the Real World

1. Khalti’s Dynamic User Profiles

   • Uses MongoDB (document store) to handle schema changes without downtime.
   • Example: Initially stored only name and email, but later added kyc_status and wallet_balance without altering the schema.

2. Pathao’s Driver Matching

   • Uses Redis (key-value) for real-time ride queues:
     • Key: ride_request:thamel:available_drivers
     • Value: Set of driver IDs { "d123", "d456" }.
   • Uses PostGIS (spatial DB) to find the nearest driver within 2km.

3. NTC’s Network Monitoring

   • Uses Cassandra (column-family) to store time-series data:
     • Columns: timestamp, latency_ms, packet_loss.
     • Query: "Show latency spikes in Kathmandu from 2023-10-01 to 2023-10-07."

---

Exam Tip

1. Compare NoSQL models in tables (like above) and explain trade-offs (e.g., "MongoDB sacrifices joins for flexibility").
2. Design questions (e.g., "Model a library system") require:
   • EER diagrams for relational parts.
   • Document structure for NoSQL (e.g., nested JSON for books + authors).
3. CAP theorem: Always relate to real systems (e.g., "eSewa uses AP because downtime is worse than stale data").
4. Emerging systems:
   • Spatial: Know R-tree and geohashing.
   • Blockchain: Mention smart contracts and consensus (PoA vs. PoW).
5. Short-answer tips:
   • NoSQL features: Schema-less, horizontal scaling, BASE model.
   • Multimedia DB: Content-based retrieval, metadata indexing.
   • Distributed DB: Replication (master-slave), fragmentation (horizontal/vertical).

Common Pitfalls:

• Forgetting eventual consistency in NoSQL (e.g., "Two users see different inventory counts").
• Overusing joins in NoSQL (denormalize instead).
• Ignoring sharding keys in design questions (always pick a high-cardinality field).