Distributed Computing & Cloud Apps: Models, MapReduce, SOA, Scalability

Core Concepts: Distributed Computing vs. Cloud Computing

What is Distributed Computing?

Distributed computing is a model where multiple independent computers (nodes) work together to solve a problem by sharing resources and workload. Unlike traditional centralized systems, no single machine controls the entire process.

mindmap
  root((Distributed Computing))
    Definition["Multiple independent nodes collaborate to solve problems"]
    Characteristics
      "No single point of failure"
      "Scalability: Add more nodes as needed"
      "Transparency: Users see a unified system"
      "Concurrency: Tasks run in parallel"
    Models
      "Client-Server (e.g., Web Browsing)"
      "Peer-to-Peer (P2P) (e.g., BitTorrent)"
      "Grid Computing (e.g., SETI@home)"
      "Cloud Computing (e.g., AWS, Google Cloud)"

Key Difference from Cloud Computing:

MapReduce: The Backbone of Cloud Data Processing

How MapReduce Works

MapReduce is a programming model (not a language) designed for processing large datasets across clusters. It divides work into two phases:

1. Map Phase: Splits data into key-value pairs and processes them in parallel.
2. Reduce Phase: Aggregates results from the Map phase.

sequenceDiagram
    participant User
    participant Mapper1
    participant Mapper2
    participant Reducer
    User->>Mapper1: Input Data (e.g., logs)
    User->>Mapper2: Input Data (e.g., logs)
    Mapper1->>Reducer: (Key, Value) pairs
    Mapper2->>Reducer: (Key, Value) pairs
    Reducer->>User: Aggregated Result (e.g., word count)

Worked Example: Word Count in eSewa Transactions

Scenario: eSewa processes millions of transactions daily. To analyze frequent payment methods, they use MapReduce:

1. Map Phase:
   • Input: ["User1: Electricity", "User2: Mobile", "User1: Electricity"]
   • Output: (Electricity, 1), (Mobile, 1), (Electricity, 1)
2. Reduce Phase:
   • Aggregates counts: (Electricity, 2), (Mobile, 1)

Why MapReduce?

• Fault Tolerance: If a node fails, another takes over.
• Scalability: Add more nodes to process larger datasets (e.g., NTC’s traffic data).

Cloud Applications: SOA and Microservices

Service-Oriented Architecture (SOA)

SOA is a design principle where applications are built as independent services that communicate via standardized protocols (e.g., SOAP, REST). Cloud apps leverage SOA for modularity.

classDiagram
    class Service {
        +processRequest()
        +returnResponse()
    }
    class Client {
        +invokeService()
    }
    Client "1" --> "*" Service : "Uses"
    Service "1" --> "1" Interface : "Exposes via"

Example in Nepal:

• eSewa: Uses SOA to integrate payment, KYC, and bank services.
  • Service 1: Authentication (Khalti API)
  • Service 2: Transaction Processing (Nabil Bank API)
  • Service 3: Notification (SMS Gateway)

Advantages:

• Reusability: Services like "Payment Gateway" can be reused.
• Agility: Update one service without affecting others (e.g., Daraz’s inventory service).

Scalability in Cloud Applications

Scalability is the ability to handle growth by adding resources dynamically. Cloud apps use:

1. Vertical Scaling: Increase power of a single machine (e.g., upgrading a server).
2. Horizontal Scaling: Add more machines (e.g., Kubernetes pods for Pathao’s ride-matching).

Real-World Example: NEPSE Stock Data

• Problem: NEPSE’s website crashes during high trading volumes.
• Solution: Deploy a load-balanced cloud cluster (AWS Auto Scaling) to distribute traffic.

Peer-to-Peer (P2P) in Cloud Apps

P2P networks eliminate central servers by distributing tasks among peers. Used in:

• File Sharing: BitTorrent (used by Daraz for software updates).
• Decentralized Clouds: IPFS (InterPlanetary File System) for censorship-resistant storage.

Example: Pathao’s Ride-Matching

• Traditionally: Central server matches drivers and riders.
• P2P Alternative: Riders/drivers connect directly via blockchain (like Uber’s early model).

Challenges and Solutions

In the Real World

1. eSewa’s Payment Processing

   • Idea Used: Distributed Transaction Processing
   • How: Uses a MapReduce-like system to validate transactions across banks in real-time. If one bank’s server fails, another node (e.g., Global IME) takes over.
   • Impact: 99.99% uptime during Dashain sales.

2. Daraz’s Order Fulfillment

   • Idea Used: Horizontal Scaling + SOA
   • How: Orders are split across microservices (Inventory, Shipping, Billing). During sales, Kubernetes auto-scales the inventory service to handle 10x traffic.
   • Impact: Processed 500,000 orders in 24 hours during Republic Day.

3. NTC’s Traffic Management

   • Idea Used: Grid Computing for Simulation
   • How: Uses Hadoop clusters to simulate traffic patterns across Kathmandu. Each node processes a different route (e.g., Ring Road, Thapathali).
   • Impact: Reduced congestion by 15% via dynamic signal timing.

Exam Tip

1. Compare Distributed vs. Cloud: Always highlight ownership (user vs. provider) and scalability (manual vs. automatic).
2. MapReduce Questions:
   • Draw the sequence diagram for Map and Reduce phases.
   • Relate to real data: eSewa transactions, NTC traffic logs.
3. SOA/Microservices:
   • Use class diagrams to show services and clients.
   • Mention protocols: REST for Daraz APIs, SOAP for government systems (e.g., MoF’s financial services).
4. Scalability:
   • Vertical vs. Horizontal: Give one example each (e.g., NEPSE’s DB upgrade vs. Pathao’s Kubernetes).
5. Challenges:
   • Link to Nepali context: "How would you secure eSewa’s P2P transactions?" → Use blockchain + encryption.