Container Orchestration: Scheduling, Scaling & Kubernetes

Key Concepts and Definitions

What is Container Orchestration?

Container orchestration is the automated process of managing containerized applications across clusters of hosts. It handles:

• Deployment: Placing containers where they should run.
• Scaling: Adding or removing containers based on demand.
• Load balancing: Distributing traffic evenly.
• Self-healing: Restarting failed containers.
• Service discovery: Connecting containers to each other.

Without orchestration, managing hundreds of containers manually would be impossible. Orchestration tools like Kubernetes (K8s) automate these tasks.

Why is Orchestration Needed?

Containers are lightweight and fast, but they introduce challenges:

• Dynamic environments: Containers can start, stop, or crash unpredictably.
• Resource constraints: Limited CPU, memory, or network bandwidth.
• Dependency management: Containers must communicate with each other.
• High availability: Applications must stay running even if some containers fail.

Orchestration solves these problems by providing a centralized control plane.

Core Components of Container Orchestration

1. Kubernetes Architecture

Kubernetes is the most popular orchestration tool. Its architecture consists of:

classDiagram
    class MasterNode {
        +API Server
        +Scheduler
        +Controller Manager
        +etcd (Key-Value Store)
    }
    class WorkerNode {
        +Kubelet
        +Container Runtime (Docker, containerd)
        +Kube-Proxy
    }
    MasterNode --> WorkerNode : "Manages"
    WorkerNode --> "Pods" : "Hosts"
    WorkerNode --> "Services" : "Exposes"
    WorkerNode --> "Volumes" : "Stores"

Key Components:

2. Scheduling in Kubernetes

The Scheduler decides where to place pods based on:

• Resource requirements: CPU, memory, GPU.
• Affinity/Anti-affinity rules: Prefer or avoid specific nodes.
• Taints and Tolerations: Restrict pods to certain nodes.
• Node labels and selectors: Match pods to nodes with specific labels (e.g., role=database).

stateDiagram-v2
    [*] --> Scheduler
    Scheduler --> CheckResourceRequirements: Pod needs 2 CPU, 4GB RAM
    CheckResourceRequirements --> EvaluateNodeA: Node A (4 CPU, 8GB RAM, role=web)
    CheckResourceRequirements --> EvaluateNodeB: Node B (8 CPU, 16GB RAM, role=db)
    EvaluateNodeA --> NodeSelector: role=db? No
    EvaluateNodeB --> NodeSelector: role=db? Yes
    NodeSelector --> AssignPod: Pod scheduled to Node B
    AssignPod --> [*]

Example: Scheduling a Pod

Suppose we have:

• A pod requiring 2 CPU cores and 4GB RAM.
• Two nodes:
  • Node A: 4 CPU, 8GB RAM, labeled role=web.
  • Node B: 8 CPU, 16GB RAM, labeled role=db.

If the pod has a node selector role=db, it will run on Node B.

3. Scaling Strategies

Orchestration automates scaling to handle traffic spikes or failures.

Types of Scaling:

Example: Horizontal Pod Autoscaler (HPA)

Suppose a microservice has:

• Current replicas: 3
• CPU target: 70% utilization
• Max replicas: 10

If the average CPU usage across pods exceeds 70%, Kubernetes adds more replicas until the load drops below 70%.

4. Service Discovery and Load Balancing

Containers are ephemeral (they can be created or destroyed). Orchestration provides:

• Services: Stable endpoints for pods (even if pods restart).
• Load Balancing: Distributes traffic across pods.

How Services Work:

sequenceDiagram
    participant User
    participant LoadBalancer
    participant Service
    participant Pod1
    participant Pod2

    User->>LoadBalancer: Request (e.g., http://esewa-payment-service)
    LoadBalancer->>Service: Forwards request
    Service->>Pod1: Routes to Pod1 (or Pod2)
    Pod1-->>Service: Returns response
    Service-->>LoadBalancer: Returns response
    LoadBalancer-->>User: Returns response

Example: eSewa’s Payment Service

• Pods: Multiple instances of the payment microservice.
• Service: A stable endpoint (esewa-payment-service) that load-balances traffic across pods.
• Load Balancer: Distributes requests to avoid overloading a single pod.

5. Self-Healing

Kubernetes automatically:

• Restarts failed containers.
• Replaces containers if they crash repeatedly.
• Reschedules pods if a node fails.

Example: Daraz Order Processing

If a pod processing orders crashes due to high traffic:

1. Kubernetes detects the failure.
2. It creates a new pod on another node.
3. Traffic is rerouted to the new pod.

Real-World Applications

1. eSewa: Microservices Orchestration

• Use Case: eSewa’s payment system uses Kubernetes to orchestrate microservices for:
  • User authentication.
  • Transaction processing.
  • Notification services.
• How Orchestration Helps:
  • Scaling: During Dashain/Tihar, transaction volumes spike. Kubernetes scales up pod replicas.
  • High Availability: If a pod fails, another takes over instantly.
  • Service Discovery: Microservices communicate via Kubernetes services (e.g., auth-service, payment-service).

2. Ncell: API Gateway Management

• Use Case: Ncell’s API gateway (for SMS, internet, and billing) uses Kubernetes to:
  • Manage thousands of API requests per second.
  • Scale horizontally during peak hours (e.g., New Year’s Eve).
  • Route traffic to the nearest data center for low latency.

3. Daraz: Order Fulfillment

• Use Case: Daraz’s order processing system uses Kubernetes to:
  • Deploy order-processing pods near warehouses (edge computing).
  • Scale pods based on real-time order volume.
  • Ensure no order is lost if a pod crashes.

Worked Example: Kubernetes Deployment for a Bank’s Loan System

Scenario:

A bank wants to deploy a loan approval microservice with:

• Requirements:
  • 2 CPU cores, 4GB RAM per pod.
  • Must run on nodes labeled role=finance.
  • Should scale to 10 replicas if CPU > 70%.
• Constraints:
  • Avoid running on nodes with taints (e.g., dedicated=gpu).

Solution:

1. Define the Pod Spec:

   apiVersion: v1
   kind: Pod
   metadata:
     name: loan-approval-pod
   spec:
     containers:
     - name: loan-service
   
       resources:
         requests:
           cpu: "2"
           memory: "4Gi"
     nodeSelector:
       role: finance
     tolerations:
     - key: "dedicated"
       operator: "Equal"
       value: "gpu"
       effect: "NoSchedule"
   

2. Set Up Horizontal Pod Autoscaler (HPA):

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: loan-service-hpa
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: loan-service
     minReplicas: 3
     maxReplicas: 10
     metrics:
     - type: Resource
       resource:
         name: cpu
         target:
           type: Utilization
           averageUtilization: 70
   

3. Deploy the Service:

   apiVersion: v1
   kind: Service
   metadata:
     name: loan-service
   spec:
     selector:
       app: loan-service
     ports:
     - protocol: TCP
       port: 80
       targetPort: 8080
     type: LoadBalancer
   

What Happens During Peak Hours?

• Step 1: CPU usage rises above 70%.
• Step 2: HPA detects this and adds more replicas (up to 10).
• Step 3: The LoadBalancer service distributes traffic across all pods.
• Step 4: If a node fails, Kubernetes reschedules pods to healthy nodes.

Advantages and Disadvantages of Container Orchestration

Advantages:

• Automation: Reduces manual intervention in deployment and scaling.
• High Availability: Ensures applications stay running even if nodes fail.
• Efficiency: Optimizes resource usage (CPU, memory, network).
• Portability: Containers can run anywhere (on-premises, cloud, hybrid).
• Scalability: Handles traffic spikes seamlessly.

Disadvantages:

• Complexity: Requires learning Kubernetes/YAML/configuration.
• Resource Overhead: Master nodes consume resources.
• Networking Challenges: Managing inter-pod communication can be tricky.
• Cost: Running large clusters can be expensive.

Comparison: Orchestration Tools

In the Real World

1. Google’s Borg (Kubernetes’ Predecessor)

   • What it does: Google uses Borg (now Kubernetes) to orchestrate millions of containers across its data centers.
   • How it uses orchestration:
     • Automatically schedules jobs (e.g., YouTube video processing) based on resource availability.
     • Scales services like Gmail or Search to handle global traffic.
     • Self-heals by restarting failed tasks on other machines.

2. WhatsApp’s Containerized Backend

   • What it does: WhatsApp uses Kubernetes to manage its real-time messaging service.
   • How it uses orchestration:
     • Deploys microservices (e.g., message routing, media storage) as containers.
     • Scales horizontally during peak usage (e.g., New Year’s Eve).
     • Ensures 99.999% uptime by rescheduling pods if a node fails.

3. Nepal’s NTC: Network Service Orchestration

   • What it does: The Nepal Telecommunications Corporation (NTC) uses container orchestration to manage its SDN (Software-Defined Networking) infrastructure.
   • How it uses orchestration:
     • Deploys network functions (e.g., firewalls, load balancers) as containers.
     • Scales services dynamically based on internet traffic patterns.
     • Uses Kubernetes to manage edge computing for rural connectivity.

Exam Tip

What to Expect in the Exam:

1. Definitions:

   • Explain container orchestration, Kubernetes components, and scheduling.
   • Define pods, services, and autoscaling.

2. Diagrams:

   • Draw the Kubernetes architecture (master/worker nodes).
   • Sketch a service discovery flow (user → load balancer → service → pod).

3. Scenario-Based Questions:

   • Given a use case (e.g., eSewa payment system), describe how Kubernetes would:
     • Schedule pods.
     • Scale horizontally.
     • Ensure high availability.
   • Example question:

     "A Daraz order-processing system needs to handle 10,000 orders/hour. Explain how Kubernetes would scale and manage this workload."

4. YAML/Configuration:

   • You may be asked to write a pod spec or HPA configuration for a given scenario.
   • Focus on key fields: resources, nodeSelector, tolerations, replicas.

5. Advantages/Disadvantages:

   • Compare Kubernetes vs. Docker Swarm or discuss trade-offs of orchestration.

How to Score Full Marks:

• Use real-world examples (e.g., eSewa, Ncell, Google) to illustrate concepts.
• Draw diagrams for architecture, scheduling, or service flows.
• Explain step-by-step how orchestration solves a problem (e.g., scaling, self-healing).
• Mention key Kubernetes objects (pods, deployments, services, HPA) in your answers.