Kernel & OS Structures: Monolithic vs Microkernel, System Calls & Kernel Modes

TAKEAWAYS:

• The kernel is the OS core managing hardware, processes, and memory, running in privileged mode (Ring 0) to execute critical tasks like scheduling and I/O.
• Monolithic kernels (e.g., Linux) bundle all OS functions into one large executable, offering speed but complexity, while microkernels (e.g., QNX) isolate services in user space for modularity and stability.
• System calls bridge user programs and kernel via traps or software interrupts, handled by the system call table and dispatcher (e.g., fork() in Linux).
• Kernel modes (Ring 0) enforce security: user mode (Ring 3) cannot directly access hardware; mode switches occur via interrupts or exceptions.
• Layered and modular designs (e.g., Windows NT) separate kernel components (e.g., scheduler, memory manager) to improve maintainability and scalability.
• Real-world tie: eSewa’s transaction processing uses a microkernel-like design to isolate payment modules, while NTC’s network routing relies on a monolithic kernel for low-latency packet handling.

1. The Kernel: OS Core and Privileged Mode

The kernel is the heart of an operating system, responsible for:

• Process management (creation, scheduling, termination).
• Memory management (allocation, protection, virtualization).
• Device drivers (hardware abstraction).
• File systems (storage and I/O).
• Security and isolation (preventing user programs from crashing the system).

Unlike user applications, the kernel runs in privileged mode (Ring 0 in x86 architecture), granting it direct access to hardware. User programs run in unprivileged mode (Ring 3), restricted by the Hardware Protection Ring mechanism.

stateDiagram-v2
    [*] --> UserMode: Ring 3 (User Programs)
    UserMode --> KernelMode: System Call/Interrupt
    KernelMode --> [*]: Return to UserMode
    note right of KernelMode
        Privileged instructions (e.g.,
        HALT, I/O operations) only
        allowed in Ring 0.
    end note

Why Ring 0?

• Prevents malicious user programs from rebooting the system or stealing data.
• Enforces security boundaries (e.g., preventing a crashed browser from taking down the OS).

| A CPU’s protection rings: Ring 0 (kernel) has full access, while Ring 3 (user) is restricted to safe instructions. |

2. Monolithic vs Microkernel Architectures

The kernel’s design structure impacts performance, scalability, and maintainability. Two dominant approaches:

How it works:

• Monolithic Kernel: All OS functions (scheduler, file system, drivers) are compiled into one binary. Example: Linux’s vmlinux image.
• Microkernel: Only essential kernel functions (memory management, process scheduling) reside in Ring 0. Everything else (e.g., GUI, network stack) runs as user-space processes communicating via message passing.

| A monolithic kernel’s source tree: All components (drivers, scheduler) are compiled into one executable. |

Worked Example: eSewa’s Payment Processing eSewa uses a microkernel-like design to isolate its payment modules:

• Kernel: Handles low-level tasks like transaction logging and bank API calls.
• User-space services:
  • Authentication service (runs in user mode).
  • Fraud detection module (can be updated independently). If the authentication service crashes, eSewa’s core payment system remains stable.

3. System Calls: User-Kernel Interface

User programs cannot directly execute kernel functions (e.g., open(), write()). Instead, they use system calls, which:

1. Trap into kernel mode via a software interrupt (e.g., int 0x80 in x86).
2. The interrupt descriptor table (IDT) routes the call to the kernel’s system call handler.
3. The kernel executes the request and returns results to the user program.

sequenceDiagram
  participant User
  participant IDT
  participant Kernel

  User->>IDT: Trigger syscall (e.g., int 0x80)
  IDT->>Kernel: Route to handler (e.g., open())
  Kernel-->>IDT: Execute request
  IDT-->>User: Return status (success/failure)

System Call Table Example (Linux x86_64):

struct syscall_table {
    void (*sys_read)(...);
    void (*sys_write)(...);
    void (*sys_open)(...);
    // ... 300+ entries
};

How it works:

1. User program calls syscall(2, "read", fd, buffer).
2. CPU triggers software interrupt (e.g., syscall instruction in x86_64).
3. Kernel’s syscall table maps the number to the correct function.
4. Kernel executes read(fd, buffer) and returns success/failure.

| A fragment of Linux’s system call table, showing entries for read, write, and exit. |

Worked Example: Pathao’s Ride Request When you request a ride on Pathao:

1. Your app calls sys_write() to send the request to Pathao’s server.
2. The kernel handles the network socket (via socket() system call).
3. Pathao’s backend processes the request in user space, while the kernel ensures fair CPU scheduling for all apps.

4. Kernel Modes and Mode Switching

The CPU enforces privilege levels via mode switches, triggered by:

• Interrupts (e.g., keyboard input, timer ticks).
• Exceptions (e.g., division by zero).
• System calls (explicit user requests).

stateDiagram-v2
    [*] --> UserMode: Ring 3 (User Programs)
    UserMode --> KernelMode: Interrupt/Exception/System Call
    KernelMode --> [*]: Return to UserMode
    note right of KernelMode
        Mode switch latency: ~100ns
        Example: IRQ0 (timer) triggers scheduler
    end note

Why mode switches matter:

• Prevent user programs from crashing the kernel (e.g., infinite loop in a browser tab won’t reboot your PC).
• Enable hardware abstraction (e.g., the kernel handles read() for both disk and network devices).

| A timeline showing how quickly the CPU switches between user and kernel modes (~100ns). |

5. Layered and Modular Kernel Designs

To improve maintainability and scalability, kernels use:

1. Layered Design (e.g., Windows NT):

   • Each layer has a well-defined interface (e.g., HAL → Kernel → File System).
   • Example: Windows’ Executive Layer (memory manager, security) sits above the kernel.

2. Modular Design (e.g., Linux):

   • Kernels are built from loadable kernel modules (LKMs) (e.g., Wi-Fi drivers).
   • Modules can be added/removed without rebooting.

Advantages of Modularity:

• Hot-plugging: Add a new driver (e.g., USB camera) without rebooting.
• Security: Isolate problematic modules (e.g., a buggy driver won’t crash the whole OS).

Worked Example: NTC’s Network Routing NTC’s routers use a modular kernel design:

• Base kernel: Handles packet forwarding and QoS (Quality of Service).
• Loadable modules:
  • MPLS support (for high-speed data transfer).
  • Firewall rules (can be updated dynamically). If a module fails (e.g., MPLS), NTC can reload it without downtime.

6. Exam Tip: Focus on These Patterns

1. Compare Architectures:

   • Always contrast monolithic (Linux) vs. microkernel (QNX) in terms of performance, fault isolation, and use cases.
   • Example question: "Why does a real-time OS like VxWorks use a microkernel?" Answer: To ensure deterministic behavior (no unpredictable crashes from user-space services).

2. System Call Flow:

   • Draw the 3-step process (user → trap → kernel → return) and label:
     • Software interrupt (e.g., syscall instruction).
     • System call table lookup.
     • Return value (e.g., errno for failure).

3. Kernel Mode vs User Mode:

   • Relate to security: "Why can’t a user program execute HALT?" Answer: Because it’s a privileged instruction only allowed in Ring 0.

4. Real-World Tie:

   • For modularity, mention eSewa’s isolated payment modules.
   • For performance, contrast Linux (monolithic) vs. Windows (hybrid) in servers.

5. Banker’s Algorithm Connection:

   • If asked about deadlocks, recall that kernel design affects resource management:
     • A microkernel isolates resource allocators (e.g., memory manager), reducing deadlock risk.
     • A monolithic kernel may require Banker’s Algorithm for complex resource pools.

| A screenshot of insmod (insert module) in Linux, showing how drivers like nvidia.ko are loaded dynamically. |