OS Security, Protection & Linux Case Study

TAKEAWAYS:

• Protection uses access matrices, rings, and capabilities to enforce least privilege (e.g., Linux’s setuid vs. capabilities).
• Security threats include malware (viruses, worms), DoS attacks, and privilege escalation—mitigated by firewalls, encryption, and sandboxing.
• Linux security relies on SELinux/AppArmor for mandatory access control (MAC), while chmod, chown, and umask handle discretionary access.
• Case study: Linux’s security model (kernel hardening, systemd, auditd) mirrors real-world protections in banks (Nepal Rastra Bank) and cloud providers (Google Cloud).
• Exam focus: Compare DAC vs. MAC, trace a buffer overflow exploit, and explain how iptables blocks SYN floods.

1. Protection Mechanisms

Protection ensures system resources (CPU, memory, files) are accessed only by authorized entities. Three core models:

A. Access Matrix Model

• Definition: A table where rows = subjects (processes/users), columns = objects (files, devices), and entries = permissions (read/write/execute).
• How it works:
  • Each cell defines whether a subject can access an object.
  • Simplified via Access Control Lists (ACLs) (e.g., chmod 755 file.txt grants owner full access, others read/execute).
  • Capability lists: Objects hold tickets (capabilities) granting access to subjects (used in Unix for file descriptors).

erDiagram
    PROCESS ||--o{ FILE : "can_access"
    PROCESS {
        string pid
        string user
    }
    FILE {
        string path
        string owner
        string permissions
    }
    PROCESS }|--|| PERMISSION : "has"
    PERMISSION {
        string type read/write/execute
    }

B. Hierarchical Protection (Rings)

• Definition: CPU privilege levels (rings) restrict operations:
  • Ring 0: Kernel (full access).
  • Ring 3: User processes (restricted).
• Example: Linux uses rings for system calls (e.g., open() traps to Ring 0).
• Real-world tie: Nepal Rastra Bank’s core banking system runs in Ring 0, while customer-facing apps (e.g., eSewa) operate in Ring 3.

stateDiagram-v2
    [*] --> Ring3: User Process
    Ring3 --> Ring0: System Call (e.g., read())
    Ring0 --> Ring3: Return Result
    Ring0 --> [*]: Kernel Mode

C. Capabilities vs. Access Control Lists (ACLs)

Worked Example:

• Scenario: A Daraz delivery app needs to read GPS data but not modify system files.
  • Solution: Grant a capability for /dev/ttyGPS (GPS device) without root access.
  • Command:

    setcap cap_location=ep delivery_app
    

2. Security Threats and Countermeasures

A. Malware Types

Real-world example:

• Pathao’s ride-hailing system faced a DDoS attack during Dashain. Mitigation:
  • Deployed iptables to block malicious IP ranges.
  • Used Cloudflare’s DDoS protection for global traffic.

B. Buffer Overflow Exploits

• How it works:
  1. Attacker overflows a fixed-size buffer (e.g., strcpy()).
  2. Overwrites return address on the stack with malicious code.
  3. Code executes with victim’s privileges (e.g., root).
• Defenses:
  • Stack Canaries: Random value before return address (triggers crash if overwritten).
  • ASLR (Address Space Layout Randomization): Randomizes memory addresses.
  • DEP (Data Execution Prevention): Marks stack as non-executable.

sequenceDiagram
    participant Attacker
    participant VulnerableProgram
    participant Kernel
    Attacker->>VulnerableProgram: Sends malicious input (e.g., 1000-byte string)
    VulnerableProgram->>VulnerableProgram: Buffer overflows, overwrites return address
    VulnerableProgram-->>Attacker: Executes shellcode (e.g., `system("/bin/sh")`)
    Kernel->>Attacker: Grants root access

Worked Example:

• Scenario: A bank’s loan processing script uses gets() (unsafe).
  • Exploit: Attacker inputs AAAA; rm -rf / to delete files.
  • Fix: Replace with fgets() and compile with -fstack-protector.

3. Linux Security Features

A. Discretionary Access Control (DAC)

• Mechanism: Owners control access via chmod, chown, umask.
  • Example: chmod 640 secret.txt → Owner: read/write, Group: read-only.
• Weakness: Users can share permissions (e.g., chmod 777 = insecure).

B. Mandatory Access Control (MAC)

• Mechanism: System enforces policies (e.g., SELinux, AppArmor).
  • SELinux: Labels processes/files with security contexts (e.g., httpd_t).
  • AppArmor: Profiles restrict programs (e.g., firefox can’t access /etc/).

classDiagram
    class SecurityContext {
        +string user
        +string role
        +string type
    }
    class Process {
        +string name
        +SecurityContext context
    }
    class File {
        +string path
        +SecurityContext context
    }
    Process "1" --> "1" SecurityContext : "has"
    File "1" --> "1" SecurityContext : "has"
    SecurityContext --> SecurityContext : "allows_access_if" Process.type == File.type

Worked Example:

• Scenario: Prevent nginx from writing to /var/www/html (web root).
  • Solution: Edit /etc/selinux/targeted/contexts/files/file_contexts:

    /var/www/html(/.*)? all_files_t:obj=rwX
    

  • Then restore contexts:

    restorecon -Rv /var/www/html
    

C. Linux Security Tools

Real-world example:

• Ncell’s network uses iptables to block ICMP floods during peak hours:

  iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT
  

4. Case Study: Linux Security Architecture

A. Kernel Hardening

• Features:
  • Stack Smashing Protector (SSP): Detects buffer overflows.
  • Kernel Address Space Layout Randomization (KASLR): Hides kernel memory.
  • Integrity Measurement Architecture (IMA): Verifies file signatures.

B. User-Space Security

• systemd: Runs services in isolated cgroups (resource limits).
• firejail: Sandboxes apps (e.g., firejail google-chrome).

C. File System Security

• Extended Attributes (xattr): Store security metadata (e.g., setfattr -n security.selinux -v "httpd_sys_content_t" file).
• Immutable Flags: Prevent deletion/modification (e.g., chattr +i /etc/passwd).

Real-world tie:

• Google Cloud uses Linux’s cgroups to limit a YouTube video encoder’s CPU to 2 cores, preventing one process from starving others.

5. Hands-On: Configuring Linux Security

Step 1: Enable SELinux

sudo sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config
sudo setenforce 1

Step 2: Block Port Scanning with iptables

iptables -A INPUT -p tcp --dport 22 -m connlimit --connlimit-above 3 -j DROP

Step 3: Audit Failed Logins

sudo auditctl -a exit,always -F arch=b64 -S execve -k execve
sudo ausearch -k execve | audit2why

In the Real World

1. eSewa (Nepal):

   • Idea Used: Mandatory Access Control (MAC) via SELinux.
   • How: eSewa’s backend servers run in a SELinux policy that restricts the httpd process from accessing /var/lib/eSewa/keys/ unless labeled eSewa_key_t. This prevents even a compromised web server from stealing transaction keys.

2. Khalti’s Payment Gateway:

   • Idea Used: Buffer Overflow Protections + ASLR.
   • How: Khalti’s backend (written in Go) uses stack canaries and ASLR to mitigate memory corruption attacks. During the 2022 Dashain sales, Khalti’s systems withheld 99.9% of exploits targeting older PHP-based payment systems.

3. NTC’s Network Monitoring:

   • Idea Used: iptables + fail2ban.
   • How: NTC’s routers drop SYN packets with invalid flags (iptables -A INPUT -p tcp --tcp-flags SYN,ACK,PSH,URG SYN -j DROP) and ban IPs after 5 failed SSH attempts (fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf).

Exam Tip

1. For short answers:
   • Define DAC vs. MAC in one sentence each.
   • List 3 Linux security tools and their uses.
2. For long answers (10 marks):
   • Trace a buffer overflow exploit (steps + defenses).
   • Compare SELinux and AppArmor (table + real-world use case).
   • Design a security policy for a scenario (e.g., "Protect a Daraz order processing system").
3. Common pitfalls:
   • Confusing chmod 755 (DAC) with SELinux (MAC).
   • Forgetting ASLR or stack canaries in exploit mitigation.
   • Overlooking real-world ties (e.g., Ncell’s iptables rules).

Pro Tip: Memorize these 3 commands for the exam:

# Check SELinux status
sestatus

# Block an IP with iptables
iptables -A INPUT -s 192.168.1.100 -j DROP

# Enable AppArmor for a process
aa-enforce /etc/apparmor.d/usr.bin.firefox