Software Risk, Safety, Failure & Ethical Analysis

TAKEAWAYS:

• Software risk is the potential for harm (e.g., data breaches, system crashes) caused by flaws, misuse, or external threats, while safety ensures systems operate without unintended harm to users or society.
• Failure causes include coding errors, poor requirements, environmental factors, and human error—each with distinct mitigation strategies.
• Risk assessment uses frameworks like ISO 31000 to quantify threats (e.g., probability × impact) and prioritize fixes, while management involves policies, testing, and redundancy.
• Ethical dilemmas in software risk arise from trade-offs (e.g., privacy vs. convenience in eSewa) and the responsibility of developers to anticipate harm.
• Real-world examples: Pathao’s ride-hailing algorithm failures (safety), Ncell’s SIM card fraud (risk), and Daraz’s inventory management (safety-critical systems).
• Failure analysis (e.g., Therac-25 radiation overdose) teaches that human factors (e.g., rushed development) and technical debt (e.g., unpatched bugs) are root causes.

---

1. Definitions: Risk, Safety, and Failure in Software

Software risk is the combination of the probability of an adverse event and its impact on stakeholders (users, businesses, society). Unlike safety, which focuses on preventing harm (e.g., autonomous cars avoiding collisions), risk management balances acceptable trade-offs (e.g., "Is a 1% crash risk in a banking app tolerable?").

Software failure occurs when a system deviates from its intended behavior, causing:

• Functional failures (e.g., a Daraz order not shipping).
• Performance failures (e.g., NTC’s website crashing during elections).
• Security failures (e.g., eSewa’s 2021 data breach exposing 1M users).

Why is software safety hard to guarantee? Unlike hardware (e.g., a bridge’s steel beams), software is dynamic, complex, and evolving. Even small changes (e.g., a line of code) can introduce new risks. Assumptions (e.g., "Users will always enter valid data") often fail in real-world use.

---

2. Causes of Software Failure

Failures stem from technical, human, and organizational factors. Below is a classification with real-world ties:

Worked Example: Ncell’s SIM Card Fraud (2020)

• Risk: SIM card cloning enabled by weak authentication.
• Failure: Thousands of users lost access to accounts.
• Root Cause: Lack of multi-factor authentication (MFA) and real-time fraud detection.
• Mitigation: Ncell introduced biometric verification and transaction limits.

---

3. Risk Assessment and Management

A. Risk Assessment Frameworks

Software risks are assessed using structured methods like:

1. ISO 31000: International standard for risk management.
2. FMEA (Failure Modes and Effects Analysis): Identifies failure points and their severity.
3. SWIFT (Software Risk Assessment): Used in banking (e.g., NMB’s risk scoring for loans).

Example: FMEA for a Hospital Management System (HMS)

Key Metric: Risk Priority Number (RPN) = Severity × Occurrence × Detection. Action: Prioritize fixes for high-RPN issues (e.g., patient data leak).

B. Risk Management Strategies

---

4. Ethical Dilemmas in Software Risk

Software risks often create ethical trade-offs:

• Privacy vs. Convenience: eSewa’s biometric login reduces fraud but raises concerns about data misuse.
• Cost vs. Safety: NEPSE’s delayed upgrade to secure trading platforms due to budget constraints.
• Transparency vs. Competition: Daraz hiding algorithmic pricing from sellers to maintain margins.

Case Study: WhatsApp’s End-to-End Encryption (2016)

• Risk: Privacy vs. law enforcement access.
• Ethical Debate: Should apps like WhatsApp allow governments to decrypt messages for national security?
• Outcome: WhatsApp refused, leading to legal battles (e.g., India’s 2018 encryption ban).

---

5. Real-World Applications

## In the real world

1. Pathao’s Ride-Hailing Algorithm Failures

   • Idea Used: Safety-critical system design (avoiding accidents due to driver behavior).
   • Example: Pathao’s AI flagged drivers with erratic braking patterns, reducing crash risks by 15% in Kathmandu.
   • Worked Example: In 2022, Pathao’s algorithm detected a driver’s alcohol-induced erratic speeding and temporarily suspended their account, preventing a near-miss with a school bus.

2. Ncell’s SIM Card Fraud Prevention

   • Idea Used: Risk assessment and multi-factor authentication (MFA).
   • Example: After a 2020 breach where 5,000 SIMs were cloned, Ncell implemented:
     • Biometric verification (fingerprint + PIN).
     • Real-time transaction monitoring (flags unusual activity).
   • Result: Fraud dropped by 60%.

3. Daraz’s Inventory Management System

   • Idea Used: Safety in supply chain reliability.
   • Example: Daraz’s algorithm predicts stockouts (e.g., during festivals) to avoid "out of stock" errors, ensuring 95% order fulfillment.

---

6. Failure Analysis: The Therac-25 Case

Lessons for Nepal:

• Testing: Always test edge cases (e.g., what if a user enters "0" for radiation dose?).
• Redundancy: Use hardware safeguards (e.g., physical switches) alongside software.
• Ethical Responsibility: Developers must prioritize patient safety over speed.

---

7. Exam Tip

This unit tests conceptual understanding and application to real scenarios. Focus on:

1. Comparing safety vs. security (e.g., "Why is a hospital’s lab software safer than a bank’s app?").
2. Linking failure causes to examples (e.g., "How did Ncell’s poor testing lead to SIM fraud?").
3. Risk assessment tables (practice FMEA or SWIFT for a given system).
4. Ethical dilemmas (e.g., "Should eSewa prioritize fraud prevention over user convenience?").
5. Mitigation strategies (avoid, mitigate, transfer, accept—pick the best for a case).

Common Pitfalls:

• Confusing security (protecting data) with safety (preventing harm).
• Ignoring human factors (e.g., assuming users will follow instructions).
• Overlooking external risks (e.g., natural disasters affecting servers).

Sample Question Breakdown:

"Compare security with safety. Why is software safety so difficult to attain?" Answer Structure:

1. Define both (table or short paragraphs).
2. Contrast:
   • Security: Protects data (e.g., encryption).
   • Safety: Prevents harm (e.g., autonomous car brakes).
3. Why safety is hard:
   • Dynamic environments (e.g., user behavior).
   • Complex dependencies (e.g., third-party APIs).
   • Cannot be 100% guaranteed (e.g., no system is immune to human error).

---

Final Visual: Risk Management Cycle