System Testing & Maintenance: Types, Processes, and Real-World Impact

Core Concepts: What is System Testing and Maintenance?

System testing and maintenance are interdependent phases in the System Development Life Cycle (SDLC) that ensure a software system meets user requirements, performs reliably, and adapts to changing needs over time.

1. System Testing: Ensuring Quality Before Deployment

System testing is the final evaluation phase before a system is released to end-users. It validates the complete system against specified requirements, identifying defects, performance bottlenecks, and usability issues.

Key Objectives:

• Verify that the system works as intended (functional testing).
• Ensure the system meets non-functional requirements (performance, security, usability).
• Confirm compatibility with hardware, software, and network environments.
• Validate user acceptance (alpha/beta testing).

Types of System Testing

Testing is categorized based on scope, purpose, and techniques. Below is a comparison table:

Testing Methodologies: Black-Box vs. White-Box

Testing approaches differ based on access to internal system details and testing focus.

1. Black-Box Testing

• Definition: Tests the system without knowledge of internal code/logic. Focuses on input-output behavior.
• Techniques:
  • Equivalence Partitioning: Divides input data into groups (e.g., valid/invalid user IDs).
  • Boundary Value Analysis: Tests edge cases (e.g., maximum loan amount in a bank system).
  • Decision Tables: Maps inputs to expected outputs (e.g., "If user is admin, grant access").
• Example:
  • Testing WhatsApp’s login system by entering random passwords to check if it rejects invalid attempts.

2. White-Box Testing

• Definition: Tests internal structures (code, paths, conditions) using knowledge of the system’s design.
• Techniques:
  • Statement Coverage: Ensures every line of code is executed at least once.
  • Branch Coverage: Tests all decision branches (e.g., if-else conditions).
  • Path Testing: Executes all possible code paths.
• Example:
  • A developer testing Google’s search algorithm by verifying how it handles synonyms in queries.

Comparison Table:

The Testing Process: A Step-by-Step Workflow

The testing process follows a structured lifecycle to ensure thorough validation. Below is a Mermaid flowchart of the process:

flowchart TD
    A["Start"] --> B["Requirements Analysis"]
    B --> C["Test Planning"]
    C --> D["Test Case Design"]
    D --> E["Test Environment Setup"]
    E --> F["Test Execution"]
    F --> G["Defect Logging"]
    G --> H["Defect Triage"]
    H -->|"Fix Required"| I["Developer Fixes"]
    H -->|"No Fix Needed"| J["Retesting"]
    I --> J
    J --> K["Regression Testing"]
    K --> L["User Acceptance Testing"]
    L --> M["Sign-Off"]
    M --> N["Deployment"]

Key Steps Explained:

1. Requirements Analysis:

   • Review system requirements (functional/non-functional) to define test scope.
   • Example: For Khalti’s wallet system, test cases must cover all transaction types (P2P, merchant, refunds).

2. Test Planning:

   • Define test strategy, resources, timelines, and entry/exit criteria.
   • Example: Allocate 2 weeks for Ncell’s app performance testing under 5G and 4G networks.

3. Test Case Design:

   • Create test cases using techniques like equivalence partitioning or decision tables.
   • Example: For Daraz’s checkout system, test cases include:
     • Valid/invalid card numbers.
     • Maximum order limits.
     • Concurrent user load (1000+ users).

4. Test Execution:

   • Run tests manually or via automation tools (e.g., Selenium for UI, JMeter for load testing).
   • Example: Automated eSewa’s API tests to check response times during Diwali sales.

5. Defect Logging and Triage:

   • Log defects in tools like Jira or Bugzilla.
   • Prioritize based on severity (e.g., critical = system crash, minor = UI glitch).

6. Regression Testing:

   • Re-test affected areas after fixes to avoid new defects.
   • Example: After fixing a bank loan approval bug, re-test all approval workflows.

7. User Acceptance Testing (UAT):

   • End-users (or a representative group) validate the system.
   • Example: NEPSE traders testing the new trading platform for accuracy.

System Maintenance: Keeping the System Alive

Maintenance is not just fixing bugs—it’s a proactive process to ensure the system remains reliable, secure, and aligned with business needs. The Software Engineering Institute (SEI) categorizes maintenance into four types:

mindmap
  root((System Maintenance))
    Corrective
      Definition: Fixing defects post-deployment.
      Example: Patching a **Khalti API vulnerability**.
    Adaptive
      Definition: Updating to **new environments** (OS, hardware, regulations).
      Example: **NTC’s system upgrade** to support 5G billing.
    Perfective
      Definition: Enhancing **performance, usability, or features**.
      Example: Adding **biometric login** to **eSewa**.
    Preventive
      Definition: **Proactive measures** to avoid future issues.
      Example: **Database optimization** in **Daraz’s inventory system**.

The Maintenance Process

1. Identify Maintenance Needs:

   • Monitor system logs, user feedback, and performance metrics.
   • Example: Pathao’s drivers report app crashes during peak hours → triggers maintenance.

2. Prioritize Requests:

   • Use a priority matrix (e.g., critical bugs > feature enhancements).
   • Example: A security flaw in Ncell’s app takes precedence over a UI tweak.

3. Implement Changes:

   • Develop fixes/enhancements in a controlled environment (e.g., staging server).
   • Example: Google Maps rolls out updates in phases to test stability.

4. Test Changes:

   • Conduct regression testing to ensure no new defects.
   • Example: After updating Merocredit’s loan calculator, test all interest rate scenarios.

5. Deploy and Monitor:

   • Release updates and monitor real-time performance.
   • Example: WhatsApp tracks message delivery rates post-update.

Measuring Maintenance Effectiveness

Effective maintenance is measured using quantitative and qualitative metrics:

Key Insight:

• A system with high MTBF and low MTTR is highly maintainable.
• Example: Google’s Gmail has an MTBF of ~99.9% uptime, meaning failures are rare.

In the Real World

System testing and maintenance are critical in Nepal’s digital economy. Here’s how real-world systems apply these concepts:

1. eSewa: Payment Gateway Testing

   • Testing: Black-box testing for transaction validation (e.g., testing if a Rs. 500 transfer succeeds or fails).
   • Maintenance: Corrective maintenance after the 2021 cyberattack, where eSewa patched vulnerabilities in its API authentication.
   • Real Example: During Dashain, eSewa’s system undergoes load testing to handle 100,000+ transactions/hour.

2. Daraz: Order Fulfillment System

   • Testing:
     • Integration testing between inventory, order processing, and logistics modules.
     • Usability testing for the seller dashboard (e.g., checking if a seller can cancel an order within 2 hours).
   • Maintenance:
     • Perfective maintenance to add AI-driven fraud detection for fake orders.
     • Adaptive maintenance to comply with Nepal’s new e-commerce tax laws.

3. Ncell: Network and Billing System

   • Testing:
     • Performance testing to ensure 4G/5G network stability during IPL matches (high data usage).
     • Security testing for SIM swap fraud prevention.
   • Maintenance:
     • Preventive maintenance via automated log analysis to predict server failures.
     • Corrective maintenance after the 2022 outage, where Ncell upgraded its CDN (Content Delivery Network).

4. NEPSE: Stock Trading Platform

   • Testing:
     • Regression testing after every market rule update (e.g., new short-selling regulations).
     • Stress testing to handle sudden price spikes (e.g., during budget announcements).
   • Maintenance:
     • Adaptive maintenance to support blockchain-based trading (pilot in 2023).
     • User acceptance testing with brokerage firms before new features are rolled out.

5. Khalti: Digital Wallet Security

   • Testing:
     • Penetration testing to simulate credit card skimming attacks.
     • White-box testing of encryption algorithms used for transactions.
   • Maintenance:
     • Preventive maintenance via quarterly security audits.
     • Corrective maintenance after the 2020 data breach, where Khalti implemented two-factor authentication (2FA).

6. Pathao: Ride-Matching Algorithm

   • Testing:
     • Load testing during Chhath festival (high demand in Kathmandu).
     • Usability testing for driver app navigation in rural areas with poor GPS.
   • Maintenance:
     • Perfective maintenance to add real-time traffic rerouting using Google Maps API.
     • Adaptive maintenance to support electric vehicle drivers.

Worked Example: Testing a Bank Loan System

Scenario: A bank in Nepal develops a new online loan approval system. We’ll design test cases for black-box and white-box testing.

1. Black-Box Testing: Functional Test Cases

2. White-Box Testing: Code Path Coverage

Assume the loan approval logic has these decision points:

stateDiagram-v2
    [*] --> CheckCIBIL
    CheckCIBIL -->|CIBIL >= 700| CheckIncome
    CheckCIBIL -->|CIBIL < 700| Reject
    CheckIncome -->|Income >= 80K| CheckAge
    CheckIncome -->|Income < 80K| Reject
    CheckAge -->|Age 25-60| Approve
    CheckAge -->|Age <25 or >60| Reject

Test Cases for Branch Coverage:

3. Maintenance Scenario: Post-Deployment Bug

Bug Report:

• Issue: Some users with CIBIL score 700-720 are wrongly rejected.
• Root Cause: The code had a hardcoded threshold of 700 instead of the new policy (680).
• Fix: Update the condition from CIBIL >= 700 to CIBIL >= 680.
• Regression Test:
  • Re-test WB-001 and TC-001 to ensure the fix works.
  • Add a new test case:

    Test Case ID	Description	Input	Expected Output
    TC-005	Borderline CIBIL approval.	CIBIL: 680, Income: Rs. 90,000	Approval (new threshold).

Modern Approaches to Testing and Maintenance

Traditional testing is reactive (fix after failure). Modern approaches emphasize:

1. Shift-Left Testing:

   • Integrate testing early in the SDLC (e.g., continuous integration/continuous deployment (CI/CD)).
   • Example: Google runs automated unit tests every time a developer pushes code.

2. DevOps and Agile Testing:

   • Automated pipelines (e.g., Jenkins, GitLab CI) run tests on every commit.
   • Example: WhatsApp uses feature flags to test new features with a subset of users before full rollout.

3. AI and Machine Learning in Testing:

   • AI-driven test case generation (e.g., Testim, Applitools).
   • Predictive maintenance using anomaly detection (e.g., Netflix’s chaos engineering).

4. Chaos Engineering:

   • Intentionally break systems to test resilience.
   • Example: Amazon runs chaos experiments to ensure its cloud services recover from failures.

Exam Tip: How to Score Full Marks

This unit is highly practical—expect diagrams, scenarios, and calculations. Here’s how to excel:

1. For Diagram-Based Questions (DFDs, Flowcharts, Sequence Diagrams)

• Always label clearly: Use standard symbols (e.g., circles for external entities, rectangles for processes).
• Example: If asked to draw a context diagram for a hospital management system, include:
  • External entities: Patients, Doctors, Insurance Companies.
  • Process: "Patient Registration System."
  • Data flows: "Patient Details → Registration System."
• Mermaid Tip: Use sequenceDiagram for protocol exchanges (e.g., login process in eSewa).

2. For Testing Methodology Questions

• Compare black-box vs. white-box in a table (as shown above).
• Mention real tools: Selenium (UI), JUnit (unit), Postman (API).
• Example Answer for "Differentiate black-box and white-box testing":

  Black-box testing treats the system as a "black box", focusing on input-output validation without internal knowledge. It uses techniques like equivalence partitioning (e.g., testing valid/invalid user logins in Khalti). In contrast, white-box testing examines internal code paths, such as branch coverage in Google’s search algorithm, to ensure all logical conditions are tested. While black-box is ideal for end-user validation, white-box is critical for developer-level debugging.

3. For Maintenance Questions

• Use the 4 types of maintenance in your answer (corrective, adaptive, perfective, preventive).
• Relate to Nepali examples:

  Adaptive maintenance is evident in NTC’s system upgrades to support 5G networks, while corrective maintenance was applied after the 2022 eSewa outage due to a DDoS attack. Perfective maintenance includes Daraz’s AI chatbot for customer support, and preventive maintenance involves regular database backups in banking systems like NMB’s core banking software.

4. For Calculation Questions (e.g., ROI, MTBF)

• Show all steps:
  • Example: Given development cost = $90,000, annual benefit = $70,000 (Year 1) + $10,000 increase each year, recurring cost = $40,000/year, calculate Net Present Value (NPV) over 5 years (assume discount rate = 10%).
  • Solution:

    Year	Benefit	Recurring Cost	Net Benefit	PV Factor (10%)	PV of Net Benefit
    1	70,000	40,000	30,000	0.909	27,270
    2	80,000	40,000	40,000	0.826	33,040
    3	90,000	40,000	50,000	0.751	37,550
    4	100,000	40,000	60,000	0.683	40,980
    5	110,000	40,000	70,000	0.621	43,470
    Total PV of Benefits				182,310
    Initial Cost				-90,000
    NPV				$92,310

5. Common Pitfalls to Avoid

• Vague answers: Always link concepts to real systems (e.g., "Like in eSewa’s payment system...").
• Ignoring non-functional testing: Mention performance, security, and usability in system testing answers.
• Forgetting maintenance types: If asked about maintenance, always list all four types with examples.
• Poor diagrams: If drawing a DFD, ensure data flows are labeled and external entities are separate.

Final Checklist Before Submission

✅ Diagrams: All flowcharts, DFDs, and sequence diagrams are labeled and clear. ✅ Examples: Every concept is tied to a Nepali or global real-world system (e.g., eSewa, Daraz, Google). ✅ Calculations: All numerical answers show step-by-step workings. ✅ Definitions: Key terms (black-box, white-box, MTBF, regression testing) are precisely defined. ✅ Exam Tips: Answers are structured to match expected marking schemes (e.g., tables for comparisons, diagrams for processes).

Real image of a server rack (like those used by NTC or Ncell for system hosting). (Image: Derrick Coetzee from Berkeley, CA, USA, CC0, via Wikimedia Commons)