Verification & Validation in Simulation: Methods, Checks & Output Analysis

Key Concepts and Definitions

1. Verification vs. Validation: The Core Difference

Verification and validation (V&V) are two distinct but complementary processes in simulation. While they both aim to ensure model reliability, they address different questions:

Visual:

graph LR
    A["Simulation Model"] --> B["Verification: Is it built right?"]
    A --> C["Validation: Does it work right?"]
    B --> D["Code reviews\nUnit testing\nTraceability"]
    C --> E["Real-world data\nExpert judgment\nStatistical tests"]

2. Verification: Ensuring the Model is Built Correctly

Verification is about checking the internal consistency of the model. It ensures that the simulation code, equations, and logic align with the intended design. Common techniques include:

A. Code Review and Debugging

• Static Analysis: Tools like linters or IDE checks (e.g., PyCharm for Python) flag syntax errors or logical inconsistencies.
• Unit Testing: Test individual components (e.g., a queue’s enqueue() method) in isolation.
• Traceability: Document how model inputs map to outputs (e.g., a bank loan simulation’s interest calculation formula).

B. Face Validation (Quick Check)

A subject-matter expert (SME) reviews the model for obvious flaws (e.g., a traffic simulation where cars disappear mid-route). This is a low-cost, high-impact step.

C. Animation and Animation Checks

• Purpose: Visualize the model’s behavior to spot anomalies (e.g., a particle moving backward in a physics simulation).
• Example: In a Daraz delivery simulation, an animation might reveal that some orders are stuck in a "processing" state indefinitely.

Visual:

flowchart LR
    A["Model Code"] --> B["Static Analysis\n(Linters, IDE)"]
    A --> C["Unit Testing\n(Test individual functions)"]
    A --> D["Traceability\n(Document input→output mapping)"]
    A --> E["Face Validation\n(SME review)"]
    A --> F["Animation\n(Visual debugging)"]

3. Validation: Ensuring the Model Represents Reality

Validation compares the model’s outputs with real-world data or expert judgment. Techniques include:

A. Historical Data Comparison

• Method: Run the simulation with past inputs and compare outputs to actual historical data.
• Example: A NTC electricity demand simulation is validated by comparing predicted peak hours with real NTC load data.

B. Sensitivity Analysis

• Purpose: Test how changes in input parameters affect outputs.
• Example: In a Khalti transaction simulation, adjust the failure rate of internet connectivity and observe how it impacts successful transactions.

C. Statistical Tests

Use statistical methods to quantify how well the model fits reality:

• Chi-Square Test: Compare simulated vs. observed frequencies (e.g., call volumes in a Ncell customer service simulation).
• Kolmogorov-Smirnov Test: Check if simulated and real data distributions match.

Visual (Statistical Validation Workflow):

flowchart TD
    A["Simulated Data"] --> B["Collect Real-World Data"]
    B --> C["Apply Chi-Square Test\n(Compare frequencies)"]
    B --> D["Apply KS Test\n(Compare distributions)"]
    C --> E["Accept/Reject Model\nBased on p-value"]
    D --> E

4. Output Analysis: Making Sense of Simulation Results

Even a validated model requires careful analysis of its outputs. Key steps:

1. Descriptive Statistics: Mean, variance, confidence intervals.
2. Time-Series Analysis: Trends over time (e.g., NEPSE stock price simulation).
3. Hypothesis Testing: Is the model’s improvement statistically significant?

Example: Validating a Pathao Driver Wait-Time Simulation

• Simulated Data: Average wait time = 5.2 minutes.
• Real Data: Average wait time = 5.0 minutes (from Pathao’s app logs).
• Test: Run a t-test to see if the difference is statistically significant (p > 0.05 → accept the model).

In the Real World

Simulation verification and validation are critical in industries where mistakes cost millions. Here’s how companies in Nepal and globally apply these ideas:

1. E-Sewa (Nepal) – Traffic Simulation for Road Projects

• Problem: E-Sewa needs to model traffic flow for new road designs in Kathmandu.
• Verification: Ensures the simulation uses correct traffic rules (e.g., priority at roundabouts).
• Validation: Compares simulated congestion times with real GPS data from Pathao drivers.
• Output: Helps decide whether to build a flyover or widen an existing road.

2. Ncell – Call Center Queuing Simulation

• Problem: Ncell wants to optimize call center staffing.
• Verification: Checks if the simulation’s queue discipline (FIFO) matches real call logs.
• Validation: Compares simulated wait times with actual customer complaints data.
• Result: Reduces average wait time from 4 minutes to 2 minutes.

3. Daraz – Order Fulfillment Pipeline Simulation

• Problem: Daraz needs to predict delays in order processing.
• Verification: Ensures the simulation’s "pick-pack-ship" steps are coded correctly.
• Validation: Uses real order data to check if simulated delays match actual delivery times.
• Output: Identifies that weather delays (not warehouse efficiency) are the biggest bottleneck.

Worked Example: Validating a Bank Loan Simulation

Scenario: A bank wants to simulate loan default rates. The model uses:

• Input: Customer credit score (300–850), loan amount, interest rate.
• Output: Probability of default.

Step 1: Verification

• Check: Does the model correctly apply the interest formula?
  • Formula: $\text{Monthly Payment}=P\times \frac{r(1+r{)}^n}{(1+r{)}^n-1}$
  • Trace: For $P=100,000$, $r=0.005$ (0.5% monthly), $n=120$: $$\text{Payment}=100,000\times \frac{0.005(1.005{)}^{120}}{(1.005{)}^{120}-1}\approx 1,054.12$$
  • Code Check: Does the simulation’s calculate_payment() function return this value?

Step 2: Validation

• Historical Data: The bank has 1,000 past loans with default rates by credit score.
• Simulation Output: Run the model 1,000 times with the same inputs.
• Comparison:

  Credit Score	Simulated Default Rate	Real Default Rate
  750+	2.1%	2.3%
  650–749	5.8%	6.0%
  <650	12.4%	11.8%

• Statistical Test: Chi-Square test shows $p=0.12$ (not significant), so the model is valid.

Step 3: Output Analysis

• Insight: The model predicts that raising the interest rate by 0.25% increases defaults by 0.8% for low-credit-score borrowers.
• Decision: The bank adjusts its risk policy accordingly.

Common Pitfalls and How to Avoid Them

Exam Tip

This unit is highly theoretical but practical. Exams often test:

1. Definitions: Clearly distinguish verification (model correctness) vs. validation (real-world accuracy).
2. Methods: Know when to use face validation, historical data comparison, or statistical tests.
3. Worked Examples: Be ready to:
   • Validate a simple queuing model (e.g., NTC call center).
   • Perform a Chi-Square test on simulated vs. real data.
   • Explain how animation helps in debugging.
4. Real-World Applications: Link concepts to Nepali examples (e.g., Khalti transactions, Daraz logistics).
5. Output Analysis: Understand how to interpret confidence intervals and hypothesis test results.

Common Exam Questions:

• "How would you verify and validate a simulation of Kathmandu traffic?"
• "Given simulated and real data, how would you statistically validate a model?"
• "What are the steps to ensure a bank loan simulation is both verified and validated?"

Key Takeaways:

• Verification = "Did we build the model right?" (Code, logic, traceability).
• Validation = "Did we build the right model?" (Real-world comparison).
• Animation and face validation are quick checks; statistical tests provide rigor.
• Output analysis turns raw simulation data into actionable insights.
• Real-world examples (E-Sewa, Ncell, Daraz) show how V&V prevents costly mistakes.