Verification & Validation in Simulation: Techniques, Metrics & Pitfalls

Key Concepts and Definitions

Verification vs. Validation

Verification asks: "Are we building the model right?" It checks if the model implementation matches the conceptual design and mathematical formulation. Validation asks: "Are we building the right model?" It assesses whether the model accurately represents the real-world system it simulates.

flowchart LR
    A["Verification"] -->|"Checks"| B["Model matches design"]
    C["Validation"] -->|"Checks"| D["Model matches reality"]
    B -->|"✓ Passes"| E["Proceed to Validation"]
    D -->|"✓ Passes"| F["Model is useful"]

Common Verification Techniques

1. Face Validation: Experts visually inspect the model to ensure it "looks right."
2. Animation Checks: Simulate the model and observe if outputs align with expectations.
3. Tracing: Compare simulation outputs with analytical solutions for simple cases.
4. Unit Testing: Test individual components (e.g., a queueing module) in isolation.

Common Validation Techniques

1. Historical Data Comparison: Run the model with past data and compare outputs to real historical outcomes.
2. Sensitivity Analysis: Vary inputs to see if outputs behave as expected.
3. Expert Opinion: Consult domain experts (e.g., traffic engineers for a road simulation).
4. Statistical Tests: Use hypothesis testing (e.g., t-tests) to compare simulation vs. real data.

Worked Example: Validating a Queueing Model for Pathao Drivers

Scenario

Pathao wants to simulate driver wait times at pickup points in Kathmandu. They built a queueing model with:

• Arrival rate (λ) = 10 drivers/hour (Poisson)
• Service rate (μ) = 12 drivers/hour (Exponential)
• Queue capacity = 5 drivers

Goal: Validate if the model predicts real-world wait times.

Step 1: Verification (Is the model built correctly?)

• Face Validation: Experts confirm the queueing logic (M/M/1/K) matches Pathao’s operations.
• Tracing: For λ=2, μ=3, the steady-state probability $P_0=1-\frac{\lambda }{\mu }=1-\frac{2}{3}=\frac{1}{3}$. The model’s code outputs $P_0=0.333$. ✅ Correct.

Step 2: Validation (Does the model match reality?)

Pathao collects real data for 1 week:

Statistical Test: Perform a paired t-test (H₀: mean difference = 0).

• Null hypothesis: Model predictions ≠ real data.
• Test statistic: $t=\frac{\bar{d}}{s_d/\sqrt{n}}$, where $\bar{d}=0.1$, $s_d=0.3$, $n=3$.
• $t=\frac{0.1}{0.3/\sqrt{3}}=0.58$. Critical t-value (α=0.05, df=2) = 4.30.
• Since $0.58<4.30$, fail to reject H₀. The model is statistically valid.

Visualizing Validation: Animation Check for a Traffic Simulation

Common Pitfalls and How to Avoid Them

In the Real World

1. eSewa’s Payment Queue Simulation

   • Idea Used: Verification via animation checks.
   • How: eSewa simulates payment processing queues to ensure no transactions are dropped during peak hours (e.g., Dashain). Animations show transaction flows, and developers verify that the queue logic matches the system design.

2. NTC’s Network Traffic Model

   • Idea Used: Validation with historical data.
   • How: NTC validates its network congestion model by comparing simulated traffic volumes against real call-drop rates during festivals (e.g., Tihar). If the model predicts 5% drops but reality shows 15%, they adjust parameters (e.g., increase server capacity in the model).

3. Daraz’s Warehouse Robot Simulation

   • Idea Used: Sensitivity analysis.
   • How: Daraz tests its automated warehouse model by varying robot speeds and order volumes. If increasing speed from 1 m/s to 1.5 m/s reduces delivery time by only 5% (instead of the expected 20%), they validate that the model’s speed parameter needs recalibration.

Worked Example: Verifying a Markov Chain for NEPSE Stock Prices

Scenario

NEPSE wants to model stock price movements as a Markov chain with 3 states:

1. Bullish (price ↑)
2. Stable (price ≈)
3. Bearish (price ↓)

Transition matrix (from a study):

      Bullish | Stable | Bearish
Bullish: 0.6    | 0.3    | 0.1
Stable:  0.2    | 0.5    | 0.3
Bearish: 0.1    | 0.4    | 0.5

Step 1: Verification (Does the matrix sum to 1?)

Check rows:

• Bullish: 0.6 + 0.3 + 0.1 = 1 ✅
• Stable: 0.2 + 0.5 + 0.3 = 1 ✅
• Bearish: 0.1 + 0.4 + 0.5 = 1 ✅

Step 2: Validation (Does it match real data?)

Collect NEPSE’s daily price changes for 100 days:

• Bullish days: 40
• Stable days: 35
• Bearish days: 25

Compare with model’s steady-state probabilities (π): Solve $\pi =\pi P$:

π₁ = 0.6π₁ + 0.2π₂ + 0.1π₃
π₂ = 0.3π₁ + 0.5π₂ + 0.4π₃
π₃ = 0.1π₁ + 0.3π₂ + 0.5π₃
π₁ + π₂ + π₃ = 1

Solution: $\pi =[0.428,0.357,0.214]$.

Comparison:

Conclusion: The model is partially valid but underestimates bearish states. Adjust transition probabilities (e.g., increase Bearish→Bearish to 0.6).

Graph: Loss Function for Model Validation

Interpretation:

• The MSE curve shows the validation error decreases until λ=15, then increases. This suggests λ=15 is the best parameter for the Pathao queueing model.

Exam Tip

1. Differentiate Verification and Validation:

   • Verification = "Are we building it right?" (e.g., code matches equations).
   • Validation = "Is it right for the purpose?" (e.g., model predicts real-world outcomes).
   • Exam Trick: Use analogies like "Verification is like proofreading a recipe; validation is like testing the cake."

2. Techniques to Remember:

   • Verification: Face validation, tracing, unit testing.
   • Validation: Historical data, sensitivity analysis, expert opinion.
   • Always link techniques to real-world examples (e.g., "NTC uses historical call data for validation").

3. Common Exam Questions:

   • "How would you verify a discrete-event simulation of a bank ATM queue?" → Answer: Use tracing (compare with analytical M/M/1 results) and animation checks.
   • "Validate a model of Kathmandu traffic using sensitivity analysis." → Answer: Vary parameters like traffic light timings and observe if congestion patterns match real data.

4. Avoid These Mistakes:

   • Confusing verification with validation.
   • Forgetting to mention statistical tests (e.g., t-tests, chi-square) for validation.
   • Ignoring real-world constraints (e.g., "The model assumes Poisson arrivals, but real data shows bursts").

5. Diagram Expectations:

   • Draw flowcharts for verification/validation processes.
   • Sketch state diagrams (e.g., Markov chains) and label transitions.
   • For queueing systems, show timelines with arrival/departure events.

Final Note: Always ask, "Does this model help solve a real problem?" If not, it fails validation. Use Nepalese examples (e.g., traffic, banking, e-commerce) to stand out in exams!