Analyzing Simulation Output: Confidence Intervals, Hypothesis Testing & Model Validation

Key Concepts & Flow

graph TD
    A["Simulation Output"] --> B["Data Collection"]
    B --> C["Statistical Analysis"]
    C --> D["Confidence Intervals"]
    C --> E["Hypothesis Testing"]
    C --> F["Model Validation"]
    D --> G["Mean, Variance, Distribution"]
    E --> H["p-values, t-tests, ANOVA"]
    F --> I["Face Validation\nTracing\nAnimation"]

1. Simulation Output Analysis: Why It Matters

Simulations generate stochastic (random) data, so raw results are meaningless without analysis. Key goals:

• Estimate performance metrics (e.g., average wait time in a queue).
• Check if results are statistically significant (not due to luck).
• Validate the model (does it match real-world behavior?).

Real-world analogy:

• Pathao’s driver dispatch system uses simulation to analyze ride wait times. If the simulation shows 95% confidence that wait times exceed 5 minutes, Pathao might adjust driver allocation.

2. Data Collection & Warm-Up Period

Before analysis, simulations must:

1. Run long enough to stabilize (avoid transient effects).
2. Discard initial "warm-up" data (e.g., first 1000 events in a queue).

graph LR
    A["Simulation Start"] --> B["Warm-Up\n(Transient Phase)"]
    B --> C["Steady State\n(Stable Data)"]
    C --> D["Output Collection"]

Worked Example: A bank’s ATM queue simulation runs for 10,000 transactions. The first 2,000 transactions show unstable wait times (customers learning the system). The remaining 8,000 are used for analysis.

3. Confidence Intervals: Estimating Uncertainty

Simulations use random numbers, so results vary per run. Confidence intervals (CI) quantify uncertainty.

How to Calculate CI for the Mean

1. Run the simulation multiple times (e.g., 30 replications).
2. Compute the sample mean ($\bar{X}$) and standard deviation (S).
3. Use the t-distribution (for small samples) or normal distribution (large samples): $$\text{CI}=\bar{X}\pm t_{\alpha /2,n-1}\cdot \frac{S}{\sqrt{n}}$$
   • $t_{\alpha /2,n-1}$: t-value for 95% confidence (from tables).
   • $n$: number of replications.

Worked Example: A Daraz order fulfillment simulation runs 20 times, yielding:

• $\bar{X}=4.2$ hours (avg. delivery time)
• $S=0.5$ hours
• $t_{0.025,19}=2.093$ (for 95% CI)

$$\text{CI}=4.2\pm 2.093\cdot \frac{0.5}{\sqrt{20}}=4.2\pm 0.23\Rightarrow [3.97,4.43]\text{ hours}$$ Interpretation: We are 95% confident the true mean delivery time is between 3.97 and 4.43 hours.

4. Hypothesis Testing: Is the Simulation Useful?

Tests whether simulation results significantly differ from a hypothesis.

Steps:

1. State hypotheses:
   • $H_0$: Null hypothesis (e.g., "mean wait time = 5 minutes").
   • $H_1$: Alternative hypothesis (e.g., "mean wait time ≠ 5 minutes").
2. Choose significance level ($\alpha =0.05$).
3. Compute test statistic (e.g., t-test for means).
4. Compare p-value to $\alpha$:
   • If $p<\alpha$, reject $H_0$ (results are significant).

Worked Example: NTC’s call center simulation tests if adding 5 more agents reduces average call wait time from 120 seconds.

• $H_0$: $\mu =120$ sec
• $H_1$: $\mu <120$ sec
• Simulation runs 25 times, yielding $\bar{X}=112$ sec, $S=8$ sec.
• t-test statistic: $$t=\frac{112-120}{8/\sqrt{25}}=-5$$
• p-value (one-tailed) ≈ 0.00001 (from t-table).
• Conclusion: $p<0.05$, so reject $H_0$. The change is statistically significant.

5. Model Validation: Does the Simulation Match Reality?

Even "correct" models can be wrong if inputs are flawed. Validation methods:

6. Common Pitfalls & Best Practices

❌ Mistakes to Avoid

• Ignoring warm-up period → Biased results.
• Using too few replications → Wide confidence intervals.
• Assuming normality → Check with Q-Q plots or Shapiro-Wilk test.

✅ Best Practices

• Use batch means for correlated data (e.g., time-series simulations).
• Plot output over time to detect trends.
• Validate before analyzing—garbage in, garbage out!

In the Real World

1. Khalti’s Payment Processing

   • Idea: Confidence intervals analyze transaction success rates.
   • How: Simulate 10,000 transactions to estimate failure rates with 95% CI. If the upper bound exceeds 0.5%, Khalti may investigate fraud detection.

2. Ncell’s Network Traffic Simulation

   • Idea: Hypothesis testing checks if adding 5G towers reduces latency.
   • How: Compare simulated latency before/after tower addition using a paired t-test.

3. Daraz’s Warehouse Optimization

   • Idea: Model validation ensures the simulation matches real order fulfillment times.
   • How: Animate the simulation and compare to actual warehouse camera footage.

Exam Tip

What Examiners Look For

1. Confidence Intervals:

   • Know the formula and when to use t-distribution vs. normal distribution.
   • Common exam question: Given simulation data, compute a 95% CI for the mean.

2. Hypothesis Testing:

   • Step-by-step: State $H_0/H_1$, compute test statistic, interpret p-value.
   • Example: "A simulation shows a new algorithm reduces processing time. Test if the improvement is significant at $\alpha =0.01$."

3. Validation:

   • Define at least 2 validation methods (e.g., face validation + historical data).
   • Link to real systems: "How would you validate a simulation of Pathao’s ride-matching system?"

4. Visuals:

   • Always plot:
     • Confidence interval bars (e.g., mean ± CI).
     • Time-series output to show steady state.
   • Mermaid tip: Draw a flowchart of your analysis steps (data → warm-up → CI → validation).

Model Answer Structure

Question: "A bank’s loan approval simulation runs 15 times, yielding means of 45 days with a standard deviation of 5 days. Compute a 95% CI for the mean approval time."

Answer:

1. Given:
   • $n=15$, $\bar{X}=45$, $S=5$
   • $t_{0.025,14}=2.145$ (from t-table)
2. CI Formula: $$\text{CI}=45\pm 2.145\cdot \frac{5}{\sqrt{15}}=45\pm 2.76$$
3. Result: $$[42.24,47.76]\text{ days}$$
4. Interpretation: "We are 95% confident the true mean approval time lies between 42.24 and 47.76 days."

Final Visual Summary

mindmap
  root((Simulation Output Analysis))
    CI["Confidence Intervals"]
      Formula["\(\bar{X} \pm t \cdot \frac{S}{\sqrt{n}}\)"]
      Example["Daraz delivery times: [3.97, 4.43] hours"]
    Hypothesis Testing
      Steps["1. State \(H_0/H_1\)\n2. Compute test stat\n3. Compare p-value"]
      Example["NTC call center: p = 0.00001 → Significant"]
    Validation
      Methods["Face\nTracing\nAnimation\nHistorical Data"]
      Real["Pathao ride times vs. simulation logs"]