Simulation Output Analysis: Confidence Intervals, Hypothesis Testing & Steady-State Detection

Key Concepts & Definitions

1. Simulation Output Data

Simulation output is the data generated from running a simulation model. It can be:

• Terminated output: Data collected over a fixed time period (e.g., simulating a bank’s daily transactions for 8 hours).
• Steady-state output: Data collected after the system reaches equilibrium (e.g., analyzing a hospital’s patient queue after initial fluctuations).

Why analyze output? Raw simulation data is noisy and requires statistical methods to extract meaningful insights.

---

2. Confidence Intervals for Simulation Output

Confidence intervals (CIs) estimate the range within which the true mean of a simulated metric lies, with a certain probability (e.g., 95%).

How to Compute Confidence Intervals

For a sample mean $\bar{X}$ with standard deviation $S$ and sample size $n$: $$\text{CI}=\bar{X}\pm t_{\alpha /2,n-1}\cdot \frac{S}{\sqrt{n}}$$

• $t_{\alpha /2,n-1}$: t-distribution critical value (from tables).
• $S$: Sample standard deviation.
• $n$: Number of replications (runs).

Example: Estimating Average Wait Time in a Queue Suppose we simulate a Pathao driver’s pickup queue 30 times, recording average wait times (in minutes):

Steps:

1. Compute $\bar{X}=4.5$ min, $S=0.8$ min.
2. For 95% CI, $t_{0.025,29}\approx 2.045$ (from t-table).
3. Margin of error (ME) = $2.045\times \frac{0.8}{\sqrt{30}}\approx 0.29$.
4. CI = 4.5 ± 0.29 → [4.21, 4.79] min.

Interpretation: We are 95% confident the true average wait time lies between 4.21 and 4.79 minutes.

---

3. Hypothesis Testing for Simulation Output

Used to compare simulation results against a hypothesis (e.g., "Does a new algorithm reduce latency?").

Steps:

1. State hypotheses:
   • $H_0$: Null hypothesis (e.g., "New algorithm has no effect").
   • $H_1$: Alternative hypothesis (e.g., "New algorithm reduces latency").
2. Choose significance level ($\alpha$): Typically 0.05.
3. Compute test statistic (e.g., t-test for means).
4. Compare with critical value or p-value.

Example: Testing a New Traffic Light Timing in Kathmandu

• Null Hypothesis ($H_0$): New timing does not reduce average wait time.
• Alternative ($H_1$): New timing reduces wait time.
• Simulate both old and new timings (30 runs each).
• Compute t-statistic for paired samples: $$t=\frac{{\bar{X}}_{\text{new}}-{\bar{X}}_{\text{old}}}{S_{\text{diff}}/\sqrt{n}}$$ If $|t|>t_{0.025,29}$, reject $H_0$.

---

4. Steady-State Detection

Many simulations (e.g., queueing systems) start with transient behavior before reaching equilibrium. We must discard initial data.

Methods:

1. Graphical Method: Plot output over time and identify where it stabilizes.

   graph LR
     A["Time Steps"] --> B["Output Value"]
     B --> C["Transient Phase"]
     C --> D["Steady-State"]

   Example: Simulating NTC’s call center queue length over time:

   • Discard data before t = 50 (where the curve flattens).

2. Statistical Tests:

   • Batch Means: Split data into batches and check for consistency.
   • Moving Averages: Smooth the output to detect stabilization.

---

5. Replication vs. Repetition

Example:

• Replication: Simulating Daraz’s order fulfillment 50 times with different random customer arrivals.
• Repetition: Running the same seed 10 times (useless for analysis).

---

6. Common Pitfalls & Best Practices

✅ Do:

• Use sufficient replications ($n\ge 30$ for CIs).
• Check for steady-state before analysis.
• Validate assumptions (e.g., normality for t-tests).

❌ Avoid:

• Using raw means without CIs.
• Ignoring transient data.
• Over-relying on small sample sizes.

---

In the Real World

1. eSewa & Khalti (Digital Payments)

   • Idea Used: Confidence Intervals for Transaction Latency
   • How? Simulate payment processing times to estimate 95% CIs for delays. If the CI for "successful transaction time" exceeds 5 seconds, the system flags a bottleneck.

2. Pathao (Ride-Hailing)

   • Idea Used: Hypothesis Testing for Driver Assignment Algorithms
   • How? Test whether a new driver-matching algorithm reduces average wait times. Run simulations with old vs. new algorithms and perform a t-test on the results.

3. NTC (Telecom Network)

   • Idea Used: Steady-State Detection for Call Volume
   • How? Simulate call arrivals over 24 hours. Discard the first 2 hours (transient phase) and analyze steady-state call volumes to optimize staffing.

---

Worked Example: Analyzing a Bank Loan Processing System

Scenario: A bank simulates loan approval times. They run 20 replications and record average processing times (in days):

Steps:

1. Compute $\bar{X}=7.5$ days, $S=0.6$ days.
2. For 90% CI, $t_{0.05,19}\approx 1.729$.
3. ME = $1.729\times \frac{0.6}{\sqrt{20}}\approx 0.23$.
4. CI = 7.5 ± 0.23 → [7.27, 7.73] days.

Decision: The bank can advertise an average processing time of 7.5 days, with 90% confidence it lies between 7.27 and 7.73 days.

---

Exam Tip

1. Confidence Intervals:

   • Always state the formula and assumptions (e.g., normality).
   • Show calculations step-by-step (e.g., t-value lookup).

2. Hypothesis Testing:

   • Clearly define $H_0$ and $H_1$.
   • Use the correct test (t-test, chi-square, etc.) based on the question.

3. Steady-State:

   • Plot the output and justify why you discard initial data.
   • Mention batch means or moving averages if asked.

4. Real-World Tie-Ins:

   • Expect questions linking to queueing systems (Pathao, banks), financial models (NEPSE), or telecom (NTC).
   • Example: "A Daraz warehouse simulates order fulfillment. How would you analyze the average time per order?" → Confidence intervals + steady-state check.

---

Bell curve with CI shaded (Image: Rohanukhade, CC0, via Wikimedia Commons)