Hypothesis Testing: Types, Steps, and Statistical Tools

TAKEAWAYS:

• Hypothesis testing is a structured method to make data-driven decisions by comparing a null hypothesis ($H_0$) against an alternative ($H_1$).
• Key steps include formulating hypotheses, selecting a significance level (α), choosing a test statistic, calculating p-values, and making a decision.
• Types of errors (Type I and Type II) and power of a test determine the reliability of conclusions.
• Businesses use hypothesis testing for A/B testing (e.g., Daraz’s ad campaigns), customer satisfaction surveys (e.g., Ncell’s Net Promoter Score), and financial forecasting (e.g., Nabil Bank’s loan approval models).
• Parametric tests (e.g., t-tests, ANOVA) assume normal distribution, while non-parametric tests (e.g., Chi-square, Mann-Whitney U) do not.
• The p-value and critical value approach are two methods to reject or fail to reject $H_0$, with α (e.g., 0.05) defining the threshold for significance.

1. What is Hypothesis Testing?

Hypothesis testing is a statistical method used to make inferences about a population based on sample data. It helps researchers or businesses test assumptions (hypotheses) to determine if observed effects are statistically significant or due to random chance.

Key Definitions

• Null Hypothesis ($H_0$): The default assumption (e.g., "There is no effect" or "The new marketing strategy does not increase sales").
• Alternative Hypothesis ($H_1$ or $H_a$): The claim we want to test (e.g., "The new strategy increases sales").
• Significance Level (α): The probability of rejecting $H_0$ when it is true (e.g., α = 0.05 means a 5% risk of a false positive).
• Test Statistic: A standardized value (e.g., z-score, t-score) calculated from sample data to compare against a critical value.
• p-value: The probability of observing the test statistic (or more extreme) if $H_0$ is true. A low p-value (≤ α) suggests rejecting $H_0$.

2. Steps in Hypothesis Testing

flowchart TD
    A["1. State Hypotheses"] --> B["2. Choose Significance Level (α)"]
    B --> C["3. Select Test Statistic"]
    C --> D["4. Collect Data & Calculate Test Statistic"]
    D --> E["5. Determine Critical Value or p-value"]
    E --> F["6. Make Decision: Reject or Fail to Reject \(H_0\)"]
    F --> G["7. Draw Conclusion"]

Step-by-Step Explanation

1. Formulate Hypotheses

   • Example: A bank (e.g., Nabil Bank) wants to test if a new loan approval process reduces processing time.
     • $H_0$: The new process does not reduce time (μ ≥ 10 days).
     • $H_1$: The new process reduces time (μ < 10 days).

2. Choose α (Significance Level)

   • Common values: 0.01 (strict), 0.05 (standard), or 0.10 (less strict).
   • Trade-off: Lower α reduces false positives but increases false negatives.

3. Select Test Statistic

   • Depends on data type and assumptions:
     • Parametric tests (normal distribution assumed):
       • t-test (small samples, population variance unknown).
       • z-test (large samples, known population variance).
       • ANOVA (compare means across >2 groups).
     • Non-parametric tests (no distribution assumption):
       • Chi-square test (categorical data, e.g., customer preferences).
       • Mann-Whitney U test (ordinal data, e.g., survey rankings).

4. Calculate Test Statistic

   • Example: Daraz tests if a new checkout button increases conversion rates.
     • Sample: 1000 users with old button (3% conversion) vs. 1000 with new button (4% conversion).
     • Use a two-proportion z-test: $$z=\frac{(p_1-p_2)}{\sqrt{p(1-p)(\frac{1}{n_1}+\frac{1}{n_2})}}$$ where $p_1=0.04$, $p_2=0.03$, $p=\frac{0.04+0.03}{2}=0.035$.

5. Determine Critical Value or p-value

   • Critical Value Approach: Compare test statistic to a critical value from a table (e.g., z = ±1.96 for α = 0.05).
   • p-value Approach: Find the probability of the test statistic under $H_0$. If p ≤ α, reject $H_0$.

6. Make Decision

   • Reject $H_0$: Strong evidence supports $H_1$ (e.g., the new checkout button works).
   • Fail to Reject $H_0$: Insufficient evidence (e.g., no significant difference in sales).

7. Draw Conclusion

   • Example: If p = 0.02 < α = 0.05, conclude the new button significantly increases conversions.

3. Types of Hypothesis Tests

4. Errors in Hypothesis Testing

Two types of errors can occur:

• Power of a Test: Probability of correctly rejecting $H_0$ (1 − β). Higher power reduces Type II errors.
• How to Increase Power:
  • Increase sample size.
  • Use a higher α (but increases Type I error).
  • Reduce variability in data.

5. Real-World Applications in Nepal

Case 1: Daraz’s A/B Testing for Ad Campaigns

• Problem: Daraz wants to test if a new ad banner increases click-through rates (CTR).
• Hypotheses:
  • $H_0$: CTR with new banner = CTR with old banner (5%).
  • $H_1$: CTR with new banner > 5%.
• Method: Two-sample z-test on 5000 users per group.
• Result: New banner shows a CTR of 5.8% (p = 0.03 < 0.05). Conclusion: Roll out the new banner.

Case 2: Ncell’s Net Promoter Score (NPS) Analysis

• Problem: Ncell wants to check if a new customer service training improves NPS.
• Hypotheses:
  • $H_0$: NPS after training = NPS before training (50).
  • $H_1$: NPS after training > 50.
• Method: Paired t-test on 200 customers surveyed before/after training.
• Result: Mean NPS increases from 50 to 58 (p = 0.001). Conclusion: Training is effective.

Case 3: Nabil Bank’s Loan Default Prediction

• Problem: Predict if a borrower will default using credit history.
• Method: Logistic regression (a form of hypothesis testing for classification).
• Hypotheses:
  • $H_0$: Credit history has no predictive power.
  • $H_1$: Credit history predicts default risk.
• Result: p < 0.01 for credit score variable. Conclusion: Use credit scores to approve/reject loans.

6. Parametric vs. Non-Parametric Tests

Example:

• Parametric: Test if Khalti’s average transaction time (in seconds) differs from eSewa’s (use t-test).
• Non-parametric: Test if Pathao’s driver ratings (1-5 stars) differ by region (use Kruskal-Wallis).

7. Decision Rules: p-value vs. Critical Value

Visual Decision Rule:

mindmap
  root((Decision Rule))
    p-value
      "p ≤ α → Reject \(H_0\)"
      "p > α → Fail to reject \(H_0\)"
    Critical Value
      "Test Statistic > Critical Value → Reject \(H_0\)"
      "Test Statistic ≤ Critical Value → Fail to reject \(H_0\)"

8. Common Mistakes to Avoid

1. Ignoring Assumptions: Using a t-test when data is not normally distributed (→ use Mann-Whitney U).
2. Multiple Testing: Running 10 tests increases Type I error risk (use Bonferroni correction).
3. P-hacking: Manipulating data to achieve p < 0.05 (unethical and invalidates results).
4. Confusing Correlation and Causation: Hypothesis testing shows association, not necessarily cause (e.g., ice cream sales and drowning deaths both rise in summer, but neither causes the other).

9. Software Tools for Hypothesis Testing

Example in Python:

from scipy.stats import ttest_ind
group1 = [12, 15, 14, 10, 13]  # Old checkout time (Daraz)
group2 = [9, 11, 10, 8, 12]    # New checkout time
t_stat, p_val = ttest_ind(group2, group1)
print(f"p-value: {p_val}")  # Output: p-value = 0.04 → Reject H0

In the Real World

1. eSewa and Khalti: Fraud Detection

   • Idea Used: Chi-square goodness-of-fit test to detect unusual transaction patterns (e.g., sudden spikes in refunds).
   • How: Compare observed transaction frequencies to expected frequencies under normal conditions. If p < 0.05, flag as potential fraud.

2. Daraz: A/B Testing for UI Changes

   • Idea Used: Two-sample t-test to compare conversion rates between old and new website layouts.
   • How: Split users randomly into two groups. If the new layout shows a significantly higher conversion rate (p < 0.05), roll it out globally.

3. Ncell: Customer Churn Prediction

   • Idea Used: Logistic regression (a hypothesis-testing framework) to predict churn based on call drop rates, bill payments, and usage data.
   • How: Test if "high call drop rate" ($H_1$) predicts churn ($H_0$: no relationship). If p < 0.01, include the variable in the model.

4. Nabil Bank: Loan Approval Models

   • Idea Used: ANOVA to compare default rates across income groups.
   • How: Test if borrowers with income < Rs. 50,000 have higher default rates than those earning > Rs. 100,000. If p < 0.05, adjust loan terms by income bracket.

5. NTC: Network Performance Testing

   • Idea Used: One-sample t-test to check if average internet speed meets regulatory standards (e.g., 50 Mbps).
   • How: Sample 50 users’ speeds. If the mean is significantly below 50 Mbps (p < 0.05), investigate network issues.

Exam Tip

How This Unit is Examined

1. Theory Questions (30%):

   • Define null/alternative hypotheses, Type I/II errors, and p-value.
   • Explain the difference between parametric and non-parametric tests.
   • Common Pitfalls: Students often confuse $H_0$ and $H_1$ or misapply tests (e.g., using ANOVA for two groups instead of a t-test).

2. Problem-Solving (50%):

   • Given: A scenario (e.g., "A company tests two advertising methods").
   • Task: Formulate hypotheses, choose a test, calculate test statistic, and interpret results.
   • Example Question:

     "A sample of 30 students has an average exam score of 75 with a standard deviation of 10. Test if the true mean is 80 at α = 0.05." Solution: Use a one-sample t-test: $$>t=\frac{75-80}{10/\sqrt{30}}=-2.74>$$ Critical t-value (df = 29, α = 0.05) = ±2.045. Since |−2.74| > 2.045, reject $H_0$.

3. Case Studies (20%):

   • Given: A real-world scenario (e.g., "Pathao wants to test if a new incentive increases driver earnings").
   • Task: Design a hypothesis test, justify your choice of test, and explain implications.
   • Tip: Always link your answer to business decisions (e.g., "If p < 0.05, Pathao should scale the incentive").

Marks Distribution in Exams

Quick Revision Checklist

• Can you formulate $H_0$ and $H_1$ for a given scenario?
• Do you know when to use parametric vs. non-parametric tests?
• Can you calculate a t-test or z-test from scratch?
• Do you understand what a p-value means and how to interpret it?
• Can you explain the consequences of Type I/II errors in a business context?

Final Worked Example: Kathmandu Traffic Congestion Study

Scenario: The Kathmandu Metropolitan City wants to test if a new traffic signal system reduces average wait times at a busy intersection.

Step 1: Formulate Hypotheses

• $H_0$: The new system does not reduce wait time (μ ≥ 2 minutes).
• $H_1$: The new system reduces wait time (μ < 2 minutes).

Step 2: Choose Test

• One-sample t-test (since we compare a sample mean to a known value).
• Assumptions: Wait times are approximately normally distributed (checked via histogram).

Step 3: Collect Data

• Sample: 40 drivers timed at the intersection.
• Sample mean ($\bar{x}$) = 1.8 minutes.
• Sample standard deviation ($s$) = 0.5 minutes.
• α = 0.05.

Step 4: Calculate Test Statistic $$t=\frac{\bar{x}-{\mu }_0}{s/\sqrt{n}}=\frac{1.8-2}{0.5/\sqrt{40}}=\frac{-0.2}{0.079}=-2.53$$

Step 5: Determine Critical Value

• Degrees of freedom (df) = n − 1 = 39.
• Critical t-value (one-tailed, α = 0.05) ≈ −1.685 (from t-table).

Step 6: Make Decision

• Since |−2.53| > 1.685, reject $H_0$.

Conclusion: The new traffic signal system significantly reduces wait times (p < 0.05). The city should implement it citywide.