Model Evaluation & Optimization: Metrics, Bias-Variance, Cross-Validation, Hyperparameter Tuning

TAKEAWAYS:

• Learn key metrics (accuracy, precision, recall, F1, RMSE) to compare models objectively, with confusion matrices as the foundation.
• Understand the bias-variance trade-off: high bias (underfitting) vs. high variance (overfitting), and how model complexity shifts this balance.
• Master model validation techniques: train/validation/test splits, k-fold cross-validation, and why random shuffling matters.
• Optimize models using hyperparameter tuning (grid/random search) and regularization (L1/L2) to prevent overfitting.
• Recognize real-world pitfalls: data leakage, class imbalance, and how to mitigate them with stratified sampling and resampling.
• Apply practical optimization: early stopping, learning rate scheduling, and ensemble methods (bagging/boosting) for better generalization.

1. Why Evaluate Models? The Core Challenge

Machine learning models are only as good as their predictive performance. Without evaluation, you cannot:

• Compare two models (e.g., SVM vs. Random Forest).
• Detect overfitting (a model that works on training data but fails on new data).
• Optimize hyperparameters (e.g., learning rate, tree depth).

Real-world analogy: Imagine a doctor diagnosing diseases. If the doctor’s "model" (diagnosis rules) works perfectly on past patients but fails on new ones, the rules are overfitting—just like a model memorizing training data instead of learning patterns.

2. Key Evaluation Metrics

Metrics quantify how well a model performs. We focus on classification and regression separately.

A. Classification Metrics

For binary classification (e.g., spam detection), the confusion matrix is the foundation:

figure: confusion_matrix_binary

Derived metrics:

• Accuracy: (TP + TN) / (TP + TN + FP + FN)
  • Problem: Misleading if classes are imbalanced (e.g., 95% spam vs. 5% ham).
• Precision: TP / (TP + FP)
  • Answer: Of all emails flagged as spam, how many were actually spam?
  • High precision = fewer false alarms (e.g., fewer "spam" emails in your inbox).
• Recall (Sensitivity): TP / (TP + FN)
  • Answer: Of all actual spam emails, how many did we catch?
  • High recall = fewer missed spam emails.
• F1-Score: Harmonic mean of precision and recall.
  • Use when precision and recall are equally important.
  • Formula: 2 * (Precision * Recall) / (Precision + Recall)

Worked Example: E-Sewa Fraud Detection E-Sewa uses a classifier to flag suspicious transactions. Suppose:

• Actual Fraud (Positive): 100 transactions
• Actual Legit (Negative): 9,900 transactions
• Model predicts:
  • TP = 80, FP = 20, FN = 20, TN = 9,880

Calculate:

• Accuracy = (80 + 9880) / (100 + 9900) = 99.6% (misleading!)
• Precision = 80 / (80 + 20) = 80% (80% of flagged transactions are fraudulent)
• Recall = 80 / (80 + 20) = 80% (80% of frauds are caught)
• F1 = 2 * (0.8 * 0.8) / (0.8 + 0.8) = 0.8 (balanced metric)

Why F1? E-Sewa prioritizes catching fraud (high recall) but also avoids false alarms (high precision). F1 balances both.

B. Regression Metrics

For predicting continuous values (e.g., house prices), we use:

• Mean Absolute Error (MAE): Average absolute difference between predicted and actual.
  • Formula: MAE = (1/n) * Σ|y_pred - y_actual|
  • Interpretable: "On average, predictions are off by X units."
• Root Mean Squared Error (RMSE): Penalizes large errors more heavily.
  • Formula: RMSE = sqrt((1/n) * Σ(y_pred - y_actual)^2)
  • Use when large errors are costly (e.g., predicting stock prices).
• R² (R-squared): Proportion of variance explained by the model.
  • Formula: 1 - (SS_res / SS_tot)
  • Ranges from 0 (worst) to 1 (perfect fit).

Worked Example: Daraz Price Prediction Daraz uses regression to predict product prices. Suppose:

• Actual prices (Y): [100, 200, 300]
• Predicted prices (Y_pred): [90, 210, 290]

Calculate RMSE:

RMSE = sqrt(((90-100)^2 + (210-200)^2 + (290-300)^2) / 3)
     = sqrt((100 + 100 + 100) / 3)
     = sqrt(100) = 10

Interpretation: Predictions are off by NPR 10 on average.

3. The Bias-Variance Trade-Off

Definition:

• Bias: Error due to overly simplistic assumptions (underfitting).
  • High bias = model fails to capture patterns.
• Variance: Error due to excessive sensitivity to training data (overfitting).
  • High variance = model memorizes noise.

Visualization:

figure: bias_variance_tradeoff

High Bias (Underfitting) --------------------> Low Bias
|                          |                     |
|                          |                     |
|                          |                     |
|                          |                     |
|                          |                     |
|                          |                     |
|                          |                     |
|                          |                     |
Low Variance (Overfitting) <------------------ High Variance

• Low bias + low variance: Ideal model (generalizes well).
• High bias: Model is too simple (e.g., linear regression for nonlinear data).
• High variance: Model fits training data too closely (e.g., deep neural network with no regularization).

Real-world Example: Pathao Ride Price Prediction

• High Bias Model: Uses only distance to predict fare (ignores time, traffic).
  • Underestimates prices in peak hours → high bias.
• High Variance Model: Uses a 1000-feature polynomial regression with all historical ride data.
  • Overfits to specific routes → high variance (fails on new routes).

How to Balance?

• Increase model complexity (e.g., switch from linear to polynomial regression) to reduce bias.
• Add regularization (L1/L2) or more training data to reduce variance.

4. Model Validation Techniques

Evaluating a model on the same training data leads to optimistic performance. Instead, we use:

A. Train-Validation-Test Split

• Training set: Train the model (70%).
• Validation set: Tune hyperparameters (15%).
• Test set: Final evaluation (15%).

Why shuffle? Without shuffling, if data is ordered by time/feature, the model may leak future information into training.

Example Split for Ncell Call Duration Prediction:

figure: train_val_test_split

[Training Data] ----> Model Training
[Validation Data] ----> Hyperparameter Tuning (e.g., tree depth)
[Test Data] -------> Final Accuracy Check

B. k-Fold Cross-Validation

• Split data into k folds (e.g., k=5).
• Train on k-1 folds, validate on the remaining fold.
• Repeat k times, average performance.

Advantage: Uses all data for both training and validation, reducing variance in estimates.

Mermaid Diagram:

flowchart TD
    A["Dataset"] --> B["Split into 5 folds"]
    B --> C["Fold 1: Train on 4, Validate on 1"]
    B --> D["Fold 2: Train on 4, Validate on 2"]
    B --> E["... Repeat 5 times"]
    E --> F["Average validation accuracy"]

Worked Example: NEPSE Stock Price Prediction Suppose we have 20 days of stock data. With 5-fold CV:

1. Fold 1: Train on days 1-4,11-20; validate on day 5.
2. Fold 2: Train on days 1-3,5-20; validate on day 4. ...
3. Average RMSE across all folds.

5. Hyperparameter Tuning

Hyperparameters are not learned from data (e.g., learning rate, tree depth). We tune them using:

• Grid Search: Exhaustively test all combinations.
• Random Search: Sample random combinations (often more efficient).
• Bayesian Optimization: Uses probabilistic models to find optimal hyperparameters faster.

Example: Tuning a Random Forest for Khalti Transaction Fraud

Grid Search Process:

1. Train model with n_estimators=50, max_depth=None.
2. Train with n_estimators=50, max_depth=5. ...
3. Compare validation accuracies and pick the best combo.

Mermaid Diagram:

flowchart TD
    A["Hyperparameter Grid"] --> B["Train Model (n_estimators=50, max_depth=5)"]
    B --> C["Evaluate on Validation Set"]
    C --> D{"Best Combo?"}
    D -->|"Yes"| E["Save Best Combo"]
    D -->|"No"| B
    E --> F["Compare All Combinations"]
    F --> G["Select Best Hyperparameters"]

6. Regularization to Prevent Overfitting

Regularization adds a penalty term to the loss function to discourage large weights:

• L1 (Lasso): Encourages sparsity (some weights = 0).
  • Useful for feature selection (e.g., identifying key factors in Daraz customer churn).
• L2 (Ridge): Shrinks weights but rarely sets them to zero.
  • Common in linear regression for NEPSE stock prediction.

Loss Function with L2 Regularization:

Loss = MSE + λ * Σ(w_i^2)

where λ is the regularization strength.

Worked Example: Bank Loan Default Prediction Suppose a model predicts loan defaults with:

• Training accuracy: 99%
• Validation accuracy: 70%

Solution: Add L2 regularization to reduce variance.

• New loss: MSE + λ * Σ(w_i^2)
• After tuning λ, validation accuracy improves to 85%.

7. Handling Class Imbalance

In real-world data, classes are often imbalanced:

• Example: Fraud detection (99% legit, 1% fraud).
• Problem: Accuracy is misleading (a dumb model predicting "legit" always gets 99% accuracy).

Solutions:

1. Resampling:
   • Oversample minority class (SMOTE).
   • Undersample majority class.
2. Class Weighting: Assign higher weights to minority class in loss function.
3. Metrics: Use precision-recall curves or F1-score instead of accuracy.

Example: NTC Call Drop Detection

• Actual drops: 1% of calls.
• Model predicts drops for 5% of calls (2% true positives, 3% false positives).
• Accuracy: 98% (misleading!).
• Precision: 2 / (2 + 3) = 0.4 (only 40% of flagged drops are real).
• Recall: 2 / (2 + 1) = 0.67 (67% of actual drops are caught).

Solution: Use SMOTE to balance classes or F1-score for evaluation.

8. Early Stopping

For iterative models (e.g., neural networks), early stopping halts training when validation performance stops improving.

Process:

1. Train model in epochs.
2. After each epoch, check validation loss.
3. Stop if loss doesn’t improve for patience epochs.

Mermaid Diagram:

Example: YouTube Recommendation System

• Train a neural network on user watch history.
• Validation loss decreases until epoch 10, then plateaus.
• Early stop at epoch 10 to save computation time.

In the Real World

1. Khalti: Fraud Detection with Precision-Recall Trade-offs

   • Idea: Uses F1-score and confusion matrices to balance catching fraud (recall) and avoiding false alarms (precision).
   • Real impact: Reduces chargebacks by 30% by tuning thresholds on the precision-recall curve.

2. Daraz: Price Prediction with RMSE Optimization

   • Idea: Uses RMSE to minimize prediction errors in dynamic pricing.
   • Real impact: Adjusts prices in real-time to maximize sales (e.g., reducing RMSE by 10% increases revenue by 5%).

3. Ncell: Call Quality Monitoring with k-Fold CV

   • Idea: Uses 5-fold cross-validation to ensure models generalize across different network conditions.
   • Real impact: Detects call drop patterns before they affect users, improving customer satisfaction.

4. NEPSE: Stock Price Forecasting with Bias-Variance Trade-off

   • Idea: Balances model complexity (e.g., LSTM vs. linear regression) to avoid underfitting (high bias) or overfitting (high variance).
   • Real impact: Reduces prediction errors by 20% compared to naive models.

Exam Tips

1. Always show a confusion matrix for classification problems. Label TP, FP, FN, TN and derive precision/recall/F1.

   • Example: If given a dataset, construct the matrix and calculate metrics step-by-step.

2. For bias-variance trade-off, draw the U-shaped curve and explain:

   • High bias: Underfitting (e.g., linear model for nonlinear data).
   • High variance: Overfitting (e.g., deep tree with no pruning).
   • Solution: Regularization or more data.

3. Train/validation/test split is non-negotiable. Mention:

   • Why shuffle data (to avoid temporal bias).
   • Why use validation set (to tune hyperparameters).

4. Hyperparameter tuning:

   • Grid search vs. random search: When to use each.
   • Example: Tuning learning_rate in neural networks or max_depth in decision trees.

5. Regularization:

   • L1 vs. L2: When to use each (L1 for feature selection, L2 for smooth weights).
   • Show the loss function with the penalty term.

6. Class imbalance:

   • Never report accuracy if classes are imbalanced.
   • Always suggest SMOTE, class weights, or precision-recall curves.

7. Early stopping:

   • Explain how it works (monitor validation loss).
   • Mention when to use it (iterative models like neural networks).

8. Worked examples:

   • For metrics, always plug in numbers from the confusion matrix.
   • For bias-variance, draw the curve and label regions.
   • For hyperparameter tuning, show the grid and pick the best combo.

Final Reminder:

• Metrics are your compass. Know when to use accuracy vs. F1 vs. RMSE.
• Validation is your shield. Always split data and use cross-validation.
• Optimization is your sword. Tune hyperparameters and regularize wisely.