Supervised Learning: Regression & Classification

TAKEAWAYS:

• Supervised learning predicts outputs from labeled data: regression for numbers (e.g., house prices), classification for categories (e.g., spam/ham).
• Linear regression fits a straight line to minimize error using least squares or gradient descent; logistic regression uses sigmoid functions for binary classification.
• Evaluation metrics like accuracy, precision, recall, and the confusion matrix quantify model performance on unseen data.
• The bias-variance trade-off balances underfitting (high bias) and overfitting (high variance) by tuning model complexity.
• Decision boundaries (e.g., SVM’s hyperplane) separate classes in feature space; kernel tricks extend this to non-linear data.
• Real-world tools like Daraz’s price prediction (regression) and eSewa’s fraud detection (classification) use these techniques daily.

1. Supervised Learning: Core Concepts

Supervised learning trains models on labeled datasets to predict outputs. It splits into two tasks:

• Regression: Predicts continuous values (e.g., salary, temperature).
• Classification: Predicts discrete labels (e.g., cat/dog, spam/not spam).

Key Difference

flowchart TD
    A["Supervised Learning"] --> B["Regression<br/>(Continuous Output)"]
    A --> C["Classification<br/>(Discrete Output)"]
    B --> D["Example: House Price Prediction"]
    C --> E["Example: Email Spam Detection"]

2. Regression: Predicting Continuous Values

Linear Regression

Fits a straight line $y=mx+c$ to minimize error between predicted and actual values.

Formula: $$y={\beta }_0+{\beta }_{1}x+ϵ$$ where:

• ${\beta }_0$: intercept,
• ${\beta }_1$: slope,
• $ϵ$: error term.

Worked Example: Daraz Price Prediction Daraz uses regression to estimate product prices based on features like brand, ratings, and discounts. Given data:

Step 1: Calculate means $$\bar{X}=\frac{4.5+3.8+4.2}{3}=4.17,\bar{Y}=\frac{1200+950+1100}{3}=1083.33$$

Step 2: Compute slope (${\beta }_1$) and intercept (${\beta }_0$) $${\beta }_1=\frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sum (X_i-\bar{X}{)}^2}=\frac{116.67}{0.33}=353.53$$ $${\beta }_0=\bar{Y}-{\beta }_1\bar{X}=1083.33-353.53\times 4.17=100.00$$ Best-fit line: $$y=353.53x+100$$ Prediction for $X=4.5$: $$y=353.53\times 4.5+100=1640.89\text{ Rs.}$$

Visualization:

Least Squares Method

Minimizes the sum of squared errors (SSE): $$SSE=\sum (y_i-{\hat{y}}_i{)}^2$$ Gradient Descent for Linear Regression Updates weights iteratively: $${\beta }_1:={\beta }_1-\alpha \frac{\partial SSE}{\partial {\beta }_1},{\beta }_0:={\beta }_0-\alpha \frac{\partial SSE}{\partial {\beta }_0}$$ where $\alpha$ is the learning rate.

3. Classification: Categorizing Data

Logistic Regression

Predicts probabilities for binary classification using the sigmoid function: $$\sigma (z)=\frac{1}{1+e^{-z}},z={\beta }_0+{\beta }_{1}x$$ Output: Probability $P(y=1)=\sigma (z)$. Threshold at 0.5 to classify.

Example: eSewa Fraud Detection eSewa uses logistic regression to flag suspicious transactions. Given features:

Worked Trace:

1. Train model on historical data to learn ${\beta }_0,{\beta }_1$.
2. For a new transaction $X=1500$: $$z={\beta }_0+{\beta }_1\times 1500,P(\text{Fraud})=\sigma (z)$$
3. If $P>0.5$, flag as fraud.

Decision Boundary:

4. Evaluation Metrics

Confusion Matrix

For binary classification:

          | Predicted: Yes | Predicted: No
----------|----------------|--------------
Actual: Yes | True Positive (TP) | False Negative (FN)
Actual: No  | False Positive (FP) | True Negative (TN)

Metrics:

• Accuracy: $\frac{TP+TN}{TP+TN+FP+FN}$
• Precision: $\frac{TP}{TP+FP}$ (How many predicted positives are correct?)
• Recall (Sensitivity): $\frac{TP}{TP+FN}$ (How many actual positives are caught?)
• F1-Score: Harmonic mean of precision and recall.

Example: Given:

Calculations:

• Accuracy: $\frac{80+90}{200}=0.85$ (85%)
• Precision: $\frac{80}{90}=0.89$ (89%)
• Recall: $\frac{80}{100}=0.80$ (80%)

5. Bias-Variance Trade-off

• High Bias (Underfitting): Model is too simple (e.g., linear regression for non-linear data).
• High Variance (Overfitting): Model fits noise (e.g., polynomial regression with degree 20).

Visualization:

Worked Example: Loan Approval A bank uses a linear model to predict loan defaults.

• Low complexity: Ignores risk factors → high bias (rejects many valid loans).
• High complexity: Captures noise → high variance (approves risky loans).

6. Advanced Classification: Decision Trees

Splits data based on feature thresholds to classify. Example: Pathao Ride Classification Features: distance, time, weather.

• If distance > 5 km → "Long Ride"
• Else if weather = "rainy" → "Short Rainy Ride"
• Else → "Short Normal Ride"

Visualization:

flowchart TD
    A["Distance > 5 km?"] -->|"Yes"| B["Long Ride"]
    A -->|"No"| C["Weather Rainy?"]
    C -->|"Yes"| D["Short Rainy Ride"]
    C -->|"No"| E["Short Normal Ride"]
    A -->|"Else"| F["Short Normal Ride"]

In the Real World

1. Daraz (Regression)

   • Idea: Linear regression predicts product prices based on seller ratings, discounts, and brand.
   • How: Uses historical sales data to adjust prices dynamically, improving profit margins.
   • Worked Example: If a seller’s rating drops from 4.5 to 3.8, Daraz’s model predicts a 15% price increase (as seen in the Daraz Price Prediction example above).

2. eSewa (Classification)

   • Idea: Logistic regression classifies transactions as fraudulent or legitimate.
   • How: Analyzes transaction amount, time, and user history to flag anomalies (e.g., sudden large transfers).
   • Worked Example: A transaction of Rs. 2000 at 3 AM with a new device triggers a 0.85 probability of fraud, prompting a manual review.

3. Ncell (Customer Churn Prediction)

   • Idea: Decision trees predict whether a subscriber will cancel their plan.
   • How: Uses features like call duration, data usage, and customer service interactions to split data into "churn" or "retain" nodes.
   • Worked Example: Subscribers with <100 call minutes/month and no data usage are 80% likely to churn, prompting targeted retention offers.

Exam Tip

• Focus on:
  • Deriving the linear regression equation from scratch (means, slope, intercept).
  • Confusion matrix metrics (precision, recall, F1-score) with numerical examples.
  • Bias-variance trade-off with real-world analogies (e.g., "a simple rule vs. a complex rule").
  • Decision trees: how splits are made and visualized.
• Common Pitfalls:
  • Mixing up precision and recall (precision = correct positives / predicted positives; recall = correct positives / actual positives).
  • Forgetting to normalize data before logistic regression or gradient descent.
  • Overcomplicating the bias-variance trade-off—stick to high/low bias and variance examples.
• Time Management:
  • Spend 30% of time on definitions (e.g., "What is logistic regression?").
  • Allocate 40% to worked examples (e.g., derive the regression line or build a confusion matrix).
  • Save 30% for comparisons (e.g., "Why is SVM better than logistic regression for non-linear data?").

Visual Recap:

mindmap
  root((Supervised Learning))
    Regression
      Linear Regression
        Least Squares
        Gradient Descent
      Example: Daraz Price Prediction
    Classification
      Logistic Regression
        Sigmoid Function
        Decision Boundary
      Decision Trees
        Splits & Nodes
        Example: Pathao Ride Classification
    Evaluation
      Confusion Matrix
        Precision, Recall, F1-Score
      Bias-Variance Trade-off
        Underfitting vs. Overfitting
    Real-World
      eSewa Fraud Detection
      Ncell Customer Churn