Linear & Logistic Regression: Models, Math & Applications

TAKEAWAYS:

• Linear regression minimizes Mean Squared Error (MSE) to fit a straight-line model $y=w_0+w_{1}x$ to data, while logistic regression uses the sigmoid function $\sigma (z)=\frac{1}{1+e^{-z}}$ for probability outputs.
• Gradient Descent iteratively updates weights $w$ via $w:=w-\alpha \frac{\partial J}{\partial w}$, where $\alpha$ is the learning rate and $J$ is the loss function.
• Logistic regression outputs probabilities (e.g., 0.8 for "spam") and uses thresholding (e.g., 0.5) to classify.
• Overfitting occurs when the model fits noise; regularization (L1/L2) or cross-validation helps mitigate it.
• Real-world uses: Khalti (fraud detection via logistic regression), Ncell (customer churn prediction), and Daraz (demand forecasting with linear regression).
• Exam focus: Derive loss functions, explain gradient descent steps, and compare linear vs. logistic regression.

---

1. Linear Regression: Predicting Continuous Outcomes

Linear regression models the relationship between a dependent variable $y$ (e.g., house price) and independent variables $x$ (e.g., area) as: $$y=w_0+w_{1}x+ϵ$$ where:

• $w_0$: bias (intercept),
• $w_1$: weight (slope),
• $ϵ$: error term (noise).

How It Works: Least Squares Method

The goal is to minimize the sum of squared errors (SSE) between predicted ($\hat{y}$) and actual ($y$) values: $$J(w_0,w_1)=\frac{1}{2m}\sum _{i=1}^m({\hat{y}}_i-y_i{)}^2$$ where $m$ = number of training examples.

Gradient Descent Update Rules: $$w_0:=w_0-\alpha \frac{\partial J}{\partial w_0},w_1:=w_1-\alpha \frac{\partial J}{\partial w_1}$$ where: $$\frac{\partial J}{\partial w_0}=\frac{1}{m}\sum ({\hat{y}}_i-y_i),\frac{\partial J}{\partial w_1}=\frac{1}{m}\sum ({\hat{y}}_i-y_i)x_i$$

Worked Example: Predicting Daraz Order Delays

Suppose Daraz tracks order delays ($y$) based on distance ($x$) from the warehouse. Given data:

Step 1: Initialize $w_0=0$, $w_1=0$, learning rate $\alpha =0.01$. Step 2: Compute predictions $\hat{y}=w_0+w_{1}x$. Step 3: Update weights for 100 iterations (pseudocode):

for epoch in range(100):
    for (x, y) in data:
        prediction = w0 + w1 * x
        error = prediction - y
        w0 -= alpha * error
        w1 -= alpha * error * x

Final weights: $w_0=0.5$, $w_1=0.3$. Prediction: For $x=8$ km, $\hat{y}=0.5+0.3\times 8=2.9$ hours.

Assumptions and Limitations

---

2. Logistic Regression: Binary Classification

Logistic regression predicts probabilities for binary outcomes (e.g., "spam" or "not spam") using the sigmoid function: $$\sigma (z)=\frac{1}{1+e^{-z}},z=w_0+w_{1}x$$ where $\sigma (z)\in (0,1)$.

Loss Function: Log Loss (Cross-Entropy)

The loss for a single example is: $$J(w)=-\left[y\log (\sigma (z))+(1-y)\log (1-\sigma (z))\right]$$ Gradient Descent Updates: $$\frac{\partial J}{\partial w_0}=\sigma (z)-y,\frac{\partial J}{\partial w_1}=(\sigma (z)-y)x$$

Worked Example: Khalti Fraud Detection

Khalti flags transactions as fraudulent ($y=1$) or legitimate ($y=0$) based on transaction amount ($x$). Given:

Step 1: Initialize $w_0=0$, $w_1=0$, $\alpha =0.1$. Step 2: Predict probability $\hat{y}=\sigma (w_0+w_{1}x)$. Step 3: Update weights (after 50 iterations):

for epoch in range(50):
    for (x, y) in data:
        z = w0 + w1 * x
        prediction = 1 / (1 + exp(-z))
        error = prediction - y
        w0 -= alpha * error
        w1 -= alpha * error * x

Final weights: $w_0=-2.0$, $w_1=0.003$. Prediction: For $x=1500$, $z=-2.0+0.003\times 1500=2.5$. $\sigma (2.5)=0.925$ → Classify as fraud (threshold = 0.5).

Decision Boundary and Thresholding

• The decision boundary is where $\sigma (z)=0.5$, i.e., $w_0+w_{1}x=0$.
• Threshold tuning: Adjust the threshold (e.g., 0.3 for imbalanced data) to prioritize precision/recall.

---

3. Comparing Linear and Logistic Regression

---

4. Real-World Applications

In Nepal

1. Khalti (Fraud Detection)

   • Idea: Logistic regression classifies transactions as fraudulent or legitimate.
   • How: Uses features like transaction amount, time, and user history to compute $P(\text{fraud}|x)$. Transactions with $P>0.7$ are flagged.
   • Impact: Reduces false positives by 30% compared to rule-based systems.

2. Ncell (Customer Churn Prediction)

   • Idea: Logistic regression predicts whether a customer will leave (churn) based on call duration, data usage, and complaints.
   • How: Models $P(\text{churn}=1|\text{features})$. Customers with $P>0.6$ are targeted for retention offers.
   • Example: A customer with 500 minutes/month, 2GB data, and 3 complaints has $P(\text{churn})=0.85$.

3. Daraz (Demand Forecasting)

   • Idea: Linear regression predicts product demand based on seasonality, promotions, and historical sales.
   • How: Fits $\text{demand}=w_0+w_1\times \text{season}+w_2\times \text{promotion}$. Helps optimize inventory.
   • Example: During Dashain, Daraz predicts a 40% increase in diya sales using linear regression on past 5 years of data.

Globally

4. Google (Ad Click Prediction)

   • Idea: Logistic regression predicts whether a user will click an ad based on features like location, time, and ad relevance.
   • How: Computes $P(\text{click}|\text{bid},\text{user history})$. Ads with $P>0.4$ are shown.

5. Netflix (Movie Recommendation)

   • Idea: Linear regression (with regularization) predicts user ratings for movies based on genre preferences and past ratings.
   • How: Models $\text{rating}=w_0+w_1\times \text{user bias}+w_2\times \text{genre bias}$.

---

5. Overfitting and Regularization

Problem: The model fits training data too closely, performing poorly on unseen data. Solutions:

1. L1 Regularization (Lasso): Adds $\lambda \sum |w_i|$ to the loss function; can shrink some weights to zero (feature selection).
2. L2 Regularization (Ridge): Adds $\lambda \sum w_i^2$; shrinks weights but keeps all features.
3. Cross-Validation: Split data into training/validation sets to tune hyperparameters (e.g., $\lambda$, learning rate).

Example: Regularized Linear Regression for NEPSE Stock Prediction Suppose we predict NEPSE index movement ($y$) using past 5 days' movements ($x_1$ to $x_5$) and a news sentiment score ($x_6$).

• Without regularization: Overfits to noise in $x_4$ (e.g., a single outlier day).
• With L2 ($\lambda =0.1$): Weights for $x_4$ shrink to near-zero, improving generalization.

---

6. Extensions: Multivariate and Polynomial Regression

Multivariate Linear Regression

Extends to multiple features: $$y=w_0+w_1{x}_1+w_2{x}_2+\cdots +w_n{x}_n$$ Example: Predicting Kathmandu traffic time ($y$) based on:

• $x_1$: Time of day (hours),
• $x_2$: Rainfall (mm),
• $x_3$: Number of accidents yesterday.

Polynomial Regression

Adds non-linear terms to capture curves: $$y=w_0+w_{1}x+w_2{x}^2+w_3{x}^3$$ Example: Modeling Ncell data usage growth over time (quadratic term captures acceleration).

---

Figure 1: Machine Learning Pipeline (Unit 3 fits between preprocessing and evaluation).

---

Figure 2: Gradient Descent Workflow for Linear/Logistic Regression.

---

7. Exam Tip: What to Expect

1. Derivations: Be ready to derive the gradient descent update for both linear and logistic regression from scratch. For example:

   • For linear regression, show how $\frac{\partial J}{\partial w_1}=\frac{1}{m}\sum ({\hat{y}}_i-y_i)x_i$.
   • For logistic regression, recall that $\frac{\partial \sigma (z)}{\partial z}=\sigma (z)(1-\sigma (z))$.

2. Interpretation: Explain the meaning of coefficients in context. For example:

   • In a linear regression for Daraz sales, $w_1=0.5$ means "each additional km from the warehouse increases delivery time by 0.5 hours."

3. Comparison Questions: Expect questions like:

   • "When would you use linear regression over logistic regression?" Answer: Use linear regression for continuous outputs (e.g., predicting temperature) and logistic for binary outcomes (e.g., spam detection).

4. Practical Scenarios: Problems may involve:

   • Threshold tuning: Given a logistic regression model for Khalti, how would you adjust the threshold to reduce false positives? Answer: Increase the threshold (e.g., from 0.5 to 0.7) to require higher confidence before flagging a transaction as fraudulent.
   • Feature scaling: Why is it important in gradient descent? Answer: Features on larger scales (e.g., house area in m² vs. price in ₹) can dominate updates. Normalize to $[0,1]$ or standardize to zero mean.

5. Code Snippets: Be familiar with Python-like pseudocode for:

   • Computing predictions,
   • Updating weights,
   • Evaluating loss.

---

8. Common Pitfalls

• Ignoring feature scaling: Gradient descent converges slowly if features have vastly different scales.
• Misinterpreting logistic regression outputs: $\sigma (z)=0.3$ means 30% probability, not "30% chance of error."
• Overfitting: Always validate on a hold-out set or use regularization.
• Assuming linearity: If the data shows a curve, try polynomial features or switch to decision trees.

---

9. Summary Checklist

Before the exam, ensure you can:

1. Write the equations for linear and logistic regression.
2. Derive the gradient descent updates for both.
3. Explain the sigmoid function and its role in classification.
4. Describe how regularization (L1/L2) helps prevent overfitting.
5. Give two real-world examples (one from Nepal, one global) for each model.
6. Interpret coefficients in context (e.g., "a ₹1000 increase in transaction amount increases fraud probability by X%").