Prediction & Regression: Models, Algorithms & Applications

Core Concepts

1. Prediction vs. Regression: Key Differences

Prediction and regression are both supervised learning techniques, but they differ in their goals and outputs:

Why it matters:

• Prediction answers "What category does this belong to?"
• Regression answers "What numerical value does this correspond to?"

2. Linear Regression: The Workhorse of Prediction

Linear regression models the relationship between a dependent variable (Y) and one or more independent variables (X) using a straight-line equation: $$Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+\cdots +{\beta }_n{X}_n+ϵ$$ where:

• ${\beta }_0$ = intercept
• ${\beta }_1,{\beta }_2,\dots$ = coefficients
• $ϵ$ = error term

How It Works: Step-by-Step

1. Data Collection: Gather historical data (e.g., NEPSE stock prices vs. inflation rates).
2. Model Training: Use Ordinary Least Squares (OLS) to minimize the sum of squared errors (SSE).
3. Equation Derivation: Solve for $\beta$ coefficients using calculus or matrix algebra.
4. Prediction: Plug new $X$ values into the equation to estimate $Y$.

Worked Example: Predicting NEPSE Index

Suppose we model the NEPSE index ($Y$) based on monthly inflation rate ($X_1$) and global oil prices ($X_2$). Given data:

Step 1: Fit the model (using software or manual calculation): $$Y=1500+20X_1+5X_2$$ Step 2: Predict April’s NEPSE index if inflation = 5.0% and oil = $72: $$Y=1500+20(5.0)+5(72)=1500+100+350=1950$$ Visualization of Fit:

graph LR
    A["Actual NEPSE (1850, 1880, 1920)"] --> B["Linear Regression Line"]
    B --> C["Predicted: 1950 for April"]

3. Logistic Regression: For Binary Outcomes

While linear regression predicts continuous values, logistic regression predicts probabilities (0 to 1) for binary outcomes (e.g., loan approval: Yes/No).

Sigmoid Function

The output is squashed using the logistic function: $$P(Y=1)=\frac{1}{1+e^{-({\beta }_0+{\beta }_{1}X)}}$$ where $P(Y=1)$ = probability of the positive class (e.g., "approved").

Worked Example: Bank Loan Approval

A bank uses credit score ($X_1$) and income ($X_2$) to predict loan approval. Given:

• If $X_1=700$ and $X_2=\$50K$, the model outputs: $$P(\text{Approved})=\frac{1}{1+e^{-(-5+0.01\times 700+0.00005\times 50000)}}=0.89$$ Decision Rule: Approve if $P>0.5$.

Visualization of Sigmoid Curve:

graph TD
    A["Linear Combination: Z = β₀ + β₁X"] --> B["Sigmoid: P(Y=1) = 1/(1+e⁻ᶻ)"]
    B --> C["Output: 0 (No) to 1 (Yes)"]

4. Evaluation Metrics

Example: If a model predicts house prices with RMSE = $15,000, it’s off by $15K on average.

In the Real World

1. NEPSE Stock Predictions

   • Company: NEPSE (Nepal Stock Exchange)
   • Idea Used: Linear Regression
   • How: Analysts use historical data (inflation, GDP growth) to predict future index movements. For example, a model trained on 2018–2022 data might forecast a 5% rise in 2024 if inflation stays below 6%.

2. Daraz Delivery Time Estimation

   • Company: Daraz (Alibaba Group)
   • Idea Used: Polynomial Regression
   • How: Daraz estimates delivery delays based on:
     • Distance from warehouse ($X_1$)
     • Number of orders in queue ($X_2$)
     • Weather conditions ($X_3$)
   • Example: If $X_1=50\text{ km}$, $X_2=1000\text{ orders}$, and rain = "Yes," the model predicts a 48-hour delay.

3. Khalti Fraud Detection

   • Company: Khalti (Nepal’s fintech)
   • Idea Used: Logistic Regression
   • How: Khalti flags suspicious transactions by calculating: $$P(\text{Fraud})=\frac{1}{1+e^{-({\beta }_0+{\beta }_1\times \text{Amount}+{\beta }_2\times \text{Location Risk})}}$$
   • Example: A $500 transfer from Kathmandu to Pokhara at 3 AM might trigger $P(\text{Fraud})=0.92$.

Advanced Regression Techniques

1. Polynomial Regression

Extends linear regression by adding non-linear terms (e.g., $X^2$, $X^3$): $$Y={\beta }_0+{\beta }_{1}X+{\beta }_2{X}^2+ϵ$$ When to Use: When the relationship between $X$ and $Y$ is curved (e.g., Daraz delivery times vs. distance).

Example:

graph LR
    A["Distance (km)"] --> B["Delivery Time (hours)"]
    B --> C["Quadratic Fit: Y = 2 + 0.5X + 0.01X²"]

2. Regularization: Ridge vs. Lasso

Example: Predicting house prices with 50 features (e.g., room count, school proximity). Lasso might zero out irrelevant features like "number of mailboxes."

Exam Tip

1. Always compare prediction vs. regression in questions asking for "supervised learning techniques."
2. For linear regression:
   • Know the equation and how to interpret coefficients.
   • Practice manual calculations (even with small datasets).
   • Remember: R² = 1 means perfect fit; RMSE = 0 means no error.
3. For logistic regression:
   • Draw the sigmoid curve and explain how it maps linear output to probabilities.
   • Know the decision threshold (usually 0.5).
4. Real-world applications:
   • Link NEPSE to regression, Khalti to classification, and Daraz to polynomial regression.
   • Expect worked examples with small datasets (3–5 rows).
5. Avoid common mistakes:
   • Don’t confuse regression (continuous) with classification (discrete).
   • Don’t assume linear regression works for non-linear data (use polynomial/non-linear models instead).

Visual Summary of Key Models

mindmap
  root((Prediction & Regression))
    Linear Regression
      Equation: Y = β₀ + β₁X
      Goal: Minimize SSE
    Logistic Regression
      Equation: P(Y=1) = 1/(1+e⁻ᶻ)
      Goal: Binary classification
    Polynomial Regression
      Extends to X², X³
      For curved relationships
    Regularization
      Ridge: L2 penalty
      Lasso: L1 penalty