Correlation & Regression: Scatterplots, Pearson’s r, Least Squares, R²

TAKEAWAYS:

• Correlation measures strength and direction of a linear relationship between two variables (–1 to +1), but does not imply causation.
• Pearson’s r is sensitive to outliers and assumes linearity; Spearman’s ρ is non-parametric.
• Linear regression fits a line $y=a+bx$ to minimize squared errors (least squares), with slope (b) and intercept (a) calculated from data.
• R² (coefficient of determination) explains % of variance in y predicted by x (0 to 1).
• Residuals (observed – predicted) reveal model fit; large residuals indicate poor predictions.
• Multiple regression extends to ≥2 predictors, but overfitting and multicollinearity must be checked.

1. Correlation: Measuring Relationships

Correlation quantifies how two variables move together. It answers:

• Direction: Positive (x↑ → y↑) or negative (x↑ → y↓)?
• Strength: Closer to ±1 = stronger relationship.

Pearson’s Correlation Coefficient (r)

For two variables X and Y, r is calculated as: $$r=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum (x_i-\bar{x}{)}^2\sum (y_i-\bar{y}{)}^2}}$$ Key Properties:

• Range: –1 (perfect negative) to +1 (perfect positive).
• Sensitive to outliers: One extreme point can distort r.
• Assumes linearity: Non-linear relationships (e.g., y = x²) give r ≈ 0.

Worked Example 1: Calculate r for Study Hours vs. Exam Scores Given data:

Steps:

1. Compute means: $\bar{X}=5.4$, $\bar{Y}=71$.
2. Calculate deviations and products: $$\sum (x_i-\bar{x})(y_i-\bar{y})=264,\sum (x_i-\bar{x}{)}^2=52.4,\sum (y_i-\bar{y}{)}^2=480$$
3. Plug into r formula: $$r=\frac{264}{\sqrt{52.4\times 480}}=0.98(\text{Strong positive})$$

Real-World Tie-In:

• eSewa App: Correlation between user rating (X) and transaction success rate (Y) helps identify which features (e.g., payment speed) drive satisfaction. A high r (e.g., 0.8) suggests improving payment processing boosts ratings.

2. Regression: Predicting Outcomes

Regression models the relationship between a dependent variable (Y) and one or more independent variables (X). The simplest case is linear regression: $$Y=a+bX+ϵ$$

• a: Intercept (value of Y when X = 0).
• b: Slope (change in Y per unit X).
• ε: Error term (residuals).

Least Squares Method

Finds the line that minimizes the sum of squared residuals (SSR): $$\text{SSR}=\sum (y_i-(a+b{x}_i){)}^2$$ Formulas for a and b: $$b=\frac{n\sum xy-\sum x\sum y}{n\sum x^2-(\sum x{)}^2},a=\bar{y}-b\bar{x}$$

Worked Example 2: Fit a Regression Line Using the same study hours data:

1. Compute sums: $$\sum x=27,\sum y=355,\sum xy=1935,\sum x^2=173,n=5$$
2. Calculate b and a: $$b=\frac{5\times 1935-27\times 355}{5\times 173-27^2}=1.1,a=71-1.1\times 5.4=0.5$$
3. Prediction: If X = 6 hours, $\hat{Y}=0.5+1.1\times 6=6.6+0.5=7.1$ (scaled to 71–95 range: 80.5).

Real-World Tie-In:

• Pathao Driver Earnings: Regression models predict daily earnings (Y) based on hours worked (X) and peak-time trips (Z). A slope b = 500 NPR/hour suggests drivers earn ₹500 more per hour during rush hours.

3. Goodness of Fit: R² and Residuals

Coefficient of Determination (R²)

Measures % of variance in Y explained by X: $$R^2=r^2=1-\frac{\text{SSR}}{\text{SST}}(\text{where SST}=\sum (y_i-\bar{y}{)}^2)$$

• R² = 0.85 → 85% of Y’s variation is explained by X.
• Never negative; higher = better fit (but not >1).

Residual Analysis

Residuals = Observed Y – Predicted Ŷ.

• Ideal: Residuals randomly scattered around 0 (no pattern).
• Red flags:
  • Non-linearity: Residuals form a curve (e.g., Y = X²).
  • Heteroscedasticity: Residuals’ spread changes with X (violates regression assumptions).

Worked Example 3: Check Residuals for Non-Linearity Given residuals from Example 1: [0.4, –0.2, –0.3, 0.2, –0.4].

• Plot: No clear pattern → linear model is appropriate.
• If residuals trended upward: Try Y = a + bX².

4. Multiple Regression: Extending to ≥2 Predictors

Models Y using multiple X variables: $$Y=a+b_1{X}_1+b_2{X}_2+\cdots +b_k{X}_k+ϵ$$ Advantages:

• Captures complex relationships (e.g., salary depends on experience, education, location). Challenges:
• Multicollinearity: Predictors X₁ and X₂ are highly correlated (e.g., ice cream sales vs. swimming pool visits).
• Overfitting: Too many predictors fit noise (use adjusted R² or cross-validation).

flowchart TD
    A["Multiple Regression Model"] --> B["Y = a + b₁X₁ + b₂X₂ + ... + bₖXₖ"]
    B --> C["Step 1: Standardize Variables"]
    B --> D["Step 2: Calculate Coefficients via Matrix Algebra"]
    B --> E["Step 3: Check Multicollinearity (VIF < 5)"]
    E --> F["If VIF > 5: Remove or Combine Predictors"]
    B --> G["Step 4: Validate with Adjusted R²"]

Worked Example 4: Predicting House Prices

Model: $$Y=100,000+200\times \text{Size}+30,000\times \text{Location}$$

• Interpretation: Each extra sqft adds ₹200; a location score of 1 adds ₹30,000.

Real-World Tie-In:

• Daraz Logistics: Multiple regression predicts delivery time (Y) using distance (X₁), weather (X₂), and warehouse location (X₃). A negative b₂ for rain suggests delays during monsoon.

5. Correlation vs. Regression: Key Differences

6. Common Pitfalls and How to Avoid Them

1. Causation ≠ Correlation

   • Example: Ice cream sales (X) and drowning incidents (Y) are correlated (r ≈ 0.9) but not causal (both rise in summer).
   • Fix: Use domain knowledge or experiments.

2. Extrapolation

   • Mistake: Predicting Y for X values outside the data range.
   • Fix: Only interpolate (predict within X’s min/max).

3. Ignoring Residuals

   • Mistake: Assuming a linear model fits if R² is high.
   • Fix: Plot residuals; switch to polynomial/logistic regression if needed.

4. Overfitting in Multiple Regression

   • Mistake: Using 10 predictors for 20 data points.
   • Fix: Use adjusted R² or regularization (ridge/lasso).

In the Real World

1. Nepal Stock Exchange (NEPSE)

   • Idea Used: Multiple Regression
   • How: Predicts stock prices (Y) using inflation rate (X₁), remittance inflow (X₂), and global oil prices (X₃). A model with R² = 0.75 explains 75% of price movements.
   • Example: If X₁ (inflation) rises by 1%, the model might predict a 0.5% drop in NEPSE index.

2. Khalti Payment App

   • Idea Used: Pearson’s r for Fraud Detection
   • How: Correlates transaction amount (X) with user location (Y) to flag anomalies. A low r between usual spending patterns and sudden large transactions triggers alerts.
   • Example: If a user normally spends ₹5,000/day in Kathmandu but suddenly spends ₹50,000 in Pokhara, r drops sharply → fraud risk.

3. NTC Traffic Route Optimization

   • Idea Used: Linear Regression for Demand Forecasting
   • How: Models hourly internet demand (Y) based on time of day (X₁), holidays (X₂), and weather (X₃). A slope b₁ = 100 Mbps/hour means demand rises by 100 Mbps each hour during peak times.
   • Example: Predicts server loads to allocate bandwidth efficiently in Kathmandu’s busy corridors.

Exam Tip

1. Always Plot Data First

   • Before calculating r or regression, sketch a scatterplot. If the relationship is non-linear, Pearson’s r is meaningless.

2. Interpret r and R² Correctly

   • r = –0.8: "Strong negative linear relationship."
   • R² = 0.64: "64% of Y’s variance is explained by X."

3. Show Work for Calculations

   • For r or regression coefficients, write out sums and formulas. Partial credit is given for correct steps.

4. Residuals Are Your Friend

   • Examiners often ask: "Is this model appropriate?" Always check residual plots.

5. Watch for Trick Questions

   • Example: "If r = 0.9 between X and Y, does Y cause X?" → Answer: No, correlation ≠ causation.

6. Multiple Regression Shortcuts

   • If given a table, use the normal equations or matrix form: $$\mathbf{b}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{Y}$$
   • For exams, memorize the two-variable formulas for a and b.

Final Visual Summary:

mindmap
  root((Correlation & Regression))
    Correlation
      Definition: Strength/Direction
      Pearson's r: Formula & Range
      Limitations: Outliers, Linearity
    Regression
      Linear Model: Y = a + bX
      Least Squares: Minimize SSR
      R²: Goodness of Fit
    Multiple Regression
      Extends to X₁, X₂, ...
      Challenges: Multicollinearity
    Real-World
      NEPSE: Stock Prediction
      Khalti: Fraud Detection
      NTC: Traffic Optimization