Correlation & Regression: Models, Analysis & Applications

Core Concepts & Definitions

1. Correlation Analysis

Definition & Purpose

• Correlation measures the strength and direction of a linear relationship between two continuous variables.
• Key metrics:
  • Pearson’s r (parametric, linear): $r=\frac{\text{Cov}(X,Y)}{s_X{s}_Y}$ (range: -1 to +1)
  • Spearman’s ρ (non-parametric, monotonic): Rank-based correlation.
  • Kendall’s τ (ordinal data, small samples).

Interpretation of r

Limitations

• Not causal: Correlation ≠ causation (e.g., ice cream sales and drowning deaths both rise in summer).
• Sensitive to outliers: Extreme values distort r.
• Linear only: Fails for non-linear relationships (e.g., $Y=X^2$).

---

2. Simple Linear Regression (SLR)

Mathematical Model

The population regression line: $$Y={\beta }_0+{\beta }_{1}X+ϵ$$

• Estimated model (OLS): $$\hat{Y}=b_0+b_{1}X$$ where:
• $b_0=\bar{Y}-b_1\bar{X}$ (intercept),
• $b_1=\frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sum (X_i-\bar{X}{)}^2}=r\cdot \frac{s_Y}{s_X}$ (slope).

Assumptions of SLR

1. Linearity: Relationship between $X$ and $Y$ is linear.
2. Independence: Observations are independent (no autocorrelation).
3. Homoscedasticity: Constant variance of errors ($\text{Var}(ϵ)={\sigma }^2$).
4. Normality: Errors $ϵ\sim N(0,{\sigma }^2)$.
5. No multicollinearity (irrelevant for SLR but critical in MLR).

Worked Example: Predicting Computer Efficiency

Problem: A manager tests a new program’s efficiency ($Y$: processing speed in ops/sec) against:

• $X_1$: Data size (MB),
• $X_2$: Number of tables used.

Given data (hypothetical):

Steps:

1. Plot scatterplots (e.g., $Y$ vs. $X_1$) to check linearity.
2. Calculate Pearson’s r between $Y$ and each predictor:
   • $r_{Y,X1}=0.85$ (strong positive),
   • $r_{Y,X2}=-0.60$ (moderate negative).
3. Fit SLR models:
   • Model 1: $\hat{Y}=20+2X_1$ ($R^2=0.72$),
   • Model 2: $\hat{Y}=80-5X_2$ ($R^2=0.36$).
4. Choose better predictor: $X_1$ explains more variance.

---

3. Multiple Linear Regression (MLR)

Model Equation

$$\hat{Y}=b_0+b_1{X}_1+b_2{X}_2+\cdots +b_k{X}_k$$

• Coefficients estimated via Ordinary Least Squares (OLS): $$\mathbf{b}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{Y}$$
• Partial regression coefficients: Measure effect of one predictor holding others constant.

ANOVA Table for Regression

Example Interpretation: From the past exam question:

• Regression SS = 740, df = 2 → MSR = 370.
• Error SS = 510, df = 17 → MSE = 30.
• F = 370 / 30 = 12.33 → p < 0.05 (significant model).

Key Metrics

1. Coefficient of Determination ($R^2$): $$R^2=1-\frac{\text{SSE}}{\text{SST}}=\frac{\text{SSR}}{\text{SST}}$$
   • Adjusted $R^2$: Penalizes extra predictors: $$R_{\text{adj}}^2=1-\left(\frac{\text{SSE}/(n-k-1)}{\text{SST}/(n-1)}\right)$$
2. Standard Error of Estimate (SEE): $$\text{SEE}=\sqrt{\text{MSE}}$$
3. t-tests for coefficients: Check if $b_i\ne 0$.

Worked Example: CVS Study

Problem: Predict Computer Vision Syndrome (CVS) score ($Y$) using:

• $X_1$: Age (years),
• $X_2$: Working hours/day.

Regression Output: $$\hat{Y}=10+0.5X_1+1.2X_2$$

• SSR = 450, SSE = 150, $n=30$, $k=2$.
• ANOVA Table:

  Source	df	SS	MS	F
  Regression	2	450	225	30
  Error	27	150	5.56
  Total	29	600

Calculations:

• $R^2=450/600=0.75$ (75% variance explained).
• Adjusted $R^2$: $$R_{\text{adj}}^2=1-\left(\frac{150/27}{600/29}\right)=0.73$$
• F-test: $F=30$ → p < 0.05 (model significant).

---

4. Model Diagnostics

Common Issues & Fixes

Residual Analysis

1. Plot residuals vs. fitted values:
   • Good: Random scatter around 0.
   • Bad: Patterns (e.g., curves, fans).
2. Normal Q-Q plot:
   • Points should lie on a straight line.

---

5. Partial and Multiple Correlation

Definitions

• Partial Correlation ($r_{12.3}$): Correlation between $X_1$ and $X_2$ removing effect of $X_3$. $$r_{12.3}=\frac{r_{12}-r_{13}{r}_{23}}{\sqrt{(1-r_{13}^2)(1-r_{23}^2)}}$$
• Multiple Correlation ($R$): Correlation between $Y$ and a set of predictors. $$R=\sqrt{1-\frac{\text{SSE}}{\text{SST}}}$$

Worked Example

Given: $r_{12}=0.5$, $r_{23}=0.6$, $r_{13}=0.7$.

Find:

1. Partial correlation $r_{12.3}$: $$r_{12.3}=\frac{0.5-(0.7)(0.6)}{\sqrt{(1-0.7^2)(1-0.6^2)}}=\frac{0.5-0.42}{\sqrt{0.51\times 0.64}}=\frac{0.08}{0.57}\approx 0.14$$ → Weak partial correlation.

2. Multiple correlation $R_{123}$: Use the formula for 3 variables: $$R_{123}^2=\frac{r_{12}^2+r_{13}^2-2r_{12}{r}_{13}{r}_{23}}{1-r_{23}^2}$$ (Derivation omitted; typically computed via software.)

---

6. Applications in Computer Science/IT

---

7. Latin Square Design (LSD) – Brief Link

(Note: LSD is covered in Unit 3, but regression is used in its analysis.)

• Purpose: Control two blocking variables (e.g., operator skill and machine type) in experiments.
• ANOVA Table (for $m\times m$ LSD):

  Source          df    SS      MS     F
  Rows (A)        m-1
  Columns (B)     m-1
  Treatments (T)  m-1
  Error           (m-1)(m-2)
  Total           m²-1
  

• Assumption: One observation per cell.

---

Exam Tip: How to Score Full Marks

1. Structured Answers

• For correlation:

  • Always state the type of correlation (Pearson/Spearman).
  • Interpret r with direction and strength.
  • Mention limitations (e.g., non-linearity, outliers).

• For regression:

  • Write the full model equation ($\hat{Y}=\dots$).
  • Show ANOVA table with SS, df, MS, F.
  • Calculate and interpret $R^2$ and adjusted $R^2$.
  • Discuss significance of coefficients (t-tests).

2. Common Pitfalls to Avoid

• Ignoring assumptions: Always check linearity, homoscedasticity, etc.
• Misinterpreting $R^2$: It’s not causal! High $R^2$ ≠ good model if predictors are nonsensical.
• Forgetting units: Report coefficients with units (e.g., "Efficiency increases by 2 ops/sec per MB of data").
• Partial vs. multiple correlation: Clearly label which is being asked.

3. Numerical Questions

• Always show steps:
  • For $R^2$: $1-\frac{\text{SSE}}{\text{SST}}$.
  • For partial correlation: Use the formula and simplify.
• Use given data: If SS/df are provided, compute MS and F directly.

4. Theoretical Questions

• Define key terms precisely:
  • "Partial correlation controls for the effect of a third variable."
  • "Multiple regression extends SLR to multiple predictors."
• Compare SLR vs. MLR:

  Aspect	Simple Linear Regression	Multiple Linear Regression
  Predictors	One ($X$)	Two or more ($X_1,X_2,\dots$)
  Equation	$\hat{Y}=b_0+b_{1}X$	$\hat{Y}=b_0+b_1{X}_1+\dots$
  Interpretation	Slope $b_1$ is total effect.	Coefficients are partial effects.
  Assumptions	Easier to check.	Harder (multicollinearity, etc.).

5. Practical Tips

• For ANOVA tables: Memorize the df formula:
  • Regression df = number of predictors.
  • Error df = $n-k-1$ (where $k$ = predictors).
• For partial correlation: If given 3 variables, always compute $r_{12.3}$ or similar.
• For exam questions:
  • If asked to "complete the ANOVA table," ensure SS adds up (SSR + SSE = SST).
  • If asked to "interpret," always link back to the context (e.g., "This means efficiency improves with data size...").

---

Summary Checklist

Before submitting, verify: ✅ Definitions are precise (e.g., "Pearson’s r measures linear association"). ✅ Formulas are correctly applied (e.g., $R^2$ calculation). ✅ Assumptions are listed for regression. ✅ Interpretations are contextual (not just numbers). ✅ Tables/figures are labeled clearly (e.g., "ANOVA Table for MLR"). ✅ Units are included where applicable (e.g., "per MB").

---

Final Note: This unit is heavily numerical—practice calculating r, regression coefficients, ANOVA tables, and partial correlations under time constraints. Use past exam questions to drill these skills! For IT/CS applications, always relate regression to real-world scenarios (e.g., predicting system performance).