Correlation & Regression: Pearson’s r, Rank Correlation, Least Squares Line

TAKEAWAYS:

• Correlation ≠ Causation: Pearson’s r measures linear relationships between two continuous variables (e.g., age vs. blood pressure), while Spearman’s rho ranks ordinal data (e.g., student preferences for DELL vs. HP).
• Regression predicts: The least-squares line $Y=a+bX$ estimates values (e.g., predicting pumpkin weight from volume or factory productivity from aptitude scores).
• Interpret r carefully: Values range from –1 (perfect negative) to +1 (perfect positive); r = 0 means no linear trend.
• Real-world ties: Kathmandu traffic routes use regression to predict congestion; eSewa’s fraud detection relies on correlation between transaction patterns.
• Exam focus: Always show calculations step-by-step (e.g., covariance, standard deviations) and interpret results in context (e.g., "A r = 0.85 suggests strong positive correlation between experience and performance").

1. Correlation: Measuring Relationships

Correlation quantifies how two variables move together. The two key methods are:

• Pearson’s r (for continuous data, e.g., age vs. blood pressure).
• Spearman’s rho (for ranked data, e.g., student preferences).

Pearson’s Correlation Coefficient (r)

Definition: $$r=\frac{\text{Cov}(X,Y)}{s_X{s}_Y}=\frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sqrt{\sum (X_i-\bar{X}{)}^2\sum (Y_i-\bar{Y}{)}^2}}$$ where:

• $\text{Cov}(X,Y)$ = covariance (how X and Y vary together).
• $s_X,s_Y$ = standard deviations of X and Y.

Key Properties:

graph TD
    A["Pearson’s *r* Properties"] --> B["Range: -1 to +1"]
    A --> C["*r* = +1: Perfect positive linear relationship"]
    A --> D["*r* = -1: Perfect negative linear relationship"]
    A --> E["*r* = 0: No linear relationship (could be nonlinear!)"]
    A --> F["Sensitive to outliers"]
    A --> G["Unitless (standardized)"]

Worked Example 1: Husband-Wife Ages Question: Calculate r for the ages of 10 husbands and wives. Interpret the result. Data: | Husband (X) | 23 | 27 | 28 | 28 | 29 | 30 | 31 | 33 | 35 | 36 | | Wife (Y) | 22 | 25 | 26 | 27 | 28 | 29 | 30 | 32 | 34 | 35 |

Step-by-Step Calculation:

1. Compute means: $$\bar{X}=\frac{23+27+\cdots +36}{10}=30,\bar{Y}=28.5$$
2. Compute covariance and standard deviations: Sums: $$\sum (X-\bar{X})(Y-\bar{Y})=100.5,\sum (X-\bar{X}{)}^2=120,\sum (Y-\bar{Y}{)}^2=105$$
3. Plug into r formula: $$r=\frac{100.5}{\sqrt{120\times 105}}=\frac{100.5}{112.25}\approx 0.9$$

Interpretation:

• r = 0.9 indicates a strong positive linear correlation between husband and wife ages. As one spouse’s age increases, the other’s tends to increase proportionally.
• Real-world tie: This mirrors how eSewa’s age verification system uses similar correlations to detect fraud (e.g., mismatched ages in transaction pairs).

Spearman’s Rank Correlation (rho)

Used for ordinal data (ranks, preferences). Formula: $$\rho =1-\frac{6\sum d_i^2}{n(n^2-1)}$$ where $d_i$ = difference in ranks for each pair, $n$ = number of items.

Worked Example 2: Student Preferences (DELL vs. HP) Question: Calculate Spearman’s rho for 10 students’ rankings of DELL and HP. Data:

Calculation: $$\sum d_i^2=110,n=10$$ $$\rho =1-\frac{6\times 110}{10(100-1)}=1-\frac{660}{990}=0.33$$

Interpretation:

• rho = 0.33 suggests a weak positive correlation between student preferences for DELL and HP. Most students don’t strongly prefer one over the other.
• Real-world tie: Pathao’s driver ratings use Spearman’s rho to correlate customer feedback scores with trip completion times.

2. Regression Analysis: Prediction and Modeling

Regression finds the best-fit line $Y=a+bX$ to predict $Y$ from $X$. Key methods:

• Least-squares regression: Minimizes the sum of squared errors.
• Interpretation: $b$ = slope (change in $Y$ per unit $X$), $a$ = intercept.

Regression Line Equation

$$b=\frac{\text{Cov}(X,Y)}{s_X^2},a=\bar{Y}-b\bar{X}$$ $$Y=a+bX$$

Worked Example 3: Blood Pressure vs. Age Question: Fit a regression line to predict blood pressure ($Y$) from age ($X$) for 10 individuals. Data:

Steps:

1. Compute means, covariance, and $s_X^2$ (from earlier: $\bar{X}=40$, $\bar{Y}=136.5$, $\text{Cov}(X,Y)=120$, $s_X^2=160$).
2. Calculate slope ($b$) and intercept ($a$): $$b=\frac{120}{160}=0.75,a=136.5-0.75\times 40=106.5$$
3. Regression equation: $$Y=106.5+0.75X$$

Interpretation:

• For every 1-year increase in age, blood pressure rises by 0.75 mmHg.
• Real-world tie: NTC uses similar models to predict network congestion based on call volumes.

3. Comparing Correlation and Regression

4. Common Pitfalls and Misconceptions

mindmap
  root((Correlation/Regression Mistakes))
    A1[Correlation ≠ Causation]
    A2[Extrapolation Danger]
    A3[Outliers Skew Results]
    A4[Nonlinear ≠ No Correlation]
    A5[Rank vs. Continuous Data]

Example:

• Causation vs. Correlation: Ice cream sales and drowning incidents are correlated (r ≈ 0.8), but neither causes the other (both rise in summer).
• Nonlinear Data: If $Y=X^2$, Pearson’s r may be low even though the relationship is strong.

## In the Real World

1. eSewa’s Fraud Detection:

   • Idea Used: Pearson’s r to correlate transaction amounts with user location/device.
   • How: Unusually high r between a user’s typical spending and a sudden large transaction flags fraud (e.g., r > 0.95 triggers review).

2. Pathao’s Driver Ratings:

   • Idea Used: Spearman’s rho to rank driver performance (e.g., trip completion time vs. customer feedback).
   • How: Drivers with rho < 0.5 between "on-time" and "clean car" metrics get retrained.

3. NTC’s Network Prediction:

   • Idea Used: Regression to predict call drops from call volume.
   • Equation: $Y=5+0.02X$ (where $Y$ = % drops, $X$ = calls/hour).
   • Impact: Helps NTC allocate towers during festivals (e.g., Dashain).

4. Daraz’s Order Fulfillment:

   • Idea Used: Regression to estimate delivery time from distance.
   • Example: For a 20 km order, $Y=2+0.15\times 20=5$ hours (intercept = warehouse processing time).

## Exam Tip

1. Always show calculations:

   • For r: Write out covariance and standard deviations explicitly.
   • For regression: Derive $a$ and $b$ step-by-step (examiners deduct for skipping steps).

2. Interpret results in context:

   • Bad: "r = 0.85".
   • Good: "r = 0.85 suggests a strong positive correlation between experience (X) and performance (Y), meaning operators with more years of experience tend to produce more defect-free parts."

3. Watch units:

   • If $X$ is in years and $Y$ in mmHg, state the slope as "0.75 mmHg per year."

4. Rank correlation shortcut:

   • For small datasets ($n<10$), compute ranks manually. For larger sets, use the formula: $$\rho =1-\frac{6\sum d_i^2}{n^3-n}$$

5. Graphs are mandatory:

   • Always sketch a scatter plot with the regression line. Label axes and highlight outliers.

Visual Summary: