Curve Fitting & Regression: Models, Least Squares, and Applications

TAKEAWAYS:

• Regression vs. Interpolation: Regression fits a model to approximate data trends (minimizing error globally), while interpolation fits a curve exactly through given points (zero error at data points).
• Least Squares Method: The foundation of regression—minimizes the sum of squared residuals to find the best-fit parameters (e.g., slope/intercept in linear regression).
• Model Selection: Choose between linear, polynomial, exponential, or logarithmic regression based on data behavior (e.g., exponential decay for radioactive data, quadratic for projectile motion).
• Applications: Regression predicts trends (e.g., stock prices, population growth), while curve fitting models physical laws (e.g., Newton’s cooling law).
• Algorithmic Workflow: Transform data (e.g., log-transform for exponential fits), solve normal equations, or use iterative methods (e.g., gradient descent) for large datasets.
• Error Metrics: Use $R^2$ (coefficient of determination) or RMSE (root mean squared error) to evaluate fit quality—higher $R^2$ (closer to 1) indicates better fit.

1. Definitions and Key Concepts

1.1 Curve Fitting

Curve fitting is the process of constructing a mathematical function (model) that best describes the relationship between variables in a dataset. Unlike interpolation, which passes exactly through given points, curve fitting aims to approximate the underlying trend with minimal error.

1.2 Regression Analysis

A statistical tool within curve fitting that estimates the relationships among variables. It predicts the dependent variable ($y$) based on one or more independent variables ($x_1,x_2,\dots$).

1.3 Least Squares Approximation

The most common method for regression, where the best-fit curve minimizes the sum of the squares of the residuals (differences between observed and predicted values). Mathematically, for a model $y=f(x,\beta )$, we minimize: $$S(\beta )=\sum _{i=1}^n[y_i-f(x_i,\beta ){]}^2$$

2. Types of Regression Models

2.1 Linear Regression

Fits a straight-line model $y=mx+c$ to data. Used when the relationship between $x$ and $y$ is approximately linear.

Example: Fit $y=mx+c$ to the data:

Solution:

1. Transform data: The exponential trend suggests a logarithmic transformation. Let $y=e^{mx+c}$, so $\ln (y)=mx+c$.

2. Compute $\ln (y)$:

   x	1.0	1.2	1.4	1.6	1.8	2.0	2.2	2.4
   $\ln (y)$	0.693	0.955	1.361	1.792	2.230	2.708	3.025	3.411

3. Apply linear regression to $\ln (y)$ vs. $x$:

   • Calculate means: $\bar{x}=1.65$, $\overline{\ln (y)}=2.028$.
   • Slope ($m$): $$m=\frac{\sum (x_i-\bar{x})(\ln (y_i)-\overline{\ln (y)})}{\sum (x_i-\bar{x}{)}^2}=\frac{10.35}{3.4}\approx 3.044$$
   • Intercept ($c$): $$c=\overline{\ln (y)}-m\bar{x}=2.028-3.044\times 1.65\approx -2.98$$

4. Final model: $$y=e^{3.044x-2.98}$$

2.2 Polynomial Regression

Fits a polynomial model $y=a_0+a_{1}x+a_2{x}^2+\cdots +a_n{x}^n$ to data. Useful for nonlinear trends (e.g., quadratic for projectile motion).

Example: Fit a quadratic curve $y=a{x}^2+bx+c$ to:

Solution:

1. Set up normal equations for $y=a{x}^2+bx+c$: $$\{\begin{array}{l}\sum y=an\sum x^2+b\sum x+nc\\ \sum xy=a\sum x^3+b\sum x^2+c\sum x\\ \sum x^{2}y=a\sum x^4+b\sum x^3+c\sum x^2\end{array}$$
2. Compute sums: $$\sum x=19,\sum x^2=91,\sum x^3=425,\sum x^4=1765,\sum y=30,\sum xy=143,\sum x^{2}y=560$$
3. Solve the system: $$\left[\begin{array}{ccc}91& 19& 5\\ 425& 91& 19\\ 1765& 425& 91\end{array}\right]\left[\begin{array}{c}a\\ b\\ c\end{array}\right]=\left[\begin{array}{c}30\\ 143\\ 560\end{array}\right]$$ Using matrix methods or substitution, we get: $$a=-0.5,b=2.5,c=1$$ Final model: $$y=-0.5x^2+2.5x+1$$
4. Prediction: At $x=2$: $$y=-0.5(4)+2.5(2)+1=-2+5+1=4$$

2.3 Exponential Regression

Fits models like $y=a{e}^{bx}$ or $y=a{b}^x$. Used for growth/decay processes (e.g., population, radioactive decay).

Example: Fit $y=a{e}^{bx}$ to:

Solution:

1. Take natural logs: $\ln (y)=\ln (a)+bx$.
2. Let $Y=\ln (y)$ and $A=\ln (a)$. Solve for $A$ and $b$ using linear regression on $(x,Y)$.
3. Compute means: $\bar{x}=4.5$, $\bar{Y}=\ln (0.647)\approx -0.435$.
4. Slope ($b$): $$b=\frac{\sum (x_i-\bar{x})(Y_i-\bar{Y})}{\sum (x_i-\bar{x}{)}^2}=\frac{-10.35}{122.5}\approx -0.0845$$
5. Intercept ($A$): $$A=\bar{Y}-b\bar{x}=-0.435-(-0.0845)(4.5)\approx 0.00025$$
6. Solve for $a$: $$a=e^A\approx e^{0.00025}\approx 1.00025$$ Final model: $$y\approx 1.00025e^{-0.0845x}$$

3. Least Squares Method: Theory and Implementation

3.1 Derivation

For a linear model $y={\beta }_0+{\beta }_{1}x$, the least squares solution is: $$\beta =(X^{T}X{)}^{-1}{X}^{T}y$$ where $X$ is the design matrix: $$X=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_n\end{array}\right],y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]$$

3.2 Steps for Polynomial Regression

1. Construct the design matrix $X$ for the polynomial degree.
2. Solve the normal equations $\beta =(X^{T}X{)}^{-1}{X}^{T}y$.
3. Evaluate the model at desired $x$ values.

3.3 Advantages/Disadvantages

4. Regression vs. Interpolation: Key Differences

mindmap
  root((Curve Fitting))
    Regression
      Approximation
      Minimizes global error
      Used for prediction/trend analysis
      Example: Stock prices, population growth
    Interpolation
      Exact fit
      Zero error at data points
      Used for precise reconstruction
      Example: CAD/CAM, signal processing

Comparison Table:

5. Algorithms and Pseudocode

5.1 Lagrange Interpolation (for comparison)

def lagrange_interpolation(x, y, x_query):
    n = len(x)
    result = 0.0
    for i in range(n):
        term = y[i]
        for j in range(n):
            if i != j:
                term *= (x_query - x[j]) / (x[i] - x[j])
        result += term
    return result

5.2 Least Squares Regression (Linear)

import numpy as np

def linear_regression(x, y):
    A = np.vstack([np.ones(len(x)), x]).T
    beta = np.linalg.lstsq(A, y, rcond=None)[0]
    return beta[0], beta[1]  # intercept, slope

6. Applications

1. Physics: Fitting experimental data to theoretical models (e.g., Hooke’s law for springs).
2. Economics: Predicting sales trends or cost functions.
3. Biology: Modeling population growth or drug concentration over time.
4. Engineering: Calibrating sensors or optimizing system parameters.

7. Error Metrics

1. Sum of Squared Residuals (SSR): $$SSR=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$
2. Coefficient of Determination ($R^2$): $$R^2=1-\frac{SSR}{SST},\text{where }SST=\sum (y_i-\bar{y}{)}^2$$
   • $R^2=1$: Perfect fit.
   • $R^2=0$: Model explains no variance.
3. Root Mean Squared Error (RMSE): $$RMSE=\sqrt{\frac{SSR}{n}}$$

8. Exam Tip

What Examiners Look For

1. Correct Model Selection:

   • Identify whether data suggests linear, polynomial, or exponential trends (e.g., exponential decay for radioactive data).
   • Justify your choice (e.g., "The data shows a multiplicative trend, so exponential regression is appropriate").

2. Mathematical Rigor:

   • Show all steps for deriving normal equations or transformations (e.g., log-transform for exponential fits).
   • For polynomial regression, explicitly write the system of equations and solve it (even if using matrix methods).

3. Numerical Accuracy:

   • Compute means, sums, and coefficients precisely. Avoid rounding intermediate steps.
   • Example: For linear regression, calculate $\sum x$, $\sum y$, $\sum xy$, and $\sum x^2$ accurately.

4. Comparison with Interpolation:

   • Clearly distinguish between the two in definitions, goals, and examples.
   • Example answer for "How does interpolation differ from regression?":

     Interpolation constructs a function that passes exactly through all given data points, ensuring zero error at those points but potentially large errors elsewhere. It is used for precise reconstruction (e.g., in computer graphics). In contrast, regression approximates the underlying trend by minimizing global error, making it robust for prediction and trend analysis but not exact at data points.

5. Programming Questions:

   • For Lagrange interpolation, show the nested loop structure and explain the term-by-term construction.
   • For least squares, demonstrate matrix operations (e.g., $(X^{T}X{)}^{-1}{X}^{T}y$) or iterative methods like gradient descent.

6. Common Pitfalls:

   • Overfitting: Avoid high-degree polynomials unless justified. Use $R^2$ or cross-validation to check.
   • Extrapolation: Never assume regression models hold outside the data range.
   • Nonlinear Models: Always transform data (e.g., log, reciprocal) before applying linear regression techniques.

Sample Exam Questions and Solutions

Q1: Fit the curve $y=a{e}^{bx}$ to the data:

Solution:

1. Take logs: $\ln (y)=\ln (a)+bx$.
2. Compute $\ln (y)$:

   x	1	2	3	4
   $\ln (y)$	0.501	0.993	1.504	1.993

3. Apply linear regression to $(x,\ln (y))$:
   • $\bar{x}=2.5$, $\overline{\ln (y)}=1.356$.
   • Slope ($b$): $$b=\frac{\sum (x_i-\bar{x})(\ln (y_i)-\overline{\ln (y)})}{\sum (x_i-\bar{x}{)}^2}=\frac{2.5}{5}=0.5$$
   • Intercept ($A=\ln (a)$): $$A=1.356-0.5\times 2.5=0.106$$
   • Thus, $a=e^{0.106}\approx 1.112$.
4. Final model: $$y=1.112e^{0.5x}$$

Q2: How does polynomial interpolation differ from least squares approximation?

Solution:

9. Practice Problems

1. Fit a second-order polynomial to the data:

   x	2	4	6	8	10
   y	1.4	2.0	2.4	2.6	2.8

2. Fit $y=a{b}^x$ to the data:

   x	0	1	2	3
   y	5	10	20	40

3. Compare the $R^2$ values for linear and quadratic fits to the data in Problem 1. Which model is better?

10. References

• Burden, R.L., & Faires, J.D. (2011). Numerical Analysis (9th ed.). Brooks/Cole.
• Chapra, S.C., & Canale, R.P. (2010). Numerical Methods for Engineers (6th ed.). McGraw-Hill.
• Wikipedia: "Curve fitting", "Polynomial interpolation"