Orthogonality, Inner Products, and Norms in Vector Spaces

Key Concepts and Definitions

1. Inner Product Spaces

An inner product space is a vector space $V$ over a field $\mathbb{F}$ (usually $\mathbb{R}$ or $\mathbb{C}$) equipped with an inner product $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{F}$ satisfying:

• Conjugate symmetry: $⟨u,v⟩=\overline{⟨v,u⟩}$
• Linearity in the first argument: $⟨au+bv,w⟩=a⟨u,w⟩+b⟨v,w⟩$
• Positive-definiteness: $⟨v,v⟩\ge 0$ and $⟨v,v⟩=0⟺v=0$.

Example (Dot Product in ${\mathbb{R}}^n$): For $u=(u_1,\dots ,u_n)$ and $v=(v_1,\dots ,v_n)$, the standard inner product is: $$⟨u,v⟩=u_1{v}_1+\cdots +u_n{v}_n.$$

2. Norms and Induced Norms

A norm $\Vert \cdot \Vert$ on $V$ is a function $V\to \mathbb{R}$ satisfying:

1. $\Vert v\Vert \ge 0$ and $\Vert v\Vert =0⟺v=0$,
2. $\Vert \alpha v\Vert =|\alpha |\Vert v\Vert$ for scalars $\alpha$,
3. $\Vert u+v\Vert \le \Vert u\Vert +\Vert v\Vert$ (triangle inequality).

An inner product induces a norm via: $$\Vert v\Vert =\sqrt{⟨v,v⟩}.$$

3. Orthogonality

Two vectors $u,v\in V$ are orthogonal if $⟨u,v⟩=0$.

• A set $\{v_1,\dots ,v_k\}$ is orthogonal if $⟨v_i,v_j⟩=0$ for $i\ne j$.
• An orthonormal set satisfies $⟨v_i,v_j⟩={\delta }_{ij}$ (Kronecker delta).

Example (Orthogonal Vectors in ${\mathbb{R}}^2$): Let $u=(1,0)$ and $v=(0,1)$. Then $⟨u,v⟩=0$, so $u\perp v$.

---

Gram-Schmidt Orthogonalization

Given a linearly independent set $\{v_1,\dots ,v_k\}$, the Gram-Schmidt process constructs an orthogonal set $\{u_1,\dots ,u_k\}$ with the same span:

1. $u_1=v_1$,
2. $u_j=v_j-{\sum }_{i=1}^{j-1}\frac{⟨v_j,u_i⟩}{⟨u_i,u_i⟩}{u}_i$ for $j=2,\dots ,k$.

Worked Example: Orthogonalize $\{v_1,v_2\}=\{(1,1,0),(0,1,1),(1,0,1)\}$ in ${\mathbb{R}}^3$ using the dot product.

Solution:

1. $u_1=v_1=(1,1,0)$.
2. $u_2=v_2-\frac{⟨v_2,u_1⟩}{⟨u_1,u_1⟩}{u}_1=(0,1,1)-\frac{1}{2}(1,1,0)=\left(-\frac{1}{2},\frac{1}{2},1\right)$.
3. $u_3=v_3-\frac{⟨v_3,u_1⟩}{⟨u_1,u_1⟩}{u}_1-\frac{⟨v_3,u_2⟩}{⟨u_2,u_2⟩}{u}_2$.

Normalization: Divide each $u_i$ by $\Vert u_i\Vert$ to get an orthonormal set.

---

Projections and Least Squares

Projection onto a Subspace

Given a subspace $W\subseteq V$ with orthonormal basis $\{w_1,\dots ,w_k\}$, the projection of $v\in V$ onto $W$ is: $${\text{proj}}_{W}v=\sum _{i=1}^k⟨v,w_i⟩w_i.$$ The residual is $v-{\text{proj}}_{W}v$, which is orthogonal to $W$.

Example (Projection in ${\mathbb{R}}^2$): Let $W=\text{span}\{(1,0)\}$. The projection of $v=(a,b)$ onto $W$ is: $${\text{proj}}_{W}v=(a,0).$$

Least Squares Solution

For an overdetermined system $Ax=b$ (where $A\in {\mathbb{R}}^{m\times n}$, $m>n$), the least squares solution minimizes $\Vert Ax-b{\Vert }_2$. It is given by: $$x^{\ast }=(A^{T}A{)}^{-1}{A}^{T}b,$$ provided $A^{T}A$ is invertible.

Worked Example: Solve $\{\begin{array}{l}x+y=1\\ 2x-y=0\\ x+2y=3\end{array}$ in the least squares sense.

Solution: Write as $Ax=b$, where: $$A=\left[\begin{array}{cc}1& 1\\ 2& -1\\ 1& 2\end{array}\right],b=\left[\begin{array}{c}1\\ 0\\ 3\end{array}\right].$$ Compute $A^{T}A$ and $A^{T}b$, then solve $(A^{T}A)x=A^{T}b$.

---

Key Properties and Theorems

---

Applications

1. Signal Processing: Orthogonal bases (e.g., Fourier series) decompose signals efficiently.
2. Machine Learning: Projections and least squares are used in regression and PCA.
3. Quantum Mechanics: Inner products define probabilities in Hilbert spaces.
4. Computer Graphics: Orthogonal transformations (e.g., rotations) preserve angles and lengths.

---

Exam Tip

1. Definitions First: Always define inner products, orthogonality, and norms clearly. Partial credit is often given for correct definitions even if calculations are wrong.
2. Gram-Schmidt Steps: Show each step explicitly. Forgetting to normalize or misapplying the projection formula costs marks.
3. Least Squares: Remember the formula $x^{\ast }=(A^{T}A{)}^{-1}{A}^{T}b$ and verify invertibility of $A^{T}A$.
4. Geometric Intuition: Draw diagrams for projections in ${\mathbb{R}}^2$ or ${\mathbb{R}}^3$. Examiners appreciate visual explanations.
5. Common Pitfalls:
   • Confusing orthogonality with linear independence.
   • Forgetting to take conjugates in complex inner products.
   • Incorrectly computing projections (e.g., missing the orthonormality condition).
6. Past Exam Patterns:
   • 2078 Fall: Prove the Cauchy-Schwarz inequality.
   • 2079 Spring: Apply Gram-Schmidt to a given set of vectors.
   • 2080 Fall: Solve a least squares problem and interpret the residual.

---

Practice Problems:

1. Show that the set $\{(1,1,0),(1,-1,1),(0,1,1)\}$ is orthogonal in ${\mathbb{R}}^3$.
2. Find the projection of $(1,2,3)$ onto the plane $x+y+z=0$.
3. Use least squares to fit a line $y=mx+c$ to the data points $(1,2),(2,3),(3,5)$.