Gram-Schmidt: The Art of Tidying Up

The Gram-Schmidt process is a straightforward algorithm used to "tidy up" a set of linearly independent vectors. It takes a "messy" basis (where vectors might be at odd angles to each other) and systematically converts it into a "clean" orthogonal basis, where every vector is at a 90-degree angle to every other vector.

This is incredibly useful in quantitative finance and data science. Orthogonal bases simplify calculations immensely. For example, in regression analysis, if your predictor variables (features) are orthogonal, it makes it much easier to understand the individual impact of each feature on the outcome.

The Two-Vector Process

Let's say we have two linearly independent vectors, v₁ and v₂. Here's how we create an orthogonal basis (u₁, u₂):

1. Step 1: The Anchor. We start by simply accepting the first vector as is. The first vector of our new basis, u₁, is just v₁.
$$\mathbf{u}_1 = \mathbf{v}_1$$
2. Step 2: Find the Shadow. We take the second vector, v₂, and find its projection onto our first new basis vector, u₁. This projection is the "shadow" that v₂ casts on the line defined by u₁.
$$\text{proj}_{\mathbf{u}_1}(\mathbf{v}_2) = \frac{\mathbf{v}_2 \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1} \mathbf{u}_1$$
3. Step 3: Find the Perpendicular Part. The original vector v₂ can be seen as the sum of its part parallel to u₁ (the projection) and its part perpendicular to u₁. To find this perpendicular part, we simply subtract the projection from the original vector. This gives us our second orthogonal basis vector, u₂.
$$\mathbf{u}_2 = \mathbf{v}_2 - \text{proj}_{\mathbf{u}_1}(\mathbf{v}_2)$$

The result is a new basis (u₁, u₂) that spans the exact same space as the original basis (v₁, v₂), but with the convenient property that $\mathbf{u}_1 \cdot \mathbf{u}_2 = 0$.