Linear Independence

Identifying and removing redundant vectors.

In our last lesson, we saw that when we took two vectors pointing in different directions, their span was the entire 2D plane. But when we took two collinear vectors, the second vector was redundant; it didn't add anything new to our span.

This idea of redundancy is central to linear algebra. A set of vectors is said to be linearly independent if no vector in the set can be written as a linear combination of the others. Conversely, a set is linearly dependent if at least one vector is a combination of the others.

The Formal Definition (The "Zero" Test)

How the Test Works

Consider the vector equation:

c_1v_1 + c_2v_2 + \dots + c_nv_n = \vec{0}

This asks: "Is there a linear combination of our vectors that results in the zero vector?" There is always one trivial solution: set all the scalars to zero. The real question is: **Is the trivial solution the *only* solution?**

If the only way to get the zero vector is the trivial solution (all $c$ 's are zero), then the set of vectors is linearly independent.
If there is *any* non-trivial solution (where at least one $c$ is not zero), then the set is linearly dependent.

Visualizing Linear Dependence

In 2D: Two vectors are linearly dependent if they are collinear. Three or more vectors in 2D are *always* linearly dependent.
In 3D: Three vectors are linearly dependent if they are coplanar (they lie on the same plane).

The Core Idea for Quants & ML

Linear dependence in your features is called multicollinearity, and it's a major headache for many statistical models, especially linear regression.

Imagine you're predicting a stock price using: (1) revenue in USD, (2) revenue in EUR, and (3) number of employees. The revenue features are almost perfectly linearly dependent. A regression model would struggle to assign importance. Identifying and removing linearly dependent features is a crucial step in building robust quantitative models.

Summary: The Efficiency Test

Linearly Independent: A set of vectors containing no redundant information. Each vector adds a new dimension to the span.
Linearly Dependent: A set of vectors containing redundant information. At least one vector is a combination of the others.

Linear Combinations and Span

Basis and Dimension