Special Matrices: The Main Characters

The Identity matrix, denoted $I$, is a square matrix with 1s on the main diagonal and 0s everywhere else.

The Identity matrix is the matrix equivalent of the number 1. Just as $1 \cdot x = x$, multiplying any matrix $A$ by the Identity matrix $I$ leaves $A$ completely unchanged.

As a transformation, the Identity matrix does nothing. It leaves all of space completely untouched.

For many (but not all!) square matrices $A$, there exists a special matrix called its inverse, denoted $A^{-1}$. The inverse is the matrix that "undoes" the transformation of $A$.

If you apply transformation $A$, and then apply its inverse $A^{-1}$, you get back to where you started. You get the "do-nothing" Identity matrix.

A square matrix has an inverse only if its transformation is reversible. This means the matrix cannot "squish" or "collapse" space into a lower dimension. A matrix that has an inverse is called invertible or non-singular. A matrix without an inverse is called non-invertible or singular.

A diagonal matrix is one where all the non-zero elements are on the main diagonal.

Diagonal matrices are the simplest transformations of all. They perform a pure scaling along each axis, with no rotation or shear. The matrix $D$ above scales the x-axis by 3, the y-axis by -2 (stretching and flipping it), and the z-axis by 5.

A symmetric matrix is a square matrix that is unchanged by a transpose. In other words, $A = A^T$.

Symmetric matrices are the superstars of quantitative finance and machine learning. Covariance matrices and correlation matrices are always symmetric. They have beautiful, powerful properties: their eigenvalues are always real, and their eigenvectors are always orthogonal.

A triangular matrix is a square matrix where all the entries either above or below the main diagonal are zero.

Triangular matrices are a huge deal in numerical computation. The entire point of the LU Decomposition is to break down a complicated matrix $A$ into the product of a Lower triangular matrix $L$ and an Upper triangular matrix $U$. This makes solving $Ax=b$ vastly more efficient for computers.

An Orthogonal Matrix is a square matrix that represents a rigid motion: a transformation that can rotate or reflect space, but cannot stretch, shrink, or shear it.

The defining feature of an orthogonal matrix, denoted $Q$, is that its columns form an orthonormal basis. This means every column vector has a length of 1 and is orthogonal to every other column.

The inverse of an orthogonal matrix is simply its transpose.

This is a phenomenal result. The difficult operation of inversion is replaced by the trivial operation of transposing. This is why many advanced numerical algorithms (like QR Decomposition and SVD) are designed to work with orthogonal matrices whenever possible.