Spectral Theorem

Spectral Theorem is also sometimes called Principal Axis Theorem


Theorem:

  • Every Symmetric Matrix can be factorized as $A = Q \Lambda Q^T$
  • with real eigenvalues $\Lambda$ and orthonormal eigenvectors in the columns of $Q$


The factorization is Eigendecomposition

  • Spectral Theorem is a special case for symmetric matrices
  • See the proof in the Symmetric Matrices article


Sum of Rank One Matrices

We can look differently at the results of Eigendecomposition of $A$

  • [math]A = Q \Lambda Q^T = \begin{bmatrix} | & & | \\ \mathbf q_1 & \cdots & \mathbf q_n \\ | & & | \\ \end{bmatrix} \begin{bmatrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_n \\ \end{bmatrix} \begin{bmatrix} - & \mathbf q_1^T & - \\ & \vdots & \\ - & \mathbf q_n^T & - \\ \end{bmatrix}[/math]
  • can represent it as $A = Q \Lambda Q^T = \sum \lambda_i \mathbf q_i \mathbf q_i^T$ - sum of Outer Products
  • each of these outer products can be seen as a Projection Matrix
  • a projection matrix is $P_i = \cfrac{\mathbf q_i \mathbf q_i^T}{\| \mathbf q_i \|^2} = \mathbf q_i \mathbf q_i^T$
  • so symmetric matrix can be represented as a combination of mutually orthogonal projection matrices


Applications

Principal Component Analysis

  • The Spectral Theorem guarantees that we will find an orthogonal basis in PCA
  • Because the Covariance Matrix $C = \cfrac{1}{n - 1} X^T X$ is symmetric and Positive-Definite


Sources