Spectral Theorem
Spectral Theorem is also sometimes called Principal Axis Theorem
- In Linear Algebra a Spectrum is a set of Eigenvectors of a matrix
’'’Theorem’’’:
- Every Symmetric Matrix can be factorized as $A = Q \Lambda Q^T$
- with real eigenvalues $\Lambda$ and orthonormal eigenvectors in the columns of $Q$
The factorization is Eigendecomposition
- Spectral Theorem is a special case for symmetric matrices
- See the proof in the Symmetric Matrices article
Sum of Rank One Matrices
We can look differently at the results of Eigendecomposition of $A$
- $A = Q \Lambda Q^T = \begin{bmatrix}
| & & | \ |\mathbf q_1 & \cdots & \mathbf q_n
| & & | \ |\end{bmatrix} \begin{bmatrix} \lambda_1 & &
& \ddots &
& & \lambda_n
\end{bmatrix} \begin{bmatrix} - & \mathbf q_1^T & -
& \vdots & \ - & \mathbf q_n^T & -
\end{bmatrix}$ - can represent it as $A = Q \Lambda Q^T = \sum \lambda_i \mathbf q_i \mathbf q_i^T$ - sum of Outer Products
- each of these outer products can be seen as a Projection Matrix
-
a projection matrix is $P_i = \cfrac{\mathbf q_i \mathbf q_i^T}{| \mathbf q_i |^2} = \mathbf q_i \mathbf q_i^T$ - so symmetric matrix can be represented as a combination of mutually orthogonal projection matrices
Applications
- The Spectral Theorem guarantees that we will find an orthogonal basis in PCA
- Because the Covariance Matrix $C = \cfrac{1}{n - 1} X^T X$ is symmetric and Positive-Definite