ML Wiki
Machine Learning Wiki - A collection of ML concepts, algorithms, and resources.

Four Fundamental Subspaces

linear-algebra

Four Fundamental Subspaces

A matrix $A$ has four subspaces:

  • Column Space $C(A)$
  • Nullspace $N(A)$
  • Row Space $C(A^T)$ of $A$ is the same as Column Space of $A^T$
  • Nullspace of $A^T$ (also called “Left Nullspace”)

<img width=”40%” src=”Image” />

Some Properties

Suppose we have an $m \times n$ matrix of rank $r$

Orthogonality

  • Nullspace of $A$ is orthogonal to the row space: $N(A) \; \bot \; C(A^T)$
  • Left nullspace of $A$ is orthogonal to the column space: $N(A^T) \; \bot \; C(A)$
  • see the proof in Space Orthogonality#Row space and Nullspace

Column Space

  • $\text{dim } C(A) = r$, there are $r$ pivot columns
  • basis: columns of $A$

Row Space

  • $\text{dim } C(A^T) = r = \text{dim } C(A)$, there are $r$ pivot rows - the same dim as for Column Space
  • Let $R$ be Row Reduced Echelon Form of $A$, then $C(A^T) = C(R^T)$
  • basis: first $r$ rows of $R$

Nullspace

Left Nullspace

  • This is the nullspace of $A^T$ ($A^T$ is $n \times m$ matrix of rank $r$)
  • $\text{dim } N(A^T) = m - r$ - there are $m$ columns, $m$ variables, and $m - r$ free variables

Singular Value Decomposition

We know how to find the basis for all the subspaces

SVD finds these bases:

  • if $A V = U \Sigma$ then
  • $\mathbf v_1, \ … \ , \mathbf v_r$ is the basis for the row space $C(A^T)$
  • $\mathbf v_{r+1}, \ … \ , \mathbf v_{n}$ is the basis for the nullspace $N(A)$
  • $\mathbf u_1, \ … \ , \mathbf u_r$ is the basis for the column space $C(A)$
  • $\mathbf u_{r+1}, \ … \ , \mathbf u_{m}$ is the basis for the left nullspace $N(A^T)$

<img width=”50%” src=”Image” />

Sources

✏️ Edit on GitHub