4.10. Singular Values

In previous section we saw diagonalization of square matrices which resulted in an eigen value decomposition of the matrix. This matrix factorization is very useful yet it is not applicable in all situations. In particular, the eigen value decomposition is useless if the square matrix is not diagonalizable or if the matrix is not square at all. Moreover, the decomposition is particularly useful only for real symmetric or Hermitian matrices where the diagonalizing matrix is an $\mathbb{F}$-unitary matrix (see Definition 4.111). Otherwise, one has to consider the inverse of the diagonalizing matrix also.

Fortunately there happens to be another decomposition which applies to all matrices and it involves just $\mathbb{F}$-unitary matrices.

4.10.1. Singular value

Definition 4.131 (Singular value)

A non-negative real number $\sigma$ is a singular value for a matrix $\mathbf{A}\in {\mathbb{F}}^{m\times n}$ if and only if there exist unit-length vectors $\mathbf{u}\in {\mathbb{F}}^m$ and $\mathbf{v}\in {\mathbb{F}}^n$ such that

$$\mathbf{A}\mathbf{v}=\sigma \mathbf{u}$$

and

$${\mathbf{A}}^H\mathbf{u}=\sigma \mathbf{v}$$

hold. The vectors $\mathbf{u}$ and $\mathbf{v}$ are called left-singular and right-singular vectors for $\sigma$ respectively.

We first present the basic result of singular value decomposition. We will not prove this result completely although we will present proofs of some aspects.

Theorem 4.138 (Existence of singular values)

For every $\mathbf{A}\in {\mathbb{F}}^{m\times n}$ with $k=min(m,n)$, there exist two $\mathbb{F}$-unitary matrices $\mathbf{U}\in {\mathbb{F}}^{m\times m}$
and $\mathbf{V}\in {\mathbb{F}}^{n\times n}$ and a sequence of real numbers

$${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_k\ge 0$$

such that

(4.14)$${\mathbf{U}}^H\mathbf{A}\mathbf{V}=\mathrm{\Sigma }$$

where

$$\mathrm{\Sigma }=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_k)\in {\mathbb{F}}^{m\times n}.$$

The non-negative real numbers ${\sigma }_i$ are the singular values of $\mathbf{A}$ as per Definition 4.131.

The sequence of real numbers ${\sigma }_i$ doesn’t depend on the particular choice of $\mathbf{U}$ and $\mathbf{V}$.

$\mathrm{\Sigma }$ is rectangular with the same size as $\mathbf{A}$. The singular values of $\mathbf{A}$ lie on the principle diagonal of $\mathrm{\Sigma }$. All other entries in $\mathrm{\Sigma }$ are zero.

It is certainly possible that some of the singular values are 0 themselves.

Remark 4.22

Since ${\mathbf{U}}^H\mathbf{A}\mathbf{V}=\mathrm{\Sigma }$ hence

(4.15)$$\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H.$$

Definition 4.132 (Singular value decomposition)

The decomposition of a matrix $\mathbf{A}\in {\mathbb{F}}^{m\times n}$ given by

(4.16)$$\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$$

is known as its singular value decomposition.

Remark 4.23

When $\mathbb{F}$ is $\mathbb{R}$ then the decomposition simplifies to

(4.17)$${\mathbf{U}}^T\mathbf{A}\mathbf{V}=\mathrm{\Sigma }$$

and

$$\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T.$$

Remark 4.24

There can be at most $k=min(m,n)$ distinct singular values of $\mathbf{A}$.

Remark 4.25

We can also write

(4.18)$$\mathbf{A}\mathbf{V}=\mathbf{U}\mathrm{\Sigma }.$$

Remark 4.26 (SVD as a sum of rank 1 matrices)

Let us expand

$$\begin{array}{r}\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\left[\begin{array}{cccc}{\mathbf{u}}_1& {\mathbf{u}}_2& \dots & {\mathbf{u}}_m\end{array}\right]\left[\begin{array}{c}{\sigma }_{ij}\end{array}\right]\left[\begin{array}{c}{\mathbf{v}}_1^H\\ {\mathbf{v}}_2^H\\ ⋮\\ {\mathbf{v}}_n^H\end{array}\right]=\sum _{i=1}^m\sum _{j=1}^n{\sigma }_{ij}{\mathbf{u}}_i{\mathbf{v}}_j^H.\end{array}$$

Remark 4.27

Alternatively, let us expand

$$\begin{array}{r}\mathrm{\Sigma }={\mathbf{U}}^H\mathbf{A}\mathbf{V}=\left[\begin{array}{c}{\mathbf{u}}_1^H\\ {\mathbf{u}}_2^H\\ ⋮\\ {\mathbf{u}}_m^H\end{array}\right]\mathbf{A}\left[\begin{array}{cccc}{\mathbf{v}}_1& {\mathbf{v}}_2& \dots & {\mathbf{v}}_m\end{array}\right]=\left[\begin{array}{c}{\mathbf{u}}_i^H\mathbf{A}{\mathbf{v}}_j\end{array}\right]\end{array}$$

This gives us

$${\sigma }_{ij}={\mathbf{u}}_i^H\mathbf{A}{\mathbf{v}}_j.$$

Following lemma verifies that $\mathrm{\Sigma }$ indeed consists of singular values of $\mathbf{A}$ as per Definition 4.131.

Lemma 4.70

Let $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$ be a singular value decomposition of $\mathbf{A}$. Then the main diagonal entries of $\mathrm{\Sigma }$ are singular values. The first $k=min(m,n)$ column vectors in $\mathbf{U}$ and $\mathbf{V}$ are left and right singular vectors of $\mathbf{A}$.

Proof. We have

$$\mathbf{A}\mathbf{V}=\mathbf{U}\mathrm{\Sigma }.$$

1. Let us expand R.H.S.

   $$\mathbf{U}\mathrm{\Sigma }=\left[\begin{array}{c}\sum _{j=1}^m{\mathbf{u}}_{ij}{\sigma }_{jk}\end{array}\right]=[u_{ik}{\sigma }_k]=\left[\begin{array}{ccccccc}{\sigma }_1{\mathbf{u}}_1& {\sigma }_2{\mathbf{u}}_2& \dots & {\sigma }_k{\mathbf{u}}_k& \mathbf{0}& \dots & \mathbf{0}\end{array}\right]$$

   where $\mathbf{0}$ columns in the end appear $n-k$ times.

2. Expanding the L.H.S. we get

   $$\mathbf{A}\mathbf{V}=\left[\begin{array}{cccc}\mathbf{A}{\mathbf{v}}_1& \mathbf{A}{\mathbf{v}}_2& \dots & \mathbf{A}{\mathbf{v}}_n\end{array}\right].$$

3. Thus by comparing both sides we get

   $$\mathbf{A}{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i\text{ for }1\le i\le k$$

   and

   $$\mathbf{A}{\mathbf{v}}_i=\mathbf{0}\text{ for }k<i\le n.$$

4. Now let us start with

   $$\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H⟹{\mathbf{A}}^H=\mathbf{V}{\mathrm{\Sigma }}^H{\mathbf{U}}^H⟹{\mathbf{A}}^H\mathbf{U}=\mathbf{V}{\mathrm{\Sigma }}^H.$$

5. Let us expand R.H.S.

   $$\mathbf{V}{\mathrm{\Sigma }}^H=\left[\begin{array}{c}\sum _{j=1}^n{v}_{ij}{\sigma }_{jk}\end{array}\right]=[v_{ik}{\sigma }_k]=\left[\begin{array}{ccccccc}{\sigma }_1{\mathbf{v}}_1& {\sigma }_2{\mathbf{v}}_2& \dots & {\sigma }_k{\mathbf{v}}_k& \mathbf{0}& \dots & \mathbf{0}\end{array}\right]$$

   where $\mathbf{0}$ columns appear $m-k$ times.

6. Expanding the L.H.S. we get

   $${\mathbf{A}}^H\mathbf{U}=\left[\begin{array}{cccc}{\mathbf{A}}^H{\mathbf{u}}_1& {\mathbf{A}}^H{\mathbf{u}}_2& \dots & {\mathbf{A}}^H{\mathbf{u}}_m\end{array}\right].$$

7. Thus by comparing both sides we get

   $${\mathbf{A}}^H{\mathbf{u}}_i={\sigma }_i{\mathbf{v}}_i\text{ for }1\le i\le k$$

   and

   $${\mathbf{A}}^H{\mathbf{u}}_i=\mathbf{0}\text{ for }k<i\le m.$$

We now consider the three cases.

1. For $m=n$, we have $k=m=n$. And we get

   $$\mathbf{A}{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i,{\mathbf{A}}^H{\mathbf{u}}_i={\sigma }_i{\mathbf{v}}_i\text{ for }1\le i\le m.$$

   Thus ${\sigma }_i$ is a singular value of $\mathbf{A}$ and ${\mathbf{u}}_i$ is a left singular vector while ${\mathbf{v}}_i$ is a right singular vector.

2. For $m<n$, we have $k=m$. We get for first $m$ vectors in $\mathbf{V}$

   $$\mathbf{A}{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i,{\mathbf{A}}^H{\mathbf{u}}_i={\sigma }_i{\mathbf{v}}_i\text{ for }1\le i\le m.$$

   Finally for remaining $n-m$ vectors in $\mathbf{V}$, we can write

   $$\mathbf{A}{\mathbf{v}}_i=\mathbf{0}.$$

   They belong to the null space of $\mathbf{A}$.

3. For $m>n$, we have $k=n$. We get for first $n$ vectors in $\mathbf{U}$

   $$\mathbf{A}{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i,{\mathbf{A}}^H{\mathbf{u}}_i={\sigma }_i{\mathbf{v}}_i\text{ for }1\le i\le n.$$

   Finally for remaining $m-n$ vectors in $\mathbf{U}$, we can write

   $${\mathbf{A}}^H{\mathbf{u}}_i=\mathbf{0}.$$

Lemma 4.71

$\mathrm{\Sigma }{\mathrm{\Sigma }}^H$ is an $m\times m$ matrix given by

$$\mathrm{\Sigma }{\mathrm{\Sigma }}^H=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0)$$

where the number of $0$’s following ${\sigma }_k^2$ is $m-k$.

Lemma 4.72

${\mathrm{\Sigma }}^H\mathrm{\Sigma }$ is an $n\times n$ matrix given by

$${\mathrm{\Sigma }}^H\mathrm{\Sigma }=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0)$$

where the number of $0$’s following ${\sigma }_k^2$ is $n-k$.

Lemma 4.73 (Rank from singular value decomposition)

Let $\mathbf{A}\in {\mathbb{F}}^{m\times n}$ have a singular value decomposition given by

$$\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H.$$

Then

$$r=\mathrm{rank}(A)=\mathrm{rank}(\mathrm{\Sigma }).$$

In other words, the rank of $\mathbf{A}$ is number of nonzero singular values of $\mathbf{A}$. Since the singular values are ordered in descending order in $\mathbf{A}$ hence, the first $r$ singular values ${\sigma }_1,\dots ,{\sigma }_r$ are nonzero.

Proof. This is a straight forward application of Lemma 4.4 and Lemma 4.5. Further since only nonzero values in $\mathrm{\Sigma }$ appear on its main diagonal hence its rank is number of nonzero singular values ${\sigma }_i$.

Corollary 4.24

Let $r=\mathrm{rank}(\mathbf{A})$. Then $\mathrm{\Sigma }$ can be split as a block matrix

$$\begin{array}{r}\mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_r& \mathbf{O}\\ \mathbf{O}& \mathbf{O}\end{array}\right]\end{array}$$

where ${\mathrm{\Sigma }}_r$ is an $r\times r$ diagonal matrix of the nonzero singular values $\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_r)$. All other sub-matrices in $\mathrm{\Sigma }$ are $\mathbf{O}$.

Lemma 4.74 (Eigen values of the Gram matrix)

The eigen values of Hermitian matrix ${\mathbf{A}}^H\mathbf{A}\in {\mathbb{F}}^{n\times n}$ are ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0$ with $n-k$ $0$’s after ${\sigma }_k^2$. Moreover the eigen vectors are the columns of $\mathbf{V}$.

Proof. Expanding the Gram matrix:

$${\mathbf{A}}^H\mathbf{A}={\left(\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\right)}^H\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{V}{\mathrm{\Sigma }}^H{\mathbf{U}}^H\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{V}{\mathrm{\Sigma }}^H\mathrm{\Sigma }{\mathbf{V}}^H.$$

1. We note that ${\mathbf{A}}^H\mathbf{A}$ is Hermitian.

2. Hence $A^{H}A$ is diagonalized by $\mathbf{V}$.

3. The diagonalization of ${\mathbf{A}}^H\mathbf{A}$ is ${\mathrm{\Sigma }}^H\mathrm{\Sigma }$.

4. Thus the eigen values of ${\mathbf{A}}^H\mathbf{A}$ are ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0$ with $n-k$ $0$’s after ${\sigma }_k^2$.

5. Clearly

   $$({\mathbf{A}}^H\mathbf{A})\mathbf{V}=\mathbf{V}({\mathrm{\Sigma }}^H\mathrm{\Sigma }).$$

6. Thus columns of $\mathbf{V}$ are the eigen vectors of ${\mathbf{A}}^H\mathbf{A}$.

Lemma 4.75 (Eigen values of the frame operator)

The eigen values of Hermitian matrix $\mathbf{A}{\mathbf{A}}^H\in {\mathbb{F}}^{m\times m}$ are ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0$ with $m-k$ $0$’s after ${\sigma }_k^2$. Moreover the eigen vectors are the columns of $\mathbf{V}$.

Proof. Expanding the frame operator:

$$\mathbf{A}{\mathbf{A}}^H=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H{\left(\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\right)}^H=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{V}{\mathrm{\Sigma }}^H{\mathbf{U}}^H=\mathbf{U}\mathrm{\Sigma }{\mathrm{\Sigma }}^H{\mathbf{U}}^H.$$

1. We note that ${\mathbf{A}}^H\mathbf{A}$ is Hermitian.

2. Hence ${\mathbf{A}}^H\mathbf{A}$ is diagonalized by $\mathbf{V}$.

3. The diagonalization of ${\mathbf{A}}^H\mathbf{A}$ is ${\mathrm{\Sigma }}^H\mathrm{\Sigma }$.

4. Thus the eigen values of ${\mathbf{A}}^H\mathbf{A}$ are ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_k^2,0,0,\dots ,0$ with $m-k$ $0$’s after ${\sigma }_k^2$.

5. Clearly

   $$(\mathbf{A}{\mathbf{A}}^H)\mathbf{U}=\mathbf{U}(\mathrm{\Sigma }{\mathrm{\Sigma }}^H).$$

6. Thus columns of $\mathbf{U}$ are the eigen vectors of $\mathbf{A}{\mathbf{A}}^H$.

Observation 4.8

The Gram matrices $\mathbf{A}{\mathbf{A}}^H$ and ${\mathbf{A}}^H\mathbf{A}$ share the same eigen values except for some extra $0$s. Their eigen values are the squares of singular values of $\mathbf{A}$ and some extra $0$s. In other words singular values of $\mathbf{A}$ are the square roots of nonzero eigen values of the Gram matrices $\mathbf{A}{\mathbf{A}}^H$ or ${\mathbf{A}}^H\mathbf{A}$.

4.10.2. The Largest Singular Value

Lemma 4.76

For every $\mathbf{u}\in {\mathbb{F}}^n$ the following holds

$$\Vert \mathrm{\Sigma }\mathbf{u}{\Vert }_2\le {\sigma }_1\Vert \mathbf{u}{\Vert }_2.$$

Also for every $\mathbf{u}\in {\mathbb{F}}^m$ the following holds

$$\Vert {\mathrm{\Sigma }}^H\mathbf{u}{\Vert }_2\le {\sigma }_1\Vert \mathbf{u}{\Vert }_2.$$

Proof. Let us expand the term $\mathrm{\Sigma }\mathbf{u}$.

$$\begin{array}{r}\left[\begin{array}{ccccc}{\sigma }_1& 0& \dots & \dots & 0\\ 0& {\sigma }_2& \dots & \dots & 0\\ ⋮& ⋮& \ddots & \dots & 0\\ 0& ⋮& {\sigma }_k& \dots & 0\\ 0& 0& ⋮& \dots & 0\end{array}\right]\left[\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_k\\ ⋮\\ u_n\end{array}\right]=\left[\begin{array}{c}{\sigma }_1{u}_1\\ {\sigma }_2{u}_2\\ ⋮\\ {\sigma }_k{u}_k\\ 0\\ ⋮\\ 0\end{array}\right]\end{array}$$

Now since ${\sigma }_1$ is the largest singular value, hence

$$|{\sigma }_i{u}_i|\le |{\sigma }_1{u}_i|\mathrm{\forall }1\le i\le k.$$

Thus

$$\sum _{i=1}^n|{\sigma }_1{u}_i{|}^2\ge \sum _{i=1}^n|{\sigma }_i{u}_i{|}^2$$

or

$${\sigma }_1^2\Vert \mathbf{u}{\Vert }_2^2\ge \Vert \mathrm{\Sigma }\mathbf{u}{\Vert }_2^2.$$

The result follows.

A simpler representation of $\mathrm{\Sigma }\mathbf{u}$ can be given using the block representation of $\mathrm{\Sigma }$ in Corollary 4.24.

Let $r=\mathrm{rank}(\mathbf{A})$. Then

$$\begin{array}{r}\mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_r& \mathbf{O}\\ \mathbf{O}& \mathbf{O}\end{array}\right].\end{array}$$

We split entries in $\mathbf{u}$ as

$$\mathbf{u}=[(u_1,\dots ,u_r)(u_{r+1}\dots u_n){]}^T.$$

Then

$$\begin{array}{r}\mathrm{\Sigma }\mathbf{u}=\left[\begin{array}{c}{\mathrm{\Sigma }}_r{\left[\begin{array}{ccc}{u}_1& \dots & u_r\end{array}\right]}^T\\ \mathbf{O}{\left[\begin{array}{ccc}{u}_{r+1}& \dots & u_n\end{array}\right]}^T\end{array}\right]={\left[\begin{array}{ccccccc}{\sigma }_1{u}_1& {\sigma }_2{u}_2& \dots & {\sigma }_r{u}_r& 0& \dots & 0\end{array}\right]}^T\end{array}$$

Thus

$$\Vert \mathrm{\Sigma }\mathbf{u}{\Vert }_2^2=\sum _{i=1}^r|{\sigma }_i{u}_i{|}^2\le {\sigma }_1\sum _{i=1}^r|u_i{|}^2\le {\sigma }_1\Vert \mathbf{u}{\Vert }_2^2.$$

2nd result can also be proven similarly.

Lemma 4.77 (Upper bounds on norms of matrix vector product)

Let ${\sigma }_1$ be the largest singular value of an $m\times n$ matrix $\mathbf{A}$. Then

$$\Vert \mathbf{A}\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

Moreover

$$\Vert {\mathbf{A}}^H\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^m.$$

Proof. We have

$$\Vert \mathbf{A}\mathbf{x}{\Vert }_2=\Vert \mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{x}{\Vert }_2=\Vert \mathrm{\Sigma }{\mathbf{V}}^H\mathbf{x}{\Vert }_2$$

since $\mathbf{U}$ is unitary. Now from previous result we have

$$\Vert \mathrm{\Sigma }{\mathbf{V}}^H\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert {\mathbf{V}}^H\mathbf{x}{\Vert }_2={\sigma }_1\Vert \mathbf{x}{\Vert }_2$$

since ${\mathbf{V}}^H$ also unitary. Thus we get the result

$$\Vert \mathbf{A}\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

Similarly

$$\Vert {\mathbf{A}}^H\mathbf{x}{\Vert }_2=\Vert \mathbf{V}{\mathrm{\Sigma }}^H{\mathbf{U}}^H\mathbf{x}{\Vert }_2=\Vert {\mathrm{\Sigma }}^H{\mathbf{U}}^H\mathbf{x}{\Vert }_2$$

since $\mathbf{V}$ is unitary. Now from previous result we have

$$\Vert {\mathrm{\Sigma }}^H{\mathbf{U}}^H\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert {\mathbf{U}}^H\mathbf{x}{\Vert }_2={\sigma }_1\Vert \mathbf{x}{\Vert }_2$$

since ${\mathbf{U}}^H$ also unitary. Thus we get the result

$$\Vert {\mathbf{A}}^H\mathbf{x}{\Vert }_2\le {\sigma }_1\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^m.$$

There is a direct connection between the largest singular value and $2$-norm of a matrix (see p-norm for Matrices).

Corollary 4.25

The largest singular value of $\mathbf{A}$ is nothing but its $2$-norm; i.e.,

$${\sigma }_1=\underset{\Vert \mathbf{u}{\Vert }_2=1}{max}\Vert \mathbf{A}\mathbf{u}{\Vert }_2.$$

4.10.3. SVD and Pseudo Inverse

Lemma 4.78

Let $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$ and let $r=\mathrm{rank}(\mathbf{A})$. Let ${\sigma }_1,\dots ,{\sigma }_r$ be the $r$ nonzero singular values of $\mathbf{A}$. Then the Moore-Penrose pseudo-inverse of $\mathrm{\Sigma }$ is an $n\times m$ matrix ${\mathrm{\Sigma }}^{†}$ given by

$$\begin{array}{r}{\mathrm{\Sigma }}^{†}=\left[\begin{array}{cc}{\mathrm{\Sigma }}_r^{-1}& \mathbf{O}\\ \mathbf{O}& \mathbf{O}\end{array}\right]\end{array}$$

where ${\mathrm{\Sigma }}_r=\mathrm{diag}({\sigma }_1,\dots ,{\sigma }_r)$.

${\mathrm{\Sigma }}^{†}$ is obtained by transposing $\mathrm{\Sigma }$ and inverting all its nonzero (positive real) values.

Proof. Straight forward application of Lemma 4.24.

Corollary 4.26

The rank of $\mathrm{\Sigma }$ and its pseudoinverse ${\mathrm{\Sigma }}^{†}$ are same; i.e.,

$$\mathrm{rank}(\mathrm{\Sigma })=\mathrm{rank}({\mathrm{\Sigma }}^{†}).$$

Proof. The number of nonzero diagonal entries in $\mathrm{\Sigma }$ and ${\mathrm{\Sigma }}^{†}$ are same.

Lemma 4.79

Let $\mathbf{A}$ be an $m\times n$ matrix and let $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$ be its singular value decomposition. Let ${\mathrm{\Sigma }}^{†}$ be the pseudoinverse of $\mathrm{\Sigma }$ as per Lemma 4.78. Then the Moore-Penrose pseudo-inverse of $\mathbf{A}$ is given by

$${\mathbf{A}}^{†}=\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H.$$

Proof. As usual we verify the requirements for a Moore-Penrose pseudo-inverse as per Definition 4.42. We note that since ${\mathrm{\Sigma }}^{†}$ is the pseudo-inverse of $\mathrm{\Sigma }$ it already satisfies necessary criteria.

First requirement:

$$\mathbf{A}{\mathbf{A}}^{†}\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{U}\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{A}.$$

Second requirement:

$${\mathbf{A}}^{†}\mathbf{A}{\mathbf{A}}^{†}=\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H=\mathbf{V}{\mathrm{\Sigma }}^{†}\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H=\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H={\mathbf{A}}^{†}.$$

We now consider

$$\mathbf{A}{\mathbf{A}}^{†}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H=\mathbf{U}\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H.$$

Thus

$${\left(\mathbf{A}{\mathbf{A}}^{†}\right)}^H={\left(\mathbf{U}\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H\right)}^H=\mathbf{U}{\left(\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}\right)}^H{\mathbf{U}}^H=\mathbf{U}\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H=\mathbf{A}{\mathbf{A}}^{†}$$

since $\mathrm{\Sigma }{\mathrm{\Sigma }}^{†}$ is Hermitian. Finally we consider

$${\mathbf{A}}^{†}\mathbf{A}=\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H=\mathbf{V}{\mathrm{\Sigma }}^{†}\mathrm{\Sigma }{\mathbf{V}}^H.$$

Thus

$${\left({\mathbf{A}}^{†}\mathbf{A}\right)}^H={\left(\mathbf{V}{\mathrm{\Sigma }}^{†}\mathrm{\Sigma }{\mathbf{V}}^H\right)}^H=\mathbf{V}{\left({\mathrm{\Sigma }}^{†}\mathrm{\Sigma }\right)}^H{\mathbf{V}}^H=\mathbf{V}{\mathrm{\Sigma }}^{†}\mathrm{\Sigma }{\mathbf{V}}^H={\mathbf{A}}^{†}\mathbf{A}$$

since ${\mathrm{\Sigma }}^{†}\mathrm{\Sigma }$ is also Hermitian. This completes the proof.

Finally we can connect the singular values of $A$ with the singular values of its pseudo-inverse.

Corollary 4.27 (Rank of pseudoinverse)

The rank of any $m\times n$ matrix $\mathbf{A}$ and its pseudoinverse ${\mathbf{A}}^{†}$ are same. i.e.

$$\mathrm{rank}(\mathbf{A})=\mathrm{rank}({\mathbf{A}}^{†}).$$

Proof. We have $\mathrm{rank}(A)=\mathrm{rank}(\mathrm{\Sigma })$. Also it is easy to verify that $\mathrm{rank}({\mathbf{A}}^{†})=\mathrm{rank}({\mathrm{\Sigma }}^{†})$. So using Corollary 4.26 completes the proof.

Lemma 4.80 (Singular values of the pseudoinverse)

Let $\mathbf{A}$ be an $m\times n$ matrix and let ${\mathbf{A}}^{†}$ be its $n\times m$ pseudoinverse as per Lemma 4.79. Let $r=\mathrm{rank}(\mathbf{A})$. Let $k=min(m,n)$ denote the number of singular values while $r$ denote the number of nonzero singular values of $\mathbf{A}$. Let ${\sigma }_1,\dots ,{\sigma }_r$ be the nonzero singular values of $\mathbf{A}$. Then the number of singular values of ${\mathbf{A}}^{†}$ is same as that of $\mathbf{A}$ and the nonzero singular values of ${\mathbf{A}}^{†}$ are

$$\frac{1}{{\sigma }_1},\dots ,\frac{1}{{\sigma }_r}$$

while all other $k-r$ singular values of ${\mathbf{A}}^{†}$ are zero.

Proof. $k=min(m,n)$ denotes the number of singular values for both $\mathbf{A}$ and ${\mathbf{A}}^{†}$.

1. Since rank of $\mathbf{A}$ and ${\mathbf{A}}^{†}$ are same, hence the number of nonzero singular values is same.

2. Now look at

   $${\mathbf{A}}^{†}=\mathbf{V}{\mathrm{\Sigma }}^{†}{\mathbf{U}}^H$$

   where

   $$\begin{array}{r}{\mathrm{\Sigma }}^{†}=\left[\begin{array}{cc}{\mathrm{\Sigma }}_r^{-1}& \mathbf{O}\\ \mathbf{O}& \mathbf{O}\end{array}\right].\end{array}$$

3. Clearly ${\mathrm{\Sigma }}_r^{-1}=\mathrm{diag}(\frac{1}{{\sigma }_1},\dots ,\frac{1}{{\sigma }_r})$.

4. Thus expanding the R.H.S. we can get

   $${\mathbf{A}}^{†}=\sum _{i=1}^r\frac{1}{{\sigma }_i}{\mathbf{v}}_i{\mathbf{u}}_i^H$$

   where ${\mathbf{v}}_i$ and ${\mathbf{u}}_i$ are first $r$ columns of $\mathbf{V}$ and $\mathbf{U}$ respectively.

5. If we reverse the order of first $r$ columns of $\mathbf{U}$ and $\mathbf{V}$ and reverse the first $r$ diagonal entries of ${\mathrm{\Sigma }}^{†}$ , the R.H.S. remains the same while we are able to express ${\mathbf{A}}^{†}$ in the standard singular value decomposition form.

6. Thus $\frac{1}{{\sigma }_1},\dots ,\frac{1}{{\sigma }_r}$ are indeed the nonzero singular values of ${\mathbf{A}}^{†}$.

4.10.4. Full Column Rank Matrices

In this subsection we consider some specific results related to the singular value decomposition of a full column rank matrix.

1. We will consider $\mathbf{A}$ to be an $m\times n$ matrix in ${\mathbb{F}}^{m\times n}$ with $m\ge n$ and $\mathrm{rank}(\mathbf{A})=n$.

2. Let $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$ be its singular value decomposition.

3. From Lemma 4.73 we observe that there are $n$ nonzero singular values of $\mathbf{A}$.

4. We will call these singular values as ${\sigma }_1,{\sigma }_2,\dots ,{\sigma }_n$.

5. We will define

   $${\mathrm{\Sigma }}_n=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_n).$$

6. Clearly $\mathrm{\Sigma }$ is an $2\times 1$ block matrix given by

   $$\begin{array}{r}\mathrm{\Sigma }=\left[\begin{array}{c}{\mathrm{\Sigma }}_n\\ \mathbf{O}\end{array}\right]\end{array}$$

   where the lower $\mathbf{O}$ is an $(m-n)\times n$ zero matrix.

7. From here we obtain that ${\mathrm{\Sigma }}^H\mathrm{\Sigma }$ is an $n\times n$ matrix given by

   $${\mathrm{\Sigma }}^H\mathrm{\Sigma }={\mathrm{\Sigma }}_n^2$$

   where

   $${\mathrm{\Sigma }}_n^2=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2).$$

Lemma 4.81

Let $\mathbf{A}$ be a full column rank matrix with singular value decomposition $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$. Then ${\mathrm{\Sigma }}^H\mathrm{\Sigma }={\mathrm{\Sigma }}_n^2=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2)$ and ${\mathrm{\Sigma }}^H\mathrm{\Sigma }$ is invertible.

Proof. Since all singular values are nonzero hence ${\mathrm{\Sigma }}_n^2$ is invertible. Thus

$${\left({\mathrm{\Sigma }}^H\mathrm{\Sigma }\right)}^{-1}={\left({\mathrm{\Sigma }}_n^2\right)}^{-1}=\mathrm{diag}(\frac{1}{{\sigma }_1^2},\frac{1}{{\sigma }_2^2},\dots ,\frac{1}{{\sigma }_n^2}).$$

Lemma 4.82

Let $\mathbf{A}$ be a full column rank matrix with singular value decomposition $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$. Let ${\sigma }_1$ be its largest singular value and ${\sigma }_n$ be its smallest singular value. Then

$${\sigma }_n^2\Vert \mathbf{x}{\Vert }_2\le \Vert {\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{x}{\Vert }_2\le {\sigma }_1^2\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

Proof. Let $\mathbf{x}\in {\mathbb{F}}^n$.

1. We have

   $$\Vert {\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{x}{\Vert }_2^2=\Vert {\mathrm{\Sigma }}_n^2\mathbf{x}{\Vert }_2^2=\sum _{i=1}^n|{\sigma }_i^2{x}_i{|}^2.$$

2. Now since

   $${\sigma }_n\le {\sigma }_i\le {\sigma }_1;$$

3. hence

   $${\sigma }_n^4\sum _{i=1}^n|x_i{|}^2\le \sum _{i=1}^n|{\sigma }_i^2{x}_i{|}^2\le {\sigma }_1^4\sum _{i=1}^n|x_i{|}^2.$$

4. Thus

   $${\sigma }_n^4\Vert \mathbf{x}{\Vert }_2^2\le \Vert {\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{x}{\Vert }_2^2\le {\sigma }_1^4\Vert \mathbf{x}{\Vert }_2^2.$$

5. Applying square roots, we get

   $${\sigma }_n^2\Vert \mathbf{x}{\Vert }_2\le \Vert {\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{x}{\Vert }_2\le {\sigma }_1^2\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

We recall from Corollary 4.7 that the Gram matrix of its column vectors $\mathbf{G}={\mathbf{A}}^H\mathbf{A}$ is full rank and invertible.

Lemma 4.83 (Norm bounds for Gram matrix vector product)

Let $\mathbf{A}$ be a full column rank matrix with singular value decomposition $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$. Let ${\sigma }_1$ be its largest singular value and ${\sigma }_n$ be its smallest singular value. Then

$${\sigma }_n^2\Vert \mathbf{x}{\Vert }_2\le \Vert {\mathbf{A}}^H\mathbf{A}\mathbf{x}{\Vert }_2\le {\sigma }_1^2\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

Proof. Expanding the Gram matrix

$${\mathbf{A}}^H\mathbf{A}=(\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H{)}^H(\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H)=\mathbf{V}{\mathrm{\Sigma }}^H\mathrm{\Sigma }{\mathbf{V}}^H.$$

Let $\mathbf{x}\in {\mathbb{F}}^n$. Let

$$\mathbf{u}={\mathbf{V}}^H\mathbf{x}⟹\Vert \mathbf{u}{\Vert }_2=\Vert \mathbf{x}{\Vert }_2.$$

Let

$$\mathbf{w}={\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{u}.$$

Then from previous lemma we have

$${\sigma }_n^2\Vert \mathbf{u}{\Vert }_2\le \Vert {\mathrm{\Sigma }}^H\mathrm{\Sigma }\mathbf{u}{\Vert }_2=\Vert \mathbf{w}{\Vert }_2\le {\sigma }_1^2\Vert \mathbf{u}{\Vert }_2.$$

Finally

$${\mathbf{A}}^H\mathbf{A}\mathbf{x}=\mathbf{V}{\mathrm{\Sigma }}^H\mathrm{\Sigma }{\mathbf{V}}^H\mathbf{x}=\mathbf{V}\mathbf{w}.$$

Thus

$$\Vert {\mathbf{A}}^H\mathbf{A}\mathbf{x}{\Vert }_2=\Vert \mathbf{w}{\Vert }_2.$$

Substituting we get

$${\sigma }_n^2\Vert \mathbf{x}{\Vert }_2\le \Vert {\mathbf{A}}^H\mathbf{A}\mathbf{x}{\Vert }_2\le {\sigma }_1^2\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

There are bounds for the inverse of Gram matrix also. First let us establish the inverse of Gram matrix.

Lemma 4.84 (SVD of inverse Gram matrix)

Let $\mathbf{A}$ be a full column rank matrix with singular value decomposition $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$. Let the singular values of $\mathbf{A}$ be ${\sigma }_1,\dots ,{\sigma }_n$. Let the Gram matrix of columns of $\mathbf{A}$ be $\mathbf{G}={\mathbf{A}}^H\mathbf{A}$. Then

$${\mathbf{G}}^{-1}=\mathbf{V}\mathrm{\Psi }{\mathbf{V}}^H$$

where

$$\mathrm{\Psi }=\mathrm{diag}(\frac{1}{{\sigma }_1^2},\frac{1}{{\sigma }_2^2},\dots ,\frac{1}{{\sigma }_n^2}).$$

Proof. We have

$$\mathbf{G}=\mathbf{V}{\mathrm{\Sigma }}^H\mathrm{\Sigma }{\mathbf{V}}^H.$$

Thus

$${\mathbf{G}}^{-1}={\left(\mathbf{V}{\mathrm{\Sigma }}^H\mathrm{\Sigma }{\mathbf{V}}^H\right)}^{-1}={\left({\mathbf{V}}^H\right)}^{-1}{\left({\mathrm{\Sigma }}^H\mathrm{\Sigma }\right)}^{-1}{\mathbf{V}}^{-1}=\mathbf{V}{\left({\mathrm{\Sigma }}^H\mathrm{\Sigma }\right)}^{-1}{\mathbf{V}}^H.$$

From Lemma 4.81 we have

$$\mathrm{\Psi }={\left({\mathrm{\Sigma }}^H\mathrm{\Sigma }\right)}^{-1}=\mathrm{diag}(\frac{1}{{\sigma }_1^2},\frac{1}{{\sigma }_2^2},\dots ,\frac{1}{{\sigma }_n^2}).$$

This completes the proof.

We can now state the bounds:

Lemma 4.85 (Norm bounds for inverse Gram matrix vector product)

Let $\mathbf{A}$ be a full column rank matrix with singular value decomposition $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$. Let ${\sigma }_1$ be its largest singular value and ${\sigma }_n$ be its smallest singular value. Then

$$\frac{1}{{\sigma }_1^2}\Vert \mathbf{x}{\Vert }_2\le \Vert {\left({\mathbf{A}}^H\mathbf{A}\right)}^{-1}\mathbf{x}{\Vert }_2\le \frac{1}{{\sigma }_n^2}\Vert \mathbf{x}{\Vert }_2\mathrm{\forall }\mathbf{x}\in {\mathbb{F}}^n.$$

Proof. From Lemma 4.84 we have

$${\mathbf{G}}^{-1}={\left({\mathbf{A}}^H\mathbf{A}\right)}^{-1}=\mathbf{V}\mathrm{\Psi }{\mathbf{V}}^H$$

where

$$\mathrm{\Psi }=\mathrm{diag}(\frac{1}{{\sigma }_1^2},\frac{1}{{\sigma }_2^2},\dots ,\frac{1}{{\sigma }_n^2}).$$

Let $\mathbf{x}\in {\mathbb{F}}^n$. Let

$$\mathbf{u}={\mathbf{V}}^H\mathbf{x}⟹\Vert \mathbf{u}{\Vert }_2=\Vert \mathbf{x}{\Vert }_2.$$

Let

$$\mathbf{w}=\mathrm{\Psi }\mathbf{u}.$$

Then

$$\Vert \mathbf{w}{\Vert }_2^2=\sum _{i=1}^n{\left|\frac{1}{{\sigma }_i^2}{u}_i\right|}^2.$$

Thus

$$\frac{1}{{\sigma }_1^2}\Vert \mathbf{u}{\Vert }_2\le \Vert \mathrm{\Psi }\mathbf{u}{\Vert }_2=\Vert \mathbf{w}{\Vert }_2\le \frac{1}{{\sigma }_n^2}\Vert \mathbf{u}{\Vert }_2.$$

Finally

$${\left({\mathbf{A}}^H\mathbf{A}\right)}^{-1}\mathbf{x}=\mathbf{V}\mathrm{\Psi }{\mathbf{V}}^H\mathbf{x}=\mathbf{V}\mathbf{w}.$$

Thus

$$\Vert {\left({\mathbf{A}}^H\mathbf{A}\right)}^{-1}\mathbf{x}{\Vert }_2=\Vert \mathbf{w}{\Vert }_2.$$

Substituting we get the result.

4.10.5. Low Rank Approximation of a Matrix

Definition 4.133 (Low rank matrix)

An $m\times n$ matrix $\mathbf{A}$ is called low rank if

$$\mathrm{rank}(\mathbf{A})\ll min(m,n).$$

A matrix is low rank if the number of nonzero singular values for the matrix is much smaller than its dimensions.

Following is a simple procedure for making a low rank approximation of a given matrix $\mathbf{A}$.

• Perform the singular value decomposition of $\mathbf{A}$ given by $\mathbf{A}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$.

• Identify the singular values of $\mathbf{A}$ in $\mathrm{\Sigma }$.

• Keep the first $r$ singular values (where $r\ll min(m,n)$ is the rank of the approximation).

• Set all other singular values to 0 to obtain $\hat{\mathrm{\Sigma }}$.

• Compute $\hat{A}=U\hat{\mathrm{\Sigma }}{V}^H$.