4.2. Matrices II

This section deals with the concepts of vector spaces associated with matrices, span, rank, invertible matrices, similar matrices, gram matrices, pseudo inverses, traces and determinants.

4.2.1. Spaces Associated with a Matrix

Definition 4.32 (Column space)

The column space of a matrix is defined as the vector space spanned by columns of the matrix. Let $\mathbf{A}$ be an $m\times n$ matrix with

$$\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{a}}_1& {\mathbf{a}}_2& \dots & {\mathbf{a}}_n\end{array}\right]$$

Then the column space is given by

$$\mathcal{C}(\mathbf{A})=\{\mathbf{x}\in {\mathbb{F}}^m|\mathbf{x}=\sum _{i=1}^n{\alpha }_i{\mathbf{a}}_i\text{for some }{\alpha }_i\in \mathbb{F}\}.$$

Definition 4.33 (Row space)

The row space of a matrix is defined as the vector space spanned by rows of the matrix.

Let $\mathbf{A}$ be an $m\times n$ matrix with

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{c}{\mathbf{a}}_1^T\\ {\mathbf{a}}_2^T\\ ⋮\\ {\mathbf{a}}_m^T\end{array}\right]\end{array}$$

Then the row space is given by

$$\mathcal{R}(\mathbf{A})=\{\mathbf{x}\in {\mathbb{F}}^n|\mathbf{x}=\sum _{i=1}^m{\alpha }_i{\mathbf{a}}_i\text{for some }{\alpha }_i\in \mathbb{F}\}.$$

4.2.2. Rank

Definition 4.34 (Column rank)

The column rank of a matrix is defined as the maximum number of columns which are linearly independent. In other words column rank is the dimension of the column space of a matrix.

Definition 4.35 (Row rank)

The row rank of a matrix is defined as the maximum number of rows which are linearly independent. In other words row rank is the dimension of the row space of a matrix.

Theorem 4.21 (Equality of row and column rank)

The column rank and row rank of a matrix are equal.

Definition 4.36 (Rank)

The rank of a matrix is defined to be equal to its column rank which is equal to its row rank.

Lemma 4.1 (Rank bounds)

For an $m\times n$ matrix $\mathbf{A}$

$$0\le \mathrm{rank}(A)\le min(m,n).$$

Lemma 4.2 (Zero rank matrix)

The rank of a matrix is 0 if and only if it is a zero matrix.

Definition 4.37 (Full rank matrix)

An $m\times n$ matrix $\mathbf{A}$ is called full rank if

$$\mathrm{rank}(\mathbf{A})=min(m,n).$$

In other words it is either a full column rank matrix or a full row rank matrix or both.

Lemma 4.3 (Rank of product of two matrices)

Let $\mathbf{A}$ be an $m\times n$ matrix and $\mathbf{B}$ be an $n\times p$ matrix then

$$\mathrm{rank}(\mathbf{A}\mathbf{B})\le min(\mathrm{rank}(\mathbf{A}),\mathrm{rank}(\mathbf{B})).$$

Lemma 4.4 (Full rank post multiplication)

Let $\mathbf{A}$ be an $m\times n$ matrix and $\mathbf{B}$ be an $n\times p$ matrix. If $\mathbf{B}$ is of rank $n$ then

$$\mathrm{rank}(\mathbf{A}\mathbf{B})=\mathrm{rank}(\mathbf{A}).$$

Lemma 4.5 (Full rank pre multiplication)

Let $\mathbf{A}$ be an $m\times n$ matrix and $\mathbf{B}$ be an $n\times p$ matrix. If $\mathbf{A}$ is of rank $n$ then

$$\mathrm{rank}(\mathbf{A}\mathbf{B})=\mathrm{rank}(\mathbf{B}).$$

Lemma 4.6 (Rank of a diagonal matrix)

The rank of a diagonal matrix is equal to the number of nonzero elements on its main diagonal.

Proof. The columns which correspond to diagonal entries which are zero are zero columns. Other columns are linearly independent. The number of linearly independent rows is also the same. Hence their count gives us the rank of the matrix.

4.2.3. Invertible Matrices

We say that an $m\times n$ matrix $\mathbf{A}$ is left-invertible if there exists an $n\times m$ matrix $\mathbf{B}$ such that $\mathbf{B}\mathbf{A}=\mathbf{I}$. We say that an $m\times n$ matrix $\mathbf{A}$ is right-invertible if there exists an $n\times m$ matrix $\mathbf{B}$ such that $\mathbf{A}\mathbf{B}=\mathbf{I}$.

We say that a square matrix $\mathbf{A}$ is invertible when there exists another square matrix $\mathbf{B}$ of same size such that $\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}$. A square matrix is invertible iff it is both left and right invertible. Inverse of a square invertible matrix is denoted by ${\mathbf{A}}^{-1}$.

A special left or right inverse is the pseudo inverse, which is denoted by ${\mathbf{A}}^{†}$.

Column space of a matrix is denoted by $\mathcal{C}(\mathbf{A})$, the null space by $\mathcal{N}(\mathbf{A})$, and the row space by $\mathcal{R}(\mathbf{A})$.

We say that a matrix is symmetric when $\mathbf{A}={\mathbf{A}}^T$, conjugate symmetric or Hermitian when ${\mathbf{A}}^H=\mathbf{A}$.

When a square matrix is not invertible, we say that it is singular. A non-singular matrix is invertible.

Definition 4.38 (Invertible matrix)

A square matrix $\mathbf{A}$ is called invertible if there exists another square matrix $\mathbf{B}$ of same size such that

$$\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}.$$

The matrix $\mathbf{B}$ is called the inverse of $\mathbf{A}$ and is denoted as ${\mathbf{A}}^{-1}$.

Lemma 4.7 (Invertibility of the inverse)

If $\mathbf{A}$ is invertible then its inverse ${\mathbf{A}}^{-1}$ is also invertible and the inverse of ${\mathbf{A}}^{-1}$ is nothing but $\mathbf{A}$.

Lemma 4.8 (Invertibility of identity matrix)

Identity matrix $\mathbf{I}$ is invertible.

Proof. We can see that

$$\mathbf{I}\mathbf{I}=\mathbf{I}⟹{\mathbf{I}}^{-1}=\mathbf{I}.$$

Lemma 4.9 (Linear independence of columns of invertible matrices)

If $\mathbf{A}$ is invertible then columns of $\mathbf{A}$ are linearly independent.

Proof. Assume $\mathbf{A}$ is invertible, then there exists a matrix $\mathbf{B}$ such that

$$\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}.$$

Assume that columns of $\mathbf{A}$ are linearly dependent. Then there exists $\mathbf{u}\ne \mathbf{0}$ such that

$$\mathbf{A}\mathbf{u}=\mathbf{0}⟹\mathbf{B}\mathbf{A}\mathbf{u}=\mathbf{0}⟹\mathbf{I}\mathbf{u}=\mathbf{0}⟹\mathbf{u}=\mathbf{0}$$

a contradiction. Hence columns of $\mathbf{A}$ are linearly independent.

Lemma 4.10 (Span of columns of invertible matrix)

If an $n\times n$ matrix $\mathbf{A}$ is invertible then columns of $\mathbf{A}$ span ${\mathbb{F}}^n$.

Proof. Assume $\mathbf{A}$ is invertible, then there exists a matrix $\mathbf{B}$ such that

$$\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}.$$

1. Now let $\mathbf{x}\in {\mathbb{F}}^n$ be any arbitrary vector.

2. We need to show that there exists $\mathbf{a}\in {\mathbb{F}}^n$ such that

   $$\mathbf{x}=\mathbf{A}\mathbf{a}.$$

3. But

   $$\mathbf{x}=\mathbf{I}\mathbf{x}=\mathbf{A}\mathbf{B}\mathbf{x}=\mathbf{A}(\mathbf{B}\mathbf{x}).$$

4. Thus if we choose $\mathbf{a}=\mathbf{B}\mathbf{x}$, then

   $$\mathbf{x}=\mathbf{A}\mathbf{a}.$$

5. Thus columns of $\mathbf{A}$ span ${\mathbb{F}}^n$.

Lemma 4.11 (Columns of invertible matrix as basis)

If $\mathbf{A}$ is invertible, then columns of $\mathbf{A}$ form a basis for ${\mathbb{F}}^n$.

Proof. In ${\mathbb{F}}^n$ a basis is a set of vectors which is linearly independent and spans ${\mathbb{F}}^n$. By Lemma 4.9 and Lemma 4.10, columns of an invertible matrix $\mathbf{A}$ satisfy both conditions. Hence they form a basis.

Lemma 4.12 (Invertibility of transpose)

If $\mathbf{A}$ is invertible than ${\mathbf{A}}^T$ is invertible.

Proof. Assume $\mathbf{A}$ is invertible, then there exists a matrix $\mathbf{B}$ such that

$$\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}.$$

Applying transpose on both sides we get

$${\mathbf{B}}^T{\mathbf{A}}^T={\mathbf{A}}^T{\mathbf{B}}^T=\mathbf{I}.$$

Thus ${\mathbf{B}}^T$ is inverse of ${\mathbf{A}}^T$ and ${\mathbf{A}}^T$ is invertible.

Lemma 4.13 (Invertibility of Hermitian transpose)

If $\mathbf{A}$ is invertible than ${\mathbf{A}}^H$ is invertible.

Proof. Assume $\mathbf{A}$ is invertible, then there exists a matrix $\mathbf{B}$ such that

$$\mathbf{A}\mathbf{B}=\mathbf{B}\mathbf{A}=\mathbf{I}.$$

Applying conjugate transpose on both sides we get

$${\mathbf{B}}^H{\mathbf{A}}^H={\mathbf{A}}^H{\mathbf{B}}^H=\mathbf{I}.$$

Thus ${\mathbf{B}}^H$ is inverse of ${\mathbf{A}}^H$ and ${\mathbf{A}}^H$ is invertible.

Lemma 4.14 (Invertibility of matrix product)

If $\mathbf{A}$ and $\mathbf{B}$ are invertible then $\mathbf{A}\mathbf{B}$ is invertible.

Proof. We note that

$$(\mathbf{A}\mathbf{B})({\mathbf{B}}^{-1}{\mathbf{A}}^{-1})=\mathbf{A}(\mathbf{B}{\mathbf{B}}^{-1}){\mathbf{A}}^{-1}=\mathbf{A}\mathbf{I}{\mathbf{A}}^{-1}=\mathbf{I}.$$

Similarly

$$({\mathbf{B}}^{-1}{\mathbf{A}}^{-1})(\mathbf{A}\mathbf{B})={\mathbf{B}}^{-1}({\mathbf{A}}^{-1}\mathbf{A})\mathbf{B}={\mathbf{B}}^{-1}\mathbf{I}\mathbf{B}=\mathbf{I}.$$

Thus ${\mathbf{B}}^{-1}{\mathbf{A}}^{-1}$ is the inverse of $\mathbf{A}\mathbf{B}$.

Lemma 4.15 (Group of invertible matrices)

The set of $n\times n$ invertible matrices under the matrix multiplication operation form a group.

Proof. We verify the properties of a group

• [Closure] If $\mathbf{A}$ and $\mathbf{B}$ are invertible then $\mathbf{A}\mathbf{B}$ is invertible. Hence the set is closed.

• [Associativity] Matrix multiplication is associative.

• [Identity element] $\mathbf{I}$ is invertible and $\mathbf{A}\mathbf{I}=\mathbf{I}\mathbf{A}=\mathbf{A}$ for all invertible matrices.

• [Inverse element] If $\mathbf{A}$ is invertible then ${\mathbf{A}}^{-1}$ is also invertible.

Thus the set of invertible matrices is indeed a group under matrix multiplication.

Lemma 4.16 (Rank of an invertible matrix)

An $n\times n$ matrix $\mathbf{A}$ is invertible if and only if it is full rank i.e.

$$\mathrm{rank}(\mathbf{A})=n.$$

Corollary 4.5 (Equality of rank of a matrix and its inverse)

The rank of an invertible matrix and its inverse are same.

4.2.4. Similar Matrices

Definition 4.39 (Similar matrix)

An $n\times n$ matrix $\mathbf{B}$ is similar to an $n\times n$ matrix $\mathbf{A}$ if there exists an $n\times n$ non-singular matrix $\mathbf{C}$ such that

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}.$$

Lemma 4.17 (Symmetry of similarity)

If $\mathbf{B}$ is similar to $\mathbf{A}$ then $\mathbf{A}$ is similar to $\mathbf{B}$. Thus similarity is a symmetric relation.

Proof. We have

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}⟹\mathbf{A}=\mathbf{C}\mathbf{B}{\mathbf{C}}^{-1}⟹\mathbf{A}=({\mathbf{C}}^{-1}{)}^{-1}\mathbf{B}{\mathbf{C}}^{-1}$$

Thus there exists a matrix $\mathbf{D}={\mathbf{C}}^{-1}$ such that

$$\mathbf{A}={\mathbf{D}}^{-1}\mathbf{B}\mathbf{D}.$$

Thus $\mathbf{A}$ is similar to $\mathbf{B}$.

Lemma 4.18 (Rank of similar matrices)

Similar matrices have same rank.

Proof. Let $\mathbf{B}$ be similar to $\mathbf{A}$. Then their exists an invertible matrix $\mathbf{C}$ such that

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}.$$

Since $\mathbf{C}$ is invertible hence we have $\mathrm{rank}(\mathbf{C})=\mathrm{rank}({\mathbf{C}}^{-1})=n$. Now using Lemma 4.4

$$\mathrm{rank}(\mathbf{A}\mathbf{C})=\mathrm{rank}(\mathbf{A})$$

and using Lemma 4.5 we have

$$\mathrm{rank}({\mathbf{C}}^{-1}(\mathbf{A}\mathbf{C}))=\mathrm{rank}(\mathbf{A}\mathbf{C})=\mathrm{rank}(\mathbf{A}).$$

Thus

$$\mathrm{rank}(\mathbf{B})=\mathrm{rank}(\mathbf{A}).$$

Lemma 4.19 (Similarity as equivalence relation)

Similarity is an equivalence relation on the set of $n\times n$ matrices.

Proof. Let $\mathbf{A},\mathbf{B},\mathbf{C}$ be $n\times n$ matrices.

1. $\mathbf{A}$ is similar to itself through an invertible matrix $\mathbf{I}$.

2. If $\mathbf{A}$ is similar to $\mathbf{B}$ then $\mathbf{B}$ is similar to itself.

3. If $\mathbf{B}$ is similar to $\mathbf{A}$ via $\mathbf{P}$ s.t. $\mathbf{B}={\mathbf{P}}^{-1}\mathbf{A}\mathbf{P}$ and $\mathbf{C}$ is similar to $\mathbf{B}$ via $\mathbf{Q}$ s.t. $\mathbf{C}={\mathbf{Q}}^{-1}\mathbf{B}\mathbf{Q}$ then $\mathbf{C}$ is similar to $\mathbf{A}$ via $\mathbf{P}\mathbf{Q}$ such that $\mathbf{C}=(\mathbf{P}\mathbf{Q}{)}^{-1}\mathbf{A}(\mathbf{P}\mathbf{Q})$.

4. Thus similarity is an equivalence relation on the set of square matrices and if $\mathbf{A}$ is any $n\times n$ matrix then the set of $n\times n$ matrices similar to $\mathbf{A}$ forms an equivalence class.

4.2.5. Gram Matrices

Definition 4.40 (Gram matrix)

The Gram matrix of columns of $\mathbf{A}$ is given by

$$\mathbf{G}={\mathbf{A}}^H\mathbf{A}.$$

Definition 4.41 (Frame operator)

The frame operator is the Gram matrix of rows of $\mathbf{A}$ is given by

$$\mathbf{F}=\mathbf{A}{\mathbf{A}}^H$$

Usually when we talk about Gram matrix of a matrix we are looking at the Gram matrix of its column vectors.

Remark 4.3 (Gram matrix and frame operators for real matrices)

For real matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, the Gram matrix of its column vectors is given by ${\mathbf{A}}^T\mathbf{A}$ and the frame operator for its row vectors is given by $\mathbf{A}{\mathbf{A}}^T$.

Following results apply equally well for the real case.

Lemma 4.20 (Linear dependence of columns and Gram matrix)

The columns of a matrix are linearly dependent if and only if the Gram matrix of its column vectors ${\mathbf{A}}^H\mathbf{A}$ is not invertible.

Proof. Let $\mathbf{A}$ be an $m\times n$ matrix and $\mathbf{G}={\mathbf{A}}^H\mathbf{A}$ be the Gram matrix of its columns.

1. If columns of $\mathbf{A}$ are linearly dependent, then there exists a vector $\mathbf{u}\ne \mathbf{0}$ such that

   $$\mathbf{A}\mathbf{u}=\mathbf{0}.$$

2. Thus

   $$\mathbf{G}\mathbf{u}={\mathbf{A}}^H\mathbf{A}\mathbf{u}=\mathbf{0}.$$

3. Hence the columns of $\mathbf{G}$ are also linearly dependent.

4. Hence $\mathbf{G}$ is not invertible.

Conversely let us assume that $\mathbf{G}$ is not invertible.

1. Thus columns of $\mathbf{G}$ are dependent.

2. There exists a vector $\mathbf{v}\ne \mathbf{0}$ such that

   $$\mathbf{G}\mathbf{v}=\mathbf{0}.$$

3. Now

   $${\mathbf{v}}^H\mathbf{G}\mathbf{v}={\mathbf{v}}^H{\mathbf{A}}^H\mathbf{A}\mathbf{v}=(\mathbf{A}\mathbf{v}{)}^H(\mathbf{A}\mathbf{v})=\Vert \mathbf{A}\mathbf{v}{\Vert }_2^2.$$

4. From previous equation, we have

   $$\Vert \mathbf{A}\mathbf{v}{\Vert }_2^2=0⟹\mathbf{A}\mathbf{v}=\mathbf{0}.$$

5. Since $\mathbf{v}\ne \mathbf{0}$ hence columns of $\mathbf{A}$ are also linearly dependent.

Corollary 4.6 (Linear independence of columns and Gram matrix)

The columns of a matrix are linearly independent if and only if the Gram matrix of its column vectors ${\mathbf{A}}^H\mathbf{A}$ is invertible.

Proof. Columns of $\mathbf{A}$ can be dependent only if its Gram matrix is not invertible. Thus if the Gram matrix is invertible, then the columns of $\mathbf{A}$ are linearly independent.

The Gram matrix is not invertible only if columns of $\mathbf{A}$ are linearly dependent. Thus if columns of $\mathbf{A}$ are linearly independent then the Gram matrix is invertible.

Corollary 4.7

Let $\mathbf{A}$ be a full column rank matrix. Then ${\mathbf{A}}^H\mathbf{A}$ is invertible.

Lemma 4.21

The null space of $\mathbf{A}$ and its Gram matrix ${\mathbf{A}}^H\mathbf{A}$ coincide; i.e.,

$$\mathcal{N}(\mathbf{A})=\mathcal{N}({\mathbf{A}}^H\mathbf{A}).$$

Proof. Let $\mathbf{u}\in \mathcal{N}(\mathbf{A})$. Then

$$\mathbf{A}\mathbf{u}=\mathbf{0}⟹{\mathbf{A}}^H\mathbf{A}\mathbf{u}=\mathbf{0}.$$

Thus

$$\mathbf{u}\in \mathcal{N}({\mathbf{A}}^H\mathbf{A})⟹\mathcal{N}(\mathbf{A})\subseteq \mathcal{N}({\mathbf{A}}^H\mathbf{A}).$$

Now let $\mathbf{u}\in \mathcal{N}({\mathbf{A}}^H\mathbf{A})$. Then

$${\mathbf{A}}^H\mathbf{A}\mathbf{u}=\mathbf{0}⟹{\mathbf{u}}^H{\mathbf{A}}^H\mathbf{A}\mathbf{u}=\mathbf{0}⟹\Vert \mathbf{A}\mathbf{u}{\Vert }_2^2=0⟹\mathbf{A}\mathbf{u}=\mathbf{0}.$$

Thus we have

$$\mathcal{N}({\mathbf{A}}^H\mathbf{A})\subseteq \mathcal{N}(\mathbf{A}).$$

Lemma 4.22

The rows of a matrix $\mathbf{A}$ are linearly dependent if and only if the Gram matrix of its row vectors $\mathbf{A}{\mathbf{A}}^H$ is not invertible.

Proof. Rows of $\mathbf{A}$ are linearly dependent, if and only if columns of ${\mathbf{A}}^H$ are linearly dependent.

There exists a vector $\mathbf{v}\ne \mathbf{0}$ s.t.

$${\mathbf{A}}^H\mathbf{v}=\mathbf{0}.$$

Thus

$$\mathbf{G}\mathbf{v}=\mathbf{A}{\mathbf{A}}^H\mathbf{v}=\mathbf{0}.$$

Since $\mathbf{v}\ne \mathbf{0}$ hence $\mathbf{G}$ is not invertible.

Converse: assuming that $\mathbf{G}$ is not invertible, there exists a vector $\mathbf{u}\ne \mathbf{0}$ s.t.

$$\mathbf{G}\mathbf{u}=\mathbf{0}.$$

Now

$${\mathbf{u}}^H\mathbf{G}\mathbf{u}={\mathbf{u}}^H\mathbf{A}{\mathbf{A}}^H\mathbf{u}=({\mathbf{A}}^H\mathbf{u}{)}^H({\mathbf{A}}^H\mathbf{u})=\Vert {\mathbf{A}}^H\mathbf{u}{\Vert }_2^2=0⟹{\mathbf{A}}^H\mathbf{u}=\mathbf{0}.$$

Since $\mathbf{u}\ne \mathbf{0}$ hence columns of ${\mathbf{A}}^H$ and consequently rows of $\mathbf{A}$ are linearly dependent.

Corollary 4.8

The rows of a matrix $\mathbf{A}$ are linearly independent if and only if the Gram matrix of its row vectors $A{A}^H$ is invertible.

Corollary 4.9

Let $\mathbf{A}$ be a full row rank matrix. Then $\mathbf{A}{\mathbf{A}}^H$ is invertible.

4.2.6. Pseudo Inverses

Definition 4.42 (Moore Penrose pseudo inverse)

Let $\mathbf{A}$ be an $m\times n$ matrix. An $n\times m$ matrix ${\mathbf{A}}^{†}$ is called its Moore-Penrose pseudo-inverse if it satisfies all of the following criteria:

• $\mathbf{A}{\mathbf{A}}^{†}\mathbf{A}=\mathbf{A}$.

• ${\mathbf{A}}^{†}\mathbf{A}{\mathbf{A}}^{†}={\mathbf{A}}^{†}$.

• ${\left(\mathbf{A}{\mathbf{A}}^{†}\right)}^H=\mathbf{A}{\mathbf{A}}^{†}$ i.e. $\mathbf{A}{\mathbf{A}}^{†}$ is Hermitian.

• $({\mathbf{A}}^{†}\mathbf{A}{)}^H={\mathbf{A}}^{†}\mathbf{A}$ i.e. ${\mathbf{A}}^{†}\mathbf{A}$ is Hermitian.

Theorem 4.22 (Existence and uniqueness of Moore Penrose pseudo inverse)

For any matrix $\mathbf{A}$ there exists precisely one matrix ${\mathbf{A}}^{†}$ which satisfies all the requirements in Definition 4.42.

We omit the proof for this. The pseudo-inverse can actually be obtained by the singular value decomposition of $\mathbf{A}$. This is shown in Lemma 4.79.

Lemma 4.23 (Moore Penrose pseudo inverse of a diagonal matrix)

Let $\mathbf{D}=\mathrm{diag}(d_1,d_2,\dots ,d_n)$ be an $n\times n$ diagonal matrix. Then its Moore-Penrose pseudo-inverse is ${\mathbf{D}}^{†}=\mathrm{diag}(c_1,c_2,\dots ,c_n)$ where

$$\begin{array}{r}{c}_i=\{\begin{array}{ll}\frac{1}{d_i}& \text{if }{d}_i\ne 0;\\ 0& \text{if }{d}_i=0.\end{array}\end{array}$$

Proof. We note that ${\mathbf{D}}^{†}\mathbf{D}=\mathbf{D}{\mathbf{D}}^{†}=\mathbf{F}=\mathrm{diag}(f_1,f_2,\dots f_n)$ where

$$\begin{array}{r}{f}_i=\{\begin{array}{ll}1& \text{if }{d}_i\ne 0;\\ 0& \text{if }{d}_i=0.\end{array}\end{array}$$

We now verify the requirements in Definition 4.42.

$$\mathbf{D}{\mathbf{D}}^{†}\mathbf{D}=\mathbf{F}\mathbf{D}=\mathbf{D}.$$

$${\mathbf{D}}^{†}\mathbf{D}{\mathbf{D}}^{†}=\mathbf{F}{\mathbf{D}}^{†}={\mathbf{D}}^{†}.$$

${\mathbf{D}}^{†}\mathbf{D}=\mathbf{D}{\mathbf{D}}^{†}=\mathbf{F}$ is a diagonal hence Hermitian matrix.

Lemma 4.24 (Moore Penrose pseudo inverse of a rectangular diagonal matrix)

Let $\mathbf{D}=\mathrm{diag}(d_1,d_2,\dots ,d_p)$ be an $m\times n$ rectangular diagonal matrix where $p=min(m,n)$. Then its Moore-Penrose pseudo-inverse is an $n\times m$ rectangular diagonal matrix ${\mathbf{D}}^{†}=\mathrm{diag}(c_1,c_2,\dots ,c_p)$ where

$$\begin{array}{r}{c}_i=\{\begin{array}{ll}\frac{1}{d_i}& \text{if }{d}_i\ne 0;\\ 0& \text{if }{d}_i=0.\end{array}\end{array}$$

Proof. We note that

$$\mathbf{F}={\mathbf{D}}^{†}\mathbf{D}=\mathrm{diag}(f_1,f_2,\dots f_n)$$

is an $n\times n$ matrix where

$$\begin{array}{r}{f}_i=\{\begin{array}{ll}1& \text{if }{d}_i\ne 0;\\ 0& \text{if }{d}_i=0;\\ 0& \text{if }i>p.\end{array}\end{array}$$

$\mathbf{G}=\mathbf{D}{\mathbf{D}}^{†}=\mathrm{diag}(g_1,g_2,\dots g_n)$ is an $m\times m$ matrix where

$$\begin{array}{r}{g}_i=\{\begin{array}{ll}1& \text{if }{d}_i\ne 0;\\ 0& \text{if }{d}_i=0;\\ 0& \text{if }i>p.\end{array}\end{array}$$

We now verify the requirements in Definition 4.42.

$$\mathbf{D}{\mathbf{D}}^{†}\mathbf{D}=\mathbf{D}\mathbf{F}=\mathbf{D}.$$

$${\mathbf{D}}^{†}\mathbf{D}{\mathbf{D}}^{†}={\mathbf{D}}^{†}\mathbf{G}={\mathbf{D}}^{†}.$$

$\mathbf{F}={\mathbf{D}}^{†}\mathbf{D}$ and $\mathbf{G}=\mathbf{D}{\mathbf{D}}^{†}$ are both real diagonal hence Hermitian matrices.

Lemma 4.25 (Moore Penrose pseudo inverse of full column rank matrices)

If $\mathbf{A}$ is full column rank then its Moore-Penrose pseudo-inverse is given by

$${\mathbf{A}}^{†}=({\mathbf{A}}^H\mathbf{A}{)}^{-1}{\mathbf{A}}^H.$$

It is a left inverse of $\mathbf{A}$.

Proof. By Corollary 4.7 ${\mathbf{A}}^H\mathbf{A}$ is invertible. First of all we verify that it is a left inverse.

$${\mathbf{A}}^{†}\mathbf{A}=({\mathbf{A}}^H\mathbf{A}{)}^{-1}{\mathbf{A}}^H\mathbf{A}=\mathbf{I}.$$

We now verify all the properties.

$$\mathbf{A}{\mathbf{A}}^{†}\mathbf{A}=\mathbf{A}\mathbf{I}=\mathbf{A}.$$

$${\mathbf{A}}^{†}\mathbf{A}{\mathbf{A}}^{†}=\mathbf{I}{\mathbf{A}}^{†}={\mathbf{A}}^{†}.$$

Hermitian properties:

$${\left(\mathbf{A}{\mathbf{A}}^{†}\right)}^H={(\mathbf{A}({\mathbf{A}}^H\mathbf{A}{)}^{-1}{\mathbf{A}}^H)}^H=(\mathbf{A}({\mathbf{A}}^H\mathbf{A}{)}^{-1}{\mathbf{A}}^H)=\mathbf{A}{\mathbf{A}}^{†}.$$

$$({\mathbf{A}}^{†}\mathbf{A}{)}^H={\mathbf{I}}^H=\mathbf{I}={\mathbf{A}}^{†}\mathbf{A}.$$

Lemma 4.26 (Moore Penrose pseudo inverse of full row rank matrices)

If $\mathbf{A}$ is full row rank then its Moore-Penrose pseudo-inverse is given by

$${\mathbf{A}}^{†}={\mathbf{A}}^H(\mathbf{A}{\mathbf{A}}^H{)}^{-1}.$$

It is a right inverse of $\mathbf{A}$.

Proof. By Corollary 4.9 $\mathbf{A}{\mathbf{A}}^H$ is invertible. First of all we verify that it is a right inverse.

$$\mathbf{A}{\mathbf{A}}^{†}=\mathbf{A}{\mathbf{A}}^H(\mathbf{A}{\mathbf{A}}^H{)}^{-1}=\mathbf{I}.$$

We now verify all the properties.

$$\mathbf{A}{\mathbf{A}}^{†}\mathbf{A}=\mathbf{I}\mathbf{A}=\mathbf{A}.$$

$${\mathbf{A}}^{†}\mathbf{A}{\mathbf{A}}^{†}={\mathbf{A}}^{†}\mathbf{I}={\mathbf{A}}^{†}.$$

Hermitian properties:

$${\left(\mathbf{A}{\mathbf{A}}^{†}\right)}^H={\mathbf{I}}^H=\mathbf{I}=\mathbf{A}{\mathbf{A}}^{†}.$$

$$({\mathbf{A}}^{†}\mathbf{A}{)}^H={({\mathbf{A}}^H(\mathbf{A}{\mathbf{A}}^H{)}^{-1}\mathbf{A})}^H={\mathbf{A}}^H(\mathbf{A}{\mathbf{A}}^H{)}^{-1}\mathbf{A}={\mathbf{A}}^{†}\mathbf{A}.$$

4.2.7. Trace

Definition 4.43 (Trace of a square matrix)

The trace of a square matrix is defined as the sum of the entries on its main diagonal. Let $\mathbf{A}$ be an $n\times n$ matrix, then

$$\mathrm{tr}(\mathbf{A})=\sum _{i=1}^n{a}_{ii}$$

where $\mathrm{tr}(\mathbf{A})$ denotes the trace of $\mathbf{A}$.

Lemma 4.27

The trace of a square matrix and its transpose are equal.

$$\mathrm{tr}(\mathbf{A})=\mathrm{tr}({\mathbf{A}}^T).$$

Lemma 4.28

Trace of sum of two square matrices is equal to the sum of their traces.

$$\mathrm{tr}(\mathbf{A}+\mathbf{B})=\mathrm{tr}(\mathbf{A})+\mathrm{tr}(\mathbf{B}).$$

Lemma 4.29 (Trace product rule)

Let $\mathbf{A}$ be an $m\times n$ matrix and $\mathbf{B}$ be an $n\times m$ matrix. Then

$$\mathrm{tr}(\mathbf{A}\mathbf{B})=\mathrm{tr}(\mathbf{B}\mathbf{A}).$$

Proof. Let $\mathbf{A}\mathbf{B}=\mathbf{C}=[c_{ij}]$. Then

$$c_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}.$$

Thus

$$c_{ii}=\sum _{k=1}^n{a}_{ik}{b}_{ki}.$$

Now

$$\mathrm{tr}(\mathbf{C})=\sum _{i=1}^m{c}_{ii}=\sum _{i=1}^m\sum _{k=1}^n{a}_{ik}{b}_{ki}=\sum _{k=1}^n\sum _{i=1}^m{a}_{ik}{b}_{ki}=\sum _{k=1}^n\sum _{i=1}^m{b}_{ki}{a}_{ik}.$$

Let $\mathbf{B}\mathbf{A}=\mathbf{D}=[d_{ij}]$. Then

$$d_{ij}=\sum _{k=1}^m{b}_{ik}{a}_{kj}.$$

Thus

$$d_{ii}=\sum _{k=1}^m{b}_{ik}{a}_{ki}.$$

Hence

$$\mathrm{tr}(D)=\sum _{i=1}^n{d}_{ii}=\sum _{i=1}^n\sum _{k=1}^m{b}_{ik}{a}_{ki}=\sum _{i=1}^m\sum _{k=1}^n{b}_{ki}{a}_{ik}.$$

This completes the proof.

Lemma 4.30 (Trace triple product rule)

Let $\mathbf{A}\in {\mathbb{F}}^{m\times n}$, $\mathbf{B}\in {\mathbb{F}}^{n\times p}$, $\mathbf{C}\in {\mathbb{F}}^{p\times m}$ be three matrices. Then

$$\mathrm{tr}(\mathbf{A}\mathbf{B}\mathbf{C})=\mathrm{tr}(\mathbf{B}\mathbf{C}\mathbf{A})=\mathrm{tr}(\mathbf{C}\mathbf{A}\mathbf{B}).$$

Proof. Let $\mathbf{A}\mathbf{B}=\mathbf{D}$. Then

$$\mathrm{tr}(\mathbf{A}\mathbf{B}\mathbf{C})=\mathrm{tr}(\mathbf{D}\mathbf{C})=\mathrm{tr}(\mathbf{C}\mathbf{D})=\mathrm{tr}(\mathbf{C}\mathbf{A}\mathbf{B}).$$

Similarly the other result can be proved.

Lemma 4.31

Trace of similar matrices is equal.

Proof. Let $\mathbf{B}$ be similar to $\mathbf{A}$. Thus

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}$$

for some invertible matrix $\mathbf{C}$. Then

$$\mathrm{tr}(\mathbf{B})=\mathrm{tr}({\mathbf{C}}^{-1}\mathbf{A}\mathbf{C})=\mathrm{tr}(\mathbf{C}{\mathbf{C}}^{-1}\mathbf{A})=\mathrm{tr}(\mathbf{A}).$$

We used Lemma 4.29.

4.2.8. Determinants

Following are some results on determinant of a square matrix $\mathbf{A}$.

Lemma 4.32 (Determinant and scalar multiplication)

$$det(\alpha \mathbf{A})={\alpha }^{n}det(\mathbf{A}).$$

Lemma 4.33

Determinant of a square matrix and its transpose are equal.

$$det(\mathbf{A})=det({\mathbf{A}}^T).$$

Lemma 4.34

Let $\mathbf{A}$ be a complex square matrix. Then

$$det({\mathbf{A}}^H)=\overline{det(\mathbf{A})}.$$

Proof. We proceed as follows:

$$det({\mathbf{A}}^H)=det({\bar{\mathbf{A}}}^T)=det(\bar{\mathbf{A}})=\overline{det(\mathbf{A})}.$$

Lemma 4.35 (Determinant product rule)

Let $\mathbf{A}$ and $\mathbf{B}$ be two $n\times n$ matrices. Then

$$det(\mathbf{A}\mathbf{B})=det(\mathbf{A})det(\mathbf{B}).$$

Lemma 4.36 (Determinant of inverse)

Let $\mathbf{A}$ be an invertible matrix. Then

$$det({\mathbf{A}}^{-1})=\frac{1}{det(\mathbf{A})}.$$

Lemma 4.37 (Determinant power rule)

$$det({\mathbf{A}}^p)={(det(\mathbf{A}))}^p.$$

Lemma 4.38 (Determinant of triangular matrices)

Determinant of a triangular matrix is the product of its diagonal entries; i.e., if $\mathbf{A}$ is upper or lower triangular matrix then

$$det(\mathbf{A})=\prod _{i=1}^n{a}_{ii}.$$

Lemma 4.39 (Determinant of diagonal matrices)

Determinant of a diagonal matrix is the product of its diagonal entries; i.e., if $\mathbf{A}$ is a diagonal matrix then

$$det(\mathbf{A})=\prod _{i=1}^n{a}_{ii}.$$

Lemma 4.40 (Determinants of similar matrices)

Determinant of similar matrices is equal.

Proof. Let $\mathbf{B}$ be similar to $\mathbf{A}$. Thus

$$\mathbf{B}={\mathbf{C}}^{-1}\mathbf{A}\mathbf{C}$$

for some invertible matrix $\mathbf{C}$. Hence

$$det(\mathbf{B})=det({\mathbf{C}}^{-1}\mathbf{A}\mathbf{C})=det({\mathbf{C}}^{-1})det(\mathbf{A})det(\mathbf{C}).$$

Now

$$det({\mathbf{C}}^{-1})det(\mathbf{A})det(\mathbf{C})=\frac{1}{det(\mathbf{C})}det(\mathbf{A})det(\mathbf{C})=det(\mathbf{A}).$$

We used Lemma 4.35 and Lemma 4.36.

Lemma 4.41

Let $\mathbf{u}$ and $\mathbf{v}$ be vectors in ${\mathbb{F}}^n$. Then

$$det(\mathbf{I}+\mathbf{u}{\mathbf{v}}^T)=1+{\mathbf{u}}^T\mathbf{v}.$$

Lemma 4.42 (Determinant of perturbation of an identity matrix)

Let $\mathbf{A}$ be a square matrix and let $ϵ\approx 0$. Then

$$det(\mathbf{I}+ϵ\mathbf{A})\approx 1+ϵ\mathrm{tr}(\mathbf{A}).$$