1.9. Matrices

This section provides some basic definitions, notation and results on the theory of matrices.

1.9.1. Basic Definitions

Definition 1.98 (Matrix)

An $m\times n$ matrix $\mathbf{A}$ is a rectangular array of numbers.

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ a_{21}& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{ms1}& a_{m2}& \dots & a_{mn}\end{array}\right].\end{array}$$

The numbers in a matrix are called its elements.

The matrix consists of $m$ rows and $n$ columns. The entry in $i$-th row and $j$-th column is referred with the notation $a_{ij}$.

If all the elements of a matrix are real, then we call it a real matrix.

If any of the elements of the matrix is complex, then we call it a complex matrix.

A matrix is often written in short as $\mathbf{A}=(a_{ij})$.

Matrices are denoted by bold capital letters $\mathbf{A}$, $\mathbf{B}$ etc.. They can be rectangular with $m$ rows and $n$ columns. Their elements or entries are referred to with small letters $a_{ij}$, $b_{ij}$ etc. where $i$ denotes the $i$-th row of matrix and $j$ denotes the $j$-th column of matrix.

Definition 1.99 (The set of matrices)

The set of all real matrices of shape $m\times n$ is denoted by ${\mathbb{R}}^{m\times n}$.

The set of all complex matrices of shape $m\times n$ is denoted by ${\mathbb{C}}^{m\times n}$.

Definition 1.100 (Square matrix)

An $m\times n$ matrix is called square matrix if $m=n$.

Definition 1.101 (Tall matrix)

An $m\times n$ matrix is called tall matrix if $m>n$ i.e. the number of rows is greater than columns.

Definition 1.102 (Wide matrix)

An $m\times n$ matrix is called wide matrix if $m<n$ i.e. the number of columns is greater than rows.

Definition 1.103 (Vector)

A vector is an $n$-tuple of numbers written as:

$$\mathbf{v}=(v_1,v_2,\dots ,v_n).$$

If all the numbers are real, then it is called a real vector belonging to the set ${\mathbb{R}}^n$. If any of the numbers is complex, then it is called a complex vector belonging to the set ${\mathbb{C}}^n$. The numbers in a vector are called its components.

Sometimes, we may use a notation without commas.

$$\mathbf{v}=\left(\begin{array}{cccc}{v}_1& v_2& \dots & v_n\end{array}\right).$$

Definition 1.104 (Column vector)

A matrix with shape $m\times 1$ is called a column vector.

Definition 1.105 (Row vector)

A matrix with shape $1\times n$ is called a row vector.

Note

It should be easy to see that ${\mathbb{R}}^{m\times 1}$ and ${\mathbb{R}}^m$ are same sets. Similarly, ${\mathbb{R}}^{1\times n}$ and ${\mathbb{R}}^n$ are same sets.

A row or column vector can easily be written as an $n$-tuple.

Definition 1.106 (Main diagonal)

Let $\mathbf{A}=[a_{ij}]$ be an $m\times n$ matrix. The main diagonal consists of entries $a_{ij}$ where $i=j$; i.e., the main diagonal is $\{a_{11},a_{22},\dots ,a_{kk}\}$ where $k=min(m,n)$.

Main diagonal is also known as leading diagonal, major diagonal primary diagonal or principal diagonal.

The entries of $\mathbf{A}$ which are not on the main diagonal are known as off diagonal entries.

Definition 1.107 (Diagonal matrix)

A diagonal matrix is a matrix (usually a square matrix) whose entries outside the main diagonal are zero.

Whenever we refer to a diagonal matrix which is not square, we will use the term rectangular diagonal matrix.

A square diagonal matrix $A$ is also written as $\mathrm{diag}(a_{11},a_{22},\dots ,a_{nn})$ which lists only the diagonal (non-zero) entries in $\mathbf{A}$.

If not specified, the square matrices will be of size $n\times n$ and rectangular matrices will be of size $m\times n$. If not specified the vectors (column vectors) will be of size $n\times 1$ and belong to either ${\mathbb{R}}^n$ or ${\mathbb{C}}^n$. Corresponding row vectors will be of size $1\times n$.

1.9.2. Matrix Operations

Definition 1.108 (Matrix addition)

Let $\mathbf{A}$ and $\mathbf{B}$ be two matrices with same shape $m\times n$. Then, their addition is defined as:

$$\mathbf{A}+\mathbf{B}=(a_{ij})+(b_{ij})\triangleq (a_{ij}+b_{ij}).$$

Definition 1.109 (Scalar multiplication)

Let $\mathbf{A}$ be a matrix of shape $m\times n$ and $\lambda$ be a scalar. The product of the matrix $\mathbf{A}$ with the scalar $\lambda$ is defined as:

$$\lambda \mathbf{A}=\mathbf{A}\lambda \triangleq (\lambda a_{ij}).$$

Theorem 1.35 (Properties of matrix addition and scalar multiplication)

Let $\mathbf{A},\mathbf{B},\mathbf{C}$ be matrices of shape $m\times n$. Let $\lambda ,\mu$ be scalars. Then:

1. Matrix addition is commutative: $\mathbf{A}+\mathbf{B}=\mathbf{B}+\mathbf{A}$.

2. Matrix addition is associative: $\mathbf{A}+(\mathbf{B}+\mathbf{C})=(\mathbf{A}+\mathbf{B})+\mathbf{C}$.

3. Addition in scalars distributes over scalar multiplication: $(\lambda +\mu )\mathbf{A}=\lambda \mathbf{A}+\mu \mathbf{A}$.

4. Scalar multiplication distributes over addition of matrices: $\lambda (\mathbf{A}+\mathbf{B})=\lambda \mathbf{A}+\lambda \mathbf{B}$.

5. Multiplication in scalars commutes with scalar multiplication: $(\lambda \mu )\mathbf{A}=\lambda (\mu \mathbf{A})$.

6. There exists a matrix with all elements being zero denoted by $\mathbf{O}$ such that $\mathbf{A}+\mathbf{O}=\mathbf{O}+\mathbf{A}=\mathbf{A}$.

7. Existence of additive inverse: $\mathbf{A}+(-1)\mathbf{A}=\mathbf{O}$.

Definition 1.110 (Matrix multiplication)

If $\mathbf{A}$ is an $m\times n$ matrix and $\mathbf{B}$ is an $n\times p$ matrix (thus, $\mathbf{A}$ has same number of columns as $\mathbf{B}$ has rows), then we define the product of $\mathbf{A}$ and $\mathbf{B}$ as:

$$\mathbf{A}\mathbf{B}\triangleq \left(\sum _{k=1}^n{a}_{ik}{b}_{kj}\right).$$

This binary operation is known as matrix multiplication. The product matrix has the shape $m\times p$. Its $i,j$-th element is $\sum _{k=1}^n{a}_{ik}{b}_{kj}$ obtained by multiplying the $i$-th row of $A$ with the $j$-th column of $B$ element by element and then summing over them.

Theorem 1.36 (Properties of matrix multiplication)

Let $\mathbf{A},\mathbf{B},\mathbf{C}$ be matrices of appropriate shape.

1. Matrix multiplication is associative: $\mathbf{A}(\mathbf{B}\mathbf{C})=(\mathbf{A}\mathbf{B})\mathbf{C}$.

2. Matrix multiplication distributes over matrix addition: $\mathbf{A}(\mathbf{B}+\mathbf{C})=\mathbf{A}\mathbf{B}+\mathbf{A}\mathbf{C}$ and $(\mathbf{A}+\mathbf{B})\mathbf{C}=\mathbf{A}\mathbf{C}+\mathbf{B}\mathbf{C}$.

1.9.3. Transpose

The transpose of a matrix $\mathbf{A}$ is denoted by ${\mathbf{A}}^T$ while the Hermitian transpose is denoted by ${\mathbf{A}}^H$. For real matrices ${\mathbf{A}}^T={\mathbf{A}}^H$.

For statements which are valid both for real and complex matrices, sometimes we might say that matrices belong to ${\mathbb{F}}^{m\times n}$ while the scalars belong to the field $\mathbb{F}$ and vectors belong to ${\mathbb{F}}^n$ where $\mathbb{F}$ refers to either the field of real numbers or the field of complex numbers. Most results from matrix analysis are written only for ${\mathbb{C}}^{m\times n}$ while still being applicable for ${\mathbb{R}}^{m\times n}$.

Identity matrix for ${\mathbb{F}}^{n\times n}$ is denoted as ${\mathbf{I}}_n$ or simply $\mathbf{I}$ whenever the size is clear from context.

Sometimes we will write a matrix in terms of its column vectors. We will use the notation

$$\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{a}}_1& {\mathbf{a}}_2& \dots & {\mathbf{a}}_n\end{array}\right]$$

indicating $n$ columns.

When we write a matrix in terms of its row vectors, we will use the notation

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{c}{\mathbf{a}}_1^T\\ {\mathbf{a}}_2^T\\ ⋮\\ {\mathbf{a}}_m^T\end{array}\right]\end{array}$$

indicating $m$ rows with ${\mathbf{a}}_i$ being column vectors whose transposes form the rows of $\mathbf{A}$.

1.9.4. Symmetric Matrices

Definition 1.111 (Symmetric matrix)

A symmetric matrix is a matrix $\mathbf{X}\in {\mathbb{F}}^{n\times n}$ which satisfies $\mathbf{X}={\mathbf{X}}^T$.

We define the set of symmetric $n\times n$ matrices as

$${\mathbb{S}}^n=\{\mathbf{X}\in {\mathbb{R}}^{n\times n}|\mathbf{X}={\mathbf{X}}^T\}.$$

1.9.5. Dot Products

The inner product or dot product of two column / row vectors $\mathbf{u}$ and $\mathbf{v}$ belonging to ${\mathbb{R}}^n$ is defined as

(1.1)$$\mathbf{u}\cdot \mathbf{v}=⟨\mathbf{u},\mathbf{v}⟩=\sum _{i=1}^n{u}_i{v}_i.$$

The inner product or dot product of two column / row vectors $\mathbf{u}$ and $\mathbf{v}$ belonging to ${\mathbb{C}}^n$ is defined as

(1.2)$$\mathbf{u}\cdot \mathbf{v}=⟨\mathbf{u},\mathbf{v}⟩=\sum _{i=1}^n{u}_i\overline{v_i}.$$

1.9.6. Block Matrices

Definition 1.112 (Block matrix)

A block matrix is a matrix whose entries themselves are matrices with following constraints

• Entries in every row are matrices with same number of rows.

• Entries in every column are matrices with same number of columns.

Let $\mathbf{A}$ be an $m\times n$ block matrix. Then

$$\begin{array}{r}\mathbf{A}=\left[\begin{array}{cccc}{\mathbf{A}}_{11}& {\mathbf{A}}_{12}& \dots & {\mathbf{A}}_{1n}\\ {\mathbf{A}}_{21}& {\mathbf{A}}_{22}& \dots & {\mathbf{A}}_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathbf{A}}_{m1}& {\mathbf{A}}_{m2}& \dots & {\mathbf{A}}_{mn}\end{array}\right]\end{array}$$

where ${\mathbf{A}}_{ij}$ is a matrix with $r_i$ rows and $c_j$ columns.

A block matrix is also known as a partitioned matrix.

Example 1.23 ($2x2$ block matrices)

Quite frequently we will be using $2x2$ block matrices.

$$\begin{array}{r}\mathbf{P}=\left[\begin{array}{cc}{\mathbf{P}}_{11}& {\mathbf{P}}_{12}\\ {\mathbf{P}}_{21}& {\mathbf{P}}_{22}\end{array}\right].\end{array}$$

An example

$$\begin{array}{r}P=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]\end{array}$$

We have

$$\begin{array}{r}{\mathbf{P}}_{11}=\left[\begin{array}{cc}a& b\\ d& e\end{array}\right]{\mathbf{P}}_{12}=\left[\begin{array}{c}c\\ f\end{array}\right]{\mathbf{P}}_{21}=\left[\begin{array}{cc}g& h\end{array}\right]{\mathbf{P}}_{22}=\left[\begin{array}{c}i\end{array}\right]\end{array}$$

• ${\mathbf{P}}_{11}$ and ${\mathbf{P}}_{12}$ have $2$ rows.

• ${\mathbf{P}}_{21}$ and ${\mathbf{P}}_{22}$ have $1$ row.

• ${\mathbf{P}}_{11}$ and ${\mathbf{P}}_{21}$ have $2$ columns.

• ${\mathbf{P}}_{12}$ and ${\mathbf{P}}_{22}$ have $1$ column.

Lemma 1.1 (Shape of a block matrix)

Let $\mathbf{A}=[{\mathbf{A}}_{ij}]$ be an $m\times n$ block matrix with ${\mathbf{A}}_{ij}$ being an $r_i\times c_j$ matrix. Then $\mathbf{A}$ is an $r\times c$ matrix where

$$r=\sum _{i=1}^m{r}_i$$

and

$$c=\sum _{j=1}^n{c}_j.$$

Sometimes it is convenient to think of a regular matrix as a block matrix whose entries are $1\times 1$ matrices themselves.

Definition 1.113 (Multiplication of block matrices)

Let $\mathbf{A}=[{\mathbf{A}}_{ij}]$ be an $m\times n$ block matrix with ${\mathbf{A}}_{ij}$ being a $p_i\times q_j$ matrices. Let $\mathbf{B}=[{\mathbf{B}}_{jk}]$ be an $n\times p$ block matrix with ${\mathbf{B}}_{jk}$ being a $q_j\times r_k$ matrices.

Then the two block matrices are compatible for multiplication and their multiplication is defined by $\mathbf{C}=\mathbf{A}\mathbf{B}=[{\mathbf{C}}_{ik}]$ where

$${\mathbf{C}}_{ik}=\sum _{j=1}^n{\mathbf{A}}_{ij}{\mathbf{B}}_{jk}$$

and ${\mathbf{C}}_{ik}$ is a $p_i\times r_k$ matrix.

Definition 1.114 (Block diagonal matrix)

A block diagonal matrix is a block matrix whose off diagonal entries are zero matrices.