7.8. Multivariate Gaussian Distribution

Definition 7.32

A random vector $X=[X_1,\dots ,X_n{]}^T$ is called Gaussian random vector if

$$⟨t,X⟩=X^{T}t=\sum _{i=1}^n{t}_i{X}_i=t_1{X}_1+\cdots +t_n{X}_n$$

follows a normal distribution for all $t=[t_1,\dots ,t_n{]}^T\in {\mathbb{R}}^n$. The components $X_1,\dots ,X_n$ are called jointly Gaussian. It is denoted by $X\sim {\mathcal{N}}_n(\mu ,\mathrm{\Sigma })$ where $\mu$ is its mean vector and $\mathrm{\Sigma }$ is its covariance matrix.

Let $X\sim {\mathcal{N}}_n(\mu ,\mathrm{\Sigma })$ be a Gaussian random vector. The subscript $n$ denotes that it takes values over the space ${\mathbb{R}}^n$. We assume that $\mathrm{\Sigma }$ is invertible. Its PDF is given by

$$f_X(x)=\frac{1}{(2\pi {)}^{n/2}det(\mathrm{\Sigma }{)}^{1/2}}\exp \{-\frac{1}{2}(x-\mu {)}^T{\mathrm{\Sigma }}^{-1}(x-\mu )\}.$$

Moments:

$$\mathbb{E}[X]=\mu \in {\mathbb{R}}^n.$$

$$\mathbb{E}[X{X}^T]=\mathrm{\Sigma }+\mu {\mu }^T.$$

$$\mathrm{Cov}[X]=\mathbb{E}[X{X}^T]-\mathbb{E}[X]\mathbb{E}[X{]}^T=\mathrm{\Sigma }.$$

Let $Y=AX+b$ where $A\in {\mathbb{R}}^{n\times n}$ is an invertible matrix and $b\in {\mathbb{R}}^n$. Then

$$Y\sim {\mathcal{N}}_n(A\mu +b,A\mathrm{\Sigma }{A}^T).$$

$Y$ is also a Gaussian random vector with the mean vector being $A\mu +b$ and the covariance matrix being $A\mathrm{\Sigma }{A}^T$. This essentially is a change in basis in ${\mathbb{R}}^n$.

The CF is given by

$${\mathrm{\Psi }}_X(j\omega )\exp (j{\omega }^{T}x-\frac{1}{2}{\omega }^T\mathrm{\Sigma }\omega ),\omega \in {\mathbb{R}}^n.$$

7.8.1. Whitening

Usually we are interested in making the components of $X$ uncorrelated. This process is known as whitening. We are looking for a linear transformation $Y=AX+b$ such that the components of $Y$ are uncorrelated. i.e. we start with

$$X\sim {\mathcal{N}}_n(\mu ,\mathrm{\Sigma })$$

and transform $Y=AX+b$ such that

$$Y\sim {\mathcal{N}}_n(0,I_n)$$

where $I_n$ is the $n$-dimensional identity matrix.

7.8.1.1. Whitening by Eigen Value Decomposition

Let

$$\mathrm{\Sigma }=E\mathrm{\Lambda }{E}^T$$

be the eigen value decomposition of $\mathrm{\Sigma }$ with $\mathrm{\Lambda }$ being a diagonal matrix and $E$ being an orthonormal basis.

Let

$${\mathrm{\Lambda }}^{\frac{1}{2}}=\mathrm{diag}({\lambda }_1^{\frac{1}{2}},\dots ,{\lambda }_n^{\frac{1}{2}}).$$

Choose $B=E{\mathrm{\Lambda }}^{\frac{1}{2}}$ and $A=B^{-1}={\mathrm{\Lambda }}^{-\frac{1}{2}}{E}^T$.
Then

$$\mathrm{Cov}(B^{-1}X)=\mathrm{Cov}(AX)={\mathrm{\Lambda }}^{-\frac{1}{2}}{E}^T\mathrm{\Sigma }E{\mathrm{\Lambda }}^{-\frac{1}{2}}=I.$$

$$\mathbb{E}[B^{-1}X]=B^{-1}\mu ⟺\mathbb{E}[B^{-1}(X-\mu )]=0.$$

Thus the random vector $Y=[B^{-1}(X-\mu )$ is a whitened vector of uncorrelated components.

7.8.1.2. Causal Whitening

We want that the transformation be causal, i.e. $A$ should be a lower triangular matrix. We start with

$$\mathrm{\Sigma }=LD{L}^T=(L{D}^{\frac{1}{2}})(D^{\frac{1}{2}}{L}^T).$$

Choose $B=L{D}^{\frac{1}{2}}$ and $A=B^{-1}=D^{-\frac{1}{2}}{L}^{-1}$. Clearly, $A$ is lower triangular.

The transformation is $Y=[B^{-1}(X-\mu )$.