22.1. Introduction

When designing a dictionary for a particular application, we have several options [70]. On the one hand we can go through the long list of analytically constructed or tunable dictionaries and select one of them as suitable for the application in concern. On the other hand, we can take up a number of real example signals from our application and try to construct a dictionary which is optimized for these.

Dictionary learning (DL) [30, 75] is a process which attempts to solve the problem of constructing a dictionary directly from the set of example signals. The atoms of a learnt dictionary come from the underlying empirical data of training set of example signals for the specific application. While analytically constructed dictionaries are typically meant for only specific applications, the learning method allows one to construct dictionaries for any family of signals which are amenable to the sparse and redundant representations model. This certainly comes at a cost. Learnt dictionaries have to be held completely in memory explicitly as they happen to be lack any structure. Thus they don’t provide an efficient implementation of analysis (${\mathcal{D}}^H\mathbf{x}$) and synthesis ($\mathcal{D}\mathbf{a}$) operators. Thus using them in applications leads to more computational costs.

---

We start by formalizing the notation for Dictionary Learning (DL).

1. We consider a set of $S$ example signals put together in a signal matrix $\mathbf{X}\in {\mathbb{C}}^{N\times S}$.

2. Consider a dictionary $\mathcal{D}\in {\mathbb{C}}^{N\times D}$.

3. Let ${\mathbf{a}}^i\in {\mathbb{C}}^D$ be the sparsest possible representation of ${\mathbf{x}}^i$ in $\mathcal{D}$ with

   $${\mathbf{x}}^i=\mathcal{D}{\mathbf{a}}^i+{\mathbf{e}}^i.$$

4. We put all ${\mathbf{a}}^i$ together in a matrix $\mathbf{A}\in {\mathbb{C}}^{D\times S}$.

5. Then we have

   $$\mathbf{X}=\mathcal{D}\mathbf{A}+\mathbf{E}$$

   where $\mathbf{E}\in {\mathbb{C}}^{N\times S}$ represents approximation error.

6. We are looking for best dictionary from the set of possible dictionaries such that we are able to get sparse representations of ${\mathbf{x}}^i$ with low approximation error.

7. We can quantify the notion of good approximation by putting an upper bound on the norm of approximation error as $\Vert {\mathbf{e}}^i{\Vert }_2\le ϵ$.

8. Combining these ideas, we introduce the notion of a sparse signal model denoted as ${\mathcal{M}}_{\mathcal{D},K,ϵ}$ which consists of a dictionary $\mathcal{D}$ providing $K$-sparse representations for a class of signals (from which the example signals are drawn) with an upper bound on approximation error given by $ϵ$.

9. The DL problem tries to learn best model $\mathcal{M}$ based on the example signals $\mathbf{X}$.

---

1. We can formulate the DL problem as an optimization problem as follows:

   (22.1)$$\begin{array}{rlrl}& \underset{\mathcal{D},\{{\mathbf{a}}^i{\}}_{i=1}^S}{\text{minimize }}& & \sum _{i=1}^S\Vert {\mathbf{a}}^i{\Vert }_0\\ & \text{subject to }& & \Vert {\mathbf{x}}^i-\mathcal{D}{\mathbf{a}}^i{\Vert }_2\le ϵ,i=1,\dots ,S.\end{array}$$

2. In this version, we are trying to minimize total sparsity of ${\mathbf{a}}^i$ while using the upper bound on approximation error as optimization constraint.

3. We are not enforcing sparsity constraint that $\Vert {\mathbf{a}}^i{\Vert }_0\le K\mathrm{\forall }1\le i\le S$.

4. Alternatively we can also write:

   (22.2)$$\begin{array}{rlrl}& \underset{\mathcal{D},\{{\mathbf{a}}^i{\}}_{i=1}^S}{\text{minimize }}& & \sum _{i=1}^S\Vert {\mathbf{x}}^i-\mathcal{D}{\mathbf{a}}^i{\Vert }_2^2\\ & \text{subject to }& & \Vert {\mathbf{a}}^i{\Vert }_0\le K,i=1,\dots ,S.\end{array}$$

5. In this version we are trying to minimize approximation error while keeping $K$-sparsity as optimization constraint.

6. We are not enforcing the constraint that $\Vert {\mathbf{e}}^i{\Vert }_2\le ϵ$.

7. Unfortunately, there doesn’t exist any computationally tractable algorithm to solve these optimization problems.

8. An alternative is to consider a heuristic iterative approach presented in Algorithm 22.1.

Algorithm 22.1 (Dictionary learning: iterative approach)

Initialization:

1. Choose an initial dictionary ${\mathcal{D}}^0$.

2. $k\leftarrow 0$ # iteration counter. \Repeat{halting criteria is true}{

Algorithm:

1. If halting criteria is met: break.

2. $k\leftarrow k+1$.

3. Obtain representations ${\mathbf{a}}^i$ for ${\mathbf{x}}^i$ with the given ${\mathcal{D}}^{k-1}$ and sparsity level $K$ using some sparse approximation algorithm.

4. Obtain new dictionary ${\mathcal{D}}^k$ from the signals ${\mathbf{x}}^i$ and their representations ${\mathbf{a}}^i$.

Some possible halting criteria are:

• Stop after a fixed number of iterations.

• Stop when $\Vert {\mathbf{e}}^i{\Vert }_2\le ϵ$ for every example.

• Stop when approximation errors stop improving.

For initialization of the algorithm, we have few options.

• We can start with a randomly generated dictionary.

• We can select $D$ examples from $\mathbf{X}$ and put them together as our starting dictionary.

• We can start with some analytical dictionary suitable for the application domain.

22.1.1. Dictionary Learning Methods

Two popular algorithms implementing this approach are MOD (Method of Optimal Directions) and K-SVD.

22.1.1.1. Method Of Optimal Directions

In MOD [34], dictionary update is formulated as

$${\mathcal{D}}^k=\text{arg}\underset{\mathcal{D}}{min}\Vert \mathbf{X}-\mathcal{D}{\mathbf{A}}^k{\Vert }_F^2$$

A straight forward least squares solution is obtained as

$${\mathcal{D}}^k=\mathbf{X}({\mathbf{A}}^k{)}^{†}.$$

22.1.1.2. K-SVD

K-SVD [1] is slightly different. Rather than recomputing whole dictionary at once by solving the LS problem, it updates atoms of $\mathcal{D}$ one by one.

1. Let $j$ be the index of atom ${\mathbf{d}}_j$ being updated.

2. Consider the error

   $${\mathbf{E}}_j=\mathbf{X}-\sum _{k,k\ne j}{\mathbf{d}}_k{\mathbf{b}}_k^T$$

3. ${\mathbf{b}}_k$ refers to the $k$-th row of $\mathbf{A}$; i.e. entries for atom ${\mathbf{d}}_k$ in every example.

4. ${\mathbf{E}}_j$ means the approximation error when the atom ${\mathbf{d}}_j$ (and corresponding entries in $\mathbf{A}$) has been dropped from consideration.

5. We can see that

   $$\mathbf{X}-\mathcal{D}\mathbf{A}=\mathbf{X}-\sum _{k=1}^D{\mathbf{d}}_k{\mathbf{b}}_k^T=\mathbf{X}-\sum _{k,k\ne j}{\mathbf{d}}_k{\mathbf{b}}_k^T-{\mathbf{d}}_j{\mathbf{b}}_j^T={\mathbf{E}}_j-{\mathbf{d}}_j{\mathbf{b}}_j^T.$$

6. Hence

   $$\Vert \mathbf{X}-\mathcal{D}\mathbf{A}{\Vert }_F^2=\Vert {\mathbf{E}}_j-{\mathbf{d}}_j{\mathbf{b}}_j^T{\Vert }_F^2.$$

7. On the L.H.S. we have total approximation error.

8. On the R.H.S. we have the same expressed in terms of ${\mathbf{E}}_j$ and ${\mathbf{d}}_j$ where ${\mathbf{E}}_j$ doesn’t depend on ${\mathbf{d}}_j$.

9. An optimal ${\mathbf{d}}_j$ is one which can minimize R.H.S.

10. This can easily be obtained by rank-1 approximation of ${\mathbf{E}}_j$.

11. The rank-1 approximation gives us both ${\mathbf{d}}_j$ as well as the entries in the $j$-th row of $\mathbf{A}$ given by ${\mathbf{b}}_j$. There is a small catch though.

12. In general the rank-1 approximation can lead to a dense ${\mathbf{b}}_j$ meaning the atom ${\mathbf{d}}_j$ gets used in many signal representations. We wish to avoid that. We don’t want ${\mathbf{d}}_j$ to appear in many signal representations.

13. For this, we identify representations ${\mathbf{a}}^i$ in which atom ${\mathbf{d}}_j$ appears, and let them be indexed by $\mathrm{\Gamma }$.

14. We then restrict ${\mathbf{E}}_j$ to signals indexed by $\mathrm{\Gamma }$ to avoid a dense ${\mathbf{b}}_j$.

15. Rank-1 approximation can be easily obtained using singular value decomposition.

16. We perform SVD of ${\mathbf{E}}_{j,\mathrm{\Gamma }}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^H$.

17. We then pick ${\mathbf{u}}_1,{\sigma }_1,{\mathbf{v}}_1$.

18. ${\mathbf{u}}_1$ is the new update for ${\mathbf{d}}_j$.

19. Further ${\sigma }_1{\mathbf{v}}_1$ is the update for ${\mathbf{b}}_{j,\mathrm{\Gamma }}$.

20. Rest of ${\mathbf{b}}_j$ is left with zero entries.

21. We repeat the process for each of the atoms in $\mathcal{D}$ to obtain next update of $\mathcal{D}$.