8.1. Mathematical Optimization

Main references for this section are [9, 17].

This section provides a general overview of optimization problems. The notion of objective or cost functions and constraint functions is introduced. Standard form for optimization problems is introduced. Several examples are discussed for equivalent forms of optimization problems. The standard form focuses on minimizing the objective function over a set of equality and inequality constraints. Maximization problems can also be converted into minimization problems by changing the sign of the objective function.

We introduce the notion of the domain of an optimization problem, feasible points or solutions, feasible set of solutions, the optimal value of the minimization problem, optimal points or solutions, optimal set etc.. We discuss when the problem may be infeasible or feasible but unbounded. We discuss situations where the problem may be feasible and have a global minimum/optimal value and yet a feasible solution may not exist.

We then discuss general requirements for the feasibility of the optimization problem and existence of an optimal solution. We introduce the notion of coercive functions which provide the Weierstrass type result for the existence of optimal solutions.

8.1.1. Optimization Problems

Let $\mathbb{V}$ be a real $n$-dimensional inner product space.

In the most general form, an optimization problem can be considered as minimizing or maximizing the value of a real valued function $f:\mathbb{V}\to \mathbb{R}$ over its domain $S=\mathrm{dom}f$.

1. We call $f$ as the objective function which is being minimized or maximized.

2. The variable $\mathbf{x}\in \mathbb{V}$ is called the optimization variable.

3. $S=\mathrm{dom}f$ is called the feasible set of solutions for the optimization problem.

4. A point $\mathbf{x}\in S$ is called a feasible point or feasible solution.

5. Our goal is to find an ${\mathbf{x}}^{\ast }\in S$ which maximizes or minimizes the objective function $f$.

Definition 8.1 (Globally optimal value and points)

Let $f:\mathbb{V}\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$.

1. The maximum value of $f$ is given by

   $$sup\{f(\mathbf{x})|\mathbf{x}\in S\}.$$

2. The minimum value of $f$ is given by

   $$inf\{f(\mathbf{x})|\mathbf{x}\in S\}.$$

3. We allow the maximum and minimum values to take $\mathrm{\infty }$ and $-\mathrm{\infty }$ values.

4. The function $f$ may or may not attain its maximum / minimum value at some point in its domain $S$.

5. ${\mathbf{x}}^{\ast }\in S$ is called a global minimum point if $f(\mathbf{x})\ge f({\mathbf{x}}^{\ast })$ for every $\mathbf{x}\in S$.

6. ${\mathbf{x}}^{\ast }\in S$ is called a global maximum point if $f(\mathbf{x})\le f({\mathbf{x}}^{\ast })$ for every $\mathbf{x}\in S$.

7. ${\mathbf{x}}^{\ast }\in S$ is called a strict global minimum point if $f(\mathbf{x})>f({\mathbf{x}}^{\ast })$ for every $\mathbf{x}\in S$ with $\mathbf{x}\ne {\mathbf{x}}^{\ast }$.

8. ${\mathbf{x}}^{\ast }\in S$ is called a strict global minimum point if $f(\mathbf{x})<f({\mathbf{x}}^{\ast })$ for every $\mathbf{x}\in S$ with $\mathbf{x}\ne {\mathbf{x}}^{\ast }$.

9. A point ${\mathbf{x}}^{\ast }\in S$ is called a global optimal point if it is either a global maximum or minimum point.

10. The maximum and minimum values of $f$ are always unique as opposed to global optimal points which may not be unique.

11. The set of global minimizers is given by

    $$\arg \min \{f(\mathbf{x})|\mathbf{x}\in S\}.$$

12. The set of global maximizers is given by

    $$\arg \max \{f(\mathbf{x})|\mathbf{x}\in S\}.$$

In a more restricted setting, an optimization problem can be considered as minimizing or maximizing the value of a real valued function $f:\mathbb{V}\to \mathbb{R}$ with $S=\mathrm{dom}f$ over a set $A$ such that $A\subseteq S$.

1. In this case, the feasible set is restricted to $A$.

2. Minimizers and maximizers are searched within this subset $A$.

3. This problem can be converted into the earlier general form by considering the restriction $\tilde{f}:\mathbb{V}\to \mathbb{R}$ of $f$ such that $A=\mathrm{dom}\tilde{f}$ and

   $$\tilde{f}(\mathbf{x})=f(\mathbf{x})\mathrm{\forall }\mathbf{x}\in A.$$

In several problems, it may not be possible to establish whether a point is globally optimal. However, it is easier to establish if a point is optimal in a neighborhood around it. Such points are called locally optimal points or extreme values.

Definition 8.2 (Local optimal points)

Let $f:\mathbb{V}\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. We say that $f(\mathbf{a})$ is a local extreme value or local optimal value of $f$ at $\mathbf{a}\in \mathrm{dom}f$ if there exists $\delta >0$ such that $f(\mathbf{x})-f(\mathbf{a})$ doesn’t change sign on $B(\mathbf{a},\delta )\cap S$.

More specifically,

1. $f(\mathbf{a})$ is a local maximum value of $f$ if for some $\delta >0$:

   $$f(\mathbf{x})\le f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},\delta )\cap S.$$

2. $f(\mathbf{a})$ is a local minimum value of $f$ if for some $\delta >0$:

   $$f(\mathbf{x})\ge f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},\delta )\cap S.$$

The point $\mathbf{x}=\mathbf{a}$ is called a local extreme point or local optimal point of $f$ or more specifically, a local maximum or a local minimum point of $f$.

1. $\mathbf{a}$ is a strict local maximum point if for some $\delta >0$:

   $$f(\mathbf{x})>f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B_d(\mathbf{a},\delta )\cap S.$$

2. $\mathbf{a}$ is a strict local minimum point of $f$ if for some $\delta >0$:

   $$f(\mathbf{x})>f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B_d(\mathbf{a},\delta )\cap S.$$

Here $B_d(\mathbf{a},\delta )$ denotes the deleted neighborhood (an open ball of radius $\delta$ excluding $\mathbf{a}$ itself).

Remark 8.1 (Local optimal points for open domain)

Let $f:\mathbb{V}\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Assume that $S$ is open.

1. If $\mathbf{a}\in S$ is a local maximum point, then there exists $r_1>0$ such that

   $$f(\mathbf{x})\le f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},r_1)\cap S.$$

2. But since $S$ is open, hence $\mathbf{a}\in \mathrm{int}S$. Thus, there exists an open ball $B(\mathbf{a},r_2)\subseteq S$.

3. Let $r=min(r_1,r_2)$.

4. Then, $B(\mathbf{a},r)\subseteq S$ and $B(\mathbf{a},r)\subseteq B(\mathbf{a},r_1)$.

5. Thus, $B(\mathbf{a},r)\subseteq B(\mathbf{a},r_1)\cap S$.

6. Thus, $f(\mathbf{x})\le f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},r)\subseteq S$.

7. Thus, the local optimality condition simplifies to looking for an open ball of radius $r$ totally contained inside the open domain $S$.

Based on this discussion, we can adjust the conditions for local optimality.

1. $\mathbf{a}$ is a local maximum point of $f$ if for some $r>0$:

   $$f(\mathbf{x})\le f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},r)\subseteq S.$$

2. $\mathbf{a}$ is a local minimum point of $f$ if for some $r>0$:

   $$f(\mathbf{x})\ge f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B(\mathbf{a},r)\subseteq S.$$

3. $\mathbf{a}$ is a strict local maximum point if for some $r>0$:

   $$f(\mathbf{x})>f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B_d(\mathbf{a},r)\subseteq S.$$

4. $\mathbf{a}$ is a strict local minimum point of $f$ if for some $r>0$:

   $$f(\mathbf{x})>f(\mathbf{a})\mathrm{\forall }\mathbf{x}\in B_d(\mathbf{a},r)\subseteq S.$$

8.1.1.1. Standard Form for Mathematical Optimization Problems

Definition 8.3 (Optimization problem standard form)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. A mathematical optimization problem of the form

(8.1)$$\begin{array}{rlrl}& \text{minimize }& & f_0(\mathbf{x})\\ & \text{subject to }& & f_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

is known as an optimization problem in its standard form.

1. $\mathbf{x}\in \mathbb{V}$ is called the optimization variable.

2. $f_0:\mathbb{V}\to \mathbb{R}$ is called the objective function or cost function.

3. The functions $f_i:\mathbb{V}\to \mathbb{R}$ are called the inequality constraint functions.

4. The corresponding inequalities $f_i(\mathbf{x})\le 0$ are called the inequality constraints.

5. The functions $h_j:\mathbb{V}\to \mathbb{R}$ are called the equality constraint functions.

6. The corresponding equations $h_j(\mathbf{x})=0$ are called the equality constraints.

7. If there are no constraints ($m=0,p=0$), the problem is called unconstrained.

8. The set

   $$\mathcal{D}\triangleq \mathrm{dom}{f}_0\cap \bigcap _{i=1}^m\mathrm{dom}{f}_i\cap \bigcap _{j=1}^p\mathrm{dom}{h}_j$$

   is called the domain of the optimization problem.

9. A point $\mathbf{x}\in \mathcal{D}$ is called feasible if it satisfies all the inequality constraints $f_i(\mathbf{x})\le 0$ and all the equality constraints $h_i(\mathbf{x})=0$.

10. The optimization problem is called feasible if there exists at least one feasible point.

11. If there are no feasible points in the domain $\mathcal{D}$, then the optimization problem is called infeasible. Naturally, if $\mathcal{D}=\varnothing$, then the problem is infeasible.

12. The set of feasible points for an optimization problem is called the feasible set or the constraint set. We shall denote the feasible set by $C$.

    $$C=\mathrm{dom}{f}_0\cap \bigcap _{i=1}^m{f}_i^{-1}(-\mathrm{\infty },0]\cap \bigcap _{j=1}^p{h}_j^{-1}(0).$$

    It is the intersection of the domain of $f_0$, the 0-sublevel sets of $f_i$ for $i=1,\dots ,m$, and the $0$-level sets of $h_j$ for $j=1,\dots ,p$.

13. Thus, if the problem is infeasible, then $C=\varnothing$.

14. The optimum value $p^{\ast }\in \bar{\mathbb{R}}$ of the optimization problem is defined as

    $$p^{\ast }=inf\{f_0(\mathbf{x})|f_i(\mathbf{x})\le 0,i=1,\dots ,m,h_j(\mathbf{x})=0,j=1,\dots ,p\}.$$

    In other words,

    $$p^{\ast }=inf\{f_0(\mathbf{x})|\mathbf{x}\in C\}.$$

    We allow $p^{\ast }$ to take the extended values $\mathrm{\infty }$ and $-\mathrm{\infty }$.

15. If the problem is infeasible, then $p^{\ast }=\mathrm{\infty }$ It is consistent with the convention that the infimum of an empty set is $\mathrm{\infty }$.

16. If $p^{\ast }=-\mathrm{\infty }$, then the problem is called unbounded below. In this case, there exists a sequence $\{{\mathbf{x}}_k\}$ of $C$ such that $lim{f}_0({\mathbf{x}}_k)=-\mathrm{\infty }$.

17. We say that ${\mathbf{x}}^{\ast }$ is an optimal point if it solves (8.1). In other words, ${\mathbf{x}}^{\ast }\in C$ and $f({\mathbf{x}}^{\ast })=p^{\ast }$.

18. The set of all optimal points is known as the optimal set denoted by $X_{\text{opt}}$.

    $$X_{\text{opt}}\triangleq \{\mathbf{x}|f_i(\mathbf{x})\le 0,i=1,\dots ,m,h_j(\mathbf{x})=0,j=1,\dots ,p,f_0(\mathbf{x})=p^{\ast }\}.$$

    In other words,

    $$X_{\text{opt}}=\{\mathbf{x}\in C|f_0(\mathbf{x})=p^{\ast }\}.$$

19. If an optimal point exists in $C$, then we say that the optimal value is attained or achieved.

20. If $X_{\text{opt}}$ is empty, then we say that the optimal value is not attained or not achieved.

21. In particular, if the problem is unbounded below, then $X_{\text{opt}}$ is indeed empty.

22. If the feasible set $C$ is not closed, then it is quite possible that The optimum value $p^{\ast }$ is finite and yet it is not attained at any feasible point. Then, there exists a sequence $\{{\mathbf{x}}_k\}$ of feasible points such that $limf({\mathbf{x}}_k)=p^{\ast }$. However, there is no $\mathbf{x}\in C$ such that $f(\mathbf{x})=p^{\ast }$.

23. A feasible point $\mathbf{x}\in C$ with $f_0(\mathbf{x})\le p^{\ast }+ϵ$ is called an $ϵ$-suboptimal point.

24. The set of all $ϵ$-suboptimal points is called the $ϵ$-suboptimal set for the problem (8.1).

25. We say that a feasible point $\mathbf{x}$ is locally optimal if there exists $r>0$ such that

    $$f_0(\mathbf{x})=inf\{f_0(\mathbf{z})|\mathbf{z}\in B[\mathbf{x},r]\cap C\}.$$

    In other words, $\mathbf{x}$ minimizes $f$ over a local neighborhood of feasible points.

26. If $\mathbf{x}$ is feasible and $f_i(\mathbf{x})=0$, we say that $i$-th inequality constraint is active at $\mathbf{x}$. Otherwise, $f_i(\mathbf{x})<0$ and we say that the $i$-th inequality constraint is inactive.

27. We say that a constraint is redundant if removing it does not change the feasible set.

28. If we choose an orthonormal basis $\mathcal{B}=\{{\mathbf{e}}_1,\dots ,{\mathbf{e}}_n\}$ for $\mathbb{V}$, then $\mathbb{V}$ is isomorphic to ${\mathbb{R}}^n$ under the bracket operator $[\cdot {]}_{\mathcal{B}}$. The optimization variable $\mathbf{x}$ has a representation $(x_1,\dots ,x_n)$ in ${\mathbb{R}}^n$ given by

    $$\mathbf{x}=\sum _{i=1}^n{x}_i{\mathbf{e}}_i.$$

29. Thus, determining $\mathbf{x}$ is same as determining its components $(x_1,\dots ,x_n)$.

30. Since there are $n$ scalar components in the representation of $\mathbf{x}$, hence we can say that the optimization problem has $n$ (scalar) variables.

In the sequel, we will be presenting a variety of optimization problems. We shall show how those problems can be converted to the standard form described above.

Example 8.1 (Box constraints standard form)

Let $f_0:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function. Consider the optimization problem

$$\begin{array}{rlrl}& \text{minimize }& & f_0(\mathbf{x})\\ & \text{subject to }& & l_i\le x_i\le u_i,& i=1,\dots ,n\end{array}$$

where $\mathbf{x}\in {\mathbb{R}}^n$ is the optimization variable. There are $n$ constraints, one on each component of $\mathbf{x}$. Each constraint gives a lower and upper bound on one component. Such constraints are known as variable bounds or box constraints.

This problem can be transformed into an equivalent problem:

$$\begin{array}{rlrl}& \text{minimize }& & f_0(\mathbf{x})\\ & \text{subject to }& & l_i-x_i\le 0,& i=1,\dots ,n\\ & \text{subject to }& & x_i-u_i\le 0,& i=1,\dots ,n.\end{array}$$

This form has $2n$ inequality constraints. We introduce the functions

$$\begin{array}{rl}& f_i(\mathbf{x})=l_i-x_i,i=1,\dots ,n\\ & \text{ and }\\ & f_i(\mathbf{x})=x_{i-n}-u_{i-n},i=n+1,\dots ,2n.\end{array}$$

Then, the problem becomes

$$\begin{array}{rlrl}& \text{minimize }& & f_0(\mathbf{x})\\ & \text{subject to }& & f_i(\mathbf{x})\le 0,& i=1,\dots ,2n.\end{array}$$

This is the optimization problem in standard form with $2n$ inequality constraints and 0 equality constraints.

Example 8.2 (Maximization problem in standard form)

Consider the problem

$$\begin{array}{rlrl}& \text{maximize }& & f_0(\mathbf{x})\\ & \text{subject to }& & f_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

If we replace $f_0$ by $-f_0$, then maximizing $f_0$ is same as minimizing $-f_0$. Thus, this problem has an equivalent form

$$\begin{array}{rlrl}& \text{minimize }& & -f_0(\mathbf{x})\\ & \text{subject to }& & f_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

Definition 8.4 (Feasibility problem)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. A mathematical optimization problem of the form

(8.2)$$\begin{array}{rlrl}& \text{find }& & \mathbf{x}\in \mathbb{V}\\ & \text{subject to }& & f_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

is called a feasibility problem.

We can convert this problem into the standard problem by introducing a cost function which is identically 0:

$$f_0(\mathbf{x})=0\mathrm{\forall }\mathbf{x}\in \mathbb{V}.$$

1. The domain reduces to

   $$\mathcal{D}=\bigcap _{i=1}^m\mathrm{dom}{f}_i\cap \bigcap _{j=1}^p\mathrm{dom}{h}_j.$$

2. The optimal value $p^{\ast }=\mathrm{\infty }$ if the problem is not feasible.

3. Otherwise, the optimal value is $p^{\ast }=0$.

4. Every feasible point is also an optimal point.

5. Every optimization problem in standard form can be converted into a feasibility problem by replacing the objective function with the function which is identically 0 everywhere.

6. In that case, the problem reduces to checking whether the inequality and equality constraints are consistent or not. In other words, whether there are some points which satisfy all the inequality and equality constraints.

8.1.2. Equivalent Problems

Definition 8.5 (Equivalent optimization problems)

We provide an informal definition of equivalent optimization problems.

Two optimization problems are called equivalent if it is possible to find the solution of one problem from the other problem and the vice versa.

The primary reason for transforming an optimization problem to an equivalent one is that it may be easier to solve an equivalent problem. If we can transform a problem into a form which has a well known closed form solution or has a well known algorithm to solve the problem, then we can solve the original problem by first solving the equivalent problem and then transforming the solution of the equivalent problem to the solution of the original problem.

1. Transform the problem to an equivalent problem with a known algorithm for solution.

2. Solve the equivalent problem.

3. Transform the solution of the equivalent problem to the solution of the original problem.

In this sense, the computational complexity of solving an optimization problem cannot be greater than the complexity of solving the equivalent problem plus the complexity of transforming the solution of the equivalent problem to the solution of the original problem.

There are several ways to obtain an equivalent optimization problem from an existing optimization problem.

8.1.2.1. Scaling

Proposition 8.1 (Equivalent problems by scaling of objective or constraint functions)

Consider the problem given by

$$\begin{array}{rlrl}& \text{minimize }& & f_0^{\prime }(\mathbf{x})={\alpha }_0{f}_0(\mathbf{x})\\ & \text{subject to }& & f_i^{\prime }(\mathbf{x})={\alpha }_i{f}_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j^{\prime }(\mathbf{x})={\beta }_j{h}_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

where ${\alpha }_i>0$ for $i=0,\dots ,m$ and ${\beta }_j\ne 0$ for $j=1,\dots ,p$.

This problem is obtained from the optimization problem in standard from in (8.1) by scaling the objective function and the inequality constraint functions by positive constants and the equality constraints by nonzero constants.

1. ${\alpha }_0$ cannot be negative otherwise it will convert a minimization problem to a maximization problem.

2. ${\alpha }_0$ cannot be zero as that will convert the optimization problem into a feasibility problem.

3. ${\alpha }_i$ for $i=1,\dots ,m$ cannot be negative as it will turn the inequality sign from $f_i(\mathbf{x})\le 0$ to $f^{\prime }(\mathbf{x})\ge 0$.

4. ${\alpha }_i$ for $i=1,\dots ,m$ cannot be zero as it will remove the corresponding constraint altogether.

5. ${\beta }_j$ cannot be zero. A zero value will discard the corresponding equality constraint.

6. The feasible sets for both problems are identical.

7. The optimal sets for both problems are identical.

8. The optimal values for both problems are not identical however (unless ${\alpha }_0=1$). Since $f_0^{\prime }$ is a scaled version of $f_0$, hence the optimal value is also scaled accordingly.

8.1.2.2. Change of Variables

Proposition 8.2 (Change of variables)

Let $\varphi :\mathbb{V}\to \mathbb{V}$ be an injective function with $\mathcal{D}\subseteq \mathrm{range}\varphi$.

For the problem (8.1), introduce the following functions

$${\tilde{f}}_i(\mathbf{z})=f_i(\varphi (\mathbf{z})),i=0,\dots ,m\text{ and }{\tilde{h}}_j(\mathbf{z})=h_j(\varphi (\mathbf{z})),j=1,\dots ,p.$$

Now, consider the problem

(8.3)$$\begin{array}{rlrl}& \text{minimize }& & {\tilde{f}}_0(\mathbf{z})\\ & \text{subject to }& & {\tilde{f}}_i(\mathbf{z})\le 0,& i=1,\dots ,m\\ & & & {\tilde{h}}_j(\mathbf{z})=0,& j=1,\dots ,p\end{array}$$

with the variable $\mathbf{z}\in \mathbb{V}$.

We say that the two problems are related by the change of variable or substitution of the variable $\mathbf{x}=\varphi (\mathbf{z})$.

Suppose ${\mathbf{z}}^{\ast }$ is the solution to (8.3), then ${\mathbf{x}}^{\ast }=\varphi ({\mathbf{z}}^{\ast })$ is a solution to (8.1).

Similarly, if ${\mathbf{x}}^{\ast }$ is a solution to (8.1), then ${\mathbf{z}}^{\ast }={\varphi }^{-1}({\mathbf{x}}^{\ast })$ is a solution to (8.3).

8.1.2.3. Transformation with Monotone Functions

Proposition 8.3 (Transformation of objective and constraint functions)

For the problem (8.1), we introduce the following:

1. Let ${\psi }_0:\mathbb{R}\to \mathbb{R}$ be a strictly increasing function over $\mathrm{range}{f}_0$.

2. Let ${\psi }_i:\mathbb{R}\to \mathbb{R}$ for $i=1,\dots ,m$ be strictly increasing functions over $\mathrm{range}{f}_i$ with ${\psi }_i(u)\le 0⟺u\le 0$.

3. Let ${\eta }_j:\mathbb{R}\to \mathbb{R}$ for $j=1,\dots ,p$ be real functions that satisfy ${\eta }_j(u)=0⟺u=0$.

4. Let ${\tilde{f}}_i(\mathbf{x})={\psi }_i(f_i(\mathbf{x}))$ for $i=0,\dots ,m$.

5. Let ${\tilde{h}}_j(\mathbf{x})={\eta }_j(h_j(\mathbf{x}))$ for $j=1,\dots ,p$.

Now, consider the problem

(8.4)$$\begin{array}{rlrl}& \text{minimize }& & {\tilde{f}}_0(\mathbf{x})\\ & \text{subject to }& & {\tilde{f}}_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & {\tilde{h}}_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

with the variable $\mathbf{x}\in \mathbb{V}$.

The problem (8.4) is equivalent to (8.1).

Example 8.3 (Least norms and least norms squared)

Consider the unconstrained least norm problem

$$\text{minimize}\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2$$

with $\mathbf{x}\in {\mathbb{R}}^n$, $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ and $\mathbf{b}\in {\mathbb{R}}^m$. We have $f_0(\mathbf{x})=\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2$. Then, $\mathrm{range}{f}_0={\mathbb{R}}_{+}$. Consider ${\psi }_0:\mathbb{R}\to \mathbb{R}$ as

$${\psi }_0(x)=x^2.$$

Then, ${\psi }_0$ is strictly increasing over ${\mathbb{R}}_{+}$. Let

$${\tilde{f}}_0(\mathbf{x})={\psi }_0(f_0(\mathbf{x}))=\Vert \mathbf{A}\mathbf{x}-\mathbf{b}{\Vert }_2^2=(\mathbf{A}\mathbf{x}-\mathbf{b}{)}^T(\mathbf{A}\mathbf{x}-\mathbf{b}).$$

Then, the least norm minimization problem is equivalent to the problem

$$\text{minimize}(\mathbf{A}\mathbf{x}-\mathbf{b}{)}^T(\mathbf{A}\mathbf{x}-\mathbf{b})$$

which is the least norm squared problem.

We note that while ${\tilde{f}}_0$ is differentiable, $f_0$ is not. It is a key difference between the two problems. Solving the least norm squared problem can be done using gradient methods since the objective function is differentiable.

8.1.2.4. Slack Variables

Proposition 8.4 (Slack variables)

For the inequality constraints in the problem (8.1), we can introduce the variables $s_i\ge 0$ so that $f_i(\mathbf{x})+s_i=0$. This way, we can convert an inequality constraint to an equality constraint and introduce simpler inequality constraints for the variables $s_i\ge 0$. The resultant problem:

(8.5)$$\begin{array}{rlrl}& \text{minimize }& & f_0(\mathbf{x})\\ & \text{subject to }& & s_i\ge 0,& i=1,\dots ,m\\ & & & f_i(\mathbf{x})+s_i=0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

where the optimization variables are $\mathbf{x}\in \mathbb{V}$ and $\mathbf{s}\in {\mathbb{R}}^m$ is equivalent to (8.1).

The variables $s_i$ are known as slack variables. $\mathbf{s}=(s_1,\dots ,s_m)$ is a vector that collects all the slack variables.

1. This form has $m$ inequality constraints and $m+p$ equality constraints.

2. It has $n+m$ optimization variables $x_1,\dots ,x_n$ and $s_1,\dots ,s_m$.

3. If $(\mathbf{x},\mathbf{s})$ is a feasible point for (8.5), then $\mathbf{x}$ is a feasible point for (8.1).

4. If $\mathbf{x}$ is a feasible point for (8.1), then we can pick $s_i=-f_i(\mathbf{x})$ to form $\mathbf{s}$ making $(\mathbf{x},\mathbf{s})$ a feasible point for (8.5).

5. $\mathbf{x}$ is an optimal point for (8.1) if and only if $(\mathbf{x},\mathbf{s})$ is an optimal point for (8.5) where $s_i=-f_i(\mathbf{x})$.

6. The slack variables measure how much slack does an inequality constraint function have at a feasible point.

7. If $s_i=0$ then, $f_i$ is active and it has no slack.

8. If $s_i>0$, then $f_i$ inactive and it has some slack.

8.1.2.5. Epigraph Polar Form

Proposition 8.5 (Epigraph polar form)

The problem (8.1) is equivalent to the optimization problem below:

(8.6)$$\begin{array}{rlrl}& \text{minimize }& & t\\ & \text{subject to }& & f_0(\mathbf{x})-t\le 0,& \\ & & & f_i(\mathbf{x})\le 0,& i=1,\dots ,m\\ & & & h_j(\mathbf{x})=0,& j=1,\dots ,p\end{array}$$

with the optimization variable $(\mathbf{x},t)\in \mathbb{V}\oplus \mathbb{R}$.

1. Note that $t$ appears only in a single inequality in (8.6).

2. At the optimal point, $f_0(\mathbf{x})=t$ must hold true. If $f(\mathbf{x})<t$, then we can always choose a lower value of $t$ as the solution.

3. If $\mathbf{x}$ is an optimal point for (8.1), then $(\mathbf{x},f(\mathbf{x}))$ must be an optimal point for (8.6).

4. Similarly, if $(\mathbf{x},t)$ is an optimal point for (8.6), then $\mathbf{x}$ must be an optimal point for (8.1) and $t=f_0(\mathbf{x})$ must hold true.

5. The two problems are indeed equivalent.

6. This problem has $m+1$ inequality constraints and $p$ equality constraints.

7. The objective function for the epigraph form is a linear function of $(\mathbf{x},t)$.

8.1.3. First Order Conditions

In this subsection, we focus on objective functions of type $f:{\mathbb{R}}^n\to \mathbb{R}$ which are differentiable.

Theorem 8.1 (First order optimality criterion for local optimal points)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Suppose that ${\mathbf{x}}^{\ast }\in \mathrm{int}S$ is a local optimal point. Assume that all the partial derivatives of $f$ exist at ${\mathbf{x}}^{\ast }$.
Then, $\mathrm{\nabla }f({\mathbf{x}}^{\ast })=\mathbf{0}$.

1. Let $i=1,\dots ,n$.

2. Consider the one-dimensional function $g_i:\mathbb{R}\to \mathbb{R}$ given by

   $$g_i(t)=f({\mathbf{x}}^{\ast }+t{\mathbf{e}}_i)$$

   where ${\mathbf{e}}_i$ is the $i$-th unit vector of ${\mathbb{R}}^n$.

3. We note that $g_i$ is differentiable at $t=0$ and

   $$g_i^{\prime }(0)=\frac{\partial f}{\partial x_i}({\mathbf{x}}^{\ast }).$$

4. Since ${\mathbf{x}}^{\ast }$ is a local optimal point of $f$, hence $t=0$ is a local optimal point of $g_i$.

5. Thus, $g_i^{\prime }(0)=0$.

6. Thus, $\frac{\partial f}{\partial x_i}({\mathbf{x}}^{\ast })=0$.

7. Since, this is true for every $i=1,\dots ,n$, hence $\mathrm{\nabla }f({\mathbf{x}}^{\ast })=\mathbf{0}$.

We mention that gradient being zero is a necessary condition; i.e., if a point is an optimal point then the gradient of $f$ at that point must be zero. It is not a sufficient condition. The gradient may be zero and yet the point may not be an optimal point.

Definition 8.6 (Stationary point)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Let ${\mathbf{x}}^{\ast }\in \mathrm{int}S$ and assume that $f$ is differentiable in some neighborhood of ${\mathbf{x}}^{\ast }$. Then, ${\mathbf{x}}^{\ast }$ is called a stationary point if $\mathrm{\nabla }f({\mathbf{x}}^{\ast })=\mathbf{0}$.

Thus, the locally optimal points are necessarily stationary points.

8.1.4. Second Order Conditions

Recall the discussion on semidefinite and definite matrices in Eigen Values.

For a twice continuously differentiable function $f$, the Hessian is symmetric. The semidefiniteness or definiteness of the Hessian provides necessary and sufficient conditions for local optimality at stationary points of $f$; i.e., the points $\mathbf{a}\in \mathrm{dom}f$ where $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

We have following necessary conditions for local optimality.

1. If $\mathbf{a}$ is a local minimum point then ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ must be positive semidefinite.

2. If $\mathbf{a}$ is a local maximum point then ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ must be negative semidefinite.

We have following sufficient conditions for local optimality.

1. If ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ is positive definite, then $\mathbf{a}$ is a strict local minimum.

2. If ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ is negative definite, then $\mathbf{a}$ is a strict local maximum.

Theorem 8.2 (Necessary second order optimality conditions)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Assume that $S$ is open. Further, assume that $f$ is twice continuously differentiable over $S$ and that $\mathbf{a}\in S$ is a stationary point.

Then, the following hold.

1. If $\mathbf{a}$ is a local minimum point of $f$ over $S$, then ${\mathrm{\nabla }}^{2}f(\mathbf{a})⪰\mathbf{O}$.

2. If $\mathbf{a}$ is a local maximum point of $f$ over $S$, then ${\mathrm{\nabla }}^{2}f(\mathbf{a})⪯\mathbf{O}$.

Proof. Assume $\mathbf{a}$ to be a local minimum point.

1. Then, there exists an open ball $B(\mathbf{a},r)\subseteq S$ such that $f(\mathbf{x})\ge f(\mathbf{a})$ for all $\mathbf{x}\in B(\mathbf{a},r)$.

2. Let $\mathbf{d}\in {\mathbb{R}}^n$ be a nonzero vector.

3. For $0<t<\frac{r}{\Vert \mathbf{d}\Vert }$, we define

   $${\mathbf{a}}_t=\mathbf{a}+t\mathbf{d}.$$

   By definition, ${\mathbf{a}}_t\in B(\mathbf{a},r)$.

4. Hence, for any $t\in (0,\frac{r}{\Vert \mathbf{d}\Vert })$, $f({\mathbf{a}}_t)\ge f(\mathbf{a})$.

5. By linear approximation theorem (Theorem 5.5), there exists a vector ${\mathbf{z}}_t\in [\mathbf{a},{\mathbf{a}}_t]$ such that

   $$f({\mathbf{a}}_t)-f(\mathbf{a})=\mathrm{\nabla }f(\mathbf{a}{)}^T({\mathbf{a}}_t-\mathbf{a})+\frac{1}{2}({\mathbf{a}}_t-\mathbf{a}{)}^T{\mathrm{\nabla }}^{2}f({\mathbf{z}}_t)({\mathbf{a}}_t-\mathbf{a}).$$

6. Since $\mathbf{a}$ is a stationary point, hence $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

7. Also, by definition of ${\mathbf{a}}_t$, ${\mathbf{a}}_t-\mathbf{a}=t\mathbf{d}$.

8. Thus,

   $$f({\mathbf{a}}_t)-f(\mathbf{a})=\frac{t^2}{2}{\mathbf{d}}^T{\mathrm{\nabla }}^{2}f({\mathbf{z}}_t)\mathbf{d}.$$

9. Since $f(\mathbf{a})$ is local minimum. Hence, $f({\mathbf{a}}_t)-f(\mathbf{a})\ge 0$.

10. Thus, for any $\mathbf{d}\in {\mathbb{R}}^n$ and any $0<t<\frac{r}{\Vert \mathbf{d}\Vert }$, we have

    $$\frac{t^2}{2}{\mathbf{d}}^T{\mathrm{\nabla }}^{2}f({\mathbf{z}}_t)\mathbf{d}\ge 0.$$

11. By the continuity of the Hessian, and the fact that ${\mathbf{z}}_t\to \mathbf{a}$ as $t\to 0^{+}$, we obtain that

    $$\frac{t^2}{2}{\mathbf{d}}^T{\mathrm{\nabla }}^{2}f(\mathbf{a})\mathbf{d}\ge 0\mathrm{\forall }\mathbf{d}\in {\mathbb{R}}^n.$$

12. Thus, ${\mathrm{\nabla }}^{2}f(\mathbf{a})⪰\mathbf{O}$.

The argument for second statement is similar. We can apply the same argument on $-f$ and recognize that if $\mathbf{a}$ is a local maximum for $f$ then it is a local minimum for $-f$.

Theorem 8.3 (Sufficient second order optimality conditions)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Assume that $S$ is open. Further, assume that $f$ is twice continuously differentiable over $S$ and that $\mathbf{a}\in S$ is a stationary point.

Then, the following hold.

1. If ${\mathrm{\nabla }}^{2}f(\mathbf{a})\mathrm{succ}\mathbf{O}$, then $\mathbf{a}$ is a strict local minimum point of $f$ over $S$.

2. If ${\mathrm{\nabla }}^{2}f(\mathbf{a})\prec \mathbf{O}$, then $\mathbf{a}$ is a strict local maximum point of $f$ over $S$.

Proof. Let $\mathbf{a}\in S$ is a stationary point satisfying ${\mathrm{\nabla }}^{2}f(\mathbf{a})\mathrm{succ}\mathbf{O}$.

1. Since the Hessian is continuous ($f$ is twice continuously differentiable), hence there exists an open ball $B(\mathbf{a},r)$ such that ${\mathrm{\nabla }}^{2}f(\mathbf{x})\mathrm{succ}\mathbf{O}$ for every $\mathbf{x}\in B(\mathbf{a},r)$.

2. By linear approximation theorem (Theorem 5.5), for any $\mathbf{x}\in B(\mathbf{a},r)$, there exists a vector $\mathbf{z}\in [\mathbf{a},\mathbf{x}]$ such that

   $$f(\mathbf{x})-f(\mathbf{a})=\frac{1}{2}(\mathbf{x}-\mathbf{a}{)}^T{\mathrm{\nabla }}^{2}f(\mathbf{z})(\mathbf{x}-\mathbf{a}).$$

3. We note that $\mathbf{z}\in B(\mathbf{a},r)$. Hence, ${\mathrm{\nabla }}^{2}f(\mathbf{z})\mathrm{succ}\mathbf{O}$.

4. Thus, $\frac{1}{2}(\mathbf{x}-\mathbf{a}{)}^T{\mathrm{\nabla }}^{2}f(\mathbf{z})(\mathbf{x}-\mathbf{a})>0$.

5. Thus, $f(\mathbf{x})-f(\mathbf{a})>0$ holds true for every $\mathbf{x}\in B(\mathbf{a},r)$.

6. Thus, $\mathbf{a}$ is a strict local minimum point for $f$ over $S$.

The proof for the second statement is similar.

Theorem 8.4 (Hessian based sufficient global optimality condition)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $\mathrm{dom}f={\mathbb{R}}^n$. Further, assume that $f$ is twice continuously differentiable over ${\mathbb{R}}^n$. Suppose that ${\mathrm{\nabla }}^{2}f(\mathbf{x})⪰\mathbf{O}$ for every $\mathbf{x}\in {\mathbb{R}}^n$. Let $\mathbf{a}\in {\mathbb{R}}^n$ be a stationary point of $f$. Then, $\mathbf{a}$ is a global minimum point of $f$.

In other words, if the Hessian of a function is continuous and always positive semidefinite, then all its stationary points are also global minimum points.

Proof. We are given that $\mathbf{a}$ is a stationary point.

1. $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

2. Let $\mathbf{x}\in {\mathbb{R}}^n$.

3. By linear approximation theorem (Theorem 5.5), there exists a vector $\mathbf{z}\in [\mathbf{a},\mathbf{x}]$ such that

   $$f(\mathbf{x})-f(\mathbf{a})=\frac{1}{2}(\mathbf{x}-\mathbf{a}{)}^T{\mathrm{\nabla }}^{2}f(\mathbf{z})(\mathbf{x}-\mathbf{a}).$$

4. By hypothesis, ${\mathrm{\nabla }}^{2}f(\mathbf{z})⪰\mathbf{O}$.

5. Thus, $f(\mathbf{x})-f(\mathbf{a})\ge 0$ for every $\mathbf{x}\in {\mathbb{R}}^n$.

6. Thus, $\mathbf{a}$ is a global minimum point of $f$.

This result is a special case for the global optimality of convex optimization problems. If a convex function is twice differentiable, then its Hessian is positive semidefinite Theorem 9.100.

8.1.4.1. Saddle Points

Definition 8.7 (Saddle point)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Assume that $S$ is open.

A stationary point $\mathbf{a}\in S$ is called a saddle point of $f$ over $U$ if it is neither a local minimum point nor a local maximum point of $f$ over $S$.

Theorem 8.5 (Sufficient condition for saddle points)

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a real valued function with $S=\mathrm{dom}f$. Assume that $S$ is open. Further, assume that $f$ is twice continuously differentiable over $S$ and that $\mathbf{a}\in S$ is a stationary point.

If ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ is an indefinite matrix, then $\mathbf{a}$ is a saddle point of $f$ over $S$.

Proof. We are given that $\mathbf{a}$ is a stationary point and ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ is indefinite.

1. Then, $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

2. Recall from Theorem 4.133 that a matrix is indefinite if and only if at least one eigenvalue is positive and at least one eigenvalue is negative.

3. Let $\lambda >0$ be a positive eigenvalue of ${\mathrm{\nabla }}^{2}f(\mathbf{a})$ and $\mathbf{v}$ be the corresponding normalized eigenvector.

4. Since $S$ is open, hence there exists $r>0$ such that $B(\mathbf{a},r)\subseteq S$.

5. Accordingly, $\mathbf{a}+t\mathbf{v}\in B(\mathbf{a},r)\subseteq S$ for every $t\in [0,r)$.

6. By quadratic approximation theorem (Theorem 5.6),

   $$f(\mathbf{a}+t\mathbf{v})=f(\mathbf{a})+\frac{t^2}{2}{\mathbf{v}}^T{\mathrm{\nabla }}^{2}f(\mathbf{a})\mathbf{v}+o(t^2\Vert \mathbf{v}{\Vert }^2).$$

7. Putting ${\mathrm{\nabla }}^{2}f(\mathbf{a})\mathbf{v}=\lambda \mathbf{v}$ and using $\Vert \mathbf{v}\Vert =1$,

   $$f(\mathbf{a}+t\mathbf{v})=f(\mathbf{a})+\frac{\lambda t^2}{2}+o(t^2).$$

8. Here $o:{\mathbb{R}}_{++}\to \mathbb{R}$ is a function satisfying $\frac{o(x)}{x}\to 0$ as $x\to 0^{+}$.

9. Thus, $\frac{o(t^2)}{t^2}\to 0$ as $t\to 0^{+}$.

10. For every $ϵ>0$, there exists $\delta >0$ such that

    $$\left|\frac{o(t^2)}{t^2}\right|<ϵ,\text{ whenever }t<\delta .$$

11. Choose $ϵ=\lambda$.

12. Then, $-\frac{\lambda t^2}{2}<o(t^2)<\frac{\lambda t^2}{2}$ for every $t<\delta$.

13. Thus, $\frac{\lambda t^2}{2}+o(t^2)>0$ for every $t<\delta$.

14. Thus, there exists $r_1=min(\delta ,r)$ such that for every $0\le t<r_1$, $f(\mathbf{a}+t\mathbf{v})>f(\mathbf{a})$.

15. Thus, $\mathbf{a}$ is not a local maximum point.

16. A similar argument with a negative eigenvalue and corresponding normalized eigenvector shows that $\mathbf{a}$ cannot be a local minimum point.

17. Thus, $\mathbf{a}$ must be a saddle point.

8.1.5. Minimization of Proper Functions

Let $\mathbb{V}$ be a real $n$-dimensional normed linear space. Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a proper function with $S=\mathrm{dom}f$. We consider the problem of minimizing $f$ over a set $A\subseteq S$.

1. $f$ is the objective function or cost function (being minimized).

2. The set $A$ is the feasible set or constraint set.

3. Any $\mathbf{x}\in A$ is a feasible solution or feasible point.

4. If there is at least one feasible point (i.e., $A$ is nonempty), the problem is feasible.

5. Otherwise, the problem is infeasible.

6. Let $p^{\ast }=\underset{\mathbf{x}\in A}{inf}f(\mathbf{x})$ be the optimal value of the minimization problem.

7. We allow $p^{\ast }$ to take values over the extended real line $\bar{\mathbb{R}}$.

8. If the minimization problem is infeasible, then $p^{\ast }=\mathrm{\infty }$.

9. If $p^{\ast }=-\mathrm{\infty }$, then the problem is unbounded below.

10. If there is some ${\mathbf{x}}^{\ast }\in A$ such that $f({\mathbf{x}}^{\ast })=p^{\ast }$, then ${\mathbf{x}}^{\ast }$ is an optimal solution or optimal point or minimizing point or minimizer or global minimum over $A$.

11. Alternatively, $f$ attains a minimum over $A$ at ${\mathbf{x}}^{\ast }$. We write this as

    $${\mathbf{x}}^{\ast }\in \underset{\mathbf{x}\in A}{\arg \min }f(\mathbf{x}).$$

12. If ${\mathbf{x}}^{\ast }$ is a unique minimizer of $f$ over $A$, then we abuse the notation and write

    $${\mathbf{x}}^{\ast }=\underset{\mathbf{x}\in A}{\arg \min }f(\mathbf{x}).$$

13. It is possible that $p^{\ast }$ is finite and yet there is no optimal point in $A$.

14. In other words, the set $\underset{\mathbf{x}\in A}{\arg \min }f(\mathbf{x})$ may be empty.

A basic question of optimization is whether an optimal solution exists.

Remark 8.2 (The set of optimal points)

The set of optimal points for the problem of minimization of a proper function $f$ over a feasible set $A$ is given by

$$\arg {\min }_{\mathbf{x}\in A}f(\mathbf{x})=A\cap f^{-1}(p^{\ast })$$

where $p^{\ast }=\underset{\mathbf{x}\in A}{inf}f(\mathbf{x})$ and $f^{-1}(y)$ denotes the level set of $f$ given by $\{\mathbf{x}\in S|f(\mathbf{x})=y\}$.

This comes directly from the fact that

1. ${\mathbf{x}}^{\ast }$ must be feasible. Hence ${\mathbf{x}}^{\ast }\in A$.

2. $f$ must attain the optimal value at ${\mathbf{x}}^{\ast }$. Thus, $p^{\ast }=f({\mathbf{x}}^{\ast })$. Hence, ${\mathbf{x}}^{\ast }\in f^{-1}(p^{\ast })$.

In other words, the optimal set is the intersection of the feasible set $A$ with the level set $f^{-1}(p^{\ast })$.

Thus, for an optimal solution to exist

1. $p^{\ast }$ must be finite.

2. The level set $f^{-1}(p^{\ast })$ must be nonempty.

3. The feasible set $A$ must be nonempty.

4. The intersection of $A$ with $f^{-1}(p^{\ast })$ must be nonempty.

8.1.5.1. Coercive Functions

Definition 8.8 (Coercive function)

A proper function $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ is called coercive over a set $A$ if for every sequence $\{{\mathbf{x}}_n\}$ of $A$ such that $\underset{k\to \mathrm{\infty }}{lim}\Vert {\mathbf{x}}_k\Vert =\mathrm{\infty }$, we have $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=\mathrm{\infty }$.

If $f$ is coercive over the entire vector space $\mathbb{V}$, we say that $f$ is coercive.

One way to think about coercive functions is that they grow rapidly at the extremes of the domain on which they are defined.

Theorem 8.6 (Level sets of coercive functions)

Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a coercive proper function. Let $a\in \mathbb{R}$. If the level set $f^{-1}(a)$ is nonempty, then it is bounded.

In other words, all nonempty level sets of a coercive function are bounded.

Proof. For contradiction, assume that there exists $a\in \mathbb{R}$ such that $A=f^{-1}(a)$ is unbounded.

1. Then, there exists a sequence $\{{\mathbf{x}}_k\}$ of $A$ such that $\underset{k\to \mathrm{\infty }}{lim}\Vert {\mathbf{x}}_k\Vert =\mathrm{\infty }$.

2. Since $f$ is coercive, hence $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=\mathrm{\infty }$.

3. But, by definition, $f({\mathbf{x}}_k)=a$.

4. Hence $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=a$.

5. We have a contradiction.

6. Hence, $f^{-1}(a)$ must be bounded.

Theorem 8.7 (Sublevel sets of coercive functions)

Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a coercive proper function with $S=\mathrm{dom}f$. For every $r\in \mathbb{R}$, define the sublevel set $S_r$ as $\{\mathbf{x}\in S|f(\mathbf{x})\le r\}$. If $S_r$ is nonempty, then it is bounded.

In other words, all nonempty sublevel sets of a coercive function are bounded.

Proof. For contradiction, assume that there exists $r\in \mathbb{R}$ such that $S_r$ is unbounded.

1. Then, there exists a sequence $\{{\mathbf{x}}_k\}$ of $S_r$ such that $\underset{k\to \mathrm{\infty }}{lim}\Vert {\mathbf{x}}_k\Vert =\mathrm{\infty }$.

2. Since $f$ is coercive, hence $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=\mathrm{\infty }$.

3. But, by definition, $f({\mathbf{x}}_k)\le r$.

4. Hence $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)\le r$.

5. We have a contradiction.

6. Hence, $S_r$ must be bounded.

8.1.5.2. Weierstrass’ Theorem

In this subsection, we examine the problem of unconstrained minimization of a proper closed function. Recall from Definition 3.64 that a function is called closed if all its sublevel sets are closed.

Theorem 8.8 (Unconstrained minimization of a proper closed function)

Let $\mathbb{V}$ be a real $n$-dim normed linear space. Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a proper closed function with $S=\mathrm{dom}f$.

Let $p^{\ast }=\underset{\mathbf{x}\in \mathbb{V}}{inf}f(\mathbf{x})$. Let $X=f^{-1}(p^{\ast })$ be the set of minimizers of $f$ (over all of $\mathbb{V}$). For every $p\in \mathbb{R}$, let $S_p$ denote the sublevel set $\{\mathbf{x}|f(\mathbf{x})\le p\}$. Then,

1. $p^{\ast }<\mathrm{\infty }$.

2. $X=\bigcap _{p>p^{\ast }}{S}_p$.

3. $X$ is closed.

In words, the minimal value $p^{\ast }$ is either finite or $-\mathrm{\infty }$. The set of minimizers is the intersection of all the (closed) sublevel sets for $p>p^{\ast }$. The set of minimizers is a closed set.

Proof. We show that $p^{\ast }<\mathrm{\infty }$.

1. Since $f$ is proper, hence $S$ is nonempty.

2. Thus, there exists $\mathbf{x}\in S$ such that $f(\mathbf{x})<\mathrm{\infty }$.

3. Hence, $p^{\ast }=\underset{\mathbf{x}\in \mathbb{V}}{inf}f(\mathbf{x})<\mathrm{\infty }$.

Let $Y=\bigcap _{p>p^{\ast }}{S}_p$. We have to show that $X=Y$.

We first show that $X\subseteq Y$.

1. let $\mathbf{x}\in X$.

2. Then, $f(\mathbf{x})=p^{\ast }$.

3. Then, $f(\mathbf{x})<p$ for every $p>p^{\ast }$.

4. Hence, $\mathbf{x}\in S_p$ for every $p>p^{\ast }$.

5. Thus, $\mathbf{x}\in Y$.

6. Thus, $X\subseteq Y$.

We now show that $Y\subseteq X$.

1. Let $\mathbf{x}\in Y$.

2. We claim that $f(\mathbf{x})=p^{\ast }$ must hold true.

3. $f(\mathbf{x})$ cannot be smaller than $p^{\ast }$ since by definition $p^{\ast }=inff(\mathbf{x})$.

4. Thus, $f(\mathbf{x})\ge p^{\ast }$ must hold true.

5. Now, for contradiction, assume that $f(\mathbf{x})>p^{\ast }$.

   1. Let $q=f(\mathbf{x})$.

   2. Let $p=\frac{p^{\ast }+q}{2}$.

   3. Then, $p^{\ast }<p<q$.

   4. Hence, $\mathbf{x}\notin S_p$.

   5. But then, $\mathbf{x}\notin Y$ since $Y\subseteq S_p$ as $p>p^{\ast }$.

   6. A contradiction.

6. Thus, the only allowed value for $f(\mathbf{x})$ is $p^{\ast }$.

7. Thus, $\mathbf{x}\in X$.

8. Thus, $Y\subseteq X$.

9. Thus,

   $$X=f^{-1}(p^{\ast })=Y=\bigcap _{p>p^{\ast }}{S}_p.$$

We show that $X$ is closed.

1. If $X$ is empty, then it is closed by definition.

2. Otherwise, $X$ is an intersection of sublevel sets.

3. Since $f$ is closed, its sublevel sets are closed.

4. Hence $S_p$ is closed for every $p$.

5. Thus, $X$ is an intersection of closed sets.

6. Thus, $X$ is closed.

Theorem 8.9 (Weierstrass’ theorem)

Let $\mathbb{V}$ be a real $n$-dim normed linear space. Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a proper closed function with $S=\mathrm{dom}f$.

Let $p^{\ast }=\underset{\mathbf{x}\in \mathbb{V}}{inf}f(\mathbf{x})$. Let $X=f^{-1}(p^{\ast })$ be the set of minimizers of $f$ (over all of $\mathbb{V}$). For every $p\in \mathbb{R}$, let $S_p$ denote the sublevel set $\{\mathbf{x}|f(\mathbf{x})\le p\}$.

Assume that one of the following conditions are true.

1. $S=\mathrm{dom}f$ is closed and bounded.

2. There exists a scalar $r\in \mathbb{R}$ such that the sublevel set $S_r=\{\mathbf{x}|f(\mathbf{x})\le r\}$ is nonempty and bounded.

3. $f$ is coercive.

Then, $X$ (the set of minimizers of $f$) is nonempty and compact.

Here is this result in a simplified language. The theorem addresses the problem of unconstrained minimization of a function.

1. If $f$ is proper and closed and its domain is closed and bounded, then its set of minimizers is nonempty and compact.

2. If $f$ is proper and closed and any sublevel set of $f$ is bounded, then its set of minimizers is nonempty and compact.

3. If $f$ is proper, closed and coercive, then its set of minimizers is nonempty and compact.

Since the set of minimizers is nonempty and compact, it is nonempty, closed and bounded.

Proof. If there exists $\mathbf{x}\in S$ such that $f(\mathbf{x})=p^{\ast }$, then $X$ is nonempty. To show that $X$ is compact, we just need to show that it is closed and bounded. By Theorem 4.66, every closed and bounded subset of $\mathbb{V}$ which is a real $n$-dimensional normed linear space is compact. By Theorem 8.8, the optimal value $p^{\ast }<\mathrm{\infty }$ and $X$, the set of minimizers of a proper and closed function, is closed. Thus, we just need to show that $X$ is nonempty and bounded.

Assume that condition (1) holds.

1. Since $f$ is proper, hence $S$ is nonempty.

2. Consider a sequence $\{{\mathbf{x}}_k\}$ of $S$ such that $\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=p^{\ast }$.

3. Since $S$ is bounded, hence $\{{\mathbf{x}}_k\}$ has a convergent subsequence by Bolzano-Weierstrass theorem (Theorem 4.69).

4. Let $\{{\mathbf{x}}_l\}$ be the convergent subsequence of $\{{\mathbf{x}}_k\}$ with $lim{\mathbf{x}}_l={\mathbf{x}}^{\ast }$.

   1. Such a sequence exists since $p^{\ast }=\underset{\mathbf{x}\in \mathbb{V}}{inf}f(\mathbf{x})$.

   2. Thus, for every $k\in \mathbb{N}$, there exists ${\mathbf{x}}_k\in S$ such that $f({\mathbf{x}}_k)-p^{\ast }<\frac{1}{k}$.

5. Since $S$ is closed, hence ${\mathbf{x}}^{\ast }\in S$.

6. Since $\{f({\mathbf{x}}_k)\}$ is convergent and $\{f({\mathbf{x}}_l)\}$ is a subsequence of $\{f({\mathbf{x}}_k)\}$, hence

   $$\underset{l\to \mathrm{\infty }}{lim}f({\mathbf{x}}_l)=\underset{k\to \mathrm{\infty }}{lim}f({\mathbf{x}}_k)=p^{\ast }.$$

7. Since $f$ is closed, hence it is lower semicontinuous (l.s.c.) at ${\mathbf{x}}^{\ast }\in S$ (see Theorem 3.119).

8. Also, note that since $\{f({\mathbf{x}}_k)\}$ is convergent, hence

   $$\underset{l\to \mathrm{\infty }}{lim inf}f({\mathbf{x}}_l)=\underset{l\to \mathrm{\infty }}{lim}f({\mathbf{x}}_l).$$

9. Since $f$ is l.s.c. at ${\mathbf{x}}^{\ast }\in S$, hence by Theorem 3.108,

   $$f({\mathbf{x}}^{\ast })\le \underset{l\to \mathrm{\infty }}{lim inf}f({\mathbf{x}}_l)=\underset{l\to \mathrm{\infty }}{lim}f({\mathbf{x}}_l)=\underset{k\to \mathrm{\infty }}{lim inf}f({\mathbf{x}}_k)=p^{\ast }.$$

10. Since $p^{\ast }$ is the infimum value of $f$, hence $f({\mathbf{x}}^{\ast })$ cannot be smaller than $p^{\ast }$.

11. Thus, $f({\mathbf{x}}^{\ast })=p^{\ast }$ must be true.

12. Since $f$ is proper and ${\mathbf{x}}^{\ast }\in S$, hence $p^{\ast }>-\mathrm{\infty }$. Thus the optimal value is finite.

13. Thus, $f({\mathbf{x}}^{\ast })=p^{\ast }$ means that ${\mathbf{x}}^{\ast }$ is an optimal solution for the minimization of $f$.

14. Thus, the set $X=f^{-1}(p^{\ast })$ is nonempty.

15. $X$ is bounded since $S$ is bounded by hypothesis.

16. Since $X$ is closed and bounded. Hence, $X$ is compact.

Assume that condition (2) holds.

1. The sublevel set $S_r$ is nonempty and bounded.

2. Since $f$ is closed, hence $S_r$ is also closed.

3. Consider the restriction of $f$ on $S_r$ given by

   $$\begin{array}{r}\tilde{f}=\{\begin{array}{ll}f(\mathbf{x}),& f(\mathbf{x})\le r\\ \mathrm{\infty },& \text{ otherwise }.\end{array}\end{array}$$

4. Then, $\mathrm{dom}\tilde{f}=S_r$. Thus, $\mathrm{dom}\tilde{f}$ is nonempty, closed and bounded.

5. Then, $\tilde{f}$ never takes the value $-\mathrm{\infty }$ and is not identically $\mathrm{\infty }$. Hence, $\tilde{f}$ is a proper function.

6. Since $f$ is closed, hence sublevel sets of $\tilde{f}$ are also closed.

   1. For any $p>r$, the sublevel set of $\tilde{f}$ is identical to $S_r$.

   2. For any $p\le r$, the sublevel set of $\tilde{f}$ is identical to $S_p$, the corresponding sublevel set of $f$.

7. Thus, $\tilde{f}$ is closed too.

8. Applying condition (1) on $\tilde{f}$, the set of minimizers of $\tilde{f}$ is nonempty and compact.

9. We also note that the minimizers of $f$ are identical to the minimizers of $\tilde{f}$.

10. Thus, the set of minimizers of $f$ is nonempty and compact.

Assume that condition (3) holds.

1. Since $f$ is proper, hence it has some nonempty sublevel sets.

2. Let $r\in \mathbb{R}$ be one such scalar such that the sublevel set $S_r=\{\mathbf{x}\in S|f(\mathbf{x})\le r\}$ is nonempty.

3. By Theorem 8.7, the nonempty sublevel sets of $f$ are bounded. Hence, $S_r$ is bounded.

4. Then, by applying condition (2), the set of minimizers of $f$ is nonempty and compact.

Corollary 8.1 (Minimizing a proper closed function over a closed set)

Let $f:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ be a proper closed function with $S=\mathrm{dom}f$. Let $A\subseteq S$ be a nonempty closed set. Consider the problem of minimizing $f$ over $A$.

Further, assume that one of the following conditions are true.

1. $A$ is bounded.

2. There exists a scalar $r\in \mathbb{R}$ such that the set $\{\mathbf{x}\in A|f(\mathbf{x})\le r\}$ is nonempty and bounded.

3. $f$ is coercive over $A$.

Then, the set of minimizers of $f$ over $A$ is nonempty and compact.

Proof. We define a restriction $g:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ of $f$ over the set $A$ as follows

$$\begin{array}{r}g(\mathbf{x})=\{\begin{array}{ll}f(\mathbf{x}),& \mathbf{x}\in A\\ \mathrm{\infty },& \text{ otherwise }.\end{array}\end{array}$$

1. $\mathrm{dom}g=A$. Thus, $\mathrm{dom}g$ is closed.

2. Since $f$ never takes the value $-\mathrm{\infty }$, hence $g$ also never takes the value $-\mathrm{\infty }$.

3. Since $A$ is nonempty, hence there exists $\mathbf{x}\in \mathbb{V}$ such that $g(\mathbf{x})<\mathrm{\infty }$.

4. Hence, $g$ is a proper function.

5. Also, note that the set $\{\mathbf{x}\in A|f(\mathbf{x})\le r\}$ is nothing but the sublevel set of $g$ for the scalar $r$.

   $$\{\mathbf{x}|g(\mathbf{x})\le r\}=A\cap \{\mathbf{x}|f(\mathbf{x})\le r\}.$$

6. Since $f$ is closed, all its sublevel sets are closed.

7. $A$ is closed, hence $A\cap \{\mathbf{x}|f(\mathbf{x})\le r\}$ is also closed.

8. Hence, all the sublevel sets of $g$ are also closed. Hence, $g$ is also closed.

9. The set of minimizers of $f$ over $A$ is nothing but the set of minimizers of $g$.

10. If $f$ is coercive over $A$, then $g$ is coercive (over its entire domain).

Thus, applying Theorem 8.9, the set of minimizers of $g$ is nonempty and compact. So is the set of minimizers of $f$ over $A$.

Corollary 8.2 (Minimizing a real valued l.s.c. function over a closed set)

Let $f:\mathbb{V}\to \mathbb{R}$ be a real valued function with $\mathrm{dom}f=\mathbb{V}$. Let $A\subseteq \mathbb{V}$ be a nonempty closed set. Assume that $f$ is lower semicontinuous over $A$. Consider the problem of minimizing $f$ over $A$.

Further, assume that one of the following conditions are true.

1. $A$ is bounded.

2. There exists a scalar $r\in \mathbb{R}$ such that the set $\{\mathbf{x}\in A|f(\mathbf{x})\le r\}$ is nonempty and bounded.

3. $f$ is coercive over $A$.

Then, the set of minimizers of $f$ over $A$ is nonempty and compact.

Proof. We define a restriction $g:\mathbb{V}\to (-\mathrm{\infty },\mathrm{\infty }]$ of $f$ over the set $A$ as follows

$$\begin{array}{r}g(\mathbf{x})=\{\begin{array}{ll}f(\mathbf{x}),& \mathbf{x}\in A\\ \mathrm{\infty },& \text{ otherwise }.\end{array}\end{array}$$

1. $\mathrm{dom}g=A$. Thus, $\mathrm{dom}g$ is closed.

2. Since $f$ is real valued hence $g$ also never takes the value $-\mathrm{\infty }$.

3. Since $A$ is nonempty, hence there exists $\mathbf{x}\in \mathbb{V}$ such that $g(\mathbf{x})<\mathrm{\infty }$.

4. Hence, $g$ is a proper function.

5. Since $f$ is l.s.c. on $A$, hence $g$ is closed.

6. Also, note that the set $\{\mathbf{x}\in A|f(\mathbf{x})\le r\}$ is nothing but the sublevel set of $g$ for the scalar $r$.

7. The set of minimizers of $f$ over $A$ is nothing but the set of minimizers of $g$.

8. If $f$ is coercive over $A$, then $g$ is coercive (over its entire domain).

Thus, applying Theorem 8.9, the set of minimizers of $g$ is nonempty and compact. So is the set of minimizers of $f$ over $A$.

8.1.6. Some Simple Problems and Technical Results

In this subsection, we provide some technical results on some simple optimization problems. These results are useful building blocks for bigger problems.

Theorem 8.10 (Nonnegativity of affine functional on nonnegative orthant)

Consider the affine function $f:{\mathbb{R}}^n\to \mathbb{R}$ given by

$$f(\mathbf{x})={\mathbf{a}}^T\mathbf{x}+b.$$

Then, $f(\mathbf{x})\ge 0$ for every $\mathbf{x}⪰\mathbf{0}$ if and only if $\mathbf{a}⪰\mathbf{0}$ and $b\ge 0$.

Proof. Assume that $\mathbf{a}⪰\mathbf{0}$ and $b\ge 0$. Then $f(\mathbf{x})$ is product and sum of nonnegative terms. Hence, $f(\mathbf{x})\ge 0$.

For the converse, assume that $f(\mathbf{x})\ge 0$ for every $\mathbf{x}⪰\mathbf{0}$.

1. If $b<0$, then for $\mathbf{x}=\mathbf{0}$, $f(\mathbf{x})=b<0$. Hence, $b$ cannot be smaller than 0.

2. Consider some $\mathbf{a}\prec \mathbf{0}$.

3. Assume that $a_i<0$ for some $i\in [1,\dots ,n]$.

4. Let $\mathbf{x}$ be such that $x_j=0$ for all $j\ne i$ and $x_i=t$ for some $t>0$.

5. Then, $f(\mathbf{x})=t{a}_i+b$.

6. By increasing $t$ suitably, we can get $f(\mathbf{x})<0$.

7. Thus, $\mathbf{a}⪰\mathbf{0}$ is also necessary to ensure that $f(\mathbf{x})\ge 0$ for all $\mathbf{x}⪰\mathbf{0}$.

Theorem 8.11 (Minimization of linear functional over unit ball)

For any $\mathbf{a}\in {\mathbb{R}}^n$, the optimal value of the problem

$$\underset{\Vert \mathbf{x}\Vert \le 1}{inf}{\mathbf{a}}^T\mathbf{x}$$

is $-\Vert \mathbf{a}\Vert$.

Proof. If $\mathbf{a}=\mathbf{0}$, then ${\mathbf{a}}^T\mathbf{x}=0$. The minimum value is 0 which is equal to $-\Vert \mathbf{a}\Vert$.

Now consider the case where $\mathbf{a}\ne \mathbf{0}$.

1. By Cauchy Schwartz inequality

   $${\mathbf{a}}^T\mathbf{x}\ge -\Vert \mathbf{a}\Vert \Vert \mathbf{x}\Vert \ge -\Vert \mathbf{a}\Vert$$

   for any $\Vert \mathbf{x}\Vert \le 1$.

2. Thus, we have established a lower bound:

   $$\underset{\Vert \mathbf{x}\Vert \le 1}{inf}{\mathbf{a}}^T\mathbf{x}\ge -\Vert \mathbf{a}\Vert .$$

3. We now show that the lower bound is indeed attained.

4. Let $\mathbf{x}=\frac{-\mathbf{a}}{\Vert \mathbf{a}\Vert }$.

5. Then $\Vert \mathbf{x}\Vert =1$. Thus, $\mathbf{x}$ belongs to the unit ball.

6. Now ${\mathbf{a}}^T\mathbf{x}=-\Vert \mathbf{a}\Vert$. Thus, the lower bound is attained.