9.12. Separation Theorems

Separation theorems provide us ways to find a separating hyperplane between two disjoint convex sets. Hyperplanes require the notion of an inner product.

Throughout this section, we assume that $\mathbb{V}$ is a finite dimensional real vector space endowed with norm $\Vert \cdot \Vert :\mathbb{V}\to \mathbb{R}$ and a real inner product $⟨\cdot ,\cdot ⟩:\mathbb{V}\times \mathbb{V}\to \mathbb{R}$. The norm in general is not necessarily induced by the inner product. The Euclidean norm is denoted by $\Vert \cdot {\Vert }_2$.

9.12.1. Types of Separating Hyperplanes

Definition 9.59 (Separating hyperplane)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. For any two subsets $C$ and $D$ of $\mathbb{V}$, a hyperplane $H$ is said to separate $C$ and $D$ if $C$ is contained in one of the closed halfspaces corresponding to $H$ and $D$ is contained in the other closed halfspace.

Let $H$ be given by

$$H=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{a}⟩=b\}$$

where $\mathbf{a}\in {\mathbb{V}}^{\ast }$ and $b\in \mathbb{R}$.

If $C$ and $D$ are separated by $H$, then either

$$⟨{\mathbf{x}}_1,\mathbf{a}⟩\le b\le ⟨{\mathbf{x}}_2,\mathbf{a}⟩\mathrm{\forall }{\mathbf{x}}_1\in C,{\mathbf{x}}_2\in D$$

or

$$⟨{\mathbf{x}}_2,\mathbf{a}⟩\le b\le ⟨{\mathbf{x}}_1,\mathbf{a}⟩\mathrm{\forall }{\mathbf{x}}_1\in C,{\mathbf{x}}_2\in D$$

holds true.

We can also say that the two sets $C$ and $D$ can be separated by a hyperplane or there exists a hyperplane separating $C$ and $D$ if there exists a nonzero vector $\mathbf{a}\in {\mathbb{V}}^{\ast }$ such that

$$\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩\le \underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩.$$

Any choice of $b\in [l,u]$ where $l=\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩$ and $u=\underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩$ describes a separating hyperplane.

This definition allows for some degenerate possibilities where both $C\subseteq H$ and $D\subseteq H$ since the closed halfspaces contain $H$ (as their boundary).

Definition 9.60 (Proper separation)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. For any two subsets $C$ and $D$ of $\mathbb{V}$, a hyperplane $H$ is said to properly separate $C$ and $D$ if $H$ separates them and both are not entirely contained in $H$; i.e., either $C⊈H$ or $D⊈H$ or both.

Proper separation still allows for the possibility that parts of $C$ or $D$ lies inside $H$. But it ensures that $C\mathrm{△}D\ne \varnothing$. $C\cap D\subseteq H$ is the common part.

If $C$ and $D$ are properly separated by $H$, then they are also separated by $H$. Hence

$$⟨{\mathbf{x}}_1,\mathbf{a}⟩\le b\le ⟨{\mathbf{x}}_2,\mathbf{a}⟩\mathrm{\forall }{\mathbf{x}}_1\in C,{\mathbf{x}}_2\in D$$

or

$$⟨{\mathbf{x}}_2,\mathbf{a}⟩\le b\le ⟨{\mathbf{x}}_1,\mathbf{a}⟩\mathrm{\forall }{\mathbf{x}}_1\in C,{\mathbf{x}}_2\in D$$

holds true. Also, there exists at least one ${\mathbf{x}}_1\in C$ and ${\mathbf{x}}_2\in D$ such that at least one of these inequalities is a strict inequality thus ensuring that either $C$ or $D$ or both are not contained entirely in $H$.

We can also say that the two sets $C$ and $D$ can be properly separated by a hyperplane or there exists a hyperplane properly separating $C$ and $D$ if there exists a nonzero vector $\mathbf{a}\in {\mathbb{V}}^{\ast }$ such that

$$\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩\le \underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩.$$

and

$$\underset{\mathbf{x}\in C}{inf}⟨\mathbf{x},\mathbf{a}⟩<\underset{\mathbf{x}\in D}{sup}⟨\mathbf{x},\mathbf{a}⟩.$$

Any choice of $b\in [l,u]$ where $l=\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩$ and $u=\underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩$ describes a properly separating hyperplane. This characterization is proved in Theorem 9.161 below.

If a convex set has a nonempty interior, then it cannot be contained in a hyperplane. Two disjoint sets one of which has a nonempty interior can indeed be properly separated.

Definition 9.61 (Strong separation)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. For any two subsets $C$ and $D$ of $\mathbb{V}$, a hyperplane $H$ is said to strongly separate $C$ and $D$ if there exists an $r>0$ such that $C+rB$ is contained in one of the open halfspaces corresponding to $H$ and $D+rB$ is contained in the other open halfspace.

Under these assumptions, one set lies in the interior of $H_{++}$ and the other set lies in the interior of $H_{--}$. Under strong separation, $C\cap D=\varnothing$ is guaranteed since $H_{++}\cap H_{--}=\varnothing$.

We can also say that the two sets $C$ and $D$ can be strongly separated by a hyperplane or there exists a hyperplane strongly separating $C$ and $D$ if there exists a nonzero vector $\mathbf{a}\in {\mathbb{V}}^{\ast }$ such that

$$\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩<\underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩.$$

Any choice of $b\in (l,u)$ where $l=\underset{\mathbf{x}\in C}{sup}⟨\mathbf{x},\mathbf{a}⟩$ and $u=\underset{\mathbf{x}\in D}{inf}⟨\mathbf{x},\mathbf{a}⟩$ describes a strongly separating hyperplane.

Strong separation is called as strict separation in [9].

Remark 9.9

Separation is translation invariant. For any $\mathbf{a}\in \mathbb{V}$ and convex sets $C,D\subseteq \mathbb{V}$,

$C$ and $D$ are separated by a hyperplane $H$ if and only if $C-\mathbf{a}$, $D-\mathbf{a}$ are separated by the hyperplane $H-\mathbf{a}$.

9.12.2. Disjoint Affine and Convex Sets

Separation between sets can be discussed at several levels. We start with a simple case where an affine set and a relatively open convex set are disjoint.

Theorem 9.160 (Disjoint affine and relatively open convex sets)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. Let $C$ be a nonempty relatively open convex set of $\mathbb{V}$. Let $M\subseteq \mathbb{V}$ be an affine subspace such that $M\cap C=\varnothing$. Then, there exists a hyperplane $H$ such that $M\subseteq H$ and one of the two open halfspaces associated with $H$ contains $C$.

Proof. Consider the case where $M$ is a hyperplane. Then, $H=M$ and the result is immediate.

1. If for contradiction, $C$ is not contained in one of the open halfspaces of $M$.

2. Then, $C\cap H\ne \varnothing$ since $H$ is the boundary of the halfspaces.

3. It will contradict our hypothesis that $C\cap M=\varnothing$.

4. Thus, $C$ must be in one of the open halfspaces of $H$.

Now, assume that $M$ is not a hyperplane.

Without loss of generality, assume that $\mathbf{0}\in M$. We can do this by picking any element $\mathbf{m}$ of $M$ and replacing $M$ by $M-\mathbf{m}$ and $C$ by $C-\mathbf{m}$. Thus, $M$ is a subspace.

The proof is constructive. We iteratively construct subspaces of increasing dimension which contain $M$ but don’t contain $C$ till we reach a hyperplane.

1. Let $\dim M=d$ where $d<n$.

2. Let $S=M$ be the initial subspace.

3. Let $r=n-1-d$ be the number of iterations (if $M$ is a hyperplane, no iterations are needed).

4. For $i=1,\dots ,r$:

   1. Replace $S$ by a subspace $S^{\prime }$ such that $\dim S^{\prime }=\dim S+1$ and $S^{\prime }\cap C=\varnothing$.

   2. Let $S=S^{\prime }$

5. We have arrived with a hyperplane $S$ such that $S\cap C=\varnothing$.

Towards this, let us look at the procedure to construct $S^{\prime }$ from $S$ such that $\dim S^{\prime }=\dim S+1$ and $S\cap C=\varnothing$. Here is the outline

1. Let $S^{\perp }$ be the orthogonal complement of $S$.

2. Pick any 2D subspace $P$ of $S^{\perp }$.

3. Pick a line $L$ in $P$ which passes through origin but doesn’t intersect with $C$; i.e. $L\cap C=\varnothing$.

4. Construct the subspace $S^{\prime }=S\oplus L$. The direct sum is valid since $S\cap L=\{\mathbf{0}\}$.

5. Note that $\dim S^{\prime }=\dim S+1$ and $S^{\prime }\cap C=\varnothing$ since $S\cap C=\varnothing$ as well as $L\cap C=\varnothing$.

We now elaborate the procedure for finding the line $L$:

1. We have $S$ as a linear subspace with $\dim S<n-1$ and $S\cap C=\varnothing$.

2. $\dim S^{\perp }=n-\dim S>n-(n-1)=1$.

3. Since $\dim S^{\perp }>1$, it contains a two dimensional subspace $P$.

4. Consider the set $C^{\prime }=P\cap (C-S)$.

5. The set $C-S$ has some properties.

   1. $C-S$ is convex since it is the sum of two convex sets.

   2. Since $\mathbf{0}\in S$, hence $C=C-\mathbf{0}\subseteq C-S$.

   3. Since $C\cap S=\varnothing$, hence $\mathbf{0}\notin C-S$.

   4. Note that due to Theorem 9.157:

      $$\mathrm{ri}(C-S)=\mathrm{ri}C-\mathrm{ri}S=C-S$$

      since $C$ is relatively open and $S$ is affine, hence relatively open.

   5. Thus, $C-S$ is relatively open.

6. Consequently, the set $C^{\prime }$ also has some properties.

   1. $C^{\prime }$ is convex since it is the intersection of two convex sets.

   2. Since $\mathbf{0}\notin C-S$, hence, $\mathbf{0}\notin C^{\prime }=P\cap (C-S)$.

   3. $C^{\prime }$ is relatively open.

      1. If $C^{\prime }$ is empty, then $C^{\prime }$ is relatively open vacuously.

      2. Otherwise, by Theorem 9.155:

      $$\mathrm{ri}{C}^{\prime }=\mathrm{ri}(P\cap (C-S))=P\cap \mathrm{ri}(C-S)=P\cap (C-S).$$

      Thus, $\mathrm{ri}{C}^{\prime }=C^{\prime }$ implies that $C^{\prime }$ is relatively open.

7. We now seek a line $L\subseteq P$ passing through the origin $\mathbf{0}$ that doesn’t intersect $C^{\prime }$.

8. We note that $L\cap C^{\prime }=\varnothing ⟹L\cap C=\varnothing$.

   1. Suppose for contradiction that $L\cap C^{\prime }=\varnothing$ but $L\cap C\ne \varnothing$.

   2. Let $\mathbf{x}\in L\cap C$.

   3. Then, $\mathbf{x}\in L\subseteq P$.

   4. Also, $\mathbf{x}\in C\subseteq C-S$.

   5. Thus, $\mathbf{x}\in C^{\prime }=P\cap (C-S)$.

   6. This means that $\mathbf{x}\in L\cap C^{\prime }$.

   7. In other words, $L\cap C^{\prime }\ne \varnothing$ which contradicts our assumption.

9. How to choose $L$? There are three possibilities:

   1. $C^{\prime }$ is empty.

   2. $C^{\prime }$ is contained in a line; i.e., $\dim \mathrm{aff}{C}^{\prime }=1$.

   3. $\mathrm{aff}{C}^{\prime }=P$ and $\dim \mathrm{aff}{C}^{\prime }=2$.

   $\mathrm{aff}{C}^{\prime }$ cannot be larger than $P$ since $C^{\prime }\subseteq P$ and $P$ is affine.

10. If $C^{\prime }$ is empty, then any line in $P$ passing through origin will do.

11. If $\mathrm{aff}C$ is a line.

    1. If $\mathrm{aff}{C}^{\prime }$ passes through the origin, we can take a line perpendicular to $\mathrm{aff}{C}^{\prime }$ passing through origin. In this case, $L\cap \mathrm{aff}{C}^{\prime }=\{\mathbf{0}\}$. Since $\mathbf{0}\notin C^{\prime }$, hence $L\cap C^{\prime }=\varnothing$.

    2. If $\mathrm{aff}{C}^{\prime }$ is a line that doesn’t pass through $\mathbf{0}$, we can simply take a line that is parallel to $\mathrm{aff}{C}^{\prime }$ and passes through $\mathbf{0}$. In this case $L\cap \mathrm{aff}{C}^{\prime }=\varnothing$. Consequently, $L\cap C^{\prime }=\varnothing$.

12. If $\mathrm{aff}{C}^{\prime }$ is the entire 2D subspace $P$.

    1. Consider the set $K=\bigcup \{t{C}^{\prime }|t>0\}$.

    2. $C^{\prime }\subseteq K$ (specifically, for $t=0$).

    3. $K$ is a convex cone containing $C^{\prime }$ but not containing $\mathbf{0}$.

    4. $K$ is relatively open since $C^{\prime }$ is relatively open.

    5. A boundary ray of $K$ doesn’t intersect with $C^{\prime }$ since $K$ is relatively open.

    6. Take $L$ to be any of the two boundary rays of $K$ and extend it to a line passing through $\mathbf{0}$.

9.12.3. Proper Separation Characterization

Theorem 9.161 (Characterization of proper separation)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be nonempty subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ properly if and only if there exists a vector $\mathbf{a}\in \mathbb{V}$ such that

$$inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}\ge sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}$$

and

$$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}.$$

Proof. Assume that there is some vector $\mathbf{a}\in \mathbb{V}$ satisfying the conditions above.

1. Since $\mathbf{a}$ satisfies the second strict inequality, hence $\mathbf{a}\ne \mathbf{0}$. Otherwise, the second inequality cannot hold.

2. Choose any $b\in \mathbb{R}$ such that

   $$\underset{\mathbf{x}\in S}{inf}\{⟨\mathbf{x},\mathbf{a}⟩\}\ge b\ge \underset{\mathbf{x}\in S}{sup}\{⟨\mathbf{x},\mathbf{a}⟩\}.$$

3. Then, the set $H=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩=b\}$ is a hyperplane.

4. Let $H_{+}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩\ge b\}$ and $H_{-}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩\le b\}$.

5. Clearly, $S\subseteq H_{+}$ and $T\subseteq H_{-}$.

6. The second strict inequality ensures that both $S$ and $T$ cannot be contained in $H$.

   1. For contradiction, assume $S\subseteq H$ and $T\subseteq H$

   2. Then $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}=b$ and $inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}=b$.

   3. Thus, the second inequality will not hold.

7. Thus, $S$ and $T$ are properly separated.

Now, assume that $S$ and $T$ are properly separated by some hyperplane $H$.

1. Let $H=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩=b\}$ be the specification of the said hyperplane.

2. Let $H_{+}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩\ge b\}$ and $H_{-}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩\le b\}$.

3. Without loss of generality, assume that $S\subseteq H_{+}$ and $T\subseteq H_{+}$.

4. Then, $⟨\mathbf{x},\mathbf{a}⟩\ge b$ for every $\mathbf{x}\in S$.

5. Thus, $inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}\ge b$.

6. Similarly, $⟨\mathbf{x},\mathbf{a}⟩\le b$ for every $\mathbf{x}\in T$.

7. Thus, $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}\le b$.

8. Thus, the first inequality is satisfied.

9. Since $H$ properly separates $S$ and $T$, hence either $S⊈H$ or $T⊈H$.

10. If $S⊈H$, then there exists $\mathbf{x}\in S$ such that $⟨\mathbf{x},\mathbf{a}⟩>b$.

11. Consequently, $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>b$.

12. If $T⊈H$, then there exists $\mathbf{x}\in T$ such that $⟨\mathbf{x},\mathbf{a}⟩<b$.

13. Consequently, $inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}<b$.

14. Combining,

    $$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}$$

    must be true.

9.12.3.1. Proper Separation : Set and Point

Remark 9.10 (Proper separation between a set and a point)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. We consider the case of proper separation between a set $S$ and a point $\mathbf{p}$. We can form a set $T=\{\mathbf{p}\}$. Then the proper separation between $S$ and $\mathbf{p}$ can be described in terms of proper separation between $S$ and $T$.

Then

$$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}=inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}=⟨\mathbf{p},\mathbf{a}⟩.$$

Hence, $S$ and $\mathbf{p}$ are properly separated if and only if there exists a vector $\mathbf{a}\in \mathbb{V}$ such that

$$inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}\ge ⟨\mathbf{p},\mathbf{a}⟩\text{ and }sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>⟨\mathbf{p},\mathbf{a}⟩.$$

We now consider a hyperplane

$$H=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{a}⟩=⟨\mathbf{p},\mathbf{a}⟩\}.$$

And let $H_{+}$ denote one of its closed half-spaces

$$H_{+}=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{a}⟩\ge ⟨\mathbf{p},\mathbf{a}⟩\}.$$

We can clearly see that

1. $\mathbf{a}\in H$.

2. $S\subseteq H_{+}$. $S$ is contained entirely in the closed half-space $H_{+}$.

3. $S⊈H$. $S$ is not contained entirely in $H$.

Thus, if a set and a point are properly separated, then there exists a hyperplane that passes through the point, contains the set in one of its half-spaces but does not fully contain the set itself.

9.12.4. Separating Hyperplane Theorems

Theorem 9.162 (Separating hyperplane theorem I)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be nonempty convex subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ properly if and only if $\mathrm{ri}S\cap \mathrm{ri}T=\varnothing$.

In other words, two sets are properly separated if and only if their relative interiors are disjoint.

Proof. Let $C=S-T$.

1. By Theorem 9.142, $\mathrm{ri}S$ and $\mathrm{ri}T$ are nonempty since $S$ and $T$ are nonempty convex sets.

2. By Theorem 9.157, $\mathrm{ri}C=\mathrm{ri}S-\mathrm{ri}T$ and it is nonempty.

3. Note that, $\mathbf{0}\notin \mathrm{ri}C$ if and only if $\mathrm{ri}S\cap \mathrm{ri}T=\varnothing$.

   1. If $\mathbf{0}\in \mathrm{ri}C$ then there exists $\mathbf{x}\in \mathrm{ri}S\cap \mathrm{ri}T$ such that $\mathbf{x}-\mathbf{x}=\mathbf{0}$.

Now assume that $\mathrm{ri}S\cap \mathrm{ri}T=\varnothing$.

1. Thus, $\mathbf{0}\notin \mathrm{ri}C$.

2. Consider the affine set $M=\{\mathbf{0}\}$.

3. Clearly, $M\cap \mathrm{ri}C=\varnothing$ and $\mathrm{ri}C$ is relatively open.

4. By Theorem 9.160, there exists a hyperplane containing $M$ such that $\mathrm{ri}C$ is a subset of one of its associated open halfspaces.

5. Let $H$ be this hyperplane given by $H=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩=0\}$. Since the hyperplane contains $M$, hence it is a subspace.

6. By Theorem 9.149, $C\subseteq \mathrm{cl}C\subseteq \mathrm{cl}\mathrm{ri}C$.

7. Thus, $C$ is contained in the corresponding closed halfspace.

8. Without loss of generality, assume that $C$ is contained in the nonnegative halfspace $H_{+}$.

9. Thus, $inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}\ge 0$.

10. This means that

    $$inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}=inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}-sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}\ge 0.$$

11. Thus,

    $$inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}\ge sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}.$$

12. Since $\mathrm{ri}C\in H_{++}$, there exists $\mathbf{x}\in C$, such that $⟨\mathbf{x},\mathbf{a}⟩>0$.

13. Thus, $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}>0$.

14. But then,

    $$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}=sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}-inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}>0.$$

15. Thus,

    $$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}.$$

16. Then, by Theorem 9.161, $S$ and $T$ are properly separated.

Now, for the converse, assume that $S$ and $T$ are properly separated. Thus, there exists a vector $\mathbf{a}\in \mathbb{V}$ such that

$$inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}\ge sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}$$

and

$$sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in S\}>inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in T\}.$$

1. From the first inequality, we get that $inf\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}\ge 0$.

2. From the second inequality, we get that $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}>0$.

3. Thus, there exists a hyperplane $H$ given by $H=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩=0\}$ such that the corresponding nonnegative closed halfspace $H_{+}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩\ge 0\}$ contains $C$.

4. Note that

   $$\mathrm{int}{H}_{+}=\mathrm{ri}{H}_{+}=H_{++}=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩>0\}.$$

5. Since $sup\{⟨\mathbf{x},\mathbf{a}⟩|\mathbf{x}\in C\}>0$, hence $H_{++}\cap C\ne \varnothing$.

6. $C\subseteq \mathrm{cl}{H}_{++}=H_{+}$ but $C⊈\mathrm{cl}{H}_{++}\setminus \mathrm{ri}{H}_{++}=H_{+}\setminus H_{++}=H$.

7. Thus, by Theorem 9.156, $\mathrm{ri}C\subseteq \mathrm{ri}{H}_{++}=H_{++}$.

8. Thus, $\mathbf{0}\notin \mathrm{ri}C$.

9. Thus, $\mathrm{ri}S\cap \mathrm{ri}T=\varnothing$.

We are done.

Corollary 9.10 (Separating hyperplane theorem II)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be nonempty disjoint convex subsets of $\mathbb{V}$; i.e., $S\cap T=\varnothing$.

Then, there exists a hyperplane that properly separates them.

Proof. Since $S\cap T=\varnothing$, hence $\mathrm{ri}S\cap \mathrm{ri}T=\varnothing$. Then, applying Theorem 9.162, there exists a hyperplane that properly separates $S$ and $T$.

This result is stronger than proposition 2.4.2 in [9].

Disjointness of convex sets is not enough for strong separation as their closures might meet.

Theorem 9.163 (Separating hyperplane theorem III)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ be a nonempty convex subset of $\mathbb{V}$. Let $\mathbf{a}\in \mathbb{V}$ be a vector. There exists a hyperplane $H$ that separates $S$ and $\mathbf{a}$ properly if and only if $\mathbf{a}\notin \mathrm{ri}S$.

In other words, a point can be properly separated from a convex set if and only if it doesn’t belong to its relative interior.

Proof. .

1. We form a set $T=\{\mathbf{a}\}$.

2. Then $\mathrm{ri}T=\{\mathbf{a}\}$ (Theorem 9.133).

3. By Theorem 9.162, $S$ and $T$ are properly separated if and only if

   $$\mathrm{ri}S\cap \mathrm{ri}T=\varnothing .$$

4. In other words,

   $$\mathrm{ri}S\cap \{\mathbf{a}\}=\varnothing .$$

5. In other words, $\mathbf{a}\notin \mathrm{ri}S$.

9.12.5. Strong Separation Characterization

Theorem 9.164 (Characterization of strong separation)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be nonempty convex subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ strongly if and only if

$$inf\{\Vert \mathbf{x}-\mathbf{y}\Vert |\mathbf{x}\in S,\mathbf{y}\in T\}>0.$$

In other words, $H$ strongly separates $S$ and $T$ if and only if $\mathbf{0}\notin \mathrm{cl}(S-T)$.

Proof. Let $C=S-T$. Then

$$inf\{\Vert \mathbf{x}-\mathbf{y}\Vert |\mathbf{x}\in S,\mathbf{y}\in T\}=inf\{\Vert \mathbf{v}\Vert |\mathbf{v}\in C\}.$$

By Theorem 4.51, $inf\{\Vert \mathbf{v}\Vert |\mathbf{v}\in C\}=0$ if and only if $\mathbf{0}\in \mathrm{cl}C$.

Thus, $inf\{\Vert \mathbf{v}\Vert |\mathbf{v}\in C\}>0$ if and only if $\mathbf{0}\notin \mathrm{cl}C$.

Assume that $S$ and $T$ are strongly separated. Let $H$ be the hyperplane that separates $S$ and $T$. Let $H_{++}$ be the positive open halfspace. Let $H_{--}$ be the negative open halfspace.

1. Then, there exists an $r>0$ such that $S+rB\subseteq H_{++}$ and $T+rB\subseteq H_{--}$.

2. Since $H_{++}\cap H_{--}=\varnothing$, hence $(S+rB)\cap (T+rB)=\varnothing$.

3. Then, for any $\mathbf{x}\in S$ and $\mathbf{y}\in T$ and every $\mathbf{u},\mathbf{v}\in B$,

   $$\Vert \mathbf{x}+r\mathbf{u}-(\mathbf{y}+r\mathbf{v})\Vert >0$$

   as $\mathbf{x}+r\mathbf{u}=\mathbf{y}+r\mathbf{v}$ would mean that $(S+rB)\cap (T+rB)\ne \varnothing$

4. Note that, $\Vert \mathbf{x}+r\mathbf{u}-(\mathbf{y}+r\mathbf{v})\Vert =\Vert (\mathbf{x}-\mathbf{y})-r(\mathbf{v}-\mathbf{u})\Vert$.

5. Then, $\Vert \mathbf{x}-\mathbf{y}\Vert \ge 2r$ must be true for every $\mathbf{x}\in S$ and $\mathbf{y}\in T$.

   1. For contradiction, assume that $\Vert \mathbf{x}-\mathbf{y}\Vert <2r$ for some $\mathbf{x}\in S$ and $\mathbf{y}\in T$.

   2. Let $\mathbf{u}=\frac{1}{2r}(\mathbf{y}-\mathbf{x})$.

   3. Let $\mathbf{v}=\frac{1}{2r}(\mathbf{x}-\mathbf{y})$.

   4. Note that $\Vert \mathbf{u}\Vert <1$ and $\Vert \mathbf{v}\Vert <1$.

   5. Thus, $\mathbf{u},\mathbf{v}\in B$.

   6. Then, $r(\mathbf{v}-\mathbf{u})=\mathbf{x}-\mathbf{y}$.

   7. Then, $\Vert (\mathbf{x}-\mathbf{y})-r(\mathbf{v}-\mathbf{u})\Vert =\Vert \mathbf{0}\Vert =0$.

   8. This contradicts the condition that $\Vert \mathbf{x}+r\mathbf{u}-(\mathbf{y}+r\mathbf{v})\Vert >0$ for every $\mathbf{x}\in S$, $\mathbf{y}\in T$ and $\mathbf{u},\mathbf{v}\in B$.

6. Thus, $inf\{\Vert \mathbf{x}-\mathbf{y}\Vert |\mathbf{x}\in S,\mathbf{y}\in T\}>0$.

Conversely, assume that $inf\{\Vert \mathbf{x}-\mathbf{y}\Vert |\mathbf{x}\in S,\mathbf{y}\in T\}>0$.

1. Let $inf\{\Vert \mathbf{x}-\mathbf{y}\Vert |\mathbf{x}\in S,\mathbf{y}\in T\}=2r$ where $r>0$.

2. Then, $\Vert \mathbf{x}-\mathbf{y}\Vert \ge 2r\mathrm{\forall }\mathbf{x}\in S,\mathbf{y}\in T$.

3. Then, $(S+rB)\cap (T+rB)=\varnothing$.

   1. For contradiction, assume that $(S+rB)\cap (T+rB)\ne \varnothing$.

   2. Let $\mathbf{a}\in (S+rB)\cap (T+rB)$.

   3. Then, there exists $\mathbf{x}\in S$ and $\mathbf{u}\in B$ such that $\mathbf{a}=\mathbf{x}+r\mathbf{u}$.

   4. And, there exists $\mathbf{y}\in T$ and $\mathbf{v}\in B$ such that $\mathbf{a}=\mathbf{y}+r\mathbf{v}$.

   5. Then, $\mathbf{x}+r\mathbf{u}=\mathbf{y}+r\mathbf{v}$.

   6. Thus, $\mathbf{x}-\mathbf{y}=r(\mathbf{v}-\mathbf{u})$.

   7. Thus,

      $$\Vert \mathbf{x}-\mathbf{y}\Vert =r\Vert \mathbf{v}-\mathbf{u}\Vert \le r(\Vert \mathbf{v}\Vert +\Vert \mathbf{u}\Vert )<r(1+1)=2r.$$

   8. This contradictions our hypothesis that $\Vert \mathbf{x}-\mathbf{y}\Vert \ge 2r\mathrm{\forall }\mathbf{x}\in S,\mathbf{y}\in T$.

4. Note that $S+rB$ and $T+rB$ are convex and disjoint.

5. Then, due to Corollary 9.10, there exists a hyperplane $H$ which separates $S+rB$ and $T+rB$ properly.

6. We can choose $H$ so that $S+rB\subseteq H_{+}$ and $T+rB\subseteq H_{-}$.

7. But then, $S+\frac{r}{2}B\in H_{++}$ and $T+\frac{r}{2}B\in H_{--}$ lie in the opposite open halfspaces.

8. Thus, $S$ and $T$ are strongly separated.

9.12.6. Strong Separation Conditions

We describe several conditions which are sufficient to achieve strong separation between two sets. See also strict separation theorem (proposition 2.4.3) of [9].

Theorem 9.165 (Strong separation: closed subtraction)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be two disjoint nonempty convex subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ strongly if $S-T$ is closed.

Proof. We proceed as follows.

1. Since $S$ and $T$ are disjoint hence $\mathbf{0}\notin S-T$.

2. Since $S-T$ is closed, hence $\mathbf{0}\notin \mathrm{cl}(S-T)=S-T$.

3. By Theorem 9.164, $S$ and $T$ are strongly separated.

Theorem 9.166 (Strong separation: closed and compact sets)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be two disjoint nonempty convex subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ strongly if $S$ is closed and $T$ is compact.

Proof. We proceed as follows.

1. Since $S$ and $T$ are disjoint hence $\mathbf{0}\notin S-T$.

2. Since $S-T$ is closed, hence $\mathbf{0}\notin \mathrm{cl}(S-T)=S-T$.

3. By Theorem 9.164, $S$ and $T$ are strongly separated.

9.12.6.1. Recession Cones

The conditions below are based on the theory of recession cones and lineality spaces of convex sets. See Recession Cones for a background.

Theorem 9.167 (Strong separation: recession cones)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be two disjoint nonempty closed and convex subsets of $\mathbb{V}$. There exists a hyperplane $H$ that separates $S$ and $T$ strongly if

$$R_S\cap R_T=L_S\cap L_T$$

where $R_X$ denotes the recession cone and $L_X$ denotes the lineality space of a set $X$.

Proof. Due to Corollary 9.18, $S-T$ is closed. Then due to Theorem 9.165, $S$ and $T$ are strongly separated.

9.12.6.2. Strongly Separating Hyperplane

Recall that the (orthogonal) projection of a vector $\mathbf{v}$ on a convex set $C$ is the vector $\mathbf{x}\in C$ which is nearest to $\mathbf{v}$ under the (Euclidean) norm. In particular, if $\mathbf{v}\in C$ then $\mathbf{v}$ is its own projection on $C$. The projection of the vector $\mathbf{0}$ on a convex set $C$ is this the vector of $C$ with the minimum norm. See Projection on Convex Sets for details.

Theorem 9.168 (Strongly separating hyperplane)

Let $\mathbb{V}$ be an $n$-dimensional real vector space. Let $S$ and $T$ be two disjoint nonempty and convex subsets of $\mathbb{V}$. Assume that $S-T$ is closed.

Consider the vector of minimum norm (projection of the origin) in $S-T$ given by $\mathbf{v}=\mathbf{s}-\mathbf{t}$ where $\mathbf{s}\in S$ and $\mathbf{t}\in T$.

1. Let $\mathbf{a}=\frac{1}{2}\mathbf{v}=\frac{1}{2}(\mathbf{s}-\mathbf{t})$.

2. Let $\mathbf{b}=\frac{1}{2}(\mathbf{s}+\mathbf{t})$.

3. Let $c=⟨\mathbf{b},\mathbf{a}⟩$.

Then the hyperplane $H$ given by

$$H=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{a}⟩=c\}$$

strongly separates $S$ and $T$. In other words,

$$⟨{\mathbf{x}}_1,\mathbf{a}⟩>c>⟨{\mathbf{x}}_2,\mathbf{a}⟩\mathrm{\forall }{\mathbf{x}}_1\in S,{\mathbf{x}}_2\in T.$$

Proof. .

1. Since $S$ and $T$ are disjoint hence $\mathbf{s}-\mathbf{t}\ne \mathbf{0}$.

2. Hence $\mathbf{a}\ne \mathbf{0}$.

3. $\mathbf{s}$ is the nearest point in $\mathrm{cl}S$ from $\mathrm{cl}T$.

4. $\mathbf{t}$ is the nearest point in $\mathrm{cl}T$ from $\mathrm{cl}S$.

5. The line segment $[\mathbf{s},\mathbf{t}]$ connects these nearest points.

6. $\mathbf{b}$ lies on this line segment.

7. Accordingly, $\mathbf{s}$ is projection of $\mathbf{b}$ in $\mathrm{cl}S$.

8. Similarly $\mathbf{t}$ is projection of $\mathbf{b}$ in $\mathrm{cl}T$.

9. Then, due to Theorem 10.7 (orthogonal projection characterization), for every $\mathbf{x}\in S$

   $$⟨\mathbf{x}-\mathbf{s},\mathbf{b}-\mathbf{s}⟩\le 0.$$

10. But

    $$\mathbf{b}-\mathbf{s}=\frac{1}{2}(\mathbf{s}+\mathbf{t})-\mathbf{s}=\frac{1}{2}(\mathbf{t}-\mathbf{s})=-\mathbf{a}.$$

11. Hence $⟨\mathbf{x}-\mathbf{s},\mathbf{a}⟩\ge 0$.

12. Hence $⟨\mathbf{x},\mathbf{a}⟩\ge ⟨\mathbf{s},\mathbf{a}⟩$.

13. Further

    $$\begin{array}{rl}⟨\mathbf{s},\mathbf{a}⟩& =⟨\mathbf{s}-\mathbf{b}+\mathbf{b},\mathbf{a}⟩\\ & =⟨\mathbf{a}+\mathbf{b},\mathbf{a}⟩\\ & =\Vert \mathbf{a}{\Vert }_2^2+⟨\mathbf{b},\mathbf{a}⟩\\ & =\Vert \mathbf{a}{\Vert }_2^2+c>c.\end{array}$$

    since $\mathbf{a}\ne \mathbf{0}$.

14. Hence for every $\mathbf{x}\in S$, we have $⟨\mathbf{x},\mathbf{a}⟩>c$.

15. A similar argument shows that for every $\mathbf{x}\in T$, we have $⟨\mathbf{x},\mathbf{a}⟩<c$.

9.12.7. Closed Convex Sets

Theorem 9.169 (Strict separation theorem)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. Let $C\subseteq \mathbb{V}$ be a nonempty closed convex set. Let $\mathbf{y}\notin C$. Then, there exists $\mathbf{p}\in {\mathbb{V}}^{\ast }$ and $\alpha \in \mathbb{R}$ such that

$$⟨\mathbf{y},\mathbf{p}⟩>\alpha \text{ and }⟨\mathbf{x},\mathbf{p}⟩\le \alpha \mathrm{\forall }\mathbf{x}\in C.$$

In other words, there exists a separating hyperplane such that $C$ is contained in one of its (closed) halfspaces and $\mathbf{y}$ is not in that halfspace (i.e., it is in the opposite open halfspace).

By choosing any $\beta \in (\alpha ,⟨\mathbf{y},\mathbf{p}⟩)$, we also have

$$⟨\mathbf{y},\mathbf{p}⟩>\beta \text{ and }⟨\mathbf{x},\mathbf{p}⟩<\beta \mathrm{\forall }\mathbf{x}\in C.$$

Proof. This is an application of strong separation.

1. Define a set $D=\{\mathbf{y}\}$.

2. Since $C$ is closed and $\mathbf{y}\notin C$, hence $\mathbf{y}$ is not a closure point of $C$.

3. Thus, there exists $r>0$ such that $B(\mathbf{y},r)\cap C=\varnothing$.

4. Thus, $d(C,D)\ge r$.

5. Then, by Theorem 9.164, $C$ and $D$ are strongly separated.

6. Let $H$ be a hyperplane which strongly separates them.

7. Then one of the closed halfspaces of $H$ contains $C$ but not $\mathbf{y}$.

8. Let $H$ be described by

   $$H=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{p}⟩=\alpha \}.$$

9. We can negate $\mathbf{p},\alpha$ if necessary so that

   $$C\subseteq H_{-}=\{\mathbf{x}\in \mathbb{V}|⟨\mathbf{x},\mathbf{p}⟩\le \alpha \}.$$

10. Accordingly, $⟨\mathbf{y},\mathbf{p}⟩>\alpha$.

Theorem 9.170 (Closed Convex = Intersection of Halfspaces)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. A closed convex set $C$ of $\mathbb{V}$ is the intersection of all the closed halfspaces that contain it.

Proof. The set $\mathbb{V}$ is closed and convex but no halfspace contains it. So the statement is vacuously true.

The empty set is closed and convex and every halfspace contains it. The intersection of all halfspaces
is the empty set. Thus, the statement is vacuously true.

Let us assume that $C$ is nonempty and not equal to $\mathbb{V}$.

1. Let $\mathbf{a}\in \mathbb{V}$ such that $\mathbf{a}\notin C$.

2. Since $C$ is closed, hence $\mathbf{a}$ is not a closure point of $C$.

3. Thus, there exists $r>0$ such that $B(\mathbf{a},r)\cap C=\varnothing$.

4. Thus, $\Vert \mathbf{a}-\mathbf{x}\Vert >0\mathrm{\forall }\mathbf{x}\in C$. Specifically, $inf\{\Vert \mathbf{a}-\mathbf{x}\Vert |\mathbf{x}\in C\}\ge r$.

5. Thus, by Theorem 9.164, $\{\mathbf{a}\}$ and $C$ are strongly separated by some hyperplane $H$.

6. Thus, one of the corresponding closed halfspaces contains $C$ but not $\mathbf{a}$.

7. In other words, for every $\mathbf{a}\notin C$, there exists a closed halfspace that doesn’t contain $\mathbf{a}$ but contains $C$.

8. Thus, the intersection of all halfspaces containing $C$ cannot contain any element from $\mathbb{V}\setminus C$.

   1. If the intersection contained an element $\mathbf{a}\notin C$, then there would be a closed halfspace containing $C$ but not $\mathbf{a}$ which would mean that the intersection of all halfspaces containing $C$ cannot contain $\mathbf{a}$.

9. Hence, the intersection of all closed halfspaces containing $C$ is exactly equal to $C$.

We now have two different characterizations of closed convex sets.

1. From Theorem 9.122, a closed and convex set is the closure of the union of all the line segments connecting the points of the set.

2. From Theorem 9.170, a closed and convex set is the intersection of all the closed half-spaces containing it.

9.12.8. Supporting Hyperplanes

Definition 9.62 (Supporting hyperplane and halfspaces)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. Let $S\subseteq \mathbb{V}$. Let ${\mathbf{x}}_0\in \mathrm{bd}S$ be a point on its boundary.

If there exists a nonzero vector $\mathbf{a}\in \mathbb{V}$ such that $⟨\mathbf{x},\mathbf{a}⟩\le ⟨{\mathbf{x}}_0,\mathbf{a}⟩$ for every $\mathbf{x}\in S$, then the hyperplane $H$ given by

$$H=\{\mathbf{x}|⟨\mathbf{x},\mathbf{a}⟩=⟨{\mathbf{x}}_0,\mathbf{a}⟩\}$$

is called a supporting hyperplane of $S$ at ${\mathbf{x}}_0$.

$H$ separates $\{{\mathbf{x}}_0\}$ and $S$ and $H$ contains ${\mathbf{x}}_0$. The halfspace $\{\mathbf{x}|⟨(\mathbf{x}-{\mathbf{x}}_0),\mathbf{a}⟩\le 0\}$ corresponding to $H$ is called a supporting halfspace of $S$ at ${\mathbf{x}}_0$.

Convex sets have this beautiful property that there exists a supporting hyperplane at every point in the boundary of the set.

Theorem 9.171 (Supporting hyperplane theorem)

Let $\mathbb{V}$ be a real $n$-dimensional vector space. Let $C$ be a nonempty convex subset of $\mathbb{V}$. Let $\mathbf{x}\in \mathrm{bd}C$ be any point in the boundary of $C$. Then, there exists a supporting hyperplane to $C$ at $\mathbf{x}$.

Proof. Consider the case where $\mathrm{int}C=\varnothing$.

1. Then, due to Theorem 9.125, $\dim \mathrm{aff}C<n$.

2. Thus, there exists a hyperplane $H$ such that $\mathrm{aff}C\subseteq H$.

3. Since $\mathbb{V}$ is finite dimensional, hence $H$ is closed (Theorem 4.200).

4. Thus, $\mathrm{cl}C\subseteq H$.

5. Thus, $\mathbf{x}\in H$.

6. This $H$ trivially separates $\{\mathbf{x}\}$ and $C$ as both are contained in $H$.

Now, assume that $\mathrm{int}C\ne \varnothing$.

1. $\dim \mathrm{aff}C=n$.

2. $\mathrm{ri}C=\mathrm{int}C$.

3. Since $\mathbf{x}\in \mathrm{bd}C$, hence the sets $\{\mathbf{x}\}$ and $\mathrm{int}C$ are disjoint.

4. $\mathrm{ri}\{\mathbf{x}\}=\{\mathbf{x}\}$ (Theorem 9.133).

5. We have $\mathrm{ri}\{\mathbf{x}\}\cap \mathrm{ri}C=\varnothing$.

6. By Theorem 9.162, there exists a hyperplane $H$ that separates $\{\mathbf{x}\}$ and $C$ properly.

7. Consequently $C$ lies entirely in one of the closed halfspaces of $H$.

Corollary 9.11

Let $\mathbb{V}$ be a real $n$-dimensional vector space. A closed convex set $C$ of $\mathbb{V}$ is the intersection of all the supporting halfspaces that contain it.