5.2. Differentiation in Banach Spaces#

We introduce the concept of differentiation in Banach spaces. Recall that Banach spaces are normed linear spaces that are complete.

5.2.1. Gateaux Differential#

Definition 5.14 (Directional derivative)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . The directional derivative of $f$ at $x \in int S$ in the direction $h \in X$ where $h \neq 0$ , denoted by $f^{'} (x; h)$ is given by

f^{'} (x; h) ≜ lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t}

whenever the limit exists. This is also known as the Gateaux differential. By convention, $f^{'} (x; 0_{X}) = 0_{Y}$ . This is consistent with the definition above.

There is no single directional derivative at a point $x$ .
The directional derivative depends on the direction $h$ .
In one dimension, there are two directional derivatives at each $x$ .
In two or more dimensions, there are infinitely many directional derivatives.
The directional derivative is a one dimensional calculation along the direction $h$ .
It is usually easy to compute the directional derivative even when the space $X$ is infinite dimensional.

Definition 5.15 (Gateaux differentiability)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . Let $U \subseteq S$ be an open set. We say that $f$ is Gateaux differentiable at $x \in U$ if the Gateaux differential $f^{'} (x; h)$ exists for every direction $h \in X$ .

Accordingly, we can define a bounded operator $T_{x} : X \to Y$ given by

T_{x} (h) ≜ lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

The operator $T$ is called the Gateaux derivative of $f$ at $x$ .

Example 5.17 (Gateaux differential of exponential function)

Let $f (x) = e^{x}$ . Then,

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{e^{x + t h} - e^{x}}{t} \\ = e^{x} lim_{t \to 0^{+}} \frac{e^{t h} - 1}{t} \\ = e^{x} lim_{t \to 0^{+}} \frac{t h}{t} = h e^{x} . \end{aligned}

We note that the Gateaux derivative depends linearly on $h$ .

Theorem 5.7 (Gateaux differential nonnegative homogeneity)

The Gateaux differential of a function $f : X \to Y$ is nonnegative homogeneous in the sense that

f^{'} (x; α h) = α f^{'} (x; h)

for every $α \in R_{+}$ and every $h \in X$ .

However, the Gateaux differential may not be additive. Thus, the Gateaux differential may fail to be linear.

Example 5.18 (Gateaux differential of absolute value function)

Let $f (x) = | x |$ . Then, the Gateaux differentials are given by

\begin{array}{r} f^{'} (x; h) = {\begin{cases} h \frac{x}{| x |} & x \neq 0; \\ | h | & x = 0. \end{cases} \end{array}

We note that the Gateaux differential of $f$ exists everywhere. However the Gateaux differential depends on $h$ in a nonlinear way at $x = 0$ . At $x \neq 0$ , the Gateaux differential depends linearly on $h$ .

Example 5.19 (Gateaux differential of square function)

Let $f (x) = x^{2}$ . Then, the Gateaux differential is given by

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \\ = lim_{t \to 0^{+}} \frac{x^{2} + t^{2} h^{2} + 2 x t h - x^{2}}{t} \\ = 2 x h . \end{aligned}

We note that the Gateaux differential is linear w.r.t. $h$ .

Example 5.20 (Gateaux differential of linear functional)

Let $f (x) = a^{T} x$ where $a \in R^{n}$ is a given fixed vector.

f^{'} (x; h) = lim_{t \to 0^{+}} \frac{a^{T} x + t a^{T} h - a^{T} x}{t} = a^{T} h .

We note that the Gateaux differential is linear w.r.t. $h$ .

Example 5.21 (Gateaux differential of simple quadratic)

Let $f (x) = x^{T} A x$ where $A \in S^{n}$ is a given symmetric matrix.

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{(x + t h)^{T} A (x + t h) - x^{T} A x}{t} \\ = lim_{t \to 0^{+}} \frac{t^{2} h^{T} A h + 2 t h^{T} A x}{t} \\ = 2 h^{T} A x = 2 x^{T} A h . \end{aligned}

We note that the Gateaux differential is linear w.r.t. $h$ .

In particular, if $f (x) = x^{T} x$ , then $f^{'} (x; h) = 2 h^{T} x = 2 x^{T} h$ .

Theorem 5.8 (Gateaux differential of a constant function)

The Gateaux differential of a constant function is zero.

Theorem 5.9 (Gateaux differential sum rule)

Gateaux differential distributes over sum.

Let $f, g : X \to Y$ both have Gateaux derivatives at $x$ in the direction $h$ . Then,

(f + g)^{'} (x; h) = f^{'} (x; h) + g^{'} (x; h) .

Also,

(f - g)^{'} (x; h) = f^{'} (x; h) - g^{'} (x; h) .

Theorem 5.10 (Gateaux differential product rule)

Let $f, g : X \to Y$ both be Gateaux differentiable at $x \in int dom f \cap dom g$ . Let $h$ be their (pointwise) product function given by

h (x) = f (x) g (x)

with $dom h = dom f \cap dom g$ . Then,

h^{'} (x; h) == (f g)^{'} (x; h) = f^{'} (x; h) g (x) + g^{'} (x; h) f (x) .

Theorem 5.11 (Gateaux differential chain rule)

Let $f : X \to Y$ and $g : Y \to Z$ be functions. Let $h : X \to Z$ be the composition of $f$ and $g$ given by $h = g \circ f$ . Let $U \subseteq dom h$ be an open set. Let $x \in U$ . Assume that $f$ is Gateaux differentiable at $x$ and $g$ is Gateaux differentiable at $f (x)$ . Then,

h^{'} (x; h) = g^{'} (f (x); f^{'} (x; h)) \forall h \in X .

We recall the little- $o$ notation. We say that a quantity $q$ is $o (t)$ if

lim_{t \to 0^{+}} \frac{q}{t} = 0.

For vector valued functions, a quantity $q$ is $o (t)$ if

lim_{t \to 0^{+}} \frac{‖ q ‖}{t} = 0.

lim_{t \to 0^{+}} \frac{q}{t} = 0 .

Proof. If $f$ is Gateaux differentiable at $x$ , then

f^{'} (x; h) = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

In terms of little-o notation,

f (x + t h) = f (x) + t f^{'} (x; h) + o (t) .

Similarly, if $g$ is Gateaux differentiable at $y$ , then

g (y + s u) = g (y) + s g^{'} (y; u) + o (s) .

Now,

\begin{aligned} h^{'} (x; h) = (g \circ f)^{'} (x; h) & = lim_{t \to 0^{+}} \frac{g (f (x + t h)) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x) + t f^{'} (x; h) + o (t)) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x) + t (f^{'} (x; h) + t^{- 1} o (t))) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x)) + t g^{'} (f (x); f^{'} (x; h) + t^{- 1} o (t)) + o (t) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{t g^{'} (f (x); f^{'} (x; h) + t^{- 1} o (t)) + o (t)}{t} \\ = lim_{t \to 0^{+}} [g^{'} (f (x); f^{'} (x; h)) + t^{- 1} o (t)) + t^{- 1} o (t)] \\ = g^{'} (f (x); f^{'} (x; h)) . \end{aligned}

Example 5.22 (Chain rule for square of inner product)

Consider the function $h (x) = (x^{T} x)^{2}$ .

Define $g (t) = t^{2}$
Define $f (x) = x^{T} x$ .
Then $h = g \circ f$ .
We have $f^{'} (x; h) = 2 h^{T} x$ .
We have $g^{'} (y; u) = 2 y u$ .
Thus,

$\begin{aligned} g^{'} (f (x); f^{'} (x; h)) & = 2 f (x) f^{'} (x; h) \\ = 2 (x^{T} x) (2 h^{T} x) \\ = 4 (h^{T} x) (x^{T} x) . \end{aligned}$

We can compute the same thing using the product rule.

We note that $h (x) = f (x) f (x)$ .
Applying the product rule:

$\begin{aligned} h^{'} (x; h) & = f^{'} (x; h) f (x) + f^{'} (x; h) f (x) \\ = 2 f^{'} (x; h) f (x) \\ = 2 (2 h^{T} x) (x^{T} x) \\ = 4 (h^{T} x) (x^{T} x) . \end{aligned}$

5.2.2. Fréchet Derivative#

Definition 5.16 (Fréchet differentiability)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . Let $U \subseteq S$ be an open set. We say that $f$ is Fréchet differentiable at $x \in U$ if there is a bounded and linear operator $T_{x} : X \to Y$ given by

T_{x} (h) = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

The operator $T_{x}$ is called the Fréchet derivative of $f$ at $x$ .

We note that $T_{x}$ depends on $x$ .

Remark 5.1 (Fréchet differentiability alternate forms)

By definition, if $f$ is Fréchet differentiable at $x$ , then it is Gateaux differentiable at $x$ . Since $T_{x}$ is linear, we can write it as

T_{x} (h) = A h

emphasizing the fact that the essential part of $T_{x}$ doesn’t depend on $h$ . $A$ may still depend on $x$ .

Using the little- $o$ notation, we can write

f (x + t h) = f (x) + t T_{x} (h) + o (t) = f (x) + t A h + o (t) .

If we set $t h = y$ , then $t \to 0$ if and only if $y \to 0$ . In particular, $‖ y ‖_{X} = t ‖ h ‖_{X} = o (t)$ . Now,

\begin{aligned} f (x + y) = f (x) + A y + o (t) \\ ⟺ f (x + y) - f (x) - A y = o (t) = o (‖ y ‖_{X}) \\ ⟺ lim_{‖ y ‖_{X} \to 0} \frac{‖ f (x + y) - f (x) - A y ‖_{Y}}{‖ y ‖_{X}} = 0 \\ ⟺ lim_{y \to 0} \frac{‖ f (x + y) - f (x) - A y ‖_{Y}}{‖ y ‖_{X}} = 0 \\ ⟺ lim_{y \to 0} \frac{‖ f (x + y) - f (x) - T_{x} (y) ‖_{Y}}{‖ y ‖_{X}} = 0. \end{aligned}

Therefore $f : X \to Y$ is Fréchet differentiable at $x \in U$ if and only if

lim_{y \to 0} \frac{f (x + y) - f (x) - T_{x} (y)}{‖ y ‖_{X}} = 0

for every $y \in X$ .

It is worthwhile to compare this definition to the definition of differentiability of $f : R^{n} \to R^{m}$ in Definition 5.1. If we put $z = x + y$ , we can rewrite the condition as

lim_{z \to x} \frac{‖ f (z) - f (x) - T_{x} (z - x) ‖_{Y}}{‖ z - x ‖_{X}} = 0.

Thus, $T_{x}$ plays the same role as the Jacobian matrix $D f (x)$ in (5.1).

Theorem 5.12 (Existence of Fréchet derivative)

The Fréchet derivative of a function $f$ exists at a point $x = a$ if and only if all Gateaux differentials of $f$ at $x$ are continuous functions of $x$ at $x = a$ .

Theorem 5.13 (Uniqueness of Fréchet derivative)

If the Fréchet derivative of a function $f$ exists at a point $x = a$ then it is unique.

5.2.3. Gradient#

Definition 5.17 (Gradient)

Let $V$ be a Hilbert space. Let $f : V \to R$ is a real valued function. Let $S = dom f$ and $U \subseteq S$ be an open set. Assume that $f$ is Fréchet differentiable at $x \in U$ . Then, the Fréchet derivative $T_{x} : V \to R$ is a bounded linear functional.

The gradient of a real valued function is denoted by $\nabla f (x)$ and $\nabla f (x) \in V^{*}$ satisfying

⟨ h, \nabla f (x) ⟩ = T_{x} (h) .

Topics in Signal Processing

Differentiation in Banach Spaces

Contents

5.2. Differentiation in Banach Spaces#

5.2.1. Gateaux Differential#

5.2.2. Fréchet Derivative#

5.2.3. Gradient#