Overview

These notes aim to provide a solid mathematical background for a deeper understanding of signal processing problems.

Algebra covers basic definitions of sets, relations, functions and sequences. It introduces general Cartesian products and axiom of choice. The concept of function is somewhat different as a function $f:A\to B$ may not be defined at every point in $A$.

Elementary Real Analysis introduces the real line $\mathbb{R}$ and explains the topology of real line in terms of neighborhoods, open sets, closed sets, interior, closure, boundary, accumulation points, covers and compact sets. Sequences and series of real numbers are introduced. The extended real line $\bar{\mathbb{R}}$ is introduced which plays an important role in optimization problems. Basic properties of real valued functions $f:X\to \mathbb{R}$ from an arbitrary set $X$ to real line $\mathbb{R}$ are introduced. The graph, epigraph, sublevel and superlevel sets are defined. Real functions of type $f:\mathbb{R}\to \mathbb{R}$ are discussed and concepts of limits and continuity are formally defined. Differentiation is introduced from analytical point of view. Several fundamental inequalities on real numbers are proved.

Metric Spaces covers the topology of metric spaces, sequences in metric spaces, functions and continuity, completeness, compactness. Real valued functions on metric spaces are discussed in detail covering concepts of closed functions, semicontinuity, limit superior and inferior.

Linear Algebra covers vector spaces, matrices, linear transformations, normed linear spaces, inner product spaces, Banach spaces, Hilbert spaces, eigen value decomposition, singular values, affine sets and transformations, and a plethora of related topics from both algebraic and analytical perspective.

Convex Sets and Functions provides an in-depth treatment of convex sets and functions. Hyperplanes, halfspaces, general convex sets, lines and line segments, rays, balls, convex hulls, orthants, simplices, polyhedra, polytopes, ellipsoids are discussed. Convex cones, conic hulls, pointed cones, proper cones, norm cones, barrier cones are described. Dual cones, polar cones and normal cones are explained. Generalized inequalities are introduced. Convexity preserving operations are described. Convex function and their properties are covered. Proofs are provided for the convexity of a large number of convex functions. Proper convex functions, indicator functions, sublevel sets, closed and convex functions, support functions, gauge functions, quasi convex functions are explained with their properties and examples. Topological properties of convex sets including closure, interior, compactness, relative interior, line segment property are covered. Different types of separation theorems are proved. First order and second order conditions for convexity of differentiable functions are proved. Operations which preserve the convexity of functions are described. Continuity of convex functions at interior and boundary points is discussed. Recession cones and lineality spaces are described. Directional derivatives for convex functions and their properties are covered. Subgradients are introduced for functions which are convex but not differentiable. Conjugate functions are developed. Smoothness and strong convexity of convex functions is discussed. Infimal convolution is introduced.

Convex Optimization develops the theory of optimization of convex functions under convex constraints. Topics covered include: basic definitions of unconstrained and constrained convex optimization problems, projections on convex sets, directions of recession, min common max crossing (MCMC) duality framework, minimax theorems, saddle point theorem, stationary points, first order and second order criteria for optimization of unconstrained functions, first order criteria for constrained optimization, descent directions methods, gradient method, gradient projection method, Farkas’ and Gordan’s lemmas, KKT conditions for minimization of smooth functions under linear equality and inequality constraints, feasible directions, tangent cones, optimality conditions based on tangent cones, minimization of smooth functions under smooth inequality and equality constraints, Fritz John and KKT conditions for the same, Lagrange multipliers, constraint qualifications, Lagrangian duality, conjugate duality, Fenchel duality theorem, linear programming, quadratic programming.