Convex optimization exam notes (Aalto University)

Theorem

• A finite intersection of halfspaces is convex.
• Cauchy-Schwarz inequality: |aHb| ≤ ||a|| · ||b||
• Each halfspace is a convex set
• The intersection of any number of convex sets is also convex
• The product of two convex functions is convex if one of them is nonnegative.
• For any cvx opt problem, any local solution is also global.

Concept

quasiconvex

Definition: A function $f$ is quasiconvex if its domain is convex and for any two points $x_1$ and $x_2$ in this domain, and for any $\lambda$ in the interval $[0,1]$, the following holds:
f ( λ x 1 + ( 1 − λ ) x 2 ) ≤ max ⁡ { f ( x 1 ) , f ( x 2 ) } f\left(\lambda x_1+(1-\lambda) x_2\right) \leq \max \left\{f\left(x_1\right), f\left(x_2\right)\right\} f(λx1​+(1−λ)x2​)≤max{f(x1​),f(x2​)}
For a function to be quasiconvex, its sublevel sets must be convex.

Jensen’s Inequality

In convex function condition. Let ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n$ be non-negative weights such that ${\lambda }_1+{\lambda }_2+\cdots +{\lambda }_n=1$. Then Jensen’s Inequality states that:
$$f\left({\lambda }_1{x}_1+{\lambda }_2{x}_2+\cdots +{\lambda }_n{x}_n\right)\le {\lambda }_{1}f\left(x_1\right)+{\lambda }_{2}f\left(x_2\right)+\cdots +{\lambda }_{n}f\left(x_n\right)$$

Problem

How to determine if a fuction it is convex, concave, quasiconvex, or quasiconcave

For single variable:
Quasiconvexity:

1. First Derivative Test (for differentiable functions): A differentiable function $f$ is quasiconvex on an interval if its derivative $f^{\prime }(x)$ does not change sign from positive to negative.

For furthermore to check convexity:
2. First Derivative Test: For functions of a single variable, if the first derivative $f^{\prime }(x)$ is non-decreasing on the interval, then $f$ is convex on that interval.
3. Second Derivative Test: For twice differentiable functions of a single variable, if the second derivative $f^{\prime \prime }(x)$ is non-negative for all $x$ in the domain, then $f$ is convex on that domain.

Quasiconcavity:
Opposite to quasiconvexity: its derivative $f^{\prime }(x)$ does not change sign from negative to positive

Concavity:
Opposite to convexity: first derivative $f^{\prime }(x)$ is non-increasing; second derivative $f^{\prime \prime }(x)$ is non-positive for all $x$ in the domain

For multi-dimension variables:
See Hessian matrix, if that is negative semidefinite -> concave, positive semidefinite-> convex.
If infinite matrix, then neither convex nor concave.

Determining quasiconvexity (quasiconcavity) directly through the Hessian matrix is generally not possible, as the concept of quasiconvexity is less rigid and more general than convexity. Here we have to check the sublevel (superlevel) sets.

Note, a linear function is always convex and concave as the for every two points equality holds:
$f(\lambda x+(1-\lambda )y)=\lambda f(x)+(1-\lambda )f(y)$

Composition: One dimension case:
$$f(x)=h(g(x))$$
$f$ is convex if $h$ is convex and nondecreasing, and $g$ is convex
$f$ is convex if $h$ is convex and nonincreasing, and $g$ is concave,
$f$ is concave if $h$ is concave and nondecreasing, and $g$ is concave
$f$ is concave if $h$ is concave and nonincreasing, and $g$ is convex

General form to standard form

In the standard form, every constraint is expressed as an equality. slack variable is crucial in converting inequality to equality.

Derive a dual problem

1. Using Lagrange multipliers for the inequality and equality constraints to form Lagrangian. The Lagrange multipliers for inequality constraints are typically constrained to be non-negative.
2. minimizing the Lagrangian with respect to the primal variables
3. The dual problem is then formulated by maximizing the dual function over all dual variables.