SVM

最新推荐文章于 2023-06-26 23:51:24 发布

Edison011235

最新推荐文章于 2023-06-26 23:51:24 发布

阅读量885

点赞数 2

CC 4.0 BY-SA版权

分类专栏：算法文章标签： svm

本文链接：https://blog.youkuaiyun.com/StanfordJava/article/details/72859327

算法专栏收录该内容

3 篇文章

订阅专栏

SVM(Support Vector Machine）是用来解决分类问题的。任何一个机器学习问题首先要考虑三个问题：

1. 模型是什么
2. 优化目标是什么
3. 算法是什么

SVM的模型： $\mathop h\nolimits_{w,b} (x) = g\left( {\mathop w\nolimits^T x + b} \right)$

g为sigmoid函数。我们考虑二元线性分类，两种不同的类别分别用1和-1表示，即 $y \in \{ 1, - 1\}$ 。 $\mathop w\nolimits^T x + b > 0$ 的样本被标记为1， $\mathop w\nolimits^T x + b < 0$ 的样本被标记为-1。

一个很朴素的想法：对于分类问题，我们想找到这样一个分类器——不仅能把数据分开，还要远远的分开，我们认为这样的分类器是好的分类器。那么怎么用数学语言来表达这一观点呢？

首先引入函数间隔和几何间隔用来精确衡量待分类样本到超平面的距离。

函数间隔：

γ ˆ (i) = y (i) (w T x (i) + b)

$\mathop {\widehat \gamma }\nolimits^{\left( i \right)} = \mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right)$

γˆ(i) $\mathop {\widehat \gamma }\nolimits^{\left( i \right)}$ 越大，分类效果越好。取所有样本最小的函数间隔作为训练集的函数间隔：

γ = min i = 1, \dots, m γ ˆ (i)

$\gamma = \mathop {\min }\limits_{i = 1, \cdots ,m} \mathop {\widehat \gamma }\nolimits^{\left( i \right)}$

几何间隔： $\mathop w\nolimits^T \left( {\mathop x\nolimits^{\left( i \right)} - \mathop \gamma \nolimits^{\left( i \right)} \frac{w}{{^{\left\| w \right\|}}}} \right) + b = 0$ 由左式得：

γ (i) = (w ∥ w ∥) T x (i) + b ∥ w ∥

$\mathop \gamma \nolimits^{\left( i \right)} = \mathop {\left( {\frac{w}{{\left\| w \right\|}}} \right)}\nolimits^T \mathop x\nolimits^{\left( i \right)} + \frac{b}{{\left\| w \right\|}}$ 上式是分类为1的样本的几何间隔，再将分类为-1的样本考虑进来，重新定义几何间隔为：

γ (i) = y (i) ⎛ ⎝ (w ∥ w ∥) T x (i) + b ∥ w ∥ ⎞ ⎠

$\mathop \gamma \nolimits^{\left( {\rm{i}} \right)} = \mathop y\nolimits^{\left( i \right)} \left( {\mathop {\left( {\frac{w}{{\left\| w \right\|}}} \right)}\nolimits^T \mathop x\nolimits^{\left( i \right)} + \frac{b}{{\left\| w \right\|}}} \right)$ 整个样本的几何间隔为：

γ = min i = 1, \dots, m γ (i)

$\gamma = \mathop {\min }\limits_{i = 1, \cdots ,m} \mathop \gamma \nolimits^{\left( {\rm{i}} \right)}$ 几何间隔等于函数间除以

∥w∥ $\left\| w \right\|$ ，当

∥w∥=1 $\left\| w \right\|=1$ 时，几何间隔等于函数间隔。

用来测量距离的尺子已经制作好了，利用这两把尺子把我们之前的朴素想法转化为数学语言，就变成了一个优化问题：

max γ, w, b γ s . t . y (i) (w T x (i) + b) \geq γ, i = 1, \dots, m ∥ w ∥ = 1

$\begin{array}{l} \mathop {\max }\nolimits_{\gamma ,w,b} \gamma \\ s.t.\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) \ge \gamma ,i = 1, \cdots ,m\\ \left\| w \right\| = 1 \end{array}$
当

∥w∥=1 $\left\| w \right\| = 1$ 时，几何间隔和函数间隔相等，上述优化问题等价于：

max γ, w, b γ ^ ∥ w ∥ s . t . y (i) (w T x (i) + b) \geq γ^, i = 1, \dots, m

$\begin{array}{l} \mathop {\max }\nolimits_{\gamma ,w,b} \frac{{\hat \gamma }}{{\left\| w \right\|}}\\ s.t.\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) \ge \hat \gamma ,i = 1, \cdots ,m \end{array}$
令

γ^=1 $\hat \gamma = 1$ （因为

wTx+b=0 $\mathop w\nolimits^T x + b = 0$ ，所以函数间隔可以等比例的缩放w,b而不改变超平面的位置），上述优化问题转化为：

min γ, w, b 1 2 ∥ w ∥ 2 s . t . y (i) (w T x (i) + b) \geq 1, i = 1, \dots, m

$\begin{array}{l} \mathop {\min }\nolimits_{\gamma ,w,b} \frac{1}{2}\mathop {\left\| w \right\|}\nolimits^2 \\ s.t.\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) \ge 1,i = 1, \cdots ,m \end{array}$
这就是SVM的优化目标，目标函数和限制条件都是凸函数，这是一个凸优化问题，凸优化满足强对偶性，原问题与对偶问题的最优解相等且等价于KKT条件的解，这也是我们花那么多努力将最初的优化问题转化为上述优化问题的原因。

限定条件的最值问题，利用拉格朗日乘子法解决。一般形式的凸优化问题有两个限定条件：一个不等式限制，一个等式限制：

min x, λ, v f 0 (x) s . t . f i (x) \leq 0, i = 1, \dots, m h i (x) = 0, i = 1, \dots, p

$\begin{array}{l} \mathop {\min }\nolimits_{x,\lambda ,v} \mathop {\rm{f}}\nolimits_0 \left( x \right)\\ s.t.\mathop f\nolimits_i \left( x \right) \le 0,i = 1, \cdots ,m\\ \mathop h\nolimits_i \left( x \right) = 0,i = 1, \cdots ,p \end{array}$ 称该问题为原问题，最优点为

x∗ $\mathop x\nolimits^ *$ ，最优值为

p∗ $\mathop p\nolimits^ *$

构造拉格朗日量：

L (x, λ, v) = f 0 (x) + \sum i = 1 m λ i f i (x) + \sum i = 1 p v i h i (x)

$L(x,\lambda ,v) = \mathop {\rm{f}}\nolimits_0 \left( x \right) + \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop f\nolimits_i \left( x \right) + \sum\limits_{i = 1}^p {\mathop v\nolimits_i } \mathop h\nolimits_i \left( x \right)$ 拉格朗日对偶函数：

g (λ, v) = inf x \in D L (x, λ, v) = inf x \in D f 0 (x) + \sum i = 1 m λ i f i (x) + \sum i = 1 p v i h i (x)

$\begin{array}{l} g(\lambda ,v) = \mathop {\inf }\limits_{x \in D} L(x,\lambda ,v)\\ = \mathop {\inf }\limits_{x \in D} \mathop {{\rm{ f}}}\nolimits_0 \left( x \right) + \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop f\nolimits_i \left( x \right) + \sum\limits_{i = 1}^p {\mathop v\nolimits_i } \mathop h\nolimits_i \left( x \right) \end{array}$

g(λ,v) $g(\lambda ,v)$ 是

L(x,λ,v) $L(x,\lambda ,v)$ 的逐点下确界。直观理解：固定

λ $\lambda$ 和v，x跑遍定义域，

Lλ,v(x) $\mathop L\nolimits_{\lambda ,v} (x)$ 值域的最小值。

对偶问题为原问题提供了下界。限制 $\mathop \lambda \nolimits_i \ge 0,i = 1, \cdots ,m$ 则：

g (λ, v) \leq p *

$g(\lambda ,v) \le \mathop p\nolimits^ *$ 证明如下：

g (λ, v) \leq f 0 (x) + \sum i = 1 m λ i f i (x) + \sum i = 1 p v i h i (x) = f 0 (x) + \sum i = 1 m λ i f i (x) \leq f 0 (x)

$\begin{array}{l} g(\lambda ,v) \le \mathop {\rm{f}}\nolimits_0 \left( x \right) + \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop f\nolimits_i \left( x \right) + \sum\limits_{i = 1}^p {\mathop v\nolimits_i } \mathop h\nolimits_i \left( x \right)\\ = \mathop {\rm{f}}\nolimits_0 \left( x \right) + \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop f\nolimits_i \left( x \right)\\ \le \mathop {\rm{f}}\nolimits_0 \left( x \right) \end{array}$
现将目标转向求解原问题的对偶问题（简化了约束），即：

max λ, v g (λ, v) s . t . λ i \geq 0, i = 1, \dots, m

$\begin{array}{l} \mathop {\max }\nolimits_{\lambda ,v} g\left( {\lambda ,v} \right)\\ s.t.\mathop \lambda \nolimits_i \ge 0,i = 1, \cdots ,m \end{array}$ 设对偶问题的最优点为

(λ∗,v∗) $\left( {\mathop \lambda \nolimits^ * ,\mathop v\nolimits^ * } \right)$ ，最优值为

d∗ $\mathop d\nolimits^ *$

凸优化满足强对偶条件，即： $\mathop d\nolimits^ * = \mathop p\nolimits^ *$

d * = g (λ *, v *) \leq f 0 (x *) + \sum i = 1 m λ * i f i (x *) + \sum i = 1 p v * i h i (x *) = f 0 (x *) + \sum i = 1 m λ * i f i (x *) \leq f 0 (x *) = p *

$\begin{array}{l} \mathop d\nolimits^ * = g\left( {\mathop \lambda \nolimits^ * ,\mathop v\nolimits^ * } \right)\\ \le \mathop {\rm{f}}\nolimits_0 \left( {\mathop x\nolimits^ * } \right) + \sum\limits_{i = 1}^m {\mathop {\mathop \lambda \nolimits^ * }\nolimits_i } \mathop f\nolimits_i \left( {\mathop x\nolimits^ * } \right) + \sum\limits_{i = 1}^p {\mathop {\mathop v\nolimits^ * }\nolimits_i } \mathop h\nolimits_i \left( {\mathop x\nolimits^ * } \right)\\ = \mathop {\rm{f}}\nolimits_0 \left( {\mathop x\nolimits^ * } \right) + \sum\limits_{i = 1}^m {\mathop {\mathop \lambda \nolimits^ * }\nolimits_i } \mathop f\nolimits_i \left( {\mathop x\nolimits^ * } \right)\\ \le \mathop {\rm{f}}\nolimits_0 \left( {\mathop x\nolimits^ * } \right)\\ = \mathop p\nolimits^ * \end{array}$

d∗=p∗ $\mathop d\nolimits^ * = \mathop p\nolimits^ *$ 意味着上述两个不等式全为等式。于是有：

\nabla x L (x, λ *, v *) = 0 \sum i = 1 m λ * i f i (x *) = 0

$\begin{array}{l} \mathop \nabla \nolimits_x L(x,\mathop \lambda \nolimits^ * ,\mathop v\nolimits^ * ) = 0\\ \sum\limits_{i = 1}^m {\mathop {\mathop \lambda \nolimits^ * }\nolimits_i } \mathop f\nolimits_i \left( {\mathop x\nolimits^ * } \right) = 0 \end{array}$ 再加上原问题与对偶问题的限定条件可得：

f i (x *) \leq 0, i = 1, \dots, m h i (x *) = 0, i = 1, \dots, p λ * i \geq 0, i = 1, \dots, m λ * i f i (x *) = 0, i = 1, \dots, m \nabla x L (x, λ *, v *) = 0

$\begin{array}{l} \mathop f\nolimits_i \left( {\mathop x\nolimits^ * } \right) \le 0,i = 1, \cdots ,m\\ \mathop h\nolimits_i \left( {\mathop x\nolimits^ * } \right) = 0,i = 1, \cdots ,p\\ \mathop {\mathop \lambda \nolimits^ * }\nolimits_i \ge 0,i = 1, \cdots ,m\\ \mathop {\mathop {\mathop \lambda \nolimits^ * }\nolimits_i f}\nolimits_i \left( {\mathop x\nolimits^ * } \right) = 0,i = 1, \cdots ,m\\ \mathop \nabla \nolimits_x L(x,\mathop \lambda \nolimits^ * ,\mathop v\nolimits^ * ) = 0 \end{array}$ 以上五个式子被称为KKT条件。

针对SVM优化目标构造拉格朗日量：

L (λ, w, b) = 1 2 ∥ w ∥ 2 - \sum i = 1 m λ i (y (i) (w T x (i) + b) - 1)

${\rm{L}}(\lambda ,w,b) = \frac{1}{2}\mathop {\left\| w \right\|}\nolimits^2 - \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \left( {\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) - 1} \right)$

\partial L ( λ , w , b ) \partial w = w - \sum i = 1 m λ i y (i) x (i) = 0 \Rightarrow w = \sum i = 1 m λ i y (i) x (i)

$\begin{array}{l} \frac{{\partial {\rm{L}}(\lambda ,w,b)}}{{\partial w}} = w - \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \mathop x\nolimits^{\left( i \right)} = 0\\ \Rightarrow w = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \mathop x\nolimits^{\left( i \right)} \end{array}$

w $w$ 是输入向量

x(i) $\mathop x\nolimits^{\left( i \right)}$ 的线性组合，回到SVM的模型：

hw,b(x)=g(wTx+b) $\mathop h\nolimits_{w,b} \left( x \right) = g(\mathop w\nolimits^T x + b)$ ，我们关心的是

wTx+b $\mathop w\nolimits^{\rm{T}} x + b$ 的大小

w T x + b = \sum i = 1 m λ i y (i) (x (i)) T x + b = \sum i = 1 m λ i y (i) ⟨ x (i), x ⟩ + b

$\begin{array}{l} \mathop w\nolimits^T x + b = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \mathop {\left( {\mathop x\nolimits^{\left( i \right)} } \right)}\nolimits^T x + b = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \left\langle {\mathop x\nolimits^{\left( i \right)} ,x} \right\rangle + b \end{array}$

wTx+b $\mathop w\nolimits^T x + b$ 是新输入的样本与训练样本的内积之和。当

λ∗i>0 ${\mathop {\mathop \lambda \nolimits^ * }\nolimits_i }>0$ ，由KKT条件

λ * i f i (x *) = 0 \Rightarrow f i (x *) = 0 \Rightarrow y (i) (w T x (i) + b) = 1

$\begin{array}{l} \mathop {\mathop {\mathop \lambda \nolimits^ * }\nolimits_i f}\nolimits_i \left( {\mathop x\nolimits^ * } \right) = 0 \Rightarrow \mathop f\nolimits_i \left( {\mathop x\nolimits^ * } \right) = 0 \Rightarrow \mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) = 1 \end{array}$ 此时

x(i) ${\mathop x\nolimits^{\left( i \right)} }$ 是取得最小间隔的样本，被称为支持向量。支持向量只占样本很小的一部分，大部分

λi=0 ${\mathop \lambda \nolimits_i }=0$ ，所以只需要计算支持向量与新输入样本的内积。

\partial L ( λ , w , b ) \partial b = \sum i = 1 m λ i y (i) = 0

$\frac{{\partial {\rm{L}}(\lambda ,w,b)}}{{\partial b}} = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0$ 对偶函数：

g (λ) = \sum i = 1 m λ i - 1 2 \sum i = 1 m \sum j = 1 m λ i λ j y (i) y (j) ⟨ x (i), x (j) ⟩

$\begin{array}{l} g(\lambda ) = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } - \frac{1}{2}\sum\limits_{i = 1}^m {\sum\limits_{j = 1}^m {\mathop \lambda \nolimits_i } } \mathop \lambda \nolimits_j \mathop y\nolimits^{\left( i \right)} \mathop y\nolimits^{\left( j \right)} \left\langle {\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)} } \right\rangle \end{array}$
把求解原问题的最小值转化为求解对偶问题的最大值：

max λ g (λ) = \sum i = 1 m λ i - 1 2 \sum i = 1 m \sum j = 1 m λ i λ j y (i) y (j) ⟨ x (i), x (j) ⟩ s . t . λ i \geq 0, i = 1, \dots, m \sum i = 1 m λ i y (i) = 0

$\begin{array}{l} \mathop {\max }\nolimits_\lambda g(\lambda ) = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } - \frac{1}{2}\sum\limits_{i = 1}^m {\sum\limits_{j = 1}^m {\mathop \lambda \nolimits_i } } \mathop \lambda \nolimits_j \mathop y\nolimits^{\left( i \right)} \mathop y\nolimits^{\left( j \right)} \left\langle {\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)} } \right\rangle \\ s.t. \mathop \lambda \nolimits_i \ge 0,i = 1, \cdots ,m\\ \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0 \end{array}$
注：为什么要加上

∑i=1mλiy(i)=0 $\sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0$ 这个约束？根据定义

g(λ) $g\left( \lambda \right)$ 是

L(λ,w,b) $L(\lambda ,w,b)$ 在定义域上的逐点下确界，如果

∑i=1mλiy(i)≠0 $\sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \ne 0$ 则

L(λ,w,b) $L(\lambda ,w,b)$ 是关于b的线性函数，无法取到最小值，

g(λ) $g\left( \lambda \right)$ 无定义。

求解上述凸优化问题，得到 $\mathop \lambda \nolimits^ *$ ，带入 $w = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} \mathop x\nolimits^{\left( i \right)}$ ，得到 $\mathop w\nolimits^ *$ ，即确定了超面的方向，从而易得：

b * = max i : y ( i ) = - 1 w * T x ( i ) + min i : y ( i ) = 1 w * T x ( i ) 2

$\begin{array}{l} \mathop b\nolimits^ * = \frac{{\mathop {\max }\nolimits_{i:\mathop y\nolimits^{\left( i \right)} = - 1} \mathop {\mathop w\nolimits^ * }\nolimits^T \mathop x\nolimits^{\left( i \right)} + \mathop {\min }\nolimits_{i:\mathop y\nolimits^{\left( i \right)} = 1} \mathop {\mathop w\nolimits^ * }\nolimits^T \mathop x\nolimits^{\left( i \right)} }}{2} \end{array}$
样本往往有很高维的特征，为了将SVM应用到高维甚至无穷维，需要借助核方法。核方法将

X ${\cal X}$ 空间中的任意两个向量

x(i),x(j) $\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)}$ 映射成另一空间的内积：

k (x (i), x (j)) = ⟨ ϕ (x (i)), ϕ (x (j)) ⟩

$\begin{array}{l} k(\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)} ) = \left\langle {\phi \left( {\mathop x\nolimits^{\left( i \right)} } \right),\phi \left( {\mathop x\nolimits^{\left( j \right)} } \right)} \right\rangle \end{array}$

ϕ $\phi$ 为核函数，如何构建合适的核函数以及如何证明一个核函数的正确性可以看Andrew Ng的讲义。

我们假定样本线性可分，如果样本线性不可分呢？需要对上述SVM进行改进，优化目标变为：L1 norm margin SVM

min γ, w, b 1 2 ∥ w ∥ 2 + C \sum i = 1 m ξ i s . t . y (i) (w T x (i) + b) \geq 1 - ξ i, i = 1, \dots, m ξ i \geq 0, i = 1, \dots, m

$\begin{array}{l} \mathop {\min }\nolimits_{\gamma ,w,b} \frac{1}{2}\mathop {\left\| w \right\|}\nolimits^2 + C\sum\limits_{i = 1}^m {\mathop \xi \nolimits_i } \\ s.t.\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) \ge 1 - \mathop \xi \nolimits_i ,i = 1, \cdots ,m\\ \mathop \xi \nolimits_i \ge 0,i = 1, \cdots ,m \end{array}$ 直观理解：当

ξi>1 $\mathop \xi \nolimits_i > 1$ 时，

y(i)(wTx(i)+b)<0 $\mathop y\nolimits^{\left( i \right)} \left( {\mathop w\nolimits^T \mathop x\nolimits^{\left( i \right)} + b} \right) < 0$ ，样本分类错误，我们允许这种情况发生，但是要对优化目标加一个正数（

C∑i=1mξi $C\sum\limits_{i = 1}^m {\mathop \xi \nolimits_i }$ ）作为惩罚。此时对偶问题变为：

max λ g (λ) = \sum i = 1 m λ i - 1 2 \sum i = 1 m \sum j = 1 m λ i λ j y (i) y (j) ⟨ x (i), x (j) ⟩ s . t . 0 \leq λ i \leq C, i = 1, \dots, m \sum i = 1 m λ i y (i) = 0

$\begin{array}{l} \mathop {\max }\nolimits_\lambda g(\lambda ) = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } - \frac{1}{2}\sum\limits_{i = 1}^m {\sum\limits_{j = 1}^m {\mathop \lambda \nolimits_i } } \mathop \lambda \nolimits_j \mathop y\nolimits^{\left( i \right)} \mathop y\nolimits^{\left( j \right)} \left\langle {\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)} } \right\rangle \\ s.t.\mathop {0 \le \lambda }\nolimits_i \le C,i = 1, \cdots ,m\\ \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0 \end{array}$ 接下来的目标就是求解该对偶问题，解得

λ∗ $\mathop \lambda \nolimits^ *$ 。

先介绍坐标上升算法：无约束条件的一个优化问题：

max α W (α 1, α 2, \dots, α m)

$\mathop {\max }\limits_\alpha W\left( {\mathop \alpha \nolimits_1 ,\mathop \alpha \nolimits_2 , \cdots ,\mathop \alpha \nolimits_m } \right)$ 坐标上升的思路：每次选取一个参数，固定其它参数不动，求得相对选定参数的最大值，不断重复这一过程，最终求得最终的最大值。将这一思路应用到修正过的SVM优化目标，因为有约束条件

∑i=1mλiy(i)=0 $\sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0$ 的存在，无法每优化一个参数。但是可以一次优化两个参数。该算法被称为SMO算法，顺序最小优化算法，所谓最小即为每次优化最小数量的参数。借助SMO算法，我们可以高效的实现SVM。

最后回答一下最开始提到的机器学习的三个问题，作为一个总结：

SVM的模型： $\mathop h\nolimits_{w,b} (x) = g\left( {\mathop w\nolimits^T x + b} \right)$
SVM的优化目标： $max λ g (λ) = \sum i = 1 m λ i - 1 2 \sum i = 1 m \sum j = 1 m λ i λ j y (i) y (j) ⟨ x (i), x (j) ⟩ s . t . 0 \leq λ i \leq C, i = 1, \dots, m \sum i = 1 m λ i y (i) = 0$ $\begin{array}{l} \mathop {\max }\nolimits_\lambda g(\lambda ) = \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } - \frac{1}{2}\sum\limits_{i = 1}^m {\sum\limits_{j = 1}^m {\mathop \lambda \nolimits_i } } \mathop \lambda \nolimits_j \mathop y\nolimits^{\left( i \right)} \mathop y\nolimits^{\left( j \right)} \left\langle {\mathop x\nolimits^{\left( i \right)} ,\mathop x\nolimits^{\left( j \right)} } \right\rangle \\ s.t.\mathop {0 \le \lambda }\nolimits_i \le C,i = 1, \cdots ,m\\ \sum\limits_{i = 1}^m {\mathop \lambda \nolimits_i } \mathop y\nolimits^{\left( i \right)} = 0 \end{array}$
求解SVM优化目标的算法：SMO算法

PS：为什么要将原问题转化为对偶问题，最重要的原因：简化约束。约束是很麻烦的，没有约束的最值问题的求解可以用梯度下降或者牛顿法很好的解决。无论原问题是不是凸优化问题，其对偶问题都是凸优化问题。区别在于：如果原问题是凸优化则对偶问题与原问题的最值相等，如果原问题是非凸的，对偶问题提供了对原问题很好的近似，而无论哪种情况，对偶问题都简化了原问题的约束，降低了求解难度。