李航《统计学习方法》SMO算法推导中的思考

1. p.128

图中，从上式到下式的推导不是很明了，困惑在于上式中右边含有$\alpha_1,\alpha_2$这样岂不是和左边的$\alpha_2$相消？若能相消，上述求偏导的过程中岂不是忽略了$v_1,v_2$是$\alpha_1,\alpha_2$的函数？答案并非如此，左边的$\alpha$与右边的$\alpha_2$相当不同。

2. 定理7.6的证明

即求最优化问题:

$\min\limits_{\alpha_1,\alpha_2}W(\alpha_1,\alpha_2)=\frac{1}{2}K_{11}\alpha_1^2+\frac{1}{2}K_22\alpha_2^2+y_1y_2K_{21}\alpha_1\alpha_2-(\alpha_1+\alpha_2)+y_1\alpha_1\sum\limits_{i=3}^{N}y_i\alpha_iK_{i1}+y_2\alpha_2\sum\limits_{i=3}^Ny_i\alpha_iK_{i2}\\ \quad s.t. \\ \qquad \alpha_1y_1+\alpha_2y_2=-\sum\limits_{i=3}^Ny_i\alpha_i=\varsigma \qquad(1)\\ \qquad 0\le\alpha_i\le C ,\quad i=1,2 \qquad(2)$
沿着约束方向未经剪辑的解为:$\alpha_2^{new,unc}=\alpha_2^{old}+\frac{y_2(E_1-E_2)}{\eta}$
其中，$\eta=K_{11}+K_{22}-2K_{21}=\|\Phi(x_1)-\Phi(x_2)\|^2\\ E_i=g(x_i)-y_i , \quad i=1,2 \\ g(x)=\sum\limits_{i=1}^{N}\alpha_iy_iK(x_i,x)+b$
$E_i$为函数$g(x_i)$对输入$x_i$的预测值与真实输出$y_i$之间的差.

证明：

引入记号变量 $v_i=\sum\limits_{j=3}^{N}\alpha_jy_jK(x_i,x_j)=g(x_i)-\sum\limits_{j=1}^{2}\alpha_jy_jK(x_i,x_j)-b, \quad i=1,2$
再由$\alpha_1y_1+\alpha_2y_2=\varsigma$和 $y_i^2=1$可将目标函数$W(\alpha_1,\alpha_2)$表示成只含$\alpha_2$的函数:

$W(\alpha_2)=\frac{1}{2}K_{11}(\varsigma-\alpha_2y_2)^2+\frac{1}{2}K_{22}\alpha_2^2+y_2K_{21}(\varsigma-\alpha_2y_2)\alpha_2-(\varsigma-\alpha_2y_2)y_1-\alpha_2+v_1(\varsigma-\alpha_2y_2)+y_2v_2\alpha_2$
对$\alpha_2$求偏导可得:$\frac{\partial W}{\partial \alpha_2}=K_{11}\alpha_2+K_{22}\alpha_2-2K_{21}\alpha_2-K_{11}\varsigma y_2+K_{21}\varsigma y_2+y_1y_2-1-v_1y_2+v_2y_2$
令其等于0,即可求出上述问题的最优解,得到:$(K_{11}+K_{22}-2K_{21})\alpha_2=y_2(y_2-y_1+\varsigma K_{11}-\varsigma K_{21}+v_1-v_2)\qquad (2-1)$
这里，不同与书上，我先求$v_1-v_2$,
由$v_i$的定义可知$v_1-v_2$为:$v_1-v_2= \lgroup g(x_1)-\sum\limits_{j=1}^{2}y_j\alpha_jK_{1j}-b\rgroup-\lgroup g(x_2)-\sum\limits_{j=1}^{2}y_j\alpha_jK_{2j}-b\rgroup\\ \qquad \quad=g(x_1)-g(x_2)+y_1\alpha_1K_{21}-y_1\alpha_1K_{11}+y_2\alpha_2K_{22}-y_2\alpha_2K_{21}\\ \because \alpha_1y_1+\alpha_2y_2=\varsigma,\quad y_1^2=1\\\therefore \alpha_1=(\varsigma-y_2\alpha_2)y_1 ,代入得，\\\qquad\quad =g(x_1)-g(x_2)+(\varsigma-y_2\alpha_2)K_{21}-(\varsigma-y_2\alpha_2)K_{11}-y_2\alpha_2K_{21}+y_2\alpha_2K_{22}\\\qquad\quad =g(x_1)-g(x_2)+\varsigma K_{21}-\varsigma K_{11}+y_2\alpha_2(K_{11}+K_{22}-2K_{21})$
值得注意的是，$v_1-v_2$中的$\alpha_1,\alpha_2$是没有更新前的$\alpha$,即$\alpha^{old}$
$\therefore v_1-v_2=g(x_1)-g(x_2)+\varsigma K_{21}-\varsigma K_{11}+y_2\alpha_2^{old}(K_{11}+K_{22}-2K_{21})$将上式代入到式（2-1）中即可求得$\alpha_2^{new,unc}=\alpha_2^{old}+\frac{y_2(E_1-E_2)}{\eta}$