机器学习4 - 牛顿法求最值

最新推荐文章于 2022-06-08 18:17:54 发布

skmygdrs

最新推荐文章于 2022-06-08 18:17:54 发布

阅读量637

点赞数 1

CC 4.0 BY-SA版权

分类专栏： machine_learning

本文链接：https://blog.youkuaiyun.com/skmygdrs/article/details/52054894

machine_learning 专栏收录该内容

9 篇文章

订阅专栏

本文详细介绍了牛顿法的数学原理及其推导过程，通过泰勒展开公式解释了牛顿法迭代更新参数的具体方式，并给出了求解多元函数极值问题的方法。

公式

θ (t + 1) = θ (t) - H - 1 \nabla θ l

${\theta ^{(t + 1)}} = {\theta ^{(t)}} - {H^{ - 1}}{\nabla _\theta }l$

海森矩阵

H i j = \partial 2 l \partial θ i \partial θ j

${H_{ij}} = {{{\partial ^2}l} \over {\partial {\theta _i}\partial {\theta _j}}}$

推导

泰勒展开式

f (x) = f (x 0) + f (1) (x o) (x - x 0) + . . . + f ( n ) ( x 0 ) n ! (x - x 0) n + f ( n + 1 ) ( x o + θ ( x - x 0 ) ) ( n + 1 ) ! (θ (x - x 0)) n + 1

$f(x) = f({x_0}) + {f^{(1)}}({x_o})(x - {x_0}) + ... + {{{f^{(n)}}({x_0})} \over {n!}}{(x - {x_0})^n} + {{{f^{(n + 1)}}\left( {{x_o} + \theta (x - {x_0})} \right)} \over {(n + 1)!}}{\left( {\theta (x - {x_0})} \right)^{n + 1}}$

多元函数泰勒展开

f (X) = f (X 0) + (X - X 0) T \nabla f (X 0) + . . . + 1 n ! [(X - X 0) T \nabla] n f (X 0) + 1 ( n + 1 ) ! [θ (X - X 0) T \nabla] n + 1 f (X 0 + θ (X - X 0))

$f(X) = f({X_0}) + {(X - {X_0})^T}\nabla f({X_0}) + ... + {1 \over {n!}}{\left[ {{{(X - {X_0})}^T}\nabla } \right]^n}f({X_0}) + {1 \over {(n + 1)!}}{\left[ {\theta {{(X - {X_0})}^T}\nabla } \right]^{n + 1}}f\left( {{X_0} + \theta (X - {X_0})} \right)$

取n=2

f (X) = f (X 0) + (X - X 0) T \nabla f (X 0) + 1 2 (X - X 0) T \nabla f (X 0) \nabla T f (X 0) (X - X 0) + o (∥ X - X 0 ∥ 2)

$f(X) = f({X_0}) + {(X - {X_0})^T}\nabla f({X_0}) + {1 \over 2}{(X - {X_0})^T}\nabla f({X_0}){\nabla ^T}f({X_0})(X - {X_0}) + o\left( {{{\left\| {X - {X_0}} \right\|}^2}} \right)$
记

∇f(X0)∇Tf(X0)=∇2f(X0)=H(X0) $\nabla f({X_0}){\nabla ^T}f({X_0}) = {\nabla ^2}f({X_0}) = H({X_0})$ 为

Hessian $Hessian$ 矩阵

\nabla X 1 2 (X - X 0) T \nabla f (X 0) \nabla T f (X 0) (X - X 0) = \nabla X 1 2 (X - X 0) T H (X 0) (X - X 0) = 1 2 \nabla X t r ((X - X 0) T H (X 0) (X - X 0)) = 1 2 \nabla X t r (X T H (X 0) X - X 0 T H (X 0) X - X T H (X 0) X 0 + X 0 T H (X 0) X 0) = 1 2 \nabla X t r (X X T H (X 0)) - \nabla X t r (X X 0 T H (X 0)) - \nabla X t r (X X 0 T H T (X 0)) = 1 2 (H (X 0) X + H T (X 0) X - H T (X 0) X 0 - H (X 0) X 0) = 1 2 (H (X 0) + H T (X 0)) (X - X 0) = H (X 0) (X - X 0)

$\eqalign{ & {\nabla _X}{1 \over 2}{(X - {X_0})^T}\nabla f({X_0}){\nabla ^T}f({X_0})(X - {X_0}) \cr & = {\nabla _X}{1 \over 2}{(X - {X_0})^T}H({X_0})(X - {X_0}) \cr & = {1 \over 2}{\nabla _X}tr\left( {{{(X - {X_0})}^T}H({X_0})(X - {X_0})} \right) \cr & = {1 \over 2}{\nabla _X}tr\left( {{X^T}H({X_0})X - {X_0}^TH({X_0})X - {X^T}H({X_0}){X_0} + {X_0}^TH({X_0}){X_0}} \right) \cr & = {1 \over 2}{\nabla _X}tr\left( {X{X^T}H({X_0})} \right) - {\nabla _X}tr\left( {X{X_0}^TH({X_0})} \right) - {\nabla _X}tr\left( {X{X_0}^T{H^T}({X_0})} \right) \cr & = {1 \over 2}\left( {H({X_0})X + {H^T}({X_0})X - {H^T}({X_0}){X_0} - H({X_0}){X_0}} \right) \cr & = {1 \over 2}\left( {H({X_0}) + {H^T}({X_0})} \right)\left( {X - {X_0}} \right) \cr & = H({X_0})(X - {X_0}) \cr}$

\nabla X f (X) = \nabla X (f (X 0) + (X - X 0) T \nabla f (X 0) + 1 2 (X - X 0) T \nabla f (X 0) \nabla T f (X 0) (X - X 0)) = \nabla f (X 0) + H (X 0) (X - X 0)

$\eqalign{ & {\nabla _X}f(X) = {\nabla _X}\left( {f({X_0}) + {{(X - {X_0})}^T}\nabla f({X_0}) + {1 \over 2}{{(X - {X_0})}^T}\nabla f({X_0}){\nabla ^T}f({X_0})(X - {X_0})} \right) \cr & = \nabla f({X_0}) + H({X_0})(X - {X_0}) \cr}$
当

f(X) $f(X)$ 取最值时