Softmax 反向传播公式推导

最新推荐文章于 2025-03-18 12:15:37 发布

Charel_CHEN

最新推荐文章于 2025-03-18 12:15:37 发布

阅读量4.9k

点赞数 3

分类专栏：深度学习与计算机视觉 caffe源代码

本文链接：https://blog.youkuaiyun.com/Charel_CHEN/article/details/81266575

版权

深度学习与计算机视觉同时被 2 个专栏收录

32 篇文章

订阅专栏

caffe源代码

13 篇文章

订阅专栏

Softmax 反向传播公式推导

标签（空格分隔）： Caffe源代码

Softmax是深度学习最常见的激活函数，能够将输入按照某一维度进行归一化，输出（0， 1）的值

Softmax函数形式为：

f (x i) = e x i \sum j = 1 K e x j

$f({x_i}) = \frac{{{e^{{x_i}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}$

现在我们对Softmax函数进行求导，分为两种情况：
（1）当 $k = i$ 时，

\partial f \partial x k = \partial f \partial x i = e x i \times \sum j = 1 K e x j - e 2 x i ( \sum j = 1 K e x j ) 2 = e x i \times ( \sum j = 1 K e x j - e x i ) ( \sum j = 1 K e x j ) 2 = e x i \sum j = 1 K e x j \times \sum j = 1 K e x j - e x i \sum j = 1 K e x j

$\frac{{\partial f}}{{\partial {x_k}}} = \frac{{\partial f}}{{\partial {x_i}}} = \frac{{{e^{{x_i}}} \times \sum\limits_{j = 1}^K {{e^{{x_j}}}} - {e^{2{x_i}}}}}{{{{(\sum\limits_{j = 1}^K {{e^{{x_j}}}} )}^2}}} = \frac{{{e^{{x_i}}} \times (\sum\limits_{j = 1}^K {{e^{{x_j}}}} - {e^{{x_i}}})}}{{{{(\sum\limits_{j = 1}^K {{e^{{x_j}}}} )}^2}}} = \frac{{{e^{{x_i}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }} \times \frac{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} - {e^{{x_i}}}}}{{\sum\limits_{j = 1}^K {{e^{{x_j}}}} }}$

显然，上面式子可以化简成如下：

\partial f \partial x k = f (x k) (1 - f (x k))

$\frac{{\partial f}}{{\partial {x_k}}} = f({x_k})(1 - f({x_k}))$

(2)当 $k \ne i$ 时

\partial f \partial x k = - e x i \times e x k ( \sum j = 1 K e x j ) 2 = - f (x i) f (x k)

$\frac{{\partial f}}{{\partial {x_k}}} = \frac{{ - {e^{{x_i}}} \times {e^{{x_k}}}}}{{{{(\sum\limits_{j = 1}^K {{e^{{x_j}}}} )}^2}}} = -f({x_i})f({x_k})$

两者结合起来就是：

\partial f \partial x k = - \sum i \neq k f (x i) f (x k) + f (x k) (1 - f (x k)) = f (x k) - \sum i = 1 K f (x k) f (x i)

$\frac{{\partial f}}{{\partial {x_k}}} = - \sum\limits_{i \ne k} {f({x_i})} f({x_k}) + f({x_k})(1 - f({x_k})) = f({x_k}) - \sum\limits_{i = 1}^K {f({x_k})} f({x_i})$

那么在Caffe代码中怎么进行反向传播的呢？