cs224n-naive-softmax-的推导与实现

最新推荐文章于 2025-03-13 19:00:00 发布

原创

最新推荐文章于 2025-03-13 19:00:00 发布 · 1.2k 阅读

1 ·

CC 4.0 BY-SA版权

This blog covers the derivation and implementation of naive softmax in the context of CS224n (2019). It starts with an introduction, followed by a detailed explanation of the formula derivation using chain rule and the gradient of the softmax function. The post concludes with a brief mention of the provided code for the implementation." 88195562,5040019,在线编译器推荐：从调试到汇编查看,"['编译器', '工具', '在线开发', '调试工具', '代码编辑']

cs224n-naive-softmax-的推导与实现

文章目录

- cs224n-naive-softmax-的推导与实现

0、简介

在cs224n（2019）第二次课后作业Assignment 2的手写作业b（推导梯度公式）和编程作业a中对naive softmax的实现。首先手写作业的推导公式，编程作业则是对这些公式的简单实现

1、公式推出

首先看下背景和问题

首先问题a中已经证明了损失函数 $\hat{y})$ ，而且有 $\hat{y}=softmax(U^Tv_c)$ , 不妨用 $\theta=U^tv_c$ ,现在有 $\hat{y}=softmax(\theta)$ , 然后用chain rule推导J关于vc的偏导数。

$\\J=-y\log{\hat{y}};\ \frac{\partial J}{\partial \hat{y}}=-\frac{y}{\hat{y}} \\ \hat{y}=softmax(\theta) \\ \text{关于softmax的求导如下：} \\ \because y=\frac{e^{x_i}}{\Sigma{e^{xj}}} \\ \frac{\partial{y}}{\partial{x_i}} = \frac{e^{x_i}(\Sigma-e^{x_i})}{\Sigma^2}=y(1-y) \\ \text{由以上softmax的导数可推：} \\ \frac{\partial{\hat{y}}}{\partial{\theta}}=\hat{y}(1-\hat{y}) \\ \frac{\partial{\theta}}{\partial{v_c}}=U^T \\ \text{由链式法则：} \\ \because \begin{aligned} \frac{\partial J}{\partial v_c} &= \frac{\partial J}{\partial \hat{y}} \frac{\partial{\hat{y}}}{\partial{\theta}} \frac{\partial \theta}{\partial v_c} =-y\frac{1}{\hat{y}}\hat{y}(1-\hat{y})U^T=-y(1-\hat{y})U^T \ \end{aligned}$

最低0.47元/天解锁文章