损失函数L对全连接层W、X、b的梯度

假设：X的维度为$s\times n$，其中s为样本数，每个样本均展平为$1\times n$的行向量；W维度为$n\times o$，其中o为全连接层的输出维度；b维度为$1\times o$；Z维度为$s\times o$，并且$Z=X\times W+b$
$$Z=\left[\begin{array}{ccc}{z}_{11}& \dots & z_{1o}\\ & \dots & \\ z_{s1}& \dots & z_{so}\end{array}\right]$$
$$X=\left[\begin{array}{ccc}{x}_{11}& \dots & x_{1n}\\ & \dots & \\ x_{s1}& \dots & x_{sn}\end{array}\right]$$
$$W=\left[\begin{array}{ccc}{w}_{11}& \dots & w_{1o}\\ & \dots & \\ w_{n1}& \dots & w_{no}\end{array}\right]$$
$$b=\left[\begin{array}{ccc}{b}_1& \dots & b_o\end{array}\right]$$

损失函数L对W的梯度：

$$\frac{\partial L}{\partial W}=X^T\times \frac{\partial L}{\partial Z}$$

$\text{证明：}$

因为

$$z_{ij}=\sum _{t=1}^n{x}_{it}{w}_{tj}+b_j$$

所以$$\frac{\partial z_{ij}}{\partial w_{kl}}=\{\begin{array}{rlrl}& x_{ik},& & \text{如果 }j=l\\ & 0,& & \text{如果 }j\ne l\end{array}$$
所以
$$\frac{\partial L}{\partial w_{kl}}=\sum _{i=1}^s\sum _{j=1}^o\frac{\partial L}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial w_{kl}}=\sum _{i=1}^s\frac{\partial L}{\partial z_{il}}{x}_{ik}$$
又因为
$$X^T\times \frac{\partial L}{\partial Z}=\left[\begin{array}{ccc}{x}_{11}& \dots & x_{s1}\\ & \dots & \\ x_{1n}& \dots & x_{sn}\end{array}\right]\times \left[\begin{array}{ccc}\frac{\partial L}{\partial z_{11}}& \dots & \frac{\partial L}{\partial z_{1o}}\\ & \dots & \\ \frac{\partial L}{\partial z_{s1}}& \dots & \frac{\partial L}{\partial z_{so}}\end{array}\right]$$
所以
$$(X^T\times \frac{\partial L}{\partial Z}{)}_{kl}=\sum _{i=1}^s\frac{\partial L}{\partial z_{il}}{x}_{ik}=\frac{\partial L}{\partial w_{kl}}$$
所以
$$\frac{\partial L}{\partial W}=X^T\times \frac{\partial L}{\partial Z}$$

损失函数L对X的梯度：

$$\frac{\partial L}{\partial X}=\frac{\partial L}{\partial Z}\times W^T$$

$\text{证明：}$

$$\frac{\partial L}{\partial x_{ij}}=\sum _{l=1}^s\sum _{k=1}^o\frac{\partial L}{\partial z_{lk}}\frac{\partial z_{lk}}{\partial x_{ij}}$$
因为

$$z_{lk}=\sum _{l=1}^n{x}_{lt}{w}_{tk}+b_k$$
所以$$\frac{\partial z_{lk}}{\partial x_{ij}}=\{\begin{array}{rlrl}& w_{jk},& & \text{如果 }l=i\\ & 0,& & \text{如果 }l\ne i\end{array}$$

所以$$\frac{\partial L}{\partial x_{ij}}=\sum _{k=1}^o\frac{\partial L}{\partial z_{ik}}{w}_{jk}$$

而$$\frac{\partial L}{\partial Z}\times W^T=\left[\begin{array}{ccc}\frac{\partial L}{\partial z_{11}}& \dots & \frac{\partial L}{\partial z_{1o}}\\ & \dots & \\ \frac{\partial L}{\partial z_{s1}}& \dots & \frac{\partial L}{\partial z_{so}}\end{array}\right]\times \left[\begin{array}{ccc}{w}_{11}& \dots & w_{n1}\\ & \dots & \\ w_{1o}& \dots & w_{no}\end{array}\right]$$

所以$$(\frac{\partial L}{\partial Z}\times W^T{)}_{ij}=\sum _{k=1}^o\frac{\partial L}{\partial z_{ik}}{w}_{jk}=\frac{\partial L}{\partial x_{ij}}$$

所以$$\frac{\partial L}{\partial X}=\frac{\partial L}{\partial Z}\times W^T$$

损失函数L对b的梯度：

$$\frac{\partial L}{\partial b}=sum(\frac{\partial L}{\partial Z},axis=0)\#逐列求和$$

$\text{证明：}$

因为
$$\frac{\partial L}{\partial b_k}=\sum _{i=1}^s\sum _{j=1}^o\frac{\partial L}{\partial z_{ij}}\frac{\partial z_{ij}}{\partial b_k}=\sum _{i=1}^s\frac{\partial L}{\partial z_{ik}}$$

所以
$$\frac{\partial L}{\partial b}=1_s\frac{\partial L}{\partial Z}=sum(\frac{\partial L}{\partial Z},axis=0)$$
其中$1_s$为s列行向量