[学习笔记]神经网络

原创已于 2024-08-19 00:16:23 修改 · 178 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#学习 #笔记 #神经网络

于 2024-08-08 13:38:47 首次发布

为什么要有激活函数?

因为假设没有激活函数,神经网络会退化成单层的矩阵运算,激活函数确保它有复杂的表达能力.

为什么要增加层数?

中间层神经网络会自主学习到重要的信息,表达能力增强.

最小化误差:

$min_\Theta J(\Theta) = min_\Theta \frac{\sum^N_{i=1} l(y_i - F_\Theta(x_i))}{N}$

梯度下降(a 学习率):

$\Theta^{new} = \Theta^{old} - a\nabla_\Theta J(\Theta)$

梯度下降法会陷入局部最优解,但由于神经网络参数量较大,故不容易出现这种情况

学习率较小会走得慢,学习率太大会震荡不达到最优解.

反向传播

$\frac{\partial L }{\partial W^{(2)}} = prod(\frac{\partial L}{\partial O},\frac{\partial O}{\partial W^{(2)}}) = \frac{\partial L}{\partial O}*h^T$

$\frac{\partial L }{\partial h} = prod(\frac{\partial L}{\partial O},\frac{\partial O}{\partial h}) = W^{(2)T}*\frac{\partial L}{\partial O}$

$\frac{\partial L }{\partial Z} = prod(\frac{\partial L}{\partial O},\frac{\partial O}{\partial h},\frac{\partial h}{\partial Z}) = \frac{\partial L}{\partial h}\bigodot \phi '(Z)$

$\frac{\partial L }{\partial W^{(1)}} = prod(\frac{\partial L}{\partial O},\frac{\partial O}{\partial h},\frac{\partial h}{\partial Z},\frac{\partial Z}{\partial W^{(1)}}) = \frac{\partial L}{\partial Z}*X^T$