LSTM为什么用sigmoid，tanh不用relu？

最新推荐文章于 2025-02-17 15:06:04 发布

原创

最新推荐文章于 2025-02-17 15:06:04 发布 · 4.7k 阅读

17 ·

CC 4.0 BY-SA版权

https://cloud.tencent.com/developer/article/1498959

1. why is an activation function not being zero-centred an disadvantage?

Say there are two parameters w1 and w2. If the gradients of two dimensions are always of the same sign, it means we can only move roughly in the direction of northeast or southwest in the parameter space.
If our goal (the optimal solution in the weights hyperspace) happens to be in the northwest, we can only move in a zig-zagging fashion to get there; i.e. very inefficient.
However, it is not clear why the gradients of the parameters would be the same sign — this may just be an empirical result for non-zero centred activation functions.

2、神经网络中的激活函数：对比ReLU与Sigmoid、Tanh的优缺点？ReLU有哪些变种？

优点：

从计算的角度上，Sigmoid和Tanh激活函数均需要计算指数，复杂度高，而ReLU只需要一个阈值即可得到激活值；
ReLU的非饱和性可以有效地解决梯度消失的问题，提供相对宽的激活边界。
ReLU的单侧抑制提供了网络的稀疏表达能力。
修正线性单元（Rectified Linear Unit，ReLU）：ReLU函数被认为有生物上的解释性