Activation Function 激活函数

最新推荐文章于 2025-02-14 15:19:59 发布

HaronW

最新推荐文章于 2025-02-14 15:19:59 发布

阅读量485

点赞数

分类专栏： Machine Learning

本文链接：https://blog.youkuaiyun.com/weixin_45286813/article/details/105647847

版权

Machine Learning 专栏收录该内容

15 篇文章

订阅专栏

Activation Function 激活函数

Intro

In parctice, we hardly ever use step function as activation function $\phi(z)$
Activation function can be divided into two types: saturated activation function and non-saturated activation function.
Non-saturated activation functions can avoid gradient disappearance and they converge faster.
Gradient disappearance means the value of function changes slightly if $∣ x ∣$ is large.

Step function

$\phi'(x) = 0$ , so we never use it.

Sigmoid Function

在这里插入图片描述
$\phi(x) = sigmoid(x) = \frac{1}{1+e^{-x}}$
$\phi'(x) = \phi(x)[1-\phi(x)]$
Sigmoid function is a saturated activation function.

tanh 双曲正切函数 (hyperbolic tangent funciton)

在这里插入图片描述
$\phi(x) = tanh(x) = \frac{e^x-e^{-x}}{e^x+e^{-x}}$
$\phi'(x) = 1 - \phi^2(x)$
The tanh function is a saturated activation function.

ReLU 线性整流函数(Rectified Linear Unit)

在这里插入图片描述
$\phi(x) = \left\{\begin{array}{l} x \ x>0 \\ 0 \ x\leq0 \end{array}\right.$
$\phi(x) = \max(0, x)$
$\phi'(x) = \left\{\begin{array}{l} 1 \ x>0 \\ 0 \ x\leq0 \end{array}\right.$
ReLU function is a non-saturated activation function.

ReLU is widely used in deep learning.

Leaky ReLU, PReLU, RReLU

The Leaky ReLU, PReLU and RReLU functions all have a tiny angle when $x < 0$ . They are non-saturated activation function.
$\phi(x) = \left\{\begin{array}{l} x \ x>0 \\ \frac xa \ x\leq0 \end{array}\right.$
$\phi(x) = \max(leak*x, x)$
$\phi'(x) = \left\{\begin{array}{l} 1 \ x>0 \\ \frac1a \ x\leq0 \end{array}\right.$
在这里插入图片描述