tf.random_normal与tf.truncated_normal的区别

tf.random_normal与tf.truncated_normal的区别

1. 定义
   The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.

2. 与tf.random_normal的不同之处

tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)outputs random values from a normal distribution.

tf.truncated_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)outputs random values from a truncated normal distribution.

3. 价值在于：
   The point for using truncated normal is to overcome saturation of tome functions like sigmoid (where if the value is too big/small, the neuron stops learning).
   即为了克服神经网络在利用某些激活函数学习时，例如sidmoid，会在值过大活过小时停止学习的缺点。

• 参考文献：
• 聊一聊深度学习的weight initialization
  夏飞：https://zhuanlan.zhihu.com/p/25110150
• truncated_normal_initializer：https://zhuanlan.zhihu.com/p/42740144