falianghuang的博客

Selective Dropout文献：Barrow E, Eastwood M, Jayne C. Selective Dropout for Deep Neural Networks[M]// Neural Information Processing. Springer International Publishing, 2016. 方法：根据dropout率来决定每层需要dropout的单

问题RNN在迭代运用状态转换操作“输入到隐状态”实现任意长序列的定长表示时，会遭遇到“对隐状态扰动过于敏感”的困境。dropoutdropout的数学形式化:y=f(W⋅d(x))y=f(W \cdot d(x)), 其中d(x)={mask∗x, if train phaseing(1−p)x,otherwised(x) = \left\{ {\begin{array}{*{20}{c}} {m

经典LSTM(1997) 1. 三个门的信息流入ft=σ(Wf⋅[ht−1,xt]+bf){f_t} = \sigma \left( {{W_f} \cdot [{h_{t - 1}},{x_t}] + {b_f}} \right) (1)it=σ(Wi⋅[ht−1,xt]+bi){i_t} = \sigma \left( {{W_i} \cdot [{h_{t - 1}},{x