Momentum, RMSprob and Adam

Gradient Descent with Momentum

Compute exponentially weighed average of gradient, and use the gradient to update weights.

Algorithm

On iteration t:

1. Compute $\operatorname {d} W$ and $\operatorname {d} b$ on current minibatch
2. $V _{\operatorname {d} W} = \beta _{1} V _{\operatorname {d} W} + \left (1 - \beta _{1} \right )\operatorname {d} W$
   $V _{\operatorname {d} b} = \beta _{1} V _{\operatorname {d} b} + \left (1 - \beta _{1} \right )\operatorname {d} b$
3. $W = W - \alpha V _{\operatorname {d} W}$
   $b = b - \alpha V _{\operatorname {d} b}$

RMSprop

On iteration t:

1. Compute $\operatorname {d} W$ and $\operatorname {d} b$ on current minibatch
2. $S _{\operatorname {d} W} = \beta _{2} S _{\operatorname {d} W} + \left (1 - \beta _{2} \right )\operatorname {d} W ^2$ (Element-wise square)
   $S _{\operatorname {d} b} = \beta _{2} S _{\operatorname {d} b} + \left (1 - \beta _{2} \right )\operatorname {d} b ^2$ (Element-wise square)
3. $W = W - \alpha \dfrac {\operatorname {d} W} {\sqrt {S _{\operatorname {d} W} + \varepsilon}}$
   $b = b - \alpha \dfrac {\operatorname {d} b} {\sqrt {S _{\operatorname {d} b} + \varepsilon}}$

Adam

Adaptive Momentum

Algorithm

On iteration t:

1. Compute $\operatorname {d} W$ and $\operatorname {d} b$ on current minibatch
2. $V _{\operatorname {d} W} = \beta _{1} V _{\operatorname {d} W} + \left (1 - \beta _{1} \right )\operatorname {d} W$
   $V _{\operatorname {d} b} = \beta _{1} V _{\operatorname {d} b} + \left (1 - \beta _{1} \right )\operatorname {d} b$
   $S _{\operatorname {d} W} = \beta _{2} S _{\operatorname {d} W} + \left (1 - \beta _{2} \right )\operatorname {d} W ^2$ (Element-wise square)
   $S _{\operatorname {d} b} = \beta _{2} S _{\operatorname {d} b} + \left (1 - \beta _{2} \right )\operatorname {d} b ^2$ (Element-wise square)
3. $V _{\operatorname {d} W} ^{\text {corrected}} = \dfrac {V _{\operatorname {d} W}} {1 - \beta _{1} ^t}$
   $V _{\operatorname {d} b} ^{\text {corrected}} = \dfrac {V _{\operatorname {d} b}} {1 - \beta _{1} ^t}$
   $S _{\operatorname {d} W} ^{\text {corrected}} = \dfrac {S _{\operatorname {d} W}} {1 - \beta _{2} ^t}$
   $S _{\operatorname {d} b} ^{\text {corrected}} = \dfrac {S _{\operatorname {d} b}} {1 - \beta _{2} ^t}$
4. $W = W - \alpha \dfrac {V _{\operatorname {d} W}} {\sqrt {S _{\operatorname {d} W} }+ \varepsilon}$
   $b = b - \alpha \dfrac {V _{\operatorname {d} b}} {\sqrt {S _{\operatorname {d} b} }+ \varepsilon}$