随机优化工具 Stochastic Optimization Techniques

@(Paper summaries)[Neural Networks|Optimization]

应用场景：训练集很大。
好处：避免取到局部最优解。

Neural networks are often trained stochastically, i.e. using a method where the objective function changes at each iteration. This stochastic variation is due to the model being trained on different data during each iteration. This is motivated by (at least) two factors: First, the dataset used as training data is often too large to fit in memory and/or be optimized over efficiently. Second, the objective function is typically nonconvex, so using different data at each iteration can help prevent the model from settling in a local minimum. Furthermore, training neural networks is usually done using only the first-order gradient of the parameters with respect to the loss function. This is due to the large number of parameters present in a neural network, which for practical purposes prevents the computation of the Hessian matrix. Because vanilla gradient descent can diverge or converge incredibly slowly if its learning rate hyperparameter is set inappropriately, many alternative methods have been proposed which are intended to produce desirable convergence with less dependence on hyperparameter settings. These methods often effectively compute and utilize a preconditioner on the gradient, adaptively change the learning rate over time or approximate the Hessian matrix.

In the following, we will use $\theta_t$ to denote some generic parameter of the model at iteration $t$, to be optimized according to some loss function $\mathcal{L}$ which is to be minimized.

Stochastic Gradient Descent

Stochastic gradient descent (SGD) simply updates each parameter by subtracting the gradient of the loss with respect to the parameter, scaled by the learning rate $\eta$, a hyperparameter. If $\eta$ is too large, SGD will diverge; if it’s too small, it will converge slowly. The update rule is simply

$$\theta_{t + 1} = \theta_t - \eta \nabla \mathcal{L}(\theta_t)$$

Momentum

In SGD, the gradient $\nabla \mathcal{L}(\theta_t)$ often changes rapidly at each iteration $t$ due to the fact that the loss is being computed over different data. This is often partially mitigated by re-using the gradient value from the previous iteration, scaled by a momentum hyperparameter $\mu$, as follows:

$$\begin{align*} v_{t + 1} &= \mu v_t - \eta \nabla \mathcal{L}(\theta_t) \\ \theta_{t + 1} &= \theta_t + v_{t+1} \end{align*}$$

It has been argued that including the previous gradient step has the effect of approximating some second-order information about the gradient.

Nesterov’s Accelerated Gradient

In Nesterov’s Accelerated Gradient (NAG), the gradient of the loss at each step is computed at $\theta_t + \mu v_t$ instead of $\theta_t$. In momentum, the parameter update could be written $\theta_{t + 1} = \theta_t + \mu v_t - \eta \nabla \mathcal{L}(\theta_t)$, so NAG effectively computes the gradient at the new parameter location but without considering the gradient term. In practice, this causes NAG to behave more stably than regular momentum in many situations. A more thorough analysis can be found in ((Sutskever, Martens, Dahl, and Hinton, “On the importance of initialization and momentum in deep learning” (ICML 2013) )). The update rules are then as follows:

vt+1θt+1=μvt−η∇(θt+μvt<