Brief overview of backward and forward

Let’s say we only feed in one data point.

• out = model:forward($x_i$) computes $f_w(x_i)$ where $f_w$ is our model with its current parameters $w$, and stores the result in out.
• loss = criterion:forward(out, $y_i$) computes the loss $\ell (f_w(x_i),y_i)$ with respect to the true value $y_i$.
• dl_dout = criterion:backward(out, $y_i$) computes $\frac{\partial \ell(...)}{\partial f_w(x_i)}$.
• model:backward($x_i$, dl_dout) computes $\frac{\partial \ell(...)}{\partial w}$ and stores this gradient in a place we have a reference to, usually called gradParameters in our code.