A few weeks ago I released some code on Github to help people understand how LSTM’s work at the implementation level. The forward pass is well explained elsewhere and is straightforward to understand, but I derived the backprop equations myself and the backprop code came without any explanation whatsoever. The goal of this post is to explain the so called backpropagation through time in the context of LSTM’s.
If you feel like anything is confusing, please post a comment below or submit an issue on Github.
Note: this post assumes you understand the forward pass of an LSTM network, as this part is relatively simple. Please read this great intro paper if you are not familiar with this, as it contains a very nice intro to LSTM’s. I follow the same notation as this paper so I recommend reading having the tutorial open in a separate browser tab for easy reference while reading this post.
Introduction
The forward pass of an LSTM node is defined as follows:
By concatenating the x(t) and h(t−1) vectors as follows:
we can re-write parts of the above as follows:
Suppose we have a loss l(t) that we wish to minimize at every time step t that depends on the hidden layer h and the label y at the current time via a loss function f :
where f can be any differentiable loss function, such as the Euclidean loss:
Our ultimate goal in this case is to use gradient descent to minimize the loss L over an entire sequence of length T :
Let’s work through the algebra of computing the loss gradient:
where w is a scalar parameter of the model (for example it may be an entry in the matrix Wgx ). Since the loss l(t)=f(h(t),y(t)) only depends on the values of the hidden layer h(t) and the label y(t) , we have by the chain rule:
where hi(t) is the scalar corresponding to the i ’th memory cell’s hidden output and M is the total number of memory cells. Since the network propagates information forwards in time, changing hi(t) will have no effect on the loss prior to time t , which allows us to write:
For notational convenience we introduce the variable L(t) that represents the cumulative loss from step t onwards:
such that L(1) is the loss for the entire sequence. This allows us to re-write the above equation as:
With this in mind, we can re-write our gradient calculation as:
Make sure you understand this last equation. The computation of dhi(t)dw follows directly follows from the forward propagation equations presented earlier. We now show how to compute dL(t)dhi(t) which is where the so called backpropagation through time comes into play.
Backpropagation through time
This variable L(t) allows us to express the following recursion:
Hence, given activation h(t) of an LSTM node at time t , we have that:
Now, we know where the first term on the right hand side dl(t)dh(t) comes from: it’s simply the elementwise derivative of the loss l(t) with respect to the activations h(t) at time t . The second term dL(t+1)dh(t) is where the recurrent nature of LSTM’s shows up. It shows that the we need the next node’s derivative information in order to compute the current current node’s derivative information. Since we will ultimately need to compute dL(t)dh(t) for all t=1,…,T , we start by computing
and work our way backwards through the network. Hence the term backpropagation through time. With these intuitions in place, we jump into the code.
Code
We now present the code that performs the backprop pass through a single node at time 1≤t≤T . The code takes as input:
top_diff_h=dL(t)dh(t)=dl(t)dh(t)+dL(t+1)dh(t)top_diff_s=dL(t+1)ds(t) .
And computes:
self.state.bottom_diff_s=dL(t)ds(t)self.state.bottom_diff_h=dL(t)dh(t−1)
whose values will need to be propagated backwards in time. The code also adds derivatives to:
self.param.wi_diff=dLdWi- …
self.param.bi_diff=dLdbi- …
since recall that we must sum the derivatives from each time step:
Also, note that we use:
dxc=dLdxc(t)
where we recall that xc(t)=[x(t),h(t−1)] . Without any further due, the code:
def top_diff_is(self, top_diff_h, top_diff_s):
# notice that top_diff_s is carried along the constant error carousel
ds = self.state.o * top_diff_h + top_diff_s
do = self.state.s * top_diff_h
di = self.state.g * ds
dg = self.state.i * ds
df = self.s_prev * ds
# diffs w.r.t. vector inside sigma / tanh function
di_input = (1. - self.state.i) * self.state.i * di
df_input = (1. - self.state.f) * self.state.f * df
do_input = (1. - self.state.o) * self.state.o * do
dg_input = (1. - self.state.g ** 2) * dg
# diffs w.r.t. inputs
self.param.wi_diff += np.outer(di_input, self.xc)
self.param.wf_diff += np.outer(df_input, self.xc)
self.param.wo_diff += np.outer(do_input, self.xc)
self.param.wg_diff += np.outer(dg_input, self.xc)
self.param.bi_diff += di_input
self.param.bf_diff += df_input
self.param.bo_diff += do_input
self.param.bg_diff += dg_input
# compute bottom diff
dxc = np.zeros_like(self.xc)
dxc += np.dot(self.param.wi.T, di_input)
dxc += np.dot(self.param.wf.T, df_input)
dxc += np.dot(self.param.wo.T, do_input)
dxc += np.dot(self.param.wg.T, dg_input)
# save bottom diffs
self.state.bottom_diff_s = ds * self.state.f
self.state.bottom_diff_x = dxc[:self.param.x_dim]
self.state.bottom_diff_h = dxc[self.param.x_dim:]Details
The forward propagation equations show that modifying s(t) affects the loss L(t) by directly changing the values of h(t) as well as h(t+1) . However, modifying s(t) affects L(t+1) only by modifying h(t+1) . Therefore, by the chain rule:
扫一扫
490
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
