Reinforcement Learning Exercise 5.10

原创于 2019-08-10 16:07:23 发布 · 274 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#reinforcement learning

reinforcement learning 专栏收录该内容

37 篇文章

订阅专栏

Exercise 5.10 Derive the weighted-average update rule (5.8) from (5.7). Follow the pattern of the derivation of the unweighted rule (2.3)

According to:
$V_{n} \doteq \frac{\sum_{k=1}^{n - 1}W_k G_k}{\sum_{k=1}^{n - 1}W_k} \text{,} \qquad n \geq 2 \qquad \text{(5.7)} \\$
and denote $C_n$ as the weights given to the first n returns. So formula (5.7) is transferred to:
$V_{n} \doteq \frac{\sum_{k=1}^{n - 1}W_k G_k}{C_{n-1}} \text{,} \qquad n \geq 2$
then we have:
$V_{n+1} \doteq \frac{\sum_{k=1}^{n}W_k G_k}{C_n} \text{,} \qquad n \geq 1$
$\begin{aligned} \therefore V_{n+1} &= \frac{\sum_{k=1}^{n - 1}W_k G_k}{C_n}+\frac{W_nG_n}{C_n}\\ &=\frac{C_{n-1}}{C_{n}} V_n +\frac{W_nG_n}{C_n} \\ &= (1 - \frac{W_n}{C_n})V_n + \frac{W_nG_n}{C_n} \\ &=V_n + \frac{W_n}{C_n}(G_n - V_n), \qquad n \geq 1, \qquad \text{(5.8)} \end{aligned}$
This derivation is very easy.