Reinforcement Learning Exercise 7.1

Exercise 7.1 In Chapter 6 we noted that the Monte Carlo error can be written as the sum of TD errors (6.6) if the value estimates don’t change from step to step. Show that the n-step error used in (7.2) can also be written as a sum TD errors (again if the value estimates don’t change) generalizing the earlier result.

Here, according to equation (7.2), the TD error is:
$${\delta }_t=G_{t:t+n}-V_{t+n-1}(S_t)$$
For $G_{t:t+n}$ there is:
$$G_{t:t+n}=\{\begin{array}{ll}{R}_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{n-1}{R}_{t+n}+{\gamma }^n{V}_{t+n-1}(S_{t+n})& (n\ge 1\text{ and }0\le t<T-n)\\ R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{T-t-1}{R}_T& (t+n\ge T)\end{array}$$
Then, for $t+n\ge T$, the Monte Carlo error is:
$$\begin{array}{rl}{G}_t-V_{t+n}(S_t)& =R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{T-t-1}{R}_T-V_{t+n}(S_t)\\ & =G_{t:t+n}-V_{t+n}(S_t)\end{array}$$
Because all states are unchanged: $V_{t+n}(s)=V_{t+n-1}(s)$, so:
$$\begin{array}{rl}{G}_t-V_{t+n}(S_t)& =G_{t:t+n}-V_{t+n-1}(S_t)\\ & ={\delta }_t\end{array}$$and for state value in any time, there is $V_t(S_t)=V_{t+x}(S_t)$. Here, $x>0$.

Similarly, for $n\ge 1$ and $0\le t<T-kn$, (here $k\ge 1$) the Monte Carlo error should be:
$$\begin{array}{rl}{G}_t-V_{t+n}(S_t)& =R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{T-t-1}{R}_T-V_{t+n}(S_t)\\ & =R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{n-1}{R}_{t+n}+{\gamma }^n{V}_{t+n-1}(S_{t+n})-V_{t+n}(S_t)\\ & -{\gamma }^n{V}_{t+n-1}(S_{t+n})+{\gamma }^n{R}_{t+n+1}+\cdots +{\gamma }^{T-t-1}{R}_T\\ & =R_{t+1}+\gamma R_{t+2}+\cdots +{\gamma }^{n-1}{R}_{t+n}+{\gamma }^n{V}_{t+n-1}(S_{t+n})-V_{t+n-1}(S_t)\\ & -{\gamma }^n{V}_{t+n-1}(S_{t+n})+{\gamma }^n{R}_{t+n+1}+\cdots +{\gamma }^{T-t-1}{R}_T\\ & ={\delta }_t+{\gamma }^n[R_{t+n+1}+\cdots +{\gamma }^{T-(t+n)-1}{R}_T-V_{t+n-1}(S_{t+n})]\\ & ={\delta }_t+{\gamma }^n[G_{t+n}-V_{t+2n}(S_{t+n})]\\ & ={\delta }_t+{\gamma }^n{\delta }_{t+n}+{\gamma }^{2n}[G_{t+2n}-V_{t+3n}(S_{t+2n})]\\ & ={\delta }_t+{\gamma }^n{\delta }_{t+n}+{\gamma }^{2n}{\delta }_{t+2n}+\cdots +{\gamma }^{kn}[G_{t+kn}-V_{t+(k+1)n}(S_{t+kn})]\\ & ={\delta }_t+{\gamma }^n{\delta }_{t+n}+{\gamma }^{2n}{\delta }_{t+2n}+\cdots +{\gamma }^{kn}{\delta }_{t+kn}\\ & +{\gamma }^{kn}[R_{t+kn+1}+\gamma R_{t+kn+2}+\cdots +{\gamma }^{T-(t+kn)-1}{R}_T-V_{t+(k+1)n}(S_{t+(k+1)n}\left)\right]\\ & =\sum _{p=0}^{p=k}{\gamma }^{pn}{\delta }_{t+pn}+{\gamma }^{kn}[G_{t+kn}-V(S_T)]\\ & =\sum _{p=0}^{p=k}{\gamma }^{pn}{\delta }_{t+pn}+{\gamma }^{kn}[G_{t+kn}-0]\\ & =\sum _{p=0}^{p=k}{\gamma }^{pn}{\delta }_{t+pn}+{\gamma }^{kn}{G}_{t+kn}\end{array}$$
Specially, if $t+kn+1=T$, then $G_{t+kn}=0$, the Monte Carlo error is:
$$G_t-V_{t+n}(S_t)=\sum _{p=0}^{p=k}{\gamma }^{pn}{\delta }_{t+pn}$$