Reinforcement Learning Exercise 3.19

Exercise 3.19 The value of an action, $q_{\pi }(s,a)$, depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state–action pair) and branching to the possible next states:

Give the equation corresponding to this intuition and diagram for the action value, $q_{\pi }(s,a)$, in terms of the expected next reward, $R_{t+1}$, and the expected next state value, $v_{\pi }(S_t+1)$, given that $S_t=s$ and $A_t=a$. This equation should include an expectation but not one conditioned on following the policy. Then give a second equation, writing out the expected value explicitly in terms of $p(s^{\prime },r|s,a)$ defined by (3.2), such that no expected value notation appears in the equation.

qπ(s,a)=Eπ(Gt∣St=s,At=a)=Eπ(Rt+1+γGt+1∣St=s,At=a)=Eπ(Rt+1∣St=s,At=a)+γEπ(Gt+1∣St=s,At=a)=Eπ(Rt+1∣St=s,At=a)+γEπ(∑k=0∞γkRt+k+2∣St=s,At=a)=Rt+1(s,a)+γ∑s′[Eπ(∑k=0∞γkRt+k+2∣St=s,At=a,St+1=s′)Pr(St+1=s′∣St=s,At=a)] \begin{aligned} q_\pi(s,a) &= \mathbb E_\pi(G_t | S_t = s, A_t = a) \\ &= \mathbb E_\pi ( R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a) \\ &= \mathbb E_\pi ( R_{t+1} | S_t= s, A_t = a) + \gamma \mathbb E_\pi ( G_{t+1} | S_t = s, A_t = a) \\ &= \mathbb E_\pi ( R_{t+1} | S_t= s, A_t = a) + \gamma \mathbb E_\pi ( \sum_{k=0}^\infty \gamma^k R_{t+k+2} | S_t = s, A_t = a) \\ &= R_{t+1}(s,a) + \gamma \sum_{s'} \bigl[ \mathbb E_\pi ( \sum_{k=0}^\infty \gamma^k R_{t+k+2} | S_t = s, A_t = a, S_{t+1} = s') Pr(S_{t+1} = s'| S_t = s, A_t = a) \bigr] \\ \end{aligned} qπ​(s,a)​=Eπ​(Gt​∣St​=s,At​=a)=Eπ​(Rt+1​+γGt+1​∣St​=s,At​=a)=Eπ​(Rt+1​∣St​=s,At​=a)+γEπ​(Gt+1​∣St​=s,At​=a)=Eπ​(Rt+1​∣St​=s,At​=a)+γEπ​(k=0∑∞​γkRt+k+2​∣St​=s,At​=a)=Rt+1​(s,a)+γs′∑​[Eπ​(k=0∑∞​γkRt+k+2​∣St​=s,At​=a,St+1​=s′)Pr(St+1​=s′∣St​=s,At​=a)]​
denote $$Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)=P_{s,s^{\prime }}^a$$
and $\because S_t$ and $A_t$ give no information to $R_{t+2+k}$
$$\begin{gathered} \therefore {\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+2}|S_t=s,A_t=a,S_{t+1}=s^{\prime })Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a) \\ \begin{array}{rl}& ={\mathbb{E}}_{\pi }(\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+2}|S_{t+1}=s^{\prime })P_{s,s^{\prime }}^a\\ & ={\upsilon }_{\pi }(S_{t+1})P_{s,s^{\prime }}^a\end{array} \end{gathered}$$
$$\therefore \begin{array}{r}{q}_{\pi }(s,a)=R_{t+1}(s,a)+\gamma \sum _{s^{\prime }}{\upsilon }_{\pi }(S_{t+1})P_{s,s^{\prime }}^a\end{array}\text{\hspace{1em}(1)}$$
Above is the first equantion.
$$\begin{array}{rl}\because R_{t+1}(s,a)& ={\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a)\\ & =\sum _{s^{\prime }}[{\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime })Pr(S_{t+1}=s^{\prime }|S_t=s,A_t=a)]\\ & =\sum _r\sum _{s^{\prime }}[{\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime })Pr(S_{t+1}=s^{\prime },R_{t+1}=r|S_t=s,A_t=a)]\end{array}$$
denote $$\begin{gathered} {\mathbb{E}}_{\pi }(R_{t+1}|S_t=s,A_t=a,S_{t+1}=s^{\prime })=R_{s,s^{\prime }}^a \\ Pr(S_{t+1}=s^{\prime },R_{t+1}=r|S_t=s,A_t=a)=p(s^{\prime },r|s,a) \end{gathered}$$
$$\therefore R_{t+1}(s,a)=\sum _r\sum _{s^{\prime }}{R}_{s,s^{\prime }}^{a}p(s^{\prime },r|s,a)$$
$$\begin{array}{rl}\because \gamma \sum _{s^{\prime }}{\upsilon }_{\pi }(S_{t+1})P_{s,s^{\prime }}^a& =\gamma \sum _r\sum _{s^{\prime }}{\upsilon }_{\pi }(S_{t+1})Pr(S_{t+1}=s^{\prime },R_{t+1}=r|S_t=s,A_t=a)\\ & =\gamma \sum _r\sum _{s^{\prime }}{\upsilon }_{\pi }(S_{t+1})p(s^{\prime },r|s,a)\end{array}$$
$$\begin{array}{rl}\therefore q_{\pi }(s,a)& =\sum _r\sum _{s^{\prime }}{R}_{s,s^{\prime }}^{a}p(s^{\prime },r|s,a)+\gamma \sum _r\sum _{s^{\prime }}{\upsilon }_{\pi }(S_{t+1})p(s^{\prime },r|s,a)\\ & =\sum _r\sum _{s^{\prime }}[R_{s,s^{\prime }}^a+\gamma {\upsilon }_{\pi }(S_{t+1})]p(s^{\prime },r|s,a)\end{array}\text{\hspace{1em}(2)}$$
This is the second equation.