Reinforcement Learning Exercise 3.22

Exercise 3.22 Consider the continuing MDP shown on to the right. The only decision to be made is that in the top state, where two actions are available, left and right. The numbers show the rewards that are received deterministically after each action. There are exactly two deterministic policies, ${\pi }_{left}$ and ${\pi }_{right}$. What policy is optimal if $\gamma =0$? If $\gamma =0.9$? If $\gamma =0.5$?

Before to solve this problem, we have to deduce the expression of $q_{\ast }(s,a)$ in terms of $R_{s,s^{\prime }}^a$ and $P_{s,s^{\prime }}^a$.
First,
$$\begin{array}{rl}{q}_{\ast }(s,a)& =\mathbb{E}[R_{t+1}+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(S_{t+1},a^{\prime })|S_t=s,A_t=a]\\ & =\sum _{s^{\prime },r}\{p(s^{\prime },r|s,a)[r+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a)]\}\\ & =\sum _{s^{\prime },r}[rp(s^{\prime },r|s,a)]+\sum _{s^{\prime },r}[p(s^{\prime },r|s,a)\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]\\ & =\sum _r[rp(r|s,a)]+\sum _{s^{\prime }}[p(s^{\prime }|s,a)\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]\\ & =\mathbb{E}(r|s,a)+\sum _{s^{\prime }}[p(s^{\prime }|s,a)\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]\\ & =\sum _{s^{\prime }}[\mathbb{E}(r|s^{\prime },s,a)p(s^{\prime }|s,a)]+\sum _{s^{\prime }}[p(s^{\prime }|s,a)\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]\\ & =\sum _{s^{\prime }}\left\{\right[\mathbb{E}(r|s^{\prime },s,a)+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })]p(s^{\prime }|s,a)\}\end{array}$$
denote $\mathbb{E}(r|s^{\prime },s,a)=R_{s,s^{\prime }}^a$ and $p(s^{\prime }|s,a)=P_{s,s^{\prime }}^a$, we get the expression we wanted
$$q_{\ast }(s,a)=\sum _{s^{\prime }}\left\{\right[R_{s,s^{\prime }}^a+\gamma \underset{a^{\prime }}{\max }{q}_{\ast }(s^{\prime },a^{\prime })\left]P_{s,s^{\prime }}^a\right\}\text{\hspace{1em}(1)}$$
Next, we name the three status in circles as $s_A$, $s_B$, $s_C$, and denote the action to left as $a_l$, the action to right as $a_r$.

According to equation (1) we can get Bellman optimality equation for $q_{\ast }$ of the three status.
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{left}}(s_A,a_l)& =\{R_{s_A,s_B}^{a_l}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_B,a)\left]\right\}{P}_{s_A,s_B}^{a_l}+\{R_{s_A,s_C}^{a_l}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_C,a)\left]\right\}{P}_{s_A,s_C}^{a_l}\\ & =[R_{s_A,s_B}^{a_l}+\gamma q_{\ast }(s_B,a)]P_{s_A,s_B}^{a_l}+[R_{s_A,s_C}^{a_r}+\gamma q_{\ast }(s_C,a)]P_{s_A,s_C}^{a_l}\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =\{R_{s_A,s_B}^{a_r}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_B,a)\left]\right\}{P}_{s_A,s_B}^{a_r}+\{R_{s_A,s_C}^{a_r}+\gamma \underset{a^{\prime }}{\max }[q_{\ast }(s_C,a)\left]\right\}{P}_{s_A,s_C}^{a_r}\\ & =[R_{s_A,s_B}^{a_r}+\gamma q_{\ast }(s_B,a)]P_{s_A,s_B}^{a_r}+[R_{s_A,s_C}^{a_r}+\gamma q_{\ast }(s_C,a)]P_{s_A,s_C}^{a_r}\\ q_{\ast }(s_B,a)& =\{R_{s_B,s_A}^a+\gamma \underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}{P}_{s_B,s_A}^a\\ q_{\ast }(s_C,a)& =\{R_{s_C,s_A}^a+\gamma \underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}{P}_{s_C,s_A}^a\end{array}$$$$\begin{gathered} \because P_{s_A,s_B}^{a_r}=0,P_{s_A,s_C}^{a_l}=0 \\ \begin{array}{rl}\therefore q_{\ast ,{\pi }_{left}}(s_A,a_l)& =[R_{s_A,s_B}^{a_l}+\gamma q_{\ast }(s_B,a)]P_{s_A,s_B}^{a_l}\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =[R_{s_A,s_C}^{a_r}+\gamma q_{\ast }(s_C,a)]P_{s_A,s_C}^{a_r}\end{array} \end{gathered}$$
Now, let’s discuss the cases in different $\gamma$.
For $\gamma =0$:
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{left}}(s_A,a_l)& =[1+0\cdot q_{\ast }(s_B,a)]\cdot 1=1\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =[0+0\cdot q_{\ast }(s_C,a)]\cdot 1=0\end{array}$$
So, ${\pi }_{left}$ is the optimal policy when $\gamma =0$.

For $\gamma =0.5$:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =\{0+0.5\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}\cdot 1\\ & =0.5\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast }(s_C,a)& =\{2+0.5\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}\cdot 1\\ & =2+0.5\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =[1+0.5\cdot q_{\ast }(s_B,a)]\cdot 1\\ & =1+0.5\cdot q_{\ast }(s_B,a)\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =[0+0.5\cdot q_{\ast }(s_C,a)]\cdot 1\\ & =0.5\cdot q_{\ast }(s_C,a)\end{array}$$
Assume $q_{\ast ,{\pi }_{left}}(s_A,a_l)\ge q_{\ast ,\pi -left}(s_C,a_l)$ then we have:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =0.5\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\\ q_{\ast }(s_C,a)& =2+0.5\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\end{array}$$
therefore,
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{left}}(s_A,a_l)& =1+0.5\cdot 0.5\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =\frac{4}{3}\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =0.5\cdot [2+0.5\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)]\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =\frac{5}{3}\end{array}$$
Here, $q_{\ast ,{\pi }_{left}}(s_A,a_l)<q_{\ast ,{\pi }_{right}}(s_C,a_l)$, conflict with the assumption, so the assumption fails.
Assume $q_{\ast ,{\pi }_{left}}(s_A,a_l)\le q_{\ast ,{\pi }_{right}}(s_C,a_l)$ then we have:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =0.5\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\\ q_{\ast }(s_C,a)& =2+0.5\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\end{array}$$
therefore,
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{right}}(s_A,a_r)& =0.5\cdot [2+0.5\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =\frac{4}{3}\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =1+0.5\cdot 0.5\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =\frac{4}{3}\end{array}$$
Here $q_{\ast ,{\pi }_{left}}(s_A,a_l)=q_{\ast ,{\pi }_{right}}(s_A,a_r)$, assumption is correct. So, both $q_{\ast ,{\pi }_{left}}(s_A,a_l)$ and $q_{\ast ,{\pi }_{right}}(s_A,a_r)$ are optimal policies for $\gamma =0.5$.

For $\gamma =0.9$:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =\{0+0.9\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}\cdot 1\\ & =0.9\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast }(s_C,a)& =\{2+0.9\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)\left]\right\}\cdot 1\\ & =2+0.9\underset{a^{\prime }}{\max }[q_{\ast ,{\pi }_{left}}(s_A,a_l),q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =[1+0.9\cdot q_{\ast }(s_B,a)]\cdot 1\\ & =1+0.9\cdot q_{\ast }(s_B,a)\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =[0+0.9\cdot q_{\ast }(s_C,a)]\cdot 1\\ & =0.9\cdot q_{\ast }(s_C,a)\end{array}$$
Assume $q_{\ast ,{\pi }_{left}}(s_A,a_l)\ge q_{\ast ,\pi -left}(s_C,a_l)$ then we have:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =0.9\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\\ q_{\ast }(s_C,a)& =2+0.9\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\end{array}$$
therefore,
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{left}}(s_A,a_l)& =1+0.9\cdot 0.9\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =\frac{100}{19}=\frac{500}{95}\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =0.9\cdot [2+0.9\cdot q_{\ast ,{\pi }_{left}}(s_A,a_l)]\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =\frac{729}{95}\end{array}$$
Here, $q_{\ast ,{\pi }_{left}}(s_A,a_l)<q_{\ast ,{\pi }_{right}}(s_C,a_l)$, conflict with the assumption, so the assumption fails.
Assume $q_{\ast ,{\pi }_{left}}(s_A,a_l)\le q_{\ast ,{\pi }_{right}}(s_C,a_l)$ then we have:
$$\begin{array}{rl}{q}_{\ast }(s_B,a)& =0.9\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\\ q_{\ast }(s_C,a)& =2+0.9\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\end{array}$$
therefore,
$$\begin{array}{rl}{q}_{\ast ,{\pi }_{right}}(s_A,a_r)& =0.9\cdot [2+0.9\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)]\\ q_{\ast ,{\pi }_{right}}(s_A,a_r)& =\frac{180}{19}\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =1+0.9\cdot 0.9\cdot q_{\ast ,{\pi }_{right}}(s_A,a_r)\\ q_{\ast ,{\pi }_{left}}(s_A,a_l)& =\frac{1648}{190}\end{array}$$
Here, $q_{\ast ,{\pi }_{left}}(s_A,a_l)<q_{\ast ,{\pi }_{right}}(s_C,a_l)$, assumption is correct. So, ${\pi }_{right}$ is the optimal policy for $\gamma =0.9$