Reinforcement Learning Exercise 5.4

Exercise 5.4 The pseudocode for Monte Carlo ES is inefficient because, for each state–action pair, it maintains a list of all returns and repeatedly calculates their mean. It would be more efficient to use techniques similar to those explained in Section 2.4 to maintain just the mean and a count (for each state–action pair) and update them incrementally. Describe how the pseudocode would be altered to achieve this.

The altered pseudocode is shown as below:
$$\begin{array}{rl}& \text{Initialize:}\\ & \pi (s)\in \mathcal{A}(s)\text{(arbitrarily), for all }s\in S\\ & Q(s,a)\in \mathbb{R}\text{(arbitrarily), for all }s\in S,a\in \mathcal{A}(s)\\ & counts(s,a)\leftarrow 0\text{, for all s }\in S,a\in \mathcal{A}(s)\\ & \text{Loop forever (for each episode):}\\ & \text{Choose }{S}_0\in \mathcal{S},A_0\in \mathcal{A}(S_0)\text{ randomly such that all pairs have probability}>0\\ & \text{Generate an episode from }{S}_0,A_0,\text{following }\pi :S_0,A_0,R_1,...,S_{T-1},A_{T-1},R_T\\ & G\leftarrow 0\\ & \text{Loop for each step of episode, }t=T-1,T-2,...,0:\\ & G\leftarrow \gamma G+R_{t+1}\\ & \text{Unless the pair }{S}_t,A_t\text{ appears in }{S}_0,A_0,S_1,A_1...,S_{t-1},A_{t-1}:\\ & counts(S_t,A_t)\leftarrow counts(S_t,A_t)+1\\ & Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\frac{(G-Q(S_t,A_t))}{count(S_t,A_t)}\\ & \pi (S_t)\leftarrow {\text{argmax}}_{a}Q(S_t,a)\end{array}$$