DRL — Policy Based Methods — Chapter 3-4 Proximal Policy Optimization

DRL — Policy Based Methods — Chapter 3-4 Proximal Policy Optimization

3.4.3 Beyond REINFORCE

1. REINFORCE works as follows: First, we initialize a random policy ${\pi }_{\theta }(a;s)$, and using the policy we collect a trajectory – or a list of (state, actions, rewards) at each time step:

$$s_1,a_1,r_1,s_2,a_2,r_2,...$$

2. Second, we compute the total reward of the trajectory $R=r_1+r_2+r_3+...$, and compute an estimate the gradient of the expected reward, $g$:

$$g=R\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)$$

3. Third, we update our policy using gradient ascent with learning rate $\alpha$:

$$\theta \leftarrow \theta +\alpha$$

4. The process then repeats.

What are the main problems of REINFORCE? There are three issues:

• The update process is very inefficient! We run the policy once, update once, and then throw away the trajectory.

• The gradient estimate gg is very noisy. By chance the collected trajectory may not be representative of the policy.

• There is no clear credit assignment. A trajectory may contain many good/bad actions and whether these actions are reinforced depends only on the final total output.

In the following concepts, we will go over ways to improve the REINFORCE algorithm and resolve all 3 issues. All of the improvements will be utilized and implemented in the PPO algorithm.

3.4.4 Noise Reduction

3.4.6 Policy Gradient Quiz

Make sure you are using the fact that ${\pi }_{\theta }(1|s_t)=\theta$, and ${\pi }_{\theta }(0|s_t)=1-\theta$. Also remember that
$$\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R_t^{\mathrm{f}\mathrm{u}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}=\sum _t\frac{{\nabla }_{\theta }{\pi }_{\theta }(a_t|s_t)}{{\pi }_{\theta }(a_t|s_t)}{R}_t^{\mathrm{f}\mathrm{u}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}}$$