强化学习（三）—— 策略学习（Policy-Based）及策略梯度（Policy Gradient）

1. 策略学习

Policy Network

• 通过策略网络近似策略函数
  $$\pi (a|s_t)\approx \pi (a|s_t;\theta )$$
• 状态价值函数及其近似
  $$V_{\pi }(s_t)=\sum _a\pi (a|s_t)Q_{\pi }(s_t,a)$$
  $$V(s_t;\theta )=\sum _a\pi (a|s_t;\theta )\cdot Q_{\pi }(s_t,a)$$
• 策略学习最大化的目标函数
  $$J(\theta )=E_S[V(S;\theta )]$$
• 依据策略梯度上升进行
  $$\theta \leftarrow \theta +\beta \cdot \frac{\partial V(s;\theta )}{\partial \theta }$$

2. 策略梯度

Policy Gradient

$$\begin{gathered} \frac{\partial V(s;\theta )}{\theta }=\sum _a{Q}_{\pi }(s,a)\frac{\partial \pi (a|s;\theta )}{\partial \theta } \\ ={\int }_a{Q}_{\pi }(s,a)\frac{\partial \pi (a|s;\theta )}{\partial \theta } \\ =\sum _a\pi (a|s;\theta )\cdot Q_{\pi }(s,a)\frac{\partial ln[\pi (a|s;\theta )]}{\partial \theta } \\ =E_{A\sim \pi (a|s;\theta )}[Q_{\pi }(s,A)\frac{\partial ln[\pi (A|s;\theta )]}{\partial \theta }] \\ \approx Q_{\pi }(s_t,a_t)\frac{\partial ln[\pi (a_t|s_t;\theta )]}{\partial \theta } \end{gathered}$$

• 观测得到状态
  $$s_t$$
• 依据策略函数随机采样动作
  $$a_t=\pi (a_t|s_t;\theta )$$
• 计算价值函数
  $$q_t=Q_{\pi }(s_t,a_t)$$
• 求取策略网络的梯度
  $$d_{\theta ,t}=\frac{\partial ln[\pi (a_t|s_t;\theta )]}{\partial \theta }|\theta ={\theta }_t$$
• 计算近似的策略梯度
  $$g(a_t,{\theta }_t)=q_t\cdot d_{\theta ,t}$$
• 更新策略网络
  $${\theta }_{t+1}={\theta }_t+\beta \cdot g(a_t,{\theta }_t)$$

3. 案例

目前没有好的方法近似动作价值函数，则未撰写案例。

by CyrusMay 2022 03 29