强化学习—— Trust Region Policy Optimization (TRPO算法)

1 Trust Region Algorithm 置信域算法

problem：
$${\theta }^{\star }=\underset{\theta }{\mathrm{argmax}}J(\theta )$$
repeat：

1. Approximation: 给定 ${\theta }_{old}$, 构建$L(\theta |{\theta }_{old})$去近似$J(\theta )$，其中$\theta$需要满足${\theta }_{old}$的置信域，即$N({\theta }_{old})$。
2. Maximization: 在置信域内，求取优化后的$\theta$：$${\theta }_{new}=\underset{\theta \in N({\theta }_{old})}{\mathrm{argmax}}L(\theta |{\theta }_{old})$$

2 Trust Region Policy Optimization (TRPO算法)

• state-value function:
  $$V_{\pi }(s)=\sum _a\pi (a|s;\theta )Q(s,a)=E_{A\pi }[Q_{\pi }(s,A)]$$
• objective function:
  $$J(\theta )=E_S[V_{\pi }(S)]$$
• approximation:
  V π ( s ) = ∑ a π ( a ∣ s ; θ ) π ( a ∣ , s ; θ o l d ) ⋅ Q π ( s , a ) ⋅ π ( a ∣ s ; θ o l d ) = E A   π ( ⋅ ∣ s ; θ o l d ) [ π ( A ∣ s ; θ ) π ( A ∣ s ; θ o l d ) ⋅ Q π ( s , A ) ] V_\pi(s)=\sum_a\frac{\pi(a|s;\theta)}{\pi(a|,s;\theta_{old})}\cdot Q_\pi(s,a)\cdot \pi(a|s;\theta_{old})=E_{A~\pi(\cdot|s;\theta_{old})}[\frac{\pi(A|s;\theta)}{\pi(A|s;\theta_{old})}\cdot Q_\pi(s,A)] Vπ​(s)=