强化学习—— Trust Region Policy Optimization (TRPO算法)

最新推荐文章于 2024-04-17 13:17:04 发布

CyrusMay

最新推荐文章于 2024-04-17 13:17:04 发布

阅读量578

点赞数 1

CC 4.0 BY-SA版权

分类专栏：强化学习文章标签：算法机器学习线性代数人工智能强化学习

本文链接：https://blog.youkuaiyun.com/Cyrus_May/article/details/125793435

16 篇文章

订阅专栏

本文详细介绍了TrustRegionPolicyOptimization (TRPO)算法，包括其核心原理：在给定旧策略θold的基础上，通过构建近似J(θ)的置信域L(θ|θold)，在该区域内寻找优化后的策略θnew。内容涉及状态值函数、目标函数的蒙特卡洛近似和实际的优化步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1 Trust Region Algorithm 置信域算法

problem：
$\theta^\star=\mathop{argmax}\limits_{\theta} J(\theta)$
repeat：

Approximation: 给定 $\theta_{old}$ , 构建 $L(\theta|\theta_{old})$ 去近似 $J(\theta)$ ，其中 $\theta$ 需要满足 $\theta_{old}$ 的置信域，即 $N(\theta_{old})$ 。
Maximization: 在置信域内，求取优化后的 $\theta$ ： $\theta_{new}=\mathop{argmax}\limits_{\theta\in N(\theta_{old})}L(\theta|\theta_{old})$

state-value function:
$V_{\pi}(s)=\sum_{a}\pi(a|s;\theta)Q(s,a)=E_{A~\pi}[Q_\pi(s,A)]$
objective function:
$J(\theta)=E_S[V_\pi(S)]$
approximation:
$V_\pi(s)=\sum_a\frac{\pi(a|s;\theta)}{\pi(a|,s;\theta_{old})}\cdot Q_\pi(s,a)\cdot \pi(a|s;\theta_{old})=E_{A~\pi(\cdot|s;\theta_{old})}[\frac{\pi(A|s;\theta)}{\pi(A|s;\theta_{old})}\cdot Q_\pi(s,A)]$
$J(\theta)=E_S[E_A[\frac{\pi(A|S;\theta)}{\pi(A|S;\theta_{old})}\cdot Q_\pi(S,A)]]$
trajectory from $\pi(a,|s;\theta_{old})$ :
$s_1,a_1,r_1,s_2,a_2,r_2,...,s_n,a_n,r_n$
Montel Carlo Approximation:
$L(\theta|\theta_{old})=\frac{1}{n}\sum_{i=1}^n \frac{\pi(a_i|s_i;\theta)}{\pi(a_i|s_i;\theta_{old})}\cdot Q_\pi(s_i,a_i)$
$Q_\pi(s_i,a_i)\approx u_i=r_i + \gamma r_{i+1}+...+\gamma^{n-i}r_n$
$\tilde{L}(\theta|\theta_{old})=\frac{1}{n}\sum_{i=1}^n \frac{\pi(a_i|s_i;\theta)}{\pi(a_i|s_i;\theta_{old})}\cdot u_i$
$\theta\in N(\theta_{old})$

option1: $||\theta-\theta_{old}||<\Delta$

option2: $\frac{1}{n}\sum_{i=1}^nKL[\pi(\cdot |s_i;\theta)||\pi(\cdot|s_i;\theta_{old})]<\Delta$

trajectory from $\pi(\cdot|s;\theta_{old})$
$s_1,a_1,r_1,s_2,a_2,r_2,...,s_n,a_n,r_n$
dicounted returns
$u_i=\sum_{k=i}^n \gamma^{k-i}\cdot r_k$
approximation:
$\tilde L(\theta|\theta_{old})=\sum_{i=1}^n \frac{\pi(a_i|s_i;\theta)}{\pi(a_i|s_i;\theta_{old})}\cdot u_i$
maximization:
$\theta_{new}=\mathop{argmax}\limits_{\theta \in N(\theta_{old})}\tilde L(\theta|\theta_{old})$
by CyrusMay 2022 07 14