Reinforcement Learning Exercise 5.6

最新推荐文章于 2025-05-03 01:00:00 发布

原创

最新推荐文章于 2025-05-03 01:00:00 发布 · 427 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#reinforcement learning

本文介绍了在使用行为策略b生成回报的情况下，状态值V(s)的公式（5.6）对于动作值Q(s,a)的类似公式。给出了从初始状态St和初始动作At开始，后续状态-动作轨迹的概率表达式，以及目标策略和行为策略下轨迹相对概率的计算。进而推导出动作值Qπ(s,a)的期望是根据重要性采样比例ρt+1:T-1和回报Gt的期望。最后分别阐述了普通重要性采样和加权重要性采样的Q(s,a)计算方式。" 136335116,17243493,使用预训练transformer模型进行文本分类实战,"['自然语言处理', '深度学习', '人工智能', '语言模型', 'Python']

Exercise 5.6 What is the equation analogous to (5.6) for action values $Q (s, a)$ instead ofstate values $V (s)$ , again given returns generated using $b$ ?

Given a starting state $S_t$ , starting action $A_t$ , the probability of the subsequent state-action trajectory, $St+1,At+1,⋯ ,STS_{t+1}, A_{t+1}, \cdots , S_T$ occurring under any policy $π\pi$ is
$\begin{aligned} &Pr(S_{t+1}, A_{t+1},\cdots, S_{T-1}, A_{T-1}, S_T \mid S_t, A_{t:T-1}\sim \pi)\\ &\qquad = p(S_{t+1} \mid S_t, A_t) \pi(A_{t+1}|S_{t+1}) \cdots p(S_{T-1} \mid S_{T-2}, A_{T-2})\pi(A_{T-1} \mid S_{T-1}) p(S_T \mid S_{T-1}, A_{T-1})\\ &\qquad =\frac {\prod_{k=t}^{T - 1} \pi(A_k \mid S_k)p(S_{k+1}\mid S_k, A_k)} {\pi(A_t \mid S_t)} \end{aligned}$

最低0.47元/天解锁文章