Reinforcement Learning Exercise 5.6

最新推荐文章于 2025-05-03 01:00:00 发布

YeXiang\^-^/

最新推荐文章于 2025-05-03 01:00:00 发布

阅读量407

点赞数

CC 4.0 BY-SA版权

分类专栏： reinforcement learning 文章标签： reinforcement learning

版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文链接：https://blog.youkuaiyun.com/ballade2012/article/details/98351732

本文介绍了在使用行为策略b生成回报的情况下，状态值V(s)的公式（5.6）对于动作值Q(s,a)的类似公式。给出了从初始状态St和初始动作At开始，后续状态-动作轨迹的概率表达式，以及目标策略和行为策略下轨迹相对概率的计算。进而推导出动作值Qπ(s,a)的期望是根据重要性采样比例ρt+1:T-1和回报Gt的期望。最后分别阐述了普通重要性采样和加权重要性采样的Q(s,a)计算方式。" 136335116,17243493,使用预训练transformer模型进行文本分类实战,"['自然语言处理', '深度学习', '人工智能', '语言模型', 'Python']

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Exercise 5.6 What is the equation analogous to (5.6) for action values $Q (s, a)$ instead ofstate values $V (s)$ , again given returns generated using $b$ ?

Given a starting state $S_t$ , starting action $A_t$ , the probability of the subsequent state-action trajectory, $St+1,At+1,⋯ ,STS_{t+1}, A_{t+1}, \cdots , S_T$ occurring under any policy $π\pi$ is
$\begin{aligned} &Pr(S_{t+1}, A_{t+1},\cdots, S_{T-1}, A_{T-1}, S_T \mid S_t, A_{t:T-1}\sim \pi)\\ &\qquad = p(S_{t+1} \mid S_t, A_t) \pi(A_{t+1}|S_{t+1}) \cdots p(S_{T-1} \mid S_{T-2}, A_{T-2})\pi(A_{T-1} \mid S_{T-1}) p(S_T \mid S_{T-1}, A_{T-1})\\ &\qquad =\frac {\prod_{k=t}^{T - 1} \pi(A_k \mid S_k)p(S_{k+1}\mid S_k, A_k)} {\pi(A_t \mid S_t)} \end{aligned}$

最低0.47元/天解锁文章

200万优质内容无限畅学

博客等级

码龄14年

37
原创

12
点赞

11
收藏

11
粉丝

关注

私信

热门文章

分类专栏

reinforcement learning 37篇

上一篇：: Reinforcement Learning Exercise 5.5

下一篇：: Reinforcement Learning Chapter 5, Example of Blackjack

最新评论

Reinforcement Learning Exercises 6.1
程序猿学长: mark
Reinforcement Learning Exercise 5.5
mymeromance: 请问这道题不用考虑probability吗？
Reinforcement Learning Exercise 5.5
Alisonyao: 我恐怕更赞同这里的做法：https://github.com/LyWangPX/Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions/blob/master/Chapter%205/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_5_r3.pdf
Reinforcement Learning Exercise 4.4
YeXiang\^-^/ 回复 Nahida_nora: Ah, thanks a lot :D. Your solution is much better. I will modify the pseudocode later. Too late for today.
Reinforcement Learning Exercise 4.9
YeXiang\^-^/ 回复 Nahida_nora: Sorry, I don't know why the policy looks like this. If you get the answer, please tell me. There is something wrong in the old code in function "policy_improvement" although it didn't affect the result. I have fixed it now so that the code can be easier to be understood. As the ties is broken between maximum status value and action, the optimal policy is not unique. You can modify the line 67 to select another action, and run the program, you will see a totally different policy. That's why I can not explain the resulting policy looks like this.

大家在看

题解丨统计每个产品的销售情况 450

最新文章

目录

展开全部

收起

评论

被折叠的条评论为什么被折叠?

到【灌水乐园】发言

查看更多评论

添加红包

成就一亿技术人!

hope_wisdom

发出的红包

实付元

使用余额支付

点击重新获取

扫码支付

钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。