本文深入探讨了强化学习中Q值迭代更新公式的推导过程，基于第3.17题的结果，详细解释了如何从状态价值函数的迭代更新公式(4.10)推导出动作价值函数q_k+1(s,a)的迭代更新公式。
Exercise 4.10 What is the analog of the value iteration update (4.10) for action values,qk+1(s,a)q_{k+1}(s, a)qk+1(s,a)?
Use the result of exercise 3.17:
Qπ(s,a)=∑s′Rs,s′aPss′a+γ∑s′[∑a′Qπ(s′,a′)π(s′,a′)]Ps,s′a
Q_\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qπ(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qπ(s′,a′)π(s′,a′)]Ps,s′a
easily, we have the iteration for qk+1(s,a)q_{k+1}(s,a)qk+1(s,a) which is analogous to the value iteration update (4.10):
qk+1(s,a)=maxa{∑s′Rs,s′aPss′a+γ∑s′[∑a′qk(s′,a′)π(s′,a′)]Ps,s′a}
q_{k+1}(s, a) = \max_a \biggl \{ \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} q_k(s',a') \pi(s',a') \bigr] P_{s,s'}^a \biggr \}
qk+1(s,a)=amax{s′∑Rs,s′aPss′a+γs′∑[a′∑qk(s′,a′)π(s′,a′)]Ps,s′a}
Reinforcement Learning Exercise 4.10
最新推荐文章于 2021-06-29 17:58:34 发布
扫一扫
9894
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
