本文详细探讨了强化学习中Q函数的迭代更新公式，基于π策略下状态动作值函数Qπ的定义，推导出了其迭代逼近公式。通过将前一次迭代的Q值估计作为输入，为理解强化学习算法的收敛性和稳定性提供了数学基础。
Exercise 4.3 What are the equations analogous to (4.3), (4.4), and (4.5) for the action-value function qπq_\piqπ and its successive approximation by a sequence of functions q0,q1,q2,...q_0, q_1, q_2, . . .q0,q1,q2,...?
According to the result of exercise 3.17, we have:
Qπ(s,a)=∑s′Rs,s′aPss′a+γ∑s′[∑a′Qπ(s′,a′)π(s′,a′)]Ps,s′a
Q_\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qπ(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qπ(s′,a′)π(s′,a′)]Ps,s′a
Let QkπQ_k^\piQkπ be the previous estimated value of QπQ_\piQπ and substitute it to the right side of the equation. For the next iteration, Qk+1πQ_{k+1}^\piQk+1π can be:
Qk+1π(s,a)=∑s′Rs,s′aPss′a+γ∑s′[∑a′Qkπ(s′,a′)π(s′,a′)]Ps,s′a
Q_{k+1}^\pi(s,a) = \sum_{s'} R_{s,s'}^a P_{ss'}^a + \gamma \sum_{s'} \bigl[ \sum_{a'} Q_k^\pi(s',a') \pi(s',a') \bigr] P_{s,s'}^a
Qk+1π(s,a)=s′∑Rs,s′aPss′a+γs′∑[a′∑Qkπ(s′,a′)π(s′,a′)]Ps,s′a
扫一扫
536
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
