在随机行走示例的左图中，首次步进仅改变V(A)的状态值，揭示了第一集中发生的事件。由于所有状态值初始化为0.5，在首次集中的中间步骤不会改变状态值。当到达终止状态A时，V(A)更新为0.45，变化量为0.05。
Exercise 6.3 From the results shown in the left graph of the random walk example it appears that the first episode results in a change in only V(A)V (A)V(A). What does this tell you about what happened on the first episode? Why was only the estimate for this one state changed? By exactly how much was it changed?
As shown in the left graph of the random walk example, all the state values are initialized to 0.5, so the intermediate steps will not change the state values in the first episode. For example, from B to A, there is:
V(B)←V(B)+α[0+γV(A)−V(B)]∴V(B)=0.5+0.1[0+1⋅0.5−0.5]=0.5
V(B) \leftarrow V(B) + \alpha[ 0 + \gamma V(A) - V(B)] \\
\therefore V(B) = 0.5 +0.1[0 +1\cdot 0.5 -0.5]=0.5
V(B)←V(B)+α[0+γV(A)−V(B)]∴V(B)=0.5+0.1[0+1⋅0.5−0.5]=0.5
Also, there is only state A changed in episode 1 shown in the left graph of the random walk example. So, the episode 1 terminate in left terminal of A, otherwise, state E should have been changed. Then, we can calculate how much the value of state A changed:
V(A)←V(A)+α[0+γV(terminal)−V(A)]∴V(A)=0.5+0.1[0+1⋅0−0.5]=0.45
V(A) \leftarrow V(A) + \alpha[ 0 + \gamma V(terminal) - V(A)] \\
\therefore V(A) = 0.5 +0.1[0 +1\cdot 0 -0.5]=0.45
V(A)←V(A)+α[0+γV(terminal)−V(A)]∴V(A)=0.5+0.1[0+1⋅0−0.5]=0.45
The changed amount is 0.05.
扫一扫
1312
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
