【论文讲义】正交策略梯度法和自动驾驶应用

【讲义】正交策略梯度法和自动驾驶应用

本文是论文Orthogonal Policy Gradient and Autonomous Driving Application的讲解讲义,本文中我们从一个关于奖励函数的回报梯度定理出发,证明了"当策略梯度向量和Q-Value向量正交时,奖励函数值为极大值",由此得出了一种实时逼近最优值的方法,并实现了这种方法且应用在了智能自动驾驶上.

---

关于奖励函数的回报梯度的定理

在此先证这个定理:在MDP(Markov Decision Process)中: 平均回报函数$\rho$和策略$\pi$及Q函数$Q^{\pi }$满足:

$$\frac{\partial \rho }{\partial \theta }=\sum _a{d}^{\pi }(s)\sum _a\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)$$

证明：

累计奖励可写作:$$V^{\pi }(s)=\sum _a\pi (s,a)Q^{\pi }(s,a)$$

可得:
$$\frac{\partial V^{\pi }(s)}{\partial \theta }=\frac{\partial }{\partial \theta }\sum _a\pi (s,a)Q^{\pi }(s,a)$$
$$=\sum _a(\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)+\pi (s,a)\frac{\partial Q^{\pi }(s,a)}{\partial \theta })$$
$$=\sum _a(\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)+\pi (s,a)\frac{\partial }{\partial \theta }(R_a^s-\rho (\pi )+\sum _{s^{\prime }}{P}_{s{s}^{\prime }}^a{V}^{\pi }(s^{\prime })))$$
$$=\sum _a(\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)+\pi (s,a)(-\frac{\partial \rho }{\partial \theta }+\sum _{s^{\prime }}{P}_{s{s}^{\prime }}^a\frac{\partial V^{\pi }(s^{\prime })}{\partial \theta }))$$

由$d^{\pi }$项累加可得:
$$\sum _s{d}^{\pi }(s)\frac{\partial \rho }{\partial \theta }=\sum _s{d}^{\pi }(s)\sum _a\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)+\sum _s{d}^{\pi }(s)\sum _a\pi (s,a)\sum _{s^{\prime }}{P}_{s{s}^{\prime }}^a\frac{\partial V^{\pi }(s^{\prime })}{\partial \theta })-\sum _s{d}^{\pi }(s)\frac{\partial V^{\pi }(s)}{\partial \theta }$$

再由$d^{\pi }$的恒定性我们可得:
$$\sum _s{d}^{\pi }(s)\frac{\partial \rho }{\partial \theta }=\sum _s{d}^{\pi }(s)\sum _a\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)+\sum _{s^{\prime }\in S}{d}^{\pi }(s^{\prime })\frac{\partial V^{\pi }(s^{\prime })}{\partial \theta }-\sum _{s\in S}{d}^{\pi }(s)\frac{\partial V^{\pi }(s)}{\partial \theta }$$
$$\Rightarrow \frac{\partial \rho }{\partial \theta }=\sum _a{d}^{\pi }(s)\sum _a\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)$$
得证.

---

有了这个定理我们不难发现,

$$\sum _a\frac{\partial \pi (s,a)}{\partial \theta }{Q}^{\pi }(s,a)=\overrightarrow{\frac{\partial \pi }{\partial \theta }}\odot \vec{Q}$$

这意味着只需要使$||\overrightarrow{\frac{\partial \pi }{\partial \theta }}\odot \vec{Q}||=0$即正交即可满足达到极值点。

---

用于自动驾驶

我们的实验环境是Torcs.《The Open Racing Car Simulator》(TORCS)是一款开源3D赛车模拟游戏.是在Linux操作系统上广受欢迎的赛车游戏.有50种车辆和20条赛道,简单的视觉效果.用C和C++写成,释放在GPL协议下.

整体架构是,由策略网络负责给出具体的驾驶操作,由Q值网络负责给出对驾驶操作达到的效果做预期评估.(这和人类的智能行为类似,是标准的Deep Reinforcement learning套路:行动能力和思考能力兼备).