Contrastive RL：Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data

ICLR 2024 Spotlight
paper

Method

对比学习结合 goal-conditioned offline 视觉强化学习实现自监督学习。考虑如下MDP设定，其中策略为$\pi (a|s,g)$，${\mathbb{P}}^{\pi (\cdot |\cdot ,g)}(s_t=s|s_0,a_0)$表示从初始状态执行策略，经过t步到达的状态概率密度。状态带折扣measure occupancy表示为$$p^{\pi (\cdot |\cdot ,g)}(s_{t+}=s\mid s_0,a_0)\triangleq (1-\gamma )\sum _{t=0}^{\infty }{\gamma }^t{\mathbb{P}}^{\pi (\cdot |\cdot ,g)}(s_t=s\mid s_0,a_0)$$
策略优化的目标便是最大化状态占有度量期望值
$${\mathbb{E}}_{p_g(g)}[p^{\pi (\cdot |\cdot ,g)}(s_{t+}=g)]={\mathbb{E}}_{p_g(g)p_0(s_0)\pi (a_0|s_0,g)}\left[p^{\pi (\cdot |\cdot ,g)}(s_{t+}\mid s_0,a_0)\right]$$

结合以往工作[C-learning]，通过对比表示学习完成述目标。首先设置价值函数$f(s,a,s_{t+})=\varphi (s,a{)}^{\top }\psi (s_{t+})$, 通过两个表征函数的内积估量当前状态动作与未来状态的关联程度。

对比强化学习目标便是区分平均未来状态$s_f^{+}$与任意采样未来状态$s_f^{-}$:
$$s_f^{+}\sim p^{\pi (\cdot |\cdot )}(s_{t+}\mid s,a)=\int p^{\pi (\cdot |\cdot ,g)}(s_{t+}\mid s,a)p^{\pi }(g\mid s,a)dg$$
$$s_f^{-}\sim p(s_{t+})=\int p^{\pi (\cdot |\cdot )}(s_{t+}|s,a)p(s,a)dads$$
采用NCE Binary loss优化价值函数
$${\mathbb{E}}_{s_f^{+}\sim p^{\pi (\cdot |\cdot )}(s_{t+}|s,a)}[\underset{{\mathcal{L}}_1(\varphi (s,a),\psi (s_f^{+}))}{\underbrace{\log \sigma (\varphi (s,a{)}^{\top }\psi (s_f^{+}))}}]+{\mathbb{E}}_{s_f^{-}\sim p(s_{t+})}[\underset{{\mathcal{L}}_2(\varphi (s,a),\psi (s_f^{-}))}{\underbrace{\log (1-\sigma (\varphi (s,a{)}^{\top }\psi (s_f^{-})))}}].$$
上述公式在offf-policy下可以重写成TD形式[C-learning]
$$\begin{array}{rl}\underset{f}{\max }{\mathbb{E}}_{(s,a)\sim p(s,a),s^{\prime }\sim p(s^{\prime }|s,a)}& [(1-\gamma )\log \sigma (f(s,a,s^{\prime }))\\ & +\log (1-\sigma (f(s,a,s_f)))\\ & +\gamma \lfloor w(s^{\prime },a^{\prime },s_f){\rfloor }_{\mathrm{sg}}\log \sigma (f(s,a,s_f))]\end{array}$$
策略的优化则是在offline设定下最大化critic $f$并结合BC正则化:
$$\underset{\pi (\cdot |\cdot ,\cdot )}{\max }{\mathbb{E}}_{p_g(g)p(s,a_{\mathrm{org}})\pi (a|s,g)}\left[\left(1-\lambda \right)\cdot f(s,a,g)+\lambda \log \pi (a_{\mathrm{orig}}\mid s,a)\right]$$

Network Architecture