Offline ：Adversarially Trained Actor Critic for Offline Reinforcement Learning

ICML 2022
paper
code
基于Stackelberg游戏博弈形式，对抗的学习actor与critic

Intro

Method

将离线RL的Stackelberg博弈表述为一个双层优化问题，学习者策略π∈Π为领导者，批评家f∈F为跟随者:
$$\begin{gathered} {\hat{\pi }}^{\ast }\in \underset{\pi \in \mathbf{II}}{\mathrm{argmax}}{\mathcal{L}}_{\mu }(\pi ,f^{\pi })\text{(1)} \\ s.t.f^{\pi }\in \mathrm{argmin}{\mathcal{L}}_{\mu }(\pi ,f)+\beta {\mathcal{E}}_{\mu }(\pi ,f) \end{gathered}$$
其中$\beta >0$，并且
$${\mathcal{L}}_{\mu }(\pi ,f):={\mathbb{E}}_{}$$