Offline:Model-Based Offline Reinforcement Learning with Uncertainty Estimation and Policy Constraint

IEEE TAI 2024
paper
code
采用集成模型下MC Dropout实现状态动作对的不确定性估计，该估计将作为奖励函数的正则化项。然后基于MMD约束实现策略优化。

Intro

面对离线RL中由分布偏移引起的高估问题，显式不确定性估计是一种有希望的方法。论文提出了一种名为MOUP（带不确定性估计和策略约束的基于模型的离线RL）的新算法，在集成网络中引入了蒙特卡洛（MC）dropout以获得可靠的不确定性估计，将最大均值差异（MMD）约束整合进策略优化中限制状态不匹配。

Method

Ensemble Dropout Network

采用集成高斯模型对环境动力学模型进行建模
T ϕ ( s t + 1 , r t ∣ s t , a t ) = N ( μ ϕ ( s t , a t ) , Σ ϕ ( s t , a t ) ) μ ϕ ( s t , a t ) = ∑ i = 1 K μ ϕ i ( s t , a t ) , Σ ϕ ( s t , a t ) = ∑ i = 1 K Σ ϕ i ( s t , a t ) . \begin{aligned} &T_{\phi}(s_{t+1},r_{t}|s_{t},a_{t})=N(\mu_{\phi}(s_{t},a_{t}),\Sigma_{\phi}(s_{t},a_{t}))\\ &\mu_{\phi}(s_{t},a_{t}) =\sum_{i=1}^{K}\mu_{\phi i}(s_{t},a_{t}), \\ &\Sigma_{\phi}\left(s_{t},a_{t}\right) =\sum_{i=1}^{K}\Sigma_{\phi^{i}}(s_{t},a_{t}). \end{aligned} ​Tϕ​(st+1​,rt​∣st​,at​)=N(μϕ​(st​,at​),Σϕ​(st​,at​))μϕ​(st​,at​)=i=1∑K​μϕi​(st​,at​),Σϕ​(s