O2O:Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning

3rd Offline Reinforcement Learning Workshop at Neural Information Processing Systems, 2022.
paper

Intro

通过对TD3-BC 中的BC正则化项系数进行调整，使得算法在实现O2O时有更好的表现。

Method

Offline

对于离线阶段的TD3+BC，策略优化目标为：
$$\pi =\arg \underset{\pi }{\max }{\mathbb{E}}_{(s,a)\sim D}[\underset{RL}{\underbrace{Q(s,\pi (s))}}-\alpha \underset{BC}{}$$