Robust Adversarial Reinforcement Learning

最新推荐文章于 2025-03-14 13:10:29 发布

Hazekiah

最新推荐文章于 2025-03-14 13:10:29 发布

阅读量1.2k

点赞数

CC 4.0 BY-SA版权

分类专栏： RL

本文链接：https://blog.youkuaiyun.com/u010909964/article/details/84405537

当前强化学习方法因测试泛化和模拟转移问题而无法很好地泛化。该工作旨在学习一种对模拟和真实世界之间建模错误及训练与测试场景不匹配具有鲁棒性的策略。通过联合学习的对手，创建困难情况以提高策略的适应性，同时引入条件价值在风险的概念，强调在最坏情况下的性能，以实现对不确定性的抵御。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

motivation

Current RL methods fail to generalize due to two issues:

test generalization
data is scarce especially in the sense of real-world data. So RL models often overfit to the training scenarios.
simulation transfer
the environment and the physics of the real-world is different from that of the experiemts. So the learned policy not necessarily transfer to the real-world setting.

Can we model the gap between the simulation and the real-world? How?

list all possible factors that may vary between the two cases.
Impractical because the space is inifinite
viewing the real-world from the simulation being applied disturbances.

The goal of this work is to learn a policy that is robust to modeling errors in simulation and mismatch between training and testing scenarios.

The basic idea is to resort to a joinly-learned adversary to apply disturbances to the system, in two ways:

the adversary create tough situations where the protagonist easily fail to gain high rewards, i.e., sample hard examples.
e.g. driving a car with two driver seats
the adversary can be endowed with domain knowledge
e.g. where exactly people want to attack the protagonist, or even the control over the environment.

Model formulation

This work is set on MDP with some variations: 1) the transition function is dependent on both actions from the adversary and the protagonist, 2) the reward function is dependent on both actions, and 3) this is a two-player zero-su