强化学习 sbs3 PPO 几个参数 n_envs n_steps batch_size n_epochs

stable-baselines3 中训练相关的几个超参数：

• n_envs : 并行的环境数目
• n_steps (int) – The number of steps to run for each environment per update (i.e. rollout buffer size is n_steps * n_envs where n_envs is number of environment copies running in parallel) NOTE: n_steps * n_envs must be greater than 1 (because of the advantage normalization) See https://github.com/pytorch/pytorch/issues/29372
• batch_size (int) – Minibatch size
• n_epochs (int) – Number of epoch when optimizing the surrogate loss

在 log 里面打印的 total_timestaps = n_envs *n_steps
从stable-baselines3 的 PPO 源代码来看，在训练的时候，外层for循环是 for循环 n_epochs 次：内层for循环是得到 batch_size 个样本。