[Reinforment Learning] Epoch and timestep

Preface

In the context of reinforcement learning (RL), the term “timestep” has a specific meaning that differs from “epoch.” Understanding these terms is crucial for interpreting how the RL algorithm operates and processes data.

Timestep:

Represents one discrete interaction: action → environment response (observation, reward, done signal).
Fundamental unit of experience in RL.

Epoch:

Represents a collection of timesteps, often aggregated before performing an update to the policy or value networks.
Helps in organizing the training process, especially in batch-based RL algorithms like PPO.
Why “Timestep” Matters:

Why timestep matters

RL algorithms rely on sequential data where each timestep’s outcome can influence future actions.
Tracking changes between consecutive timesteps (like delta_pitch) helps in understanding the dynamics and progression of the agent’s actions.

Epochs in RL:

While timesteps are about individual actions, epochs in RL organize these actions into manageable batches for updating the model.
For example, after collecting a certain number of timesteps, the agent may perform gradient updates to improve the policy based on the aggregated experience.

Visual Representation:

Epoch
│
├─ Timestep 1: Action A1 → Observation O1, Reward R1
├─ Timestep 2: Action A2 → Observation O2, Reward R2
├─ Timestep 3: Action A3 → Observation O3, Reward R3
│
└─ Policy Update based on collected timesteps