PPO keeps TRPO's 'don't move too far' principle but enforces it with a cheap first-order trick: clip the probability ratio so the objective stops rewarding big moves.

r_t(θ) = π_θ(a_t|s_t) ÷ π_old(a_t|s_t) — how much more likely the new policy makes the action actually taken at step t.
Â_t — that action's advantage (usually GAE): did it beat the average?
clip(r_t, 1−ε, 1+ε) — pin the ratio to a small band (ε ≈ 0.2).
min(r_t·Â_t, clipped·Â_t) — take the pessimistic option, killing the incentive to push r_t past the band.
𝔼 over a_t ~ π_old(·|s_t) — the expectation is taken over actions a_t drawn from the old policy in each state s_t; in practice it is the empirical mean over the batch of (s_t, a_t) timesteps collected by running π_old.

No constraint, no second-order math — just a clipped objective optimised with ordinary SGD for a few epochs per batch. It is simple, robust across an enormous range of tasks, and is the algorithm behind most continuous-control results and the RLHF stage of modern chat models.

Reinforcement learning explorer: an infinite-canvas taxonomy, equation and notation references, and a dual-lens Venn diagram.

Reinforcement learning explorer