Proximal Policy Optimization (PPO)

One-line definition

PPO (Schulman et al., 2017) is a policy-gradient algorithm that keeps each gradient update close to the previous policy by clipping the ratio of new to old action probabilities. It is the default RL algorithm for continuous control, large-scale games, and RLHF in 2026.

Why it matters

PPO is the most-used deep RL algorithm for several reasons:

• Stable across domains. Robotics, Atari, locomotion, language model fine-tuning all use it with similar hyperparameters.
• Simple: no second-order Hessian-vector products (unlike TRPO).
• Mini-batch friendly: samples can be reused for several epochs per rollout.
• Backbone of RLHF: ChatGPT, Claude, Llama-Instruct were all trained with PPO on reward signals.

The objective

For a policy ${\pi }_{\theta }$ and old policy ${\pi }_{{\theta }_{\text{old}}}$ collected during the latest rollout, define the probability ratio:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}.$$

PPO maximizes the clipped surrogate:

$$L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )\cdot A_t)\right]$$

with advantage estimate $A_t$ (typically GAE) and clip parameter $\epsilon =0.2$.

The min ensures: when the advantage is positive, the gradient is bounded above by $1+\epsilon$. Preventing the new policy from over-committing to the action; when negative, bounded below by $1-\epsilon$.

Full PPO objective

In addition to $L^{\text{CLIP}}$, PPO usually includes:

• Value function loss: $L^{\text{VF}}=(V_{\varphi }(s_t)-R_t{)}^2$ trained jointly.
• Entropy bonus: $L^{\text{S}}=H[{\pi }_{\theta }(\cdot \mid s_t)]$. Encourages exploration.

Total:

$$L=L^{\text{CLIP}}-c_1{L}^{\text{VF}}+c_2{L}^{\text{S}}.$$

Standard $c_1=0.5$, $c_2=0.01$.

The training loop

For each iteration:

1. Rollout: run policy ${\pi }_{{\theta }_{\text{old}}}$ in the environment for $T$ steps × $N$ parallel actors. Collect $(s,a,r,s^{\prime })$.
2. Compute advantages: $A_t$ via GAE using $V_{\varphi }$.
3. Optimize: SGD on $L$ for $K$ epochs over the rollout data, in mini-batches.
4. Update: ${\theta }_{\text{old}}\leftarrow \theta$ at the start of next iteration.

Typical: $T=2048$ per actor, $N=8$–$64$ actors, $K=4$–$10$ epochs, mini-batch size 64–256, $\epsilon =0.2$, $\gamma =0.99$, $\lambda =0.95$, learning rate $3\times 10^{-4}$.

Why clipping helps

Without clipping, naive policy gradient with off-policy samples (multiple SGD epochs over the same rollout) can cause runaway updates: a single very-positive advantage gets multiplied by an unbounded ratio, the policy moves drastically, the next sample is from a much-changed distribution, and learning collapses.

Clipping bounds the per-step policy change. The min in the surrogate keeps the algorithm from “overshooting” in either direction.

PPO for RLHF

In RLHF (see RLHF and DPO):

• State: prompt + partial response.
• Action: next token.
• Reward: scalar from the reward model at end of response, plus per-token KL penalty to a frozen reference model.
• Policy: the LLM being aligned.

The KL penalty is critical. It prevents the policy from drifting into reward-model-exploiting nonsense. Without it, PPO will find adversarial token sequences that maximize the reward model but produce gibberish.

PPO vs. alternatives

Algorithm	When to use
PPO	Default for on-policy with continuous or discrete actions; RLHF
SAC	Off-policy continuous control with sample efficiency required
DQN	Discrete actions, off-policy data abundant
DPO (LLM)	Direct preference optimization without explicit reward model. Increasingly displacing PPO for LLM alignment
TRPO	When KL constraint matters more than simplicity

Common pitfalls

• Wrong advantage normalization. Normalize advantages within each mini-batch (zero mean, unit std); without, learning is unstable.
• Using too many epochs per rollout. $K>10$ leads to too much off-policy drift; clipping stops bias but variance grows.
• Forgetting the value function loss coefficient. $c_1=0.5$ keeps actor and critic on similar scales.
• Logging episodic reward only. Watch entropy, KL between consecutive policies, value-function explained variance, clip fraction. Each diagnoses different failure modes.
• Treating PPO as parameter-free. It has many hyperparameters; defaults work but tuning gives ~2× sample efficiency.

Related

• Policy gradient. Foundational algorithm.
• RLHF and DPO. LLM-alignment context.