Proximal Policy Optimization (PPO)

Summary

PPO (Schulman et al., 2017) is a policy-gradient algorithm that keeps each gradient update close to the previous policy by clipping the ratio of new to old action probabilities. It is the default RL algorithm for continuous control, large-scale games, and RLHF in 2026.

PPO is the most-used deep RL algorithm for several reasons:

• Stable across domains. Robotics, Atari, locomotion, language model fine-tuning all use it with similar hyperparameters.
• Simple: no second-order Hessian-vector products (unlike TRPO).
• Mini-batch friendly: samples can be reused for several epochs per rollout.
• Backbone of RLHF: ChatGPT, Claude, Llama-Instruct were all trained with PPO on reward signals.

The objective

For a policy ${\pi }_{\theta }$ and old policy ${\pi }_{{\theta }_{\text{old}}}$ collected during the latest rollout, define the probability ratio:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}.$$

PPO maximizes the clipped surrogate:

$$L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )\cdot A_t)\right]$$

with advantage estimate $A_t$ (typically GAE) and clip parameter $\epsilon =0.2$.

For one sampled action, the min creates a one-sided plateau. With $A_t>0$, increasing the action’s probability stops improving this term once $r_t>1+\epsilon$. With $A_t<0$, decreasing its probability stops improving the term once $r_t<1-\epsilon$.

Clipped surrogate geometry

Where does PPO remove the incentive to move an action probability farther?

Full PPO objective

In addition to $L^{\text{CLIP}}$, PPO usually includes:

• Value function loss: $L^{\text{VF}}=(V_{\varphi }(s_t)-R_t{)}^2$ trained jointly.
• Entropy bonus: $L^{\text{S}}=H[{\pi }_{\theta }(\cdot \mid s_t)]$. Encourages exploration.

Total:

$$L=L^{\text{CLIP}}-c_1{L}^{\text{VF}}+c_2{L}^{\text{S}}.$$

Standard $c_1=0.5$, $c_2=0.01$.

The training loop

For each iteration:

1. Rollout: run policy ${\pi }_{{\theta }_{\text{old}}}$ in the environment for $T$ steps × $N$ parallel actors. Collect $(s,a,r,s^{\prime })$.
2. Compute advantages: $A_t$ via GAE using $V_{\varphi }$.
3. Optimize: SGD on $L$ for $K$ epochs over the rollout data, in mini-batches.
4. Update: ${\theta }_{\text{old}}\leftarrow \theta$ at the start of next iteration.

Typical: $T=2048$ per actor, $N=8$–$64$ actors, $K=4$–$10$ epochs, mini-batch size 64–256, $\epsilon =0.2$, $\gamma =0.99$, $\lambda =0.95$, learning rate $3\times 10^{-4}$.

Why clipping helps

Without clipping, repeated SGD epochs over one rollout can cause runaway policy updates. A large positive advantage gets multiplied by an unbounded ratio. The policy then moves too far from the data distribution, and learning can collapse.

Clipping removes the objective’s incentive to keep changing a sampled action beyond the sign-dependent threshold. It is not a hard constraint on the new policy: shared parameters and other samples can still move it farther, so implementations monitor approximate KL and may stop optimization early.

PPO for RLHF

In RLHF (see RLHF and DPO):

• State: prompt + partial response.
• Action: next token.
• Reward: scalar from the reward model at end of response, plus per-token KL penalty to a frozen reference model.
• Policy: the LLM being aligned.

The KL penalty is critical. It prevents the policy from drifting into reward-model-exploiting nonsense. Without it, PPO will find adversarial token sequences that maximize the reward model but produce gibberish.

PPO vs. alternatives

Algorithm	When to use
PPO	Default for on-policy with continuous or discrete actions; RLHF
SAC	Off-policy continuous control with sample efficiency required
DQN	Discrete actions, off-policy data abundant
DPO (LLM)	Direct preference optimization without explicit reward model. Increasingly displacing PPO for LLM alignment
TRPO	When KL constraint matters more than simplicity

Common pitfalls

• Wrong advantage normalization. Normalize advantages within each mini-batch (zero mean, unit std); without, learning is unstable.
• Using too many epochs per rollout. $K>10$ leads to too much off-policy drift; clipping stops bias but variance grows.
• Forgetting the value function loss coefficient. $c_1=0.5$ keeps actor and critic on similar scales.
• Logging episodic reward only. Watch entropy, KL between consecutive policies, value-function explained variance, clip fraction. Each diagnoses different failure modes.
• Treating PPO as parameter-free. It has many hyperparameters; defaults work but tuning gives ~2× sample efficiency.

Related

• Policy gradient. Foundational algorithm.
• RLHF and DPO. LLM-alignment context.