Policy gradient methods

Summary

Policy gradient methods directly parametrize a policy ${\pi }_{\theta }(a\mid s)$ and optimize $\theta$ by ascending the gradient of expected return:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\cdot R_t\right].$$

This is the policy gradient theorem (Sutton et al., 1999).

Policy gradient methods are the foundation of modern continuous-control RL (SAC, PPO), of large-scale RL (AlphaGo’s policy net, OpenAI Five), and of LLM alignment (RLHF uses PPO). They are essential whenever:

• The action space is continuous (no easy ${\max }_a$).
• A stochastic policy is desirable (exploration, multi-modal optimal policies).
• You can write a clean, differentiable policy parametrization.

REINFORCE

The simplest policy gradient (Williams, 1992):

$${\nabla }_{\theta }J={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\cdot G_t\right]$$

where $G_t={\sum }_{k=t}^T{\gamma }^{k-t}{r}_k$ is the return-to-go from step $t$.

Sample a trajectory by running the policy. For each step, compute $\log {\pi }_{\theta }(a_t\mid s_t)$ and weight by the actual return. Backprop. Take a gradient step.

Pros: extremely general. Works for any differentiable policy. Cons: high variance. Return $G_t$ is a noisy estimate of expected return; gradient estimator dominates by random rollout luck.

Variance reduction: baselines

Subtract any state-dependent baseline $b(s_t)$ from $G_t$. Does not change the expected gradient (so no bias) but can drastically reduce variance:

$${\nabla }_{\theta }J=\mathbb{E}\left[\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\cdot (G_t-b(s_t))\right].$$

Optimal baseline: a value function $V(s_t)$. This leads directly to actor-critic.

Learning objective

How does the sign of advantage change the sampled action’s probability?

SAMPLED ACTION · COMPARE WITH BASELINE

Observed
G_t is the return-to-go after sampling action a_t in state s_t.

Expected
b(s_t) is an action-independent baseline, commonly V(s_t).

Weight
A_t = G_t − b(s_t).

+ · BETTER THAN EXPECTED

Advantage
A_t > 0.

Gradient-ascent update
Move along the score direction: increase log π(a_t | s_t) and reinforce the sampled action.

0 · MATCHED EXPECTATION

Advantage
A_t = 0.

Gradient-ascent update
This sample contributes zero to the policy gradient.

− · WORSE THAN EXPECTED

Advantage
A_t < 0.

Gradient-ascent update
Reverse the score direction: decrease log π(a_t | s_t) and suppress the sampled action.

Actor-critic

Two networks:

• Actor ${\pi }_{\theta }(a\mid s)$. The policy.
• Critic $V_{\varphi }(s)$. Value function estimator.

Use the critic as the baseline. Define the advantage $A(s,a)=Q(s,a)-V(s)\approx G_t-V_{\varphi }(s_t)$. Update:

$${\nabla }_{\theta }J=\mathbb{E}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\cdot A_t\right].$$

Critic is trained by TD: $V_{\varphi }(s_t)\leftarrow V_{\varphi }(s_t)+\alpha [r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)]$.

This is the basic A2C / A3C algorithm.

Generalized Advantage Estimation (GAE)

Instead of one-step or full-return advantages, use a weighted sum:

$$A_t^{\text{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l},{\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t).$$

$\lambda \in [0,1]$ trades bias for variance. $\lambda =0$ → one-step TD. $\lambda =1$ → Monte-Carlo. Standard $\lambda =0.95$ in PPO.

Trust regions and PPO

Naive policy gradient updates can drastically change the policy in one step, leading to collapse. Trust Region Policy Optimization (TRPO; Schulman et al., 2015) constrains the KL divergence between old and new policies. PPO (Schulman et al., 2017) replaces the constraint with a clipped surrogate objective:

$$L^{\text{PPO}}(\theta )=\mathbb{E}\left[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )A_t)\right]$$

with $r_t(\theta )={\pi }_{\theta }(a_t\mid s_t)/{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)$ and $\epsilon =0.2$ typical.

PPO is the dominant policy gradient algorithm in 2026: simpler than TRPO, robust, well-understood, used in RLHF.

Off-policy actor-critic

For sample efficiency in continuous control, use a replay buffer with importance sampling correction or Q-function critics. DDPG, TD3, SAC are the standard off-policy actor-critic algorithms. SAC adds a maximum-entropy bonus that encourages exploration.

Common pitfalls

• High variance. Always use a baseline; always normalize advantages within a batch (subtract mean, divide by std).
• No entropy bonus. Policies collapse to deterministic; add $-\beta \cdot H({\pi }_{\theta })$ to the loss to encourage exploration.
• Reusing samples without importance correction. On-policy methods (REINFORCE, PPO) sample from the current policy; using stale samples introduces bias. PPO’s clipping caps the staleness.
• Mismatch between rollout and learning. GPU-driven RL often has the policy version drift between rollout and gradient update; PPO’s IS ratio handles small mismatches.
• Treating policy gradient as universally low-variance. It isn’t. Q-learning often has lower variance for discrete-action problems.

Related

• Q-learning. Value-based alternative.
• RLHF and DPO. PPO applied to LLM alignment.
• PPO. Algorithm-level details.