Skip to content
mlmentorship

Reward shaping

Modify learning signals without accidentally changing the task, creating reward hacking, or hiding specification failure.

Published · 5 min read ·Specialist ·Intermediate

Visual quick review

Visual first · depth when needed

See why potential-based shaping can give dense progress feedback without making a waypoint loop profitable, while an arbitrary positive event bonus can be farmed and change the task.

Preparing the visual…

Summary

Sparse or delayed rewards (robotics, games, long-horizon agents, recommender objectives) make credit assignment hard: the agent rarely sees a signal, so it rarely learns. Reward shaping adds auxiliary feedback to densify that signal. The danger is that changing the reward can change the optimal policy, and a badly shaped reward produces confident, well-optimized behavior that does the wrong thing: circling near a waypoint, farming easy interactions, or maximizing proxy engagement while destroying long-term value.

A shaped reward has the form

Potential-based shaping

The safe construction makes a difference of a potential function :

This shifts value estimates while preserving the set of optimal policies under standard assumptions. It rewards progress toward useful states without redefining the final objective, which is why it is the default when you must shape at all.

Learning objective

Why can one progress signal preserve the task while another creates a reward loop?

Potential differences cancel on backtracking, but event bonuses accumulate Two panels use the same finite episodic path with discount gamma equal to 1. In the safe panel, start, waypoint, and terminal goal have potentials negative 2, negative 1, and 0. Potential shaping gives plus 1 from start to waypoint, plus 1 from waypoint to goal, and negative 1 when backtracking from waypoint to start. A backtrack followed by another advance therefore adds 0. The original goal-path return is 10 and the shaped return is 12, a fixed plus 2 boundary shift from this start. In the unsafe panel, entering the waypoint pays an arbitrary plus 1 event bonus, but leaving pays 0. Every loop through the waypoint earns another plus 1 without completing the goal, so the proxy can change the preferred behavior. SAFE · POTENTIAL DIFFERENCE finite episode · γ = 1 START Φ = −2 WAYPOINT Φ = −1 GOAL terminal · Φ = 0 r = 0 F = +1 r = +10 F = +1 backtrack F = −1 · advance F = +1 · lap = 0 goal path: original = 0 + 10 = 10 shaped = (0 + 1) + (10 + 1) = 12 · fixed boundary shift +2 UNSAFE · ARBITRARY EVENT BONUS same states · no potential START task incomplete WAYPOINT entry bonus +1 GOAL task reward +10 BONUS +1 reward +10 leave +0 · re-enter +1 · every lap earns +1 LOOP RETURN GROWS WITH VISITS high shaped reward no longer proves task progress
Read it this way: compare the dashed return edges. With potential differences, the +1 earned approaching the waypoint is repaid as −1 when the agent backtracks, so a lap adds zero. A standalone waypoint bonus has no departure penalty, so repeated visits keep paying without reaching the goal. The example uses γ = 1 and zero terminal potential; for general discounted returns, the intermediate terms still telescope to boundary potentials. Original schematic checked against Ng, Harada, and Russell (1999) and the episodic analysis by Grześ (2017).

Design procedure

  1. Write down the true objective and the behavior you will not accept.
  2. Identify why credit assignment is hard.
  3. Prefer state potentials or demonstrations over arbitrary event bonuses.
  4. Check whether a policy can maximize the shaped reward without doing the task.
  5. Evaluate on the original reward and independent guardrails.
  6. Anneal or remove the shaping once it is no longer needed.

In an interview

Explain sparse credit assignment, potential-based shaping, reward hacking, and how you would red-team the proxy. The senior move is to separate optimization failure from specification failure: a perfectly optimized bad reward is not an algorithm bug, it is a spec bug.

Common confusions

  • “More detailed reward is always better.” More terms mean more loopholes and unstable scales.
  • “Human preference solves specification.” Preference data still has annotator, coverage, and manipulation limits.
  • “The training reward is the evaluation.” Evaluate against independent task outcomes and safety constraints.

Related: policy gradient, PPO, and RLHF and DPO.

Found an error or stale claim? Report a correction or send a private note.