Preference data and reward models

Summary

Preference data records comparative judgments among model outputs, and a reward model learns a scoring function that predicts those judgments for training or evaluation.

RLHF diagrams often compress the hardest part into “collect preferences.” The signal depends on the sampled prompts and responses, the judges, the rubric, and the treatment of disagreement. It also changes as the policy moves away from the reward model’s training distribution.

A powerful optimizer amplifies measurement defects. Better preference-data design can matter more than another RL algorithm.

Learning objective: explain why reward-model validation on collected comparisons does not guarantee reliable rewards after the policy is optimized.

Proxy shift

Why can a validated reward model fail during policy optimization?

1 · FIT ON SAMPLED RESPONSES

Coverage is chosen
Prompts, policy checkpoints, temperatures, and failure slices determine which response pairs appear.

Judgments define the target
Annotators apply a rubric, including ties, both-bad labels, and disagreements.

The proxy learns locally
The reward model learns which sampled response wins; held-out pairs test that same measurement process.

Known evidence
Accuracy, calibration, and slice results describe behavior on represented comparisons.

2 · OPTIMIZE INTO NEW RESPONSES

The policy follows the proxy
Updates increase outputs that the fixed reward model scores highly.

The response distribution moves
New outputs can leave the comparisons on which the proxy was trained and validated.

Search exposes shortcuts
The optimizer can find verbosity, style, or other features that raise predicted reward without improving the rubric's target.

Required response
Audit optimized outputs, refresh on-policy comparisons, and revalidate by slice before trusting the next update.

Read it this way: read the left panel as a bounded measurement claim, then move right. Validation says the reward model predicts judgments on represented comparisons; optimization changes which responses it sees and searches specifically for high scores. That is why held-out pair accuracy cannot close the loop: inspect optimized outputs, collect fresh comparisons, and recalibrate the proxy. This original synthesis is informed by Ziegler et al. (2019), Ouyang et al. (2022), and Gao et al. (2023).

Constructing comparisons

For prompt $x$ , sample responses $y_{a}$ and $y_{b}$ from policies and decoding settings chosen to expose meaningful differences. Ask an annotator which response better satisfies an explicit rubric, allows a tie, or marks both unacceptable.

Pairwise judgments are often easier than absolute scores, but sampling determines value. Comparing one obviously broken answer with one strong answer teaches less than a near-boundary pair that isolates factuality, instruction following, style, or safety.

Useful sampling strategies include:

policy checkpoints across training;
multiple temperatures and seeds;
uncertainty or disagreement sampling;
known failure slices;
adversarially generated pairs;
human and model-written alternatives;
active selection near the current reward boundary.

Annotator protocol

Define the target construct before collecting labels. “Which is better?” mixes correctness, relevance, verbosity, style, and safety differently across annotators.

A sound protocol includes:

rubric and precedence among criteria;
examples and counterexamples;
tie and both-bad options;
qualification and calibration tasks;
blind randomization of response order and model identity;
duplicate items for consistency;
escalation for high-impact ambiguity;
slice-level agreement and quality monitoring;
privacy and worker-welfare constraints.

Disagreement is data. It can reveal ambiguous prompts, plural user values, or a broken rubric rather than a bad annotator.

Reward-model objective

A common Bradley-Terry form models:

P (y_{w} ≻ y_{l} ∣ x) = σ (r_{ϕ} (x, y_{w}) - r_{ϕ} (x, y_{l})) .

The loss encourages the preferred response to receive higher reward. Only differences are identified; the absolute reward scale is arbitrary without further calibration.

Evaluate the reward model

Use held-out prompts and behavioral families. Measure:

pairwise accuracy and calibration;
performance by criterion and slice;
agreement with independent humans;
order, length, style, and verbosity bias;
out-of-distribution behavior;
adversarially optimized outputs;
ranking stability across policy checkpoints;
uncertainty and abstention quality.

High random-split accuracy can coexist with failure on a new policy because response distribution changes during optimization.

Reward hacking and shift

Once a policy optimizes the reward model, it searches for outputs the model scores highly, including shortcuts absent from ordinary samples. Mitigations include on-policy data refresh, adversarial sampling, reward ensembles, uncertainty penalties, KL constraints, independent verifiers, and held-out audits.

None remove the need to inspect what the optimized policy discovers.

Common confusions

“More preference pairs always help.” Redundant easy pairs add little signal.
“Annotator disagreement is noise.” It can expose underspecified values or prompts.
“Reward accuracy equals reward quality.” Calibration, slice behavior, and optimization robustness matter.
“The reward score is utility.” It is a learned proxy with arbitrary scale and bounded coverage.
“DPO removes preference-data problems.” It removes the explicit reward model, not bias or coverage in the comparisons.
“Model graders eliminate humans.” They inherit model biases and require independent calibration.

In an interview

Describe prompt and response sampling, rubric, annotator quality, disagreement, objective, held-out families, calibration, policy shift, reward hacking, and the online outcome the proxy is meant to improve.