Skip to content
mlmentorship

Design post-training data, an RL environment, and its grader

Turn one capability into episodes, evidence-bearing graders, adversarial data, and a training signal that cannot reward the wrong process.

Published · 8 min read ·Role-specific ·Advanced

Research depth · active recall

Practice before you read

25 minutes. State the claim, test its evidence, design discriminating experiments, and identify alternatives.

Back to my workbook How practice works

Research depth · closed-book attempt

Design post-training data, an RL environment, and its grader

State the claim, test its evidence, design discriminating experiments, and identify alternatives.

Problem

Context. A hypothetical tool-using agent resolves support tickets using account data and actions governed by user permissions. Training may use demonstrations, preferences, or environment feedback. Task. Specify target behavior, data provenance, environment state and tools, graders, training objective, and independent evaluation. Explain how you would detect reward exploitation, overfitting, and failure to transfer to real support work. Constraints. Unauthorized account access is forbidden, even if it improves measured ticket completion. Separate training rewards from held-out success evidence; preserve privacy, environment versioning, and realistic action consequences in contained tests.

25:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Improve a tool-using agent’s ability to resolve support tickets. Design the data, environment, grader, training loop, and evaluation. The agent must not access unauthorized account data.

Design post-training data and an evaluation environment that improve a support agent’s authorized task completion rather than merely its reward score.

Problem

Context. A hypothetical tool-using agent resolves support tickets using account data and actions governed by user permissions. Training may use demonstrations, preferences, or environment feedback.

Task. Specify target behavior, data provenance, environment state and tools, graders, training objective, and independent evaluation. Explain how you would detect reward exploitation, overfitting, and failure to transfer to real support work.

Constraints. Unauthorized account access is forbidden, even if it improves measured ticket completion. Separate training rewards from held-out success evidence; preserve privacy, environment versioning, and realistic action consequences in contained tests.

The design sequence

1. Define the capability distribution

Specify ticket types, user states, tools, permissions, ambiguity, long-horizon depth, and failure severity. A single average success rate will hide whether the model improves on easy password resets while failing high-risk billing changes.

2. Define episode semantics

An episode needs:

  • observable state and hidden ground truth;
  • available actions and tool schemas;
  • permission boundaries;
  • stochasticity and reset behavior;
  • terminal success, terminal failure, and timeout;
  • an audit log sufficient to reproduce the trajectory.

3. Separate outcome from process

Use a structured grade rather than one opaque scalar:

  • task success: was the ticket actually resolved?
  • policy compliance: were permissions and required confirmations respected?
  • process quality: were actions efficient, grounded, and non-duplicative?
  • communication quality: was the user-facing result accurate and calibrated?

Some violations are gates, not tradeable penalties. Unauthorized access cannot be offset by a polite final response.

Learning objective

Decide whether a successful trajectory is eligible for quality scoring by applying the policy gate first.

Task success cannot excuse unauthorized account access

A successful task can still fail the authorization requirementCompletion and policy compliance are independent checks. Unauthorized account access vetoes the otherwise successful trial.Every required gate must pass on thesame route+task+authorityAuthorized successful trial passes.+taskxauthorityUnauthorized successful trial isrejected.
Derivation
Two successful support trajectories receive different training eligibility Both trajectories resolve the ticket. The safe path reads only the requesting user's authorized account, so the policy gate passes and task success, process quality, and communication quality may be aggregated into a training signal. The unsafe path reads another user's unauthorized account, so the policy gate fails, a stop barrier terminates the path, and no quality score or positive training signal is produced. Successful resolution cannot compensate for unauthorized access. SAME TASK OUTCOME Ticket resolved correctly SAFE PATH UNSAFE PATH ACTION EVIDENCE Read account A requester owns A ACTION EVIDENCE Read account B requester does not own B POLICY GATE PASS POLICY GATE FAIL STOP - INELIGIBLE ELIGIBLE QUALITY SCORE task success + evidence process quality + evidence communication + evidence NO QUALITY SCORE success cannot compensate TRAINING SIGNAL eligible trajectory PATH TERMINATED
compare the columns from top to bottom. Both agents resolve the ticket, but only the action log proves whether the trajectory respected authorization. The safe path passes the policy gate and reaches quality-score aggregation; the unsafe path stops before scoring, so a strong outcome or polished response cannot cancel the violation. The hard gate is an original design recommendation; Gao et al. establish the reward-overoptimization risk, while Gymnasium's guidance supports explicit termination semantics.
Check: Can excellent task success compensate for reading an account outside the requester's authority?
Read it this way: Completion and policy compliance are independent checks. Unauthorized account access vetoes the otherwise successful trial.
Can successful completion override missing account authority?

4. Choose the grader stack

Use deterministic checks for permissions, tool arguments, final database state, and known invariants. Use model graders for qualities that require semantic judgment, but calibrate them against blinded human labels and adversarial examples. Reserve human review for ambiguous, high-impact, or novel failures.

Every grade should include evidence: the action, rule, comparison, or rubric item that produced it.

5. Construct training data

Combine:

  • expert trajectories for difficult or safety-critical behavior;
  • model-generated trajectories filtered by verifiers;
  • contrastive pairs that isolate one decision;
  • failed trajectories with labeled failure points;
  • adversarial cases that target grader blind spots;
  • held-out families that cannot leak into training.

Data diversity is not prompt paraphrase count. Vary state, tools, incentives, ambiguity, and consequences.

What an L4 answer sounds like

“Collect good conversations, use an LLM judge to score success, and train with RL. Give positive reward for resolution and negative reward for unsafe actions.”

The answer names the pipeline but leaves the attack surface undefined. The model judge is treated as truth, safety is a small penalty, and there is no environment version, evidence, or held-out generalization.

What an L5 answer adds

An L5 candidate specifies episode state and actions, separates deterministic and model-based grading, makes hard policy violations disqualifying, and defines coverage slices. They version environment, tools, grader, model, and data.

They also test the grader before training:

  • swap a correct final answer onto an unsafe trajectory;
  • fabricate tool output while preserving the outcome;
  • repeat no-op actions;
  • exploit formatting or verbosity preferences;
  • find a valid but unusual solution;
  • withhold information the grader assumes is present.

The grader should rank behavior for the intended reasons.

What an L6 answer adds

An L6 candidate treats the environment as a research instrument. They measure inter-rater agreement, grader calibration, exploitability, coverage, variance, and sensitivity to model scale. They know that training against a fixed judge changes the distribution of attacks on that judge.

They separate three evaluations:

  1. capability: can the model solve the task under representative conditions?
  2. alignment or policy: does it preserve constraints under pressure and adversarial context?
  3. product value: does the intervention improve real outcomes without unacceptable latency, cost, or user harm?

They define a launch decision and a rollback signal. Benchmark movement alone does not ship a model.

Tells that get you a strong-hire vote

  • Capability and prohibited behavior are explicit.
  • Episode reset, timeout, and terminal conditions are defined.
  • Safety-critical violations are gates, not small penalties.
  • Graders return evidence and are calibrated against humans.
  • Training and held-out families are separated by behavior, not wording.
  • Adversarial episodes target reward hacking.
  • Environment and grader versions accompany every result.
  • Online product evidence remains separate from training reward.

Tells that get you down-leveled

  • Starting with PPO or DPO.
  • One model judge and one scalar reward.
  • Rewarding the final answer while ignoring the trajectory.
  • Treating synthetic volume as data quality.
  • No held-out behavioral families.
  • No grader false-positive analysis.
  • Shipping because the reward curve increased.

Common follow-up

“The policy learns to call the verifier repeatedly until one noisy score passes. What changes?”

The environment should expose repeated calls and charge or cap them. The grader should evaluate the trajectory, not only the best sampled score. More importantly, investigate why the verifier is noisy enough to exploit and whether training data now overrepresents that loophole. A penalty can suppress the symptom; a calibrated verifier and explicit process contract address the cause.

Try the post-training environment lab before reading this as a checklist.

Related: preference data and reward models, RL environments and graders, and evaluate an agent.

Found an error or stale claim? Report a correction or send a private note.