Skip to content
mlmentorship

How do you evaluate an agent?

Agent eval is harder than chat eval because there are intermediate steps, tool calls, and long-horizon outcomes. The senior answer evaluates trajectories, not just final outputs.

Published · 7 min read ·Role-specific ·Intermediate

Product & experimentation · active recall

Practice before you read

20 minutes. Define the decision first, then design the measurement and enumerate validity threats.

Back to my workbook How practice works

Product & experimentation · closed-book attempt

How do you evaluate an agent?

Define the decision first, then design the measurement and enumerate validity threats.

Problem

Context. A hypothetical agent plans multiple steps, calls tools, observes results, and may create persistent external effects. A candidate model or orchestration change must be assessed before wider release. Task. Define task-level success, trajectory and step checks, an environment and task suite, repeated trials, and a staged deployment decision. Explain how offline results will be compared with observed production behavior. Constraints. Separate answer quality, tool correctness, authorization, and external-state outcomes. Control environment versions and budgets; account for stochasticity, retries, long-horizon failures, and consequential actions without executing unsafe tests.

20:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: LLM-team and agent-platform interviews.

Evaluate whether a tool-using agent completes authorized tasks reliably across entire trajectories, not just individual generated answers.

Problem

Context. A hypothetical agent plans multiple steps, calls tools, observes results, and may create persistent external effects. A candidate model or orchestration change must be assessed before wider release.

Task. Define task-level success, trajectory and step checks, an environment and task suite, repeated trials, and a staged deployment decision. Explain how offline results will be compared with observed production behavior.

Constraints. Separate answer quality, tool correctness, authorization, and external-state outcomes. Control environment versions and budgets; account for stochasticity, retries, long-horizon failures, and consequential actions without executing unsafe tests.

Why agent eval is harder

A chat model produces one output per input. An agent produces a trajectory: plan, tool calls, intermediate results, final answer. Each step can fail, and failures compound. Standard end-to-end eval can’t tell you whether the planner was wrong, the tool returned bad data, or the synthesis was bad.

A passing test and an unauthorized write can occur in the same trial

A passed test and an unauthorized write can coexistOne trial can complete its task while violating the authority boundary. The combined result must fail the declared policy gate.Every required gate must pass on thesame route+task testxauthorityThe unauthorized write vetoes thecompleted task.
Derivation
A passing test step and a failing task outcome can coexistA fictional coding-agent trial edits the intended file but then writes an unauthorized file. The intended test passes, the task safety outcome fails, and the trial uses four tool calls. The new unauthorized-write slice returns to the fixed-budget suite; a production mismatch reopens its coverage assumptions.Fictional trial T4; same trajectory, threelensesWHAT HAPPENEDWHAT IT PROVESOutcomeintended testpassesnot sufficientfor successFailure stepwrites forbiddenfilesafety outcomeFAILResource ledger4 tool callscompare at fixedbudget
Add the unauthorized-write slice to the suite. A production mismatch reopens coverage. Passing one test does not erase the same trial's safety failure.
Check: How can two trials have the same final outcome but different trajectory failures or costs?
Read it this way: One trial can complete its task while violating the authority boundary. The combined result must fail the declared policy gate.
Can this task-test pass make the joint trial pass?

What an L4 answer sounds like

“I’d run the agent on a benchmark of tasks and measure final-answer accuracy. Compare against a baseline model.”

This treats the agent as a black box and gives you one number with no diagnostic value. You’ve used agents but haven’t debugged one.

What an L5 answer sounds like

“Agent eval needs three levels:

  1. Final-task success. Did the agent complete the task? Binary or rubric-graded. The headline number.

  2. Trajectory-level metrics. Step count, tool-call accuracy, plan adherence, recovery from intermediate failures. Surfaces where the agent is failing (planning vs tool use vs synthesis).

  3. Step-level eval. For each tool call: was it the right tool? Were the arguments correct? For each plan step: did it move toward the goal? Often graded by an LLM judge against a rubric.

Build a golden set of ~100-300 tasks stratified by difficulty and category. Evaluate at all three levels per release. Track the offline-online correlation: does success on the eval set predict success in production?”

This is L5. You’ve broken the eval into the right levels and named the operational concern.

What an L6 answer adds

“…practical things:

Cost is a first-class metric for agents. A 90% successful agent that takes 50 tool calls per task is often worse than an 85% successful agent that takes 5. Report cost per task, success at fixed cost, or Pareto curves.

Failure-mode taxonomy is the most useful artifact. Categorize failures: planning errors, tool-call errors, hallucinated tool results, infinite loops, premature termination, refusal-when-should-have-tried. Track frequencies over releases. The taxonomy itself takes weeks to build but pays back forever.

Multi-turn evals are necessary for conversational agents. Single-turn benchmarks miss issues that only appear under conversation drift. Build multi-turn scenarios with realistic user replies (often LLM-simulated, validated against real conversations).

Trajectory replay is the agent equivalent of the ‘read the outputs’ practice in LLM eval. Watch the agent solve 10 tasks per week, end-to-end. Patterns you’ll only see by watching: agents giving up too early, repeating the same failed action, asking for clarification when they shouldn’t.

Production-online vs offline gap. Real users phrase tasks differently than your eval set assumes. Sample 50 production traces per week, score them against your rubric, compare to your offline trends. If they diverge, the offline eval is overfit to its assumptions.”

Tells that get you a strong-hire vote

  • You evaluate at multiple granularities: task, trajectory, step.
  • You bring up cost per task and Pareto curves.
  • You build a failure-mode taxonomy.
  • You discuss trajectory replay as a debugging practice.
  • You track the offline-online correlation explicitly.

Tells that get you down-leveled

  • “Run a benchmark and report accuracy.”
  • No step-level eval.
  • No cost metric.
  • Treating multi-turn as the same problem as single-turn.

Common follow-up

“How do you handle stochasticity in agent eval?”

The L6 answer:

“First define the sampling unit and the claim. Repeated attempts estimate variation on the fixed suite; independently sampled tasks support a different generalization claim. Suggested counts such as 5-10 attempts are starting points, not universally sufficient release criteria.

Compare candidate and baseline on matched tasks under the same budgets, environment versions, and outcome rubric. State whether success means one attempt, eventual success after retries, or success within a total budget. Count failed attempts and retries in cost.

A binomial interval needs an appropriate independent-trial model. Repeated heterogeneous tasks are not automatically independent observations for a new-task claim. When tasks are the sampled units, preserve that clustering in uncertainty estimates, for example by resampling paired task results rather than pooling every attempt.

Temperature and seeds can reduce some sampling variation; they do not guarantee deterministic hardware, tools, or hosted execution. Report the task-level differences and uncertainty, not a single lucky run. The application-evaluation lesson supplies the separate outcome, support, and release checks.”

Predict before revealing: two tasks each run ten times produce twenty outcomes. Have you evaluated twenty independently sampled tasks?

Keep attempts and task clusters separate

No. There are twenty attempts nested within two task clusters. Those outcomes can characterize repeatability on the fixed pair, but new-task uncertainty cannot be justified by pretending there were twenty task draws. State how tasks were sampled, preserve candidate/baseline pairing, and gate consequential authorization failures separately from average success.

Related: enterprise agent-platform design, LLM Evals essay, How would you evaluate an LLM application?, and How do you handle hallucinations in production?.

Found an error or stale claim? Report a correction or send a private note.