How would you build evals for a coding assistant?

Asked in: LLM-team interviews at companies building coding products.

Evaluate a coding assistant’s usefulness and risks using representative tasks rather than a single code-generation benchmark score.

Problem

Context. A hypothetical coding product proposes completions, explains code, or edits repositories, and its model is changing. You must decide whether a candidate version is better for developers.

Task. Design offline tasks, verification checks, product measurements, slice analysis, and a staged release decision. Explain which failures unit tests, acceptance rates, and standard coding benchmarks cannot independently establish.

Constraints. Separate product modes, prevent evaluation leakage, and preserve private repository boundaries. Measure correctness, latency, security, and downstream developer work; accepting a suggestion does not prove that it is correct.

What an L4 answer sounds like

“Use HumanEval and MBPP. Run the generated code and check if it passes the tests.”

This is leaderboard eval, useful for vendor selection but useless for a product. You’ve consumed benchmarks but haven’t built one.

What an L5 answer sounds like

“Code eval has a special property: ground truth is executable. Lean into it.

Three layers:

1. Functional checks via tests. Bind the task, repository/environment version, and test oracle. Passing establishes the tested properties, not correctness outside that coverage. Public suites and internal user-derived tasks have different sampling and contamination risks; neither format alone makes the oracle complete.

2. Code quality. Use static analysis, a specific review rubric, and human-reviewed samples for properties outside the functional oracle. LLM judges can miss or invent defects; validate their agreement and failure modes instead of assuming they catch everything else.

3. Real-task evaluation. Acceptance, edits, and completion time are candidate product signals. Define the task endpoint, include review and repair, and use an appropriate controlled comparison before attributing productivity or business value to the assistant.

Public benchmarks are useful for vendor selection. Real-task metrics are what you ship on.”

Learning objective: choose the evaluation layer that can answer a coding-assistant claim, and state what evidence that layer still cannot provide.

This is L5. You’ve named the three layers and distinguished benchmarks from product evals.

What an L6 answer adds

“…practical things:

Test contamination is rampant. HumanEval, MBPP, and many other benchmarks are in training data. Inflated benchmark scores tell you nothing about generalization. Use newer or held-out benchmarks (LiveCodeBench rotates problems), or build internal benchmarks from non-public sources.

Multi-file, multi-step tasks are where benchmarks fall behind. SWE-bench (full repo task completion) is closer to real product use than HumanEval. The gap between HumanEval scores and SWE-bench scores is substantial: a model can be at the top of HumanEval and useless on real codebases.

Tool use matters. Modern coding agents use search, file read/write, test execution, debugger. Eval should test the agent, not just the model: can it find the right files, write the right tests, run them, iterate?

Per-language slicing matters. A model can be excellent at Python, mediocre at Java, terrible at Rust. Aggregate scores hide this. Slice by language at minimum; by codebase pattern (web vs systems vs ML) for maturity.

Inventory failure modes rather than assuming a winner. Invented APIs are one class; incorrect requirements, incomplete edits, regressions, authorization errors, and inadequate tests are others. Measure their frequencies on representative tasks. Execution and static checks have coverage limits.

For online metrics: acceptance can precede later rewrites, and stopped editing can mean completion, abandonment, or interruption. Use an explicit verified task outcome and total developer effort rather than treating either proxy as a universal productivity measure. The application-evaluation framework separates success evidence from release criteria.”

Tells that get you a strong-hire vote

• You distinguish benchmarks (vendor selection) from product metrics (shipping).
• You bring up test contamination as a benchmark gotcha.
• You mention SWE-bench as a more realistic evaluator.
• You measure invented APIs and other failure modes instead of assuming their relative frequency.
• You discuss slicing by language and codebase type.

Tells that get you down-leveled

• HumanEval as the main signal.
• No mention of contamination.
• No tool-use evaluation for agents.
• Treating “model” and “product” as interchangeable.

Common follow-up

“How would you measure code quality beyond passing tests?”

The L6 answer:

“Three signals: (1) static analysis (linter scores, complexity metrics, type-check pass), (2) LLM judges scoring code on a rubric (readability, idiomaticity, test coverage of generated code), validated against human judgments, and (3) human review on a sample. Code quality is fuzzier than correctness, so absolute scores matter less than tracking deltas across releases. A regression in lint warnings or in judge scores is a signal worth investigating even if functional correctness is unchanged.”

Related: LLM Evals essay, How would you evaluate an LLM application?, Evaluate an agent.