Skip to content
mlmentorship

How do you A/B test a chatbot?

Chatbot A/B testing has all the hard parts of regular A/B testing plus delayed feedback, conversational state, and metrics that are hard to define.

Published · 7 min read ·Core ·Intermediate

Product & experimentation · active recall

Practice before you read

20 minutes. Define the decision first, then design the measurement and enumerate validity threats.

Back to my workbook How practice works

Product & experimentation · closed-book attempt

How do you A/B test a chatbot?

Define the decision first, then design the measurement and enumerate validity threats.

Problem

Context. A candidate chatbot differs from the current version, and users may exchange several dependent messages across sessions. The product needs evidence of better task outcomes before a broad release. Task. Choose a randomization unit, primary outcome, guardrails, exposure logging, sample-size assumptions, and a launch rule. Explain how session continuity, carryover, user feedback, and delayed outcomes affect interpretation. Constraints. Do not count correlated turns as independent users or substitute thumbs-up rates for task success without validation. Predeclare stopping and harm boundaries, preserve privacy, and distinguish assignment from the treatment actually delivered.

20:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: LLM-team and product-ML interviews.

Design an online chatbot experiment whose randomization, outcomes, and safety checks reflect complete user conversations.

Problem

Context. A candidate chatbot differs from the current version, and users may exchange several dependent messages across sessions. The product needs evidence of better task outcomes before a broad release.

Task. Choose a randomization unit, primary outcome, guardrails, exposure logging, sample-size assumptions, and a launch rule. Explain how session continuity, carryover, user feedback, and delayed outcomes affect interpretation.

Constraints. Do not count correlated turns as independent users or substitute thumbs-up rates for task success without validation. Predeclare stopping and harm boundaries, preserve privacy, and distinguish assignment from the treatment actually delivered.

What an L4 answer sounds like

“Send some traffic to the new model, some to the old, compare thumbs up/down or session length.”

This is right at the slogan level. Each piece has problems. You’ve heard A/B testing but haven’t tested a stateful product.

What an L5 answer sounds like

“Three things to get right:

1. Unit of randomization. Per-request randomization gives the most statistical power but breaks conversation continuity (the same user might get model A and B in alternating turns, which corrupts both). Per-conversation or per-user randomization keeps state coherent but reduces N.

2. Metric definition. Thumbs up/down are sparse and biased toward dissatisfaction. Better signals:

  • Implicit: regeneration rate, edit-distance from response to user follow-up, copy-rate, abandonment within N turns
  • Outcome: did the user complete their task (depends on what the bot does), did they return next session
  • Explicit (sparse but high-signal): post-conversation satisfaction surveys, NPS

3. Statistical setup: pre-commit to primary metric and sample size. Account for novelty effects (run for at least 2-4 weeks). Cut by user segment and by query type to catch slice regressions.”

Unit of randomization

Keep one conversation on one model to compare its outcome

Keep the whole conversation in one experimental cellTurn-by-turn switching mixes model effects inside one outcome. Conversation-level assignment keeps this illustrative three-turn conversation consistently on model A.One conversation encloses onemeasured outcomeConversation assignment: A, A, Aremain togetherAAAone outcomeTurn switching: A, B, A contaminateone outcomeABAone outcome
Derivation
Per-request versus per-conversation chatbot assignment Two panels show the same three-turn conversation. In the first, each request is randomized separately, producing Model A, then Model B, then Model A; a dashed warning states that the conversation outcome mixes both treatments. In the second, the conversation is assigned once to Model B, so all three turns use Model B and the outcome is attributable to one treatment. PER REQUEST: assign again at each turn USER TURN MODEL 1 · “Plan a trip” 2 · “Make it cheaper” 3 · “Book option two” Model A Model B Model A MIXED EXPOSURE One conversation outcome cannot isolate A or B. PER CONVERSATION: assign once Conversation 42 → Model B USER TURN MODEL 1 · “Plan a trip” 2 · “Make it cheaper” 3 · “Book option two” Model B Model B Model B COHERENT EXPOSURE · outcome belongs to B
follow each row from the user turn to its assigned model. Re-randomizing requests produces an A–B–A conversation, so a conversation-level outcome mixes treatments. Assigning conversation 42 once keeps every turn on B and makes its outcome attributable to B.
Check: Which assignment lets one conversation-level outcome be attributed to one model?
Read it this way: Turn-by-turn switching mixes model effects inside one outcome. Conversation-level assignment keeps this illustrative three-turn conversation consistently on model A.
Can this one conversation’s outcome isolate A after its middle turn uses B?

This is L5. You’ve named the three concrete problems and the typical fixes.

What an L6 answer adds

“…practical things that bite:

Implicit metrics drift over time. Users change behavior in response to model behavior. A model that ‘wins’ at week 1 because it surfaces options the old one didn’t may ‘lose’ at week 8 because users adapted to the old patterns. Track both short-term and long-term metrics; gate on long-term where feasible.

Cost is a first-class metric. A 1% lift on satisfaction at 3x the inference cost is usually a regression. Report cost per conversation alongside quality metrics; gate launches on Pareto improvement.

Heterogeneous treatment effects matter. Power users react very differently than casual users. Slice your A/B by user-tier, query-complexity, language, region. The aggregate average can hide a regression on the most valuable cohort.

Sample ratio mismatch (SRM) check is mandatory. If your randomization is broken, the assignment ratio won’t match expected; you’ll see ghost effects from cohort imbalance, not from the model.

Multiple comparisons: looking at 30 metrics, some will be ‘significant’ by chance. Pre-specify a primary metric. Treat the rest as guardrails (one-sided tests with stricter thresholds) or apply Bonferroni / Benjamini-Hochberg.

Holdback experiments for long-term metrics: keep a small fraction of users on the old model permanently to detect slow drift in the new model that A/B tests miss.”

Tells that get you a strong-hire vote

  • You name the unit-of-randomization trade-off explicitly.
  • You discuss implicit vs explicit signals and their biases.
  • You bring up novelty effects with a 2-4 week observation horizon.
  • You mention cost as a first-class metric.
  • You insist on slice analysis and SRM checks.

Tells that get you down-leveled

  • “Just measure thumbs up/down.”
  • Per-request randomization without flagging the state-breaking issue.
  • No discussion of statistical power.
  • No mention of cost.

Common follow-up

“What if your primary metric is statistically insignificant but you have positive directional signal across all guardrails?”

The L6 answer:

“First, check power: did you have enough N to detect the size of the effect that matters to you? If yes, the result is genuinely null and you should not ship; positive directional signal across guardrails is what you’d see by random chance roughly half the time. If no, run longer. Don’t rationalize a non-significant primary as a win because the secondaries ‘look right’; this is one of the most common decision errors in A/B-driven product teams.”

Related: A/B testing for ML systems, How would you evaluate an LLM application?.

Found an error or stale claim? Report a correction or send a private note.