Skip to content
mlmentorship

How do you scope an ambiguous problem?

Scoping is a core senior skill. The interview tests how you reduce ambiguity before choosing a model or system design.

Published · 6 min read ·Core ·Mixed

Project deep-dive · active recall

Practice before you read

20 minutes. Give a two-minute overview, then pressure-test decisions, failures, evidence, and ownership.

Back to my workbook How practice works

Project deep-dive · closed-book attempt

How do you scope an ambiguous problem?

Give a two-minute overview, then pressure-test decisions, failures, evidence, and ownership.

Problem

Context. A requester attributes poor generated answers to model size, while selected complaint cases have the invented cause assignments below. Task. Clarify the desired outcome, identify the first deciding uncertainty, and propose a small investigation with predicted results. State a success criterion, scope boundary, owner, and conditions for changing direction. Constraints. The complaint sample is selected, not a representative estimate of all traffic. Do not convert its counts into a causal claim that one intervention solves the whole product problem. A requester wants a larger generator because answers are poor. In this fictional triage, 50 selected complaint cases have disjoint assigned causes: | Assigned cause | Cases | | --- | ---: | | Missing source information | 30 | | Retrieval omission | 15 | | Answer synthesis | 5 | What should the first investigation change? Which conclusion cannot be drawn from these counts?

20:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.
Reading depth

Turn an ambiguous request for a larger model into a bounded investigation with an explicit decision and success criterion.

Problem

Context. A requester attributes poor generated answers to model size, while selected complaint cases have the invented cause assignments below.

Task. Clarify the desired outcome, identify the first deciding uncertainty, and propose a small investigation with predicted results. State a success criterion, scope boundary, owner, and conditions for changing direction.

Constraints. The complaint sample is selected, not a representative estimate of all traffic. Do not convert its counts into a causal claim that one intervention solves the whole product problem.

A requester wants a larger generator because answers are poor. In this fictional triage, 50 selected complaint cases have disjoint assigned causes:

Assigned causeCases
Missing source information30
Retrieval omission15
Answer synthesis5

What should the first investigation change? Which conclusion cannot be drawn from these counts?

Picture: Find the Decision-Changing Unknown

Use a bounded check to decide scope, not the complaint sample alone

A complaint sample proposes a scope; a bounded check tests itThe observed complaints concern one slice. A discriminating sample from other slices decides whether the proposed scope should widen.Each claim needs its own observedevidencecomplaint slice+other slices?
Derivation
A complaint sample suggests a question; a bounded check decides the next scopeA fictional selected complaint sample contains thirty source, fifteen retrieval, and five synthesis complaints. It is not a representative prevalence estimate. The named missing fact is whether permitted, current source evidence exists for ten sampled cases. A predeclared one-day availability check determines a bounded v1 source repair or a stop, rather than automatically proposing a model change.Fictional selected sample: 30 source / 15retrieval / 5 synthesisMissing fact: permitted, currentevidence exists for 10 sampledcases?One-day availability checkpredeclare evidence ruleselected sample != prevalenceEVIDENCE FOUNDNOT AVAILABLEv1: source repairmodel tuningdeferredowner + reviewdatestop thisproposalor change scoperecord missingfact
The selected complaints do not establish population prevalence. Resolve the named availability question within the agreed timebox. Its receipt changes scope or stops the proposal.
Check: What exact fact can change the scope? Which pre-agreed result would stop a model change?
Read it this way: The observed complaints concern one slice. A discriminating sample from other slices decides whether the proposed scope should widen.
What observation would justify widening beyond the complaint slice?

Work the Triage

. Under this taxonomy, a generator-only change directly targets the five synthesis cases. The other 45 motivate source/retrieval investigation, not an assumed larger-model fix.

First validate the categories and source availability against an authoritative version. If required facts are absent but permitted to obtain, scope ingestion; if already available, revisit retrieval and triage. If necessary inputs cannot lawfully be obtained, narrow the task or stop.

The sample estimates neither overall failure prevalence nor causal improvement. Real failures can overlap; the disjoint categories are an exercise assumption.

Make the Decision Artifact

Name the affected population, outcome/guardrails, inputs/outputs/labels, data quality, owner, timebox and deciding uncertainty. Record in-scope work, explicit non-goals, deferred work and the result that means proceed, rescope or stop. Agree a decision date and review authority; duration depends on risk and evidence cost, not a universal week.

Change 1: Labels or Logging Are Missing

New task. Reliable outcome evidence does not exist.

Picture and change. Scope instrumentation, representative labeling or a smaller decision first.

Check. A convenient proxy does not automatically identify the intended outcome.

Use when. Evidence availability is the biggest unknown.

Change 2: A Quality Promise Is Demanded

New task. Stakeholders request a guarantee before discovery.

Picture and change. Give conditional ranges, assumptions, a decision date and staged scope; escalate incompatible constraints honestly.

Check. Do not say yes externally while hiding an incompatible internal plan.

Use when. Commitment pressure conflicts with unresolved uncertainty.

Failure: Architecture Without a Stop Rule

An elaborate design does not define non-goals, required evidence or when discovery should end.

Transfer Check

When is coding a valid first scoping action?

Reasoned Answer

When a bounded prototype targets a named uncertainty and its result changes scope or allocation. It need not build the complete system.

Six Scoping Steps

  1. Ask requesters about actual users, outcomes and failure costs.
  2. Define the decision, inputs, outputs, truth and evaluation; allow non-ML.
  3. Inspect data availability, quality, labels and permission.
  4. Compare the simplest reasonable rule/model baseline.
  5. Identify shift, noise, regulation, cost, latency and other invalidating risks.
  6. Agree a bounded one-page decision with the responsible stakeholders.

Use a Real Incident

State your actual negotiation, framing and test contribution, what changed and what remained uncertain. The hypothetical triage is not a story to claim as personal experience.

Check the Scope

Can another person reconstruct the boundary, baseline, biggest unknown, test, owner and end date? Make v1, later work and exclusions explicit; document open questions instead of concealing them.

Missing Evidence

Add the missing measurement, risk, non-goal or decision rule before assuming model choice answers the request.

Common follow-up

If asked for quality before discovery, state assumptions and the evidence needed to narrow the range. Distinguish an estimate, a target and an accepted commitment.

Sources

Related: The 5 things every applied scientist interview is testing for, senior through senior-principal ML scope, How do you decide what to work on?.

Found an error or stale claim? Report a correction or send a private note.