Skip to content
mlmentorship

Negative sampling strategies: what actually matters

Choice of negatives often matters more than choice of model. The senior answer ranks the strategies (in-batch, hard, BM25-mined, model-mined) and explains the trade-offs.

Published · 6 min read ·Role-specific ·Advanced

ML breadth · active recall

Practice before you read

8 minutes. Explain the mechanism, why it works, when it fails, and one alternative.

Back to my workbook How practice works

ML breadth · closed-book attempt

Negative sampling strategies: what actually matters

Explain the mechanism, why it works, when it fails, and one alternative.

08:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

30-second answer map

Visual first · depth when needed

Distinguish negative hardness from label validity: mine candidates the current model confuses with the positive, but apply a negative gradient only after evidence says the candidate is truly non-relevant.

Preparing the visual…

Asked in: recsys, retrieval, and embedding-training interviews.

The L4 candidate samples random items as negatives. The L6 candidate explains why hard negatives dominate quality and how to mine them without breaking training.

Why negatives matter

In contrastive training (two-tower retrieval, embedding learning), the model sees one positive (the query and its true match) and N negatives per training example. The model learns to push positive scores up and negative scores down. The choice of negatives determines what the model learns to distinguish.

Random negatives are easy to push apart; the model trivially scores them low and learns little. Hard negatives, ones that look like positives but aren’t, force the model to learn fine-grained distinctions.

Learning objective

Separate “hard for the model” from “safe to label negative.”

Three negative candidates with different hardness and label validity For the query capital of Australia, a random passage about photosynthesis has low model similarity and is a verified nonmatch, so pushing its score down is correct but gives a weak gradient. A mined passage saying Sydney is Australia's largest city has high similarity but does not answer the query, so pushing it down gives a useful strong gradient. A mined passage saying Canberra is Australia's capital has the highest similarity and is actually relevant despite lacking a label. Treating it as negative would push a valid answer down, so it must be excluded or reviewed. The diagram concludes that mining selects by model confusion while a separate relevance gate authorizes the negative label. QUERY “What is the capital of Australia?”known positive: Canberra CANDIDATE SOURCE + PASSAGETRAINING DECISION RANDOM · LOW SIMILARITY“Photosynthesis converts sunlight…”VERIFIED NONMATCH ✓PUSH DOWNcorrect · weak signal BM25 / MODEL-MINED · HIGH SIMILARITY“Australia's largest city is Sydney…”VERIFIED NONMATCH ✓PUSH DOWNcorrect · strong signaluseful hard negative: plausible, but does not answer MINED · HIGHEST SIMILARITY“Canberra is Australia's capital…”UNLABELED, ACTUALLY RELEVANT ✕DO NOT PUSHexclude or reviewfalse negative: hardness came from being a valid answer MINE BY CONFUSION · LABEL BY RELEVANCE
Read it this way: compare the last two rows. Both rank near the query, so both are hard for the current model; only the Sydney passage is known not to answer it. Pushing that verified nonmatch down teaches a fine distinction. Pushing the unlabeled Canberra passage down teaches the opposite of the task. Mine by similarity, then use labels, answer checks, or a stronger teacher to filter before assigning the negative gradient. Original schematic checked against Dense Passage Retrieval, RocketQA, and the Sentence Transformers MS MARCO training guide.

What an L4 answer sounds like

“Sample random items from the corpus as negatives.”

Right baseline, missing the most important quality lever. You’ve trained one retrieval model, the textbook way.

What an L5 answer sounds like

“Several strategies, in order of typical effectiveness:

  1. In-batch negatives: for each (query, positive) in a batch, treat all other positives in the batch as negatives. Free, parallelizable. Good baseline.

  2. Random negatives from corpus: sample uniformly. Cheap but easy; model learns coarse distinctions.

  3. BM25-mined hard negatives: for each query, retrieve top-K candidates with BM25, treat non-relevant ones as hard negatives. They have lexical overlap but aren’t the answer; force the model to learn semantic precision.

  4. Model-mined hard negatives: use the current model (or an earlier checkpoint) to retrieve candidates; non-positive top hits are hard negatives. Requires periodic re-mining as the model improves.

  5. Curriculum: start with easy (random) negatives, progressively add harder ones.

The biggest single quality lever is moving from random to BM25-mined hard negatives. Subsequent gains from model-mined and curriculum are smaller.”

This is L5. Five strategies, ranked by impact.

What an L6 answer adds

“…practical things:

Too-hard negatives break training. If the negatives are actual positives that happen not to be labeled (false negatives), gradients pull the model in conflicting directions. Symptoms: training loss plateaus or diverges. Mitigation: filter mined negatives by label coverage, or use a margin loss that’s robust to label noise.

Negative count per positive matters. More negatives per positive (large batch contrastive, MoCo-style queue, large negative sample) consistently improves quality up to a saturation point. Engineering effort to enable larger negative pools (gradient accumulation, queue-based negatives) usually pays off.

Distillation from a stronger model into a two-tower can replace negative mining for some use cases. Train the two-tower to mimic a cross-encoder’s scores on (query, candidate) pairs. The cross-encoder implicitly handles the hard-negative problem.

For LLM embedding training, the modern recipe (E5, BGE, NV-Embed) uses a mix: in-batch negatives + hard mined negatives + a contrastive loss + sometimes a knowledge-distillation loss from a teacher cross-encoder. The exact weighting matters less than having all three sources.

Domain matters a lot. Code retrieval, legal retrieval, and conversational retrieval each have different ‘hard’ patterns. Mine domain-specific hard negatives; don’t expect general-purpose techniques to transfer cleanly.”

Tells that get you a strong-hire vote

  • You name at least four strategies and rank them by typical impact.
  • You bring up BM25-mined hard negatives as the highest-leverage step.
  • You mention false-negative leakage as the failure mode of aggressive mining.
  • You discuss distillation from cross-encoders as an alternative.
  • You acknowledge larger negative pools as a separate quality lever.

Tells that get you down-leveled

  • “Random negatives” with no further detail.
  • Suggesting in-batch negatives are the goal rather than the baseline.
  • No discussion of false-negative leakage.
  • No knowledge of curriculum or distillation.

Common follow-up

“How would you mine hard negatives without polluting your training set with false negatives?”

The L6 answer:

“Three patterns. (1) Mine candidates with the model, then filter out any candidate that has a known label (positive or negative) from the labeled set. (2) Use a stronger model (cross-encoder, larger LM) to score the mined candidates and exclude those scoring above a threshold (likely false negatives). (3) Use a margin loss (margin > 0 between positive and negative scores) that’s somewhat tolerant of weak negatives. In practice, (1) + (3) is the common recipe. (2) is heavier but worth it for high-stakes domains.”

Related: Two-tower vs cross-encoder: when to use which?, Designing a RAG system that actually works, Cross-entropy and softmax.

Found an error or stale claim? Report a correction or send a private note.