Skip to content
mlmentorship

Two-tower vs cross-encoder: when to use which?

The recsys / search architecture decision that comes up in every retrieval interview. The right answer is 'both, in sequence.'

Published · 9 min read ·Role-specific ·Advanced

ML breadth · active recall

Practice before you read

8 minutes. Explain the mechanism, why it works, when it fails, and one alternative.

Back to my workbook How practice works

ML breadth · closed-book attempt

Two-tower vs cross-encoder: when to use which?

Explain the mechanism, why it works, when it fails, and one alternative.

Problem

Context. A retrieval service must match queries against many items, then produce a short relevance-ranked result list. Two-tower models encode queries and items separately; cross-encoders jointly process each query-item pair. Task. Compare their representation, training, retrieval, and serving trade-offs, then justify a design for the workload. Explain how a two-stage design would be evaluated if you choose one. Constraints. Account for candidate recall, latency, index freshness, memory, and training-serving consistency. Do not assume either architecture is universally more accurate; distinguish measured relevance from representational capacity.

08:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: recsys, search, and retrieval interviews.

Choose how to score query-item relevance when a large candidate collection must be searched under serving constraints.

Problem

Context. A retrieval service must match queries against many items, then produce a short relevance-ranked result list. Two-tower models encode queries and items separately; cross-encoders jointly process each query-item pair.

Task. Compare their representation, training, retrieval, and serving trade-offs, then justify a design for the workload. Explain how a two-stage design would be evaluated if you choose one.

Constraints. Account for candidate recall, latency, index freshness, memory, and training-serving consistency. Do not assume either architecture is universally more accurate; distinguish measured relevance from representational capacity.

Jointly encode only retrieved pairs; missing candidates remain unreachable

Joint encoding spends work only on the retrieved candidate setA two-tower stage retrieves K=20 from 1,000 items. The cross-encoder scores those twenty pairs; it cannot rescue an item outside them.The downstream decision sees only itsavailable identitiesCorpus: one mark twenty items1000 available identities; one mark20Joint scoring: one mark one pair20 available identities; one mark 1
Derivation
Retrieve reusable vectors, then jointly encode only the surviving pairsA toy six-item corpus has independently stored item vectors. One query vector retrieves IDs two, four, six, so only three query/item pairs receive fresh joint scoring. A token-level example pairs query red with red running shoes in I4. Missing item I3 cannot be restored by the reranker.Toy corpus N=6; retained K=3OFFLINE: six reusable item vectorsI1I2I3I4I5I6Query vector: "red shoes"vector search -> IDs {2,4,6}Only these three pairs pay for joint encodingquery + I2JOINT scorequery + I4JOINT scorequery + I6JOINT scoreExample interaction in pair I4:query "red" <-> item "red runningshoes"score can inspect tokens togetherMissing I3 cannot be restored by reranking{2,4,6}.Candidate recall bounds the final result.
All six items can reuse offline vectors, but only the three retrieved IDs get joint token interaction here. The reranker cannot recover an item outside that candidate set.
Check: Why jointly encode only K pairs? Can reranking recover an item outside those candidates?
Read it this way: A two-tower stage retrieves K=20 from 1,000 items. The cross-encoder scores those twenty pairs; it cannot rescue an item outside them.
Can joint encoding rescue an item absent from the twenty retrieved candidates?

What an L4 answer sounds like

“Two-tower is faster but less accurate. Cross-encoder is more accurate but slower. So you use two-tower when speed matters and cross-encoder when accuracy matters.”

Right idea, missing the architectural point. You’ve heard the comparison but haven’t built either.

What an L5 answer sounds like

“These solve different problems and the right answer is usually ‘both, in sequence.’

Two-tower independently encodes queries and items, allowing item-vector reuse. Search cost and recall depend on catalog size, dimension, hardware, index parameters, filters, and freshness. A single-vector interaction compresses pair detail; it does not imply a universal latency or relevance score. Keep query and item/index versions compatible, as in the two-tower mechanism.

Cross-encoder takes (query, item) as a joint input, runs them through the same model, and outputs a relevance score. Strictly more expressive because it can model cross-attention between query and item tokens. The disadvantage: you have to score every (query, item) pair at serving time, which is O(N) per query, infeasible for large catalogs.

The standard production architecture is two-stage: use a two-tower for retrieval (reduce billions to ~thousands of candidates), then a cross-encoder (or other reranker) for ranking (reorder the top thousands down to the final ~10). This gets you the speed of two-tower for the bulk of the work and the accuracy of cross-encoder where it matters.

A cross-encoder can improve bounded candidate scoring, but it cannot recover an omitted relevant item. Measure candidate recall, final relevance, cost, and latency separately; the extra stage is worthwhile only when the observed gain justifies its budget.”

This is L5. You’ve named the architectures, the trade-offs, the standard composition, and the operational reality.

What an L6 answer sounds like

“…and a few more things that come up in practice:

Representation does not uniquely determine the loss. Contrastive, pointwise, pairwise, or listwise objectives require appropriate labels and candidate sets. A two-tower objective need not be symmetric, and separately scored pairs need not be statistically independent.

The ‘right’ two-tower architecture has subtleties. Modern variants include ColBERT (multi-vector representations, one embedding per token, late interaction at scoring time), which gets some of the accuracy of cross-encoder while keeping retrieval tractable; and SPLADE (sparse representations) for hybrid lexical-dense retrieval. These bridge the gap.

Negative semantics need auditing. Hard-negative mining is a candidate intervention, not an automatic gain. A plausible retrieved item can be another valid positive; exclude or model such cases instead of training them as known negatives. Compare mining policies with matched evaluation and budgets.

Cross-encoder distillation back into two-tower is a real lever. You can train your two-tower to mimic the cross-encoder’s scores on a labeled set, gaining some of the cross-encoder’s quality without its serving cost.

Late-interaction models (ColBERT and similar) are increasingly common as a third option. Multi-vector retrieval, late interaction at scoring, a middle ground between two-tower and cross-encoder. More memory-intensive but better quality.

For RAG specifically: the standard stack is bi-encoder retrieval (top 50-100) → cross-encoder rerank (top 5-10) → LLM generation. Each stage is its own engineering and quality optimization.”

This is L6. You have connected the textbook comparison to newer methods such as ColBERT, SPLADE, and distillation, plus the practical training discipline of hard negative mining.

The tells that get you a strong-hire vote

  • You frame it as two stages, not two choices.
  • You bring up hard negative mining as the actual quality lever for two-tower.
  • You mention ColBERT or other late-interaction models as a third architecture.
  • You discuss distillation from cross-encoder to two-tower.
  • You connect to RAG: this is the same architectural decision in disguise.

The tells that get you down-leveled

  • Treating it as a binary choice (one or the other).
  • “Cross-encoder is slow” without quantifying (“would need to score every item per query, which is O(N)”).
  • No mention of reranking as a standard architectural step.
  • No knowledge of training-data subtleties (hard negatives).
  • Confusing two-tower with siamese networks (they’re related but two-tower has different towers for query and item, often with different inputs).

A common follow-up

“How would you train a good two-tower from scratch?”

The L6 answer:

“Three phases:

  1. Warm-up with in-batch negatives: train with simple contrastive loss where every other item in the batch is a negative. Gets you a reasonable starting point quickly.
  2. Hard negative mining: use the warm-up model (or a separate strong model) to mine hard negatives, items that have high similarity score to the query but are not the gold positive. Train with these as negatives. This is where quality really comes from.
  3. Optional: distillation from a cross-encoder: train a strong cross-encoder on a labeled set, score (query, candidate) pairs, then train the two-tower to match those scores. Distillation often outperforms direct supervised training.

Throughout: be careful about the eval set. Standard MTEB benchmarks are public and over-fit; you want a domain-specific eval set built from production samples (or prospective production samples).”

If you can have this conversation fluently, you’re at strong-senior depth.

Related: Designing a RAG system that actually works, Design YouTube’s recommender.

Found an error or stale claim? Report a correction or send a private note.