TF-IDF and BM25

Summary

Lexical scoring combines analyzed token matches with corpus statistics, not a direct measurement of relevance. BM25 separates diminishing frequency gain from document-length normalization; TF-IDF uses a declared weighting and vector-normalization convention. Neither lexical, dense, nor fused retrieval is guaranteed to win every query slice.

Term frequency and its problem

Raw counts can be replaced with damped weights, such as $1 + lo g f$ for positive frequency and zero otherwise. This heuristic limits growth relative to raw count; it does not prove that repeated words are relevant. The log-TF weight is still unbounded, unlike BM25’s fixed-length frequency factor.

Inverse document frequency

One unsmoothed TF-IDF convention uses natural-log inverse document frequency:

idf (t) = lo g \frac{N}{df ( t )},

Here $1 \leq df (t) \leq N$ counts indexed documents containing the analyzed term. Dot-product sums and cosine-normalized TF-IDF vectors define different scorers; state which is intended.

BM25: the production formula

For $k_{1} > 0$ , $0 \leq b \leq 1$ , and positive average field length, use:

BM25 (q, d) = t \in q \sum idf (t) \cdot \frac{f ( t , d ) ( k _{1} + 1 )}{f ( t , d ) + k _{1} ( 1 - b + b \frac{∣ d ∣}{avgdl} )},

$f (t, d)$ counts term occurrences; $∣ d ∣$ and $a v g d l$ use the index’s field-length convention.
$k_{1}$ controls saturation; the displayed factor approaches $k_{1} + 1$ at fixed length.
$b$ controls the length adjustment; $b = 0$ removes it.

Sum each distinct analyzed query term once here; repeated-query weighting is a separate convention. An absent term contributes zero, including empty fields. Robertson’s IDF $lo g ((N - df + 0.5) / (df + 0.5))$ can be negative. Lucene’s log-one-plus variant is $lo g (1 + (N - df + 0.5) / (df + 0.5))$ and stays positive for indexed terms.

Fix $k_{1} = 1.2$ , $b = 0.75$ , and positive IDF one. At average length, frequencies $1, 3, 10$ give factors $1, 11/7, 55/28$ , approaching ceiling $2.2$ . At frequency three, half/average/double length gives $1.76, 1.571429, 1.294118$ .

One query term, two controls

Repeated matches saturate while extra document length can reduce their score

Derivation

Read it this way: the upper bars isolate frequency gain below the 2.20 ceiling. The lower bars hold frequency fixed while length changes. Multiply by the chosen term's IDF. Original calculations follow the Stanford IR textbook, Robertson and Zaragoza, and Lucene's documented variant.

Check: Why does term-frequency gain diminish? Why does a longer document reduce the factor at fixed frequency?

Read it this way: With k1=1.2 and b=0.75, the term-frequency factor saturates. Doubling relative document length reduces the factor at fixed frequency.
Do extra nonmatching words improve the same four-match score?

Remove Length Normalization

New task: Stop penalizing field length under a justified field-specific requirement.

Picture and change: Set $b = 0$ , keeping the other factors fixed.

Check: All three frequency-three documents now contribute $11/7$ under IDF one, regardless of their length.

Use when: Evaluate the changed field semantics and query slices; normalization is not universally beneficial.

Add Semantic Coverage

New task: Retrieve paraphrases sharing no analyzed query token.

Picture and change: Compare query expansion or dense candidates, then evaluate fusion and exact-identifier protection.

Check: Under RRF with constant sixty, a lexical-only rank-one document scores $1/61$ . A document ranked second lexically and first densely scores $1/62 + 1/61$ , potentially displacing it at a top-one cutoff.

Use when: Candidate coverage, not an assumed superiority of hybrid retrieval, is the measured bottleneck.

TF-IDF vs BM25

	TF saturation	Length normalization	Tunable	Status
TF-IDF	Convention-dependent; log-TF is unbounded	Optional vector normalization	Configurable weighting/analysis	Lexical vector scorer
BM25	Displayed factor tends to $k_{1} + 1$	Explicit via $b$	$k_{1}, b$ , field policies	Lexical ranking scorer

Their scores depend on the analyzer, corpus, field statistics, and normalization policy.

Lexical vs dense (and why hybrid wins)

	Lexical (BM25)	Dense (embeddings)
Matches	Analyzed tokens	Learned representation similarities
IDs/paraphrases/domain shift	Evaluate task-specific behavior	Evaluate task-specific behavior
Index	Typically inverted postings	Exact or approximate vector search

Rank-based RRF and weighted-score fusion use different assumptions; raw scores are not automatically comparable across retrievers. Reranking cannot recover candidates absent from its input set.

What an interviewer expects you to say

Reconstruct the six bar values, name the IDF/analyzer convention, and identify whether candidate coverage or ordering needs improvement.

Common confusions

Rarity as relevance. Corpus rarity does not establish semantic importance.
Implicit synonym guarantees. BM25 matches analyzed terms; explicit expansion/analyzer policies can introduce synonyms.
Negative-IDF confusion. A bounded positive TF factor does not imply increasing total score when its multiplier is negative.