Naive Bayes

Summary

Naive Bayes models $p (y ∣ x) \propto p (y) \prod_{j} p (x_{j} ∣ y)$ , assuming features $x_{j}$ are conditionally independent given the class $y$ . Trained by counting (closed-form MLE for each conditional).

Naive Bayes is the cheapest possible probabilistic classifier. Closed-form MLE, $O (n d)$ training, $O (d)$ prediction, no hyperparameter tuning. Despite the obviously wrong independence assumption, it works remarkably well as a baseline on:

Text classification (spam filtering, topic categorization, sentiment): bag-of-words features.
Tiny-data classification where logistic regression overfits.
Initial baselines that should be beaten before claiming victory with a fancier model.

It is also conceptually important. The canonical example of a generative classifier (model $p (x, y)$ ) versus the discriminative logistic regression (model $p (y ∣ x)$ directly).

The model

By Bayes’ rule:

p (y ∣ x) = \frac{p ( y ) p ( x ∣ y )}{p ( x )} .

The “naive” assumption: $p (x ∣ y) = \prod_{j} p (x_{j} ∣ y)$ . Then for prediction:

\overset{y}{^} = ar g y max p (y) j \prod p (x_{j} ∣ y) = ar g y max lo g p (y) + j \sum lo g p (x_{j} ∣ y) .

Sum logs to avoid underflow.

Variants

Variant	$p (x_{j} ∣ y)$	Use case
Multinomial	Multinomial over token counts	Text (bag-of-words)
Bernoulli	Bernoulli (binary) per feature	Text (presence/absence)
Gaussian	$N (μ_{j, y}, σ_{j, y}^{2})$	Continuous features
Categorical	Categorical (multinoulli)	Discrete features

Training

Just count.

For multinomial NB on text:

$p (y = c) = count (c) / N$ .
$p (word w ∣ y = c) = \frac{count ( w in class c ) + α}{total tokens in class c + α V}$ (with Laplace / additive smoothing $α$ , typically 1.0; $V$ = vocabulary size).

Without smoothing, any unseen word in a class gives $p (x ∣ y) = 0$ and $p (y ∣ x) = 0$ , breaking inference.

Why the independence assumption isn’t fatal

Even though words are clearly correlated, naive Bayes can still rank classes correctly. The independence assumption gives biased probability estimates (overconfident. Predicted probabilities tend to be near 0 or 1) but the argmax is often right.

For pure classification accuracy, NB is competitive. For calibrated probabilities, prefer logistic regression with proper regularization.

Learning objective

Multiplying evidence ratios requires conditional independence within each class

Derivation

Model score: count every token separately

For two equally likely classes, suppose a text model learned these illustrative likelihood ratios for a document containing “free prize.”

Prior odds	1 : 1
“free”	× 8
“prize”	× 7
NB odds	56 : 1 spam

1 × 8 × 7 = 56 → choose spam

Reality check: evidence can overlap

“Free” and “prize” may come from the same promotional phrase, so their occurrences can remain correlated even after the class is known.

NB assumes	two conditionally independent contributions
Text may contain	one underlying signal expressed twice
Class decision	spam can still win
Probability	56 : 1 can overstate certainty

use the argmax; distrust uncalibrated probability

Read it this way: multiply down the left ledger first: naive Bayes gives each observed token its own likelihood-ratio contribution, so the class score can separate sharply. Then read the right panel: if correlated tokens repeat one underlying signal, multiplying them as independent evidence exaggerates the odds. The winning class can remain correct even when the reported probability is overconfident. The numbers are an original illustrative calculation, checked against the Introduction to Information Retrieval treatment of multinomial naive Bayes and scikit-learn's official model guidance.

Check: What independence assumption permits multiplying 8 by 7? Why do those numbers alone not identify the correct joint odds?

Read it this way: Prior odds 1:4 become 1:1 after likelihood ratio 2 and another ratio 2. Correlated evidence cannot be multiplied as independent evidence without justification.
What assumption permits the second likelihood ratio to multiply the first?

Generative vs. discriminative

Naive Bayes models the joint $p (x, y)$ . Logistic regression models $p (y ∣ x)$ directly. Asymptotic results (Ng & Jordan, 2002):

For small $n$ , NB usually wins (less variance from the strong assumption).
For large $n$ , logistic catches up and surpasses NB (the assumption hurts at large scale).

Where it shows up in 2026

Spam filters in low-resource embedded systems.
Quick text baselines before training a transformer.
Document filtering in retrieval pipelines (cheap pre-filter).

For most modern NLP, neural classifiers dominate. NB persists in resource-constrained settings and as a reliable benchmark.

Common pitfalls

Forgetting to smooth. Without Laplace smoothing, any test document with a vocabulary token never seen in a class gets that class’s posterior set to 0.
Using NB on highly correlated features. Probabilities become very poorly calibrated; predicted class can still be okay but never trust the probability.
Mixing variants. Multinomial NB on continuous features is wrong; use Gaussian NB (or discretize first).
Comparing NB on raw counts vs. tfidf vs. binarized. Different preprocessing changes the model class; compare apples to apples.