Maximum likelihood estimation

Summary

Maximum likelihood chooses a parameter that maximizes the probability or density assigned to the observed data. The data stay fixed while candidate parameters vary. For independent observations with a common model $p (x ∣ θ)$ ,

\hat{θ}_{MLE} = ar g θ max i = 1 \prod n p (x_{i} ∣ θ) = ar g θ max i = 1 \sum n lo g p (x_{i} ∣ θ) .

The logarithm preserves the maximizer while turning products into sums. For continuous observations, the factors are densities, not probabilities of exact values. Dependent data require their joint or conditional likelihood rather than an unjustified independent product.

For seven successes and three failures under an independent Bernoulli model,

L (p) = p^{7} (1 - p)^{3}, ℓ^{'} (p) = \frac{7}{p} - \frac{3}{1 - p} .

Setting the derivative to zero gives $7 (1 - p) = 3 p$ , hence $p = 0.7$ . The second derivative is negative for $0 < p < 1$ , so this is the unique maximum for these mixed outcomes. The diagram rescales likelihood by its peak only to make the shape readable.

Learning objective

Hold the observations fixed while comparing candidate probabilities

Derivation

Read it this way: keep the observations fixed while varying p. Seven successes contribute p⁷; three failures contribute (1 − p)³. Their product peaks at 0.7. Relative likelihood is a parameter score, not a normalized parameter distribution.

Check: What changes as the marker moves along the curve? Why is the curve not by itself a posterior distribution over p?

Read it this way: For three successes in five independent trials, likelihood is p^3(1-p)^2. Its maximum occurs at p=3/5, not at a changing data sample.
Why do the two failures penalize the 0.9 candidate?

Properties

Under appropriate identification, sampling, regularity, and existence conditions, MLE can be consistent and asymptotically normal:

n (θ - θ^{*}) \Rightarrow N (0, I (θ^{*})^{- 1}) .

Here $I$ is Fisher information per observation and must be nonsingular for this expression. Efficiency statements compare regular estimators under their stated asymptotic framework; they do not rule out every pointwise superefficient estimator. Small samples, boundary parameters, growing dimension, or misspecification require separate analysis.

Common cases

Model	MLE solution
Gaussian mean (known $σ^{2}$ )	$\overset{μ}{^} = \overset{x}{ˉ}$
Gaussian variance	$\overset{σ}{^}^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}$ (biased; sample-variance uses $n - 1$ )
Bernoulli ( $p$ )	$\overset{p}{^} = fraction of successes$
Categorical	empirical class frequencies
Linear regression with independent equal-variance Gaussian noise	Least squares; the displayed inverse formula needs full column rank
Logistic regression	Iterative optimization; complete separation can prevent a finite optimum

Connection to cross-entropy and KL

For categorical $p_{θ} (x)$ and empirical distribution $\overset{p}{^}_{data}$ , the negative log-likelihood divided by $n$ is

- \frac{1}{n} i \sum lo g p_{θ} (x_{i}) = H (\overset{p}{^}_{data}, p_{θ}) = H (\overset{p}{^}_{data}) + KL (\overset{p}{^}_{data} ∥ p_{θ}) .

The empirical entropy is constant as model parameters vary, so both objectives select the same optimum. Gaussian regression produces squared-error minimization when its noise-variance and independence assumptions are appropriate.

MLE vs MAP

Add Prior Evidence

New task: Incorporate a declared prior rather than reporting only the likelihood maximizer.

Picture and change: A $Beta (2, 2)$ prior multiplies the Bernoulli likelihood by a factor proportional to $p (1 - p)$ . The posterior is $Beta (9, 5)$ .

Check: Its interior mode is $8/12 = 2/3$ , not the MLE $0.7$ ; its mean is $9/14$ .

Use when: Prior assumptions belong in the model. MAP maximizes posterior density in the chosen parameterization; it is not generally invariant under reparameterization. Gaussian and Laplace parameter priors produce quadratic and absolute-value penalties under their specified scales.

Change the Observation Unit

New task: Fit a binomial count of seven successes rather than one particular ordered sequence.

Picture and change: Multiply by $(7 10)$ to account for compatible sequences.

Check: The factor equals $120$ and does not depend on $p$ , so the maximizer remains $0.7$ .

Use when: Aggregation changes the recorded observation but preserves the model’s sufficient counts. If dependence changes instead, multiplying independent factors need not be valid.

Common pitfalls

Wrong variance denominator. With an unknown Gaussian mean, the variance MLE divides residual squares by $n$ . The unbiased sample-variance estimator uses $n - 1$ under its assumptions.
Missing identification. Parameters producing the same observation distribution need not have unique estimates.
Missing existence. All-success Bernoulli data maximize at $p = 1$ if the boundary is allowed. Restricting the parameter to $(0, 1)$ leaves only a supremum.
Unqualified misspecification claims. Suitable convergence conditions may select a KL-best approximation, not the true data-generating mechanism.

Reconstruct

Is the plotted likelihood a posterior over the parameter?

No. It scores fixed data across parameter values and has not included a prior or normalized over parameters. A constant rescaling preserves the likelihood maximizer; a nonconstant prior can change the posterior mode.