Exponential family

Summary

A distribution is in the exponential family if its density / mass function can be written as:

p (x ∣ η) = h (x) exp (η^{⊤} T (x) - A (η))

with natural parameter $η$ , sufficient statistic $T (x)$ , base measure $h (x)$ , and log-partition $A (η)$ (which normalizes).

Most distributions you use day-to-day are exponential family: Gaussian, Bernoulli, categorical, Poisson, Beta, Gamma, Dirichlet, geometric, exponential. Recognizing them as such gives you free results:

MLE is closed-form when the natural parameter is unconstrained: just match sample moments to model moments.
Conjugate priors exist and are themselves exponential family.
Sufficient statistics $T (x)$ contain all the data’s information about $θ$ . You can summarize a dataset by $\sum_{i} T (x_{i})$ and forget the rest.
Generalized linear models (GLMs) are linear regression generalized to exponential-family responses.

The canonical form

Given the form above:

$T (x)$ are the sufficient statistics (Bernoulli: $T (x) = x$ ; Gaussian: $T (x) = (x, x^{2})$ ).
$η$ are the natural parameters (Bernoulli: $η = lo g \frac{p}{1 - p}$ , the logit; Gaussian: $η = (μ / σ^{2}, - 1/ (2 σ^{2}))$ ).
$A (η)$ is the log-partition function; its derivatives with respect to the natural parameter give the mean and covariance of $T (x)$ :

\nabla_{η} A (η) = E_{η} [T (X)], \nabla_{η}^{2} A (η) = Cov_{η} (T (X)) .

This is why MLE via moment-matching works: the gradient of the log-likelihood is “data sufficient stat minus model expected sufficient stat.”

For independent observations in a regular exponential family, an interior MLE satisfies that moment equation. Such a finite interior optimum need not exist; boundary data, support restrictions, or nonidentifiability require separate treatment. The MLE lesson distinguishes a likelihood objective from a guarantee of existence or uniqueness.

Learning objective

Keep the success count and sample size; their ratio estimates the mean

Derivation

Read it this way: first replace the six observations by the single count S = 4. In the sample log-likelihood, every parameter-dependent data term now passes through S. Differentiating gives S − n∇A(η); when an interior MLE exists, setting this to zero matches the empirical statistic S/n to the model moment Eη[T(X)]. For Bernoulli data, that gives p̂ = 4/6. Original schematic checked against Berkeley Stat 210A notes and Minnesota Stat 5421 notes.

Check: What must remain known alongside S if samples can have different lengths?

Read it this way: For Bernoulli observations, five trials and three successes give maximum-likelihood mean 3/5. Their order does not change this likelihood.
Can the Bernoulli likelihood distinguish these two orders under the stated independent model?

Common members

Distribution	Sufficient stat $T (x)$	Natural parameter $η$
Bernoulli( $p$ )	$x$	$lo g (p / (1 - p))$ (logit)
Categorical( $π$ )	one-hot $(x)$	$lo g π$ (log-probabilities)
Gaussian( $μ, σ^{2}$ )	$(x, x^{2})$	$(μ / σ^{2}, - 1/ (2 σ^{2}))$
Poisson( $λ$ )	$x$	$lo g λ$
Beta( $α, β$ )	$(lo g x, lo g (1 - x))$	$(α - 1, β - 1)$
Gamma( $α, β$ )	$(lo g x, x)$	$(α - 1, - β)$

Generalized linear models

A GLM combines a linear predictor $η = X β$ with an exponential-family response distribution. The link function maps the linear predictor to the natural parameter:

Response	GLM	Link
Continuous (Gaussian)	linear regression	identity
Binary (Bernoulli)	logistic regression	logit
Count (Poisson)	Poisson regression	log
Categorical	multinomial logistic	softmax
Time-to-event (Exponential, Weibull)	survival models	log

Logistic regression is a GLM with Bernoulli response and logit link. This unifies the entire family of “regression-style” classifiers.

Properties to remember

Convexity: $A (η)$ is convex in $η$ . So negative log-likelihood is convex, and there is a unique MLE.
Sufficient statistics: by Pitman-Koopman-Darmois theorem, exponential families are essentially the only distributions with finite-dimensional sufficient statistics independent of $n$ .
Conjugacy: exponential families have conjugate priors, also in the exponential family.
Maximum entropy: the exponential family with sufficient statistics $T$ matching given moments is the maximum-entropy distribution under those constraints.

Common pitfalls

Forgetting that the Cauchy distribution is not exponential family. Heavy tails break the sufficient-statistic property.
Confusing “exponential” (the distribution) with “exponential family” (the class). The Exp( $λ$ ) distribution is one member.
Treating natural parameters as the same as canonical parameters. A Gaussian’s natural parameters are $(μ / σ^{2}, - 1/ (2 σ^{2}))$ , not $(μ, σ^{2})$ . Some software libraries default to one or the other; check.