Gaussian mixture models

Summary

A Gaussian mixture model assumes data is generated by sampling a component $k\in \{1,\dots ,K\}$ from a categorical distribution with weights ${\pi }_k$ and then sampling $x\sim \mathcal{N}({\mu }_k,{\Sigma }_k)$:

$$p(x)=\sum _{k=1}^K{\pi }_k\mathcal{N}(x;{\mu }_k,{\Sigma }_k).$$

GMMs are the canonical:

• Soft clustering algorithm (each point has a posterior responsibility distribution over components, not a hard assignment).
• Continuous density estimator with adjustable expressiveness via $K$.
• Building block for HMMs (state emissions), VAEs (mixture priors), and many speech / vision systems.
• EM teaching example. The simplest non-trivial latent-variable model with closed-form EM updates.

The latent variable view

Define a one-hot latent $z\in \{1,\dots ,K\}$ with $p(z=k)={\pi }_k$. Then $p(x\mid z=k)=\mathcal{N}({\mu }_k,{\Sigma }_k)$. Marginalize: $p(x)={\sum }_k{\pi }_k\mathcal{N}(x;{\mu }_k,{\Sigma }_k)$.

The latent $z$ is what makes GMM a soft-clustering algorithm: posterior $p(z=k\mid x)$ tells you how much “responsibility” component $k$ has for $x$.

Learning objective

See how one observation updates more than one Gaussian

Training: EM

Initialize ${\pi }_k,{\mu }_k,{\Sigma }_k$ (k-means++ for means, identity for covariances). Iterate until log-likelihood stabilizes.

E-step: for each point $x_i$ and component $k$:

$$r_{ik}=p(z_i=k\mid x_i)=\frac{{\pi }_k\mathcal{N}(x_i;{\mu }_k,{\Sigma }_k)}{{\sum }_j{\pi }_j\mathcal{N}(x_i;{\mu }_j,{\Sigma }_j)}.$$

M-step:

$$N_k=\sum _i{r}_{ik},{\pi }_k=\frac{N_k}{n},{\mu }_k=\frac{1}{N_k}\sum _i{r}_{ik}{x}_i,{\Sigma }_k=\frac{1}{N_k}\sum _i{r}_{ik}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^{\top }.$$

Covariance structure

Trade expressiveness for parameter count by restricting ${\Sigma }_k$:

Form	Parameters per component	Use case
Full	$d(d+1)/2$	Default; expressive
Diagonal	$d$	High-d data, fewer parameters
Spherical ${\sigma }_k^{2}I$	1	Approximates k-means with soft assignments
Tied (shared across components)	$d(d+1)/2$ once	Used in linear discriminant analysis

Choosing $K$

Same problem as k-means. Standard approaches:

• Information criteria: BIC ($\log n\cdot \text{params}-2\log L$) or AIC ($2\cdot \text{params}-2\log L$). Pick $K$ minimizing.
• Cross-validation log-likelihood.
• Dirichlet process mixture (infinite GMM): non-parametric Bayesian alternative that lets $K$ grow with data.

GMM vs. k-means

Property	k-means	GMM
Cluster shape	Spherical, equal radius	Arbitrary ellipsoid (full $\Sigma$)
Assignment	Hard (0/1)	Soft (probability over components)
Parameters	$K$ centroids	$K$ centroids + $K$ covariances + $K$ weights
Compute per iter	$O(nKd)$	$O(nK{d}^2)$ for full $\Sigma$
Soft probabilities	No	Yes
Density estimation	No (Voronoi cells)	Yes

GMM is k-means generalized: k-means = GMM with shared identity covariance, equal weights, and hard assignments.

Use cases

• Soft clustering with ellipsoidal clusters.
• Density estimation for low-d continuous data.
• Anomaly detection: low-likelihood points are anomalies.
• Speaker diarization: GMMs over MFCC features (legacy speech).
• Background subtraction in video: GMM per pixel.
• Mixture of experts gating in neural networks (related but distinct).

For high-dimensional data: pure GMMs struggle (covariance estimation requires $O(d^2)$ parameters per component). Use diagonal covariance, dimensionality reduction, or normalizing-flow alternatives.

Common pitfalls

• Covariance singularities. A component centered on a single point has ${\Sigma }_k\to 0$ and log-likelihood diverges. Add jitter ${\Sigma }_k+\epsilon I$.
• Initialization. Random init often gives degenerate clusters; use k-means++ centroids as initial means.
• Reading mixture weights as cluster sizes. ${\pi }_k$ is the prior; effective cluster size is $N_k={\sum }_i{r}_{ik}$.
• Comparing GMM and k-means on cluster counts. GMM components and k-means clusters are not directly comparable; soft mass distributes differently.

Related

• Expectation-Maximization. The training algorithm.
• k-means clustering. Special case of GMM.