Factor analysis and probabilistic PCA

Summary

Factor analysis explains shared covariance through latent Gaussian factors and feature-specific measurement noise. Probabilistic PCA ties those noise variances together. Recognize the model when several measurements share variation but have different residual uncertainties.

Given two measurements, infer one hidden factor without treating either measurement as exact. Use the teaching fixture $W = (2, 1)^{⊤}$ , zero mean, noise variances $(1, 4)$ , and observation $x = (2, 1)^{⊤}$ .

The generative model

Let $z \in R^{k}$ , $x \in R^{d}$ , and $W \in R^{d \times k}$ . The factor and measurement noise are independent:

z \sim N (0, I), x ∣ z \sim N (W z + μ, Ψ) .

$μ$ is the observation mean; $Ψ$ is diagonal with strictly positive entries. The loadings $W$ send each factor into the measured features. Marginalizing the factors gives

x \sim N (μ, W W^{⊤} + Ψ) .

Learning objective

Shared factors explain covariance; independent noise adds only diagonal variance

Derivation

Two-feature teaching fixture
Shared WWᵀ	Noise Ψ	Total C
[[4, 2], [2, 1]]	[[1, 0], [0, 4]]	[[5, 2], [2, 5]]

Read it this way: shared factor paths create both variance and covariance; each noise path adds only its feature's variance. The table applies that decomposition to two features. PPCA ties noise variances, but its fitted principal subspace already matches PCA at finite noise. Deterministic projection is a separate zero-noise limit. The original drawing retains its Tipping-Bishop (1999) and Ghahramani-Hinton (1996) provenance.

Check: Why can independent unique noise alter variances but not explain cross-feature covariance?

Read it this way: For one unit-variance factor with loadings (1,2), the shared covariance is [[1,2],[2,4]]. Independent noise adds diagonal terms only.
Can independent asset noise add an off-diagonal covariance term?

Work the posterior

The shared contribution is $W W^{⊤} = [4221]$ . Adding $Ψ$ gives $C = [5225]$ , whose eigenvalues are $7$ and $3$ . The off-diagonal value $2$ comes entirely from the shared factor.

Completing the square in the Gaussian log density gives posterior covariance $S$ and mean $m$ :

S = (I + W^{⊤} Ψ^{- 1} W)^{- 1}, m = S W^{⊤} Ψ^{- 1} (x - μ) .

Here the prior contributes precision $1$ ; the measurements contribute $4$ and $1/4$ . Therefore

S = (1 + 4 + 1/4)^{- 1} = 4/21, m = (4/21) (4 + 1/4) = 17/21.

The noisier second feature supplies less precision. The prior shrinks the mean below the unregularized projection coefficient $1$ . Neither the inferred coordinate nor its name establishes a causal explanation.

FA vs PPCA vs PCA: it’s all about $Ψ$

Model	Noise covariance $Ψ$	Consequence
Factor analysis	Positive diagonal	Separate residual variances
Probabilistic PCA	$σ^{2} I$	Shared residual variance; spectral likelihood fit
Classical PCA	No probabilistic noise requirement	Minimum squared reconstruction error

Tie the measurement noise

New task: require $Ψ = I$ while keeping $W$ and $x$ fixed.

Picture and change: replace the unequal noise entries, then recompute both covariance and posterior precision.

Check: $C = [5222]$ , $S = 1/6$ , and $m = 5/6$ . Reusing $17/21$ would retain the wrong measurement weights.

Use when: one shared residual scale is an intended modeling restriction.

Predict an unobserved feature

New task: observe only $x_{1} = 2$ and predict $x_{2}$ .

Picture and change: condition using only the observed loading row and noise entry. Keep the missing feature’s independent noise when predicting it.

Check: $z ∣ x_{1}$ has mean $4/5$ and variance $1/5$ . Consequently, $x_{2} ∣ x_{1}$ has mean $4/5$ and variance $1/5 + 4 = 21/5$ . Using only $1/5$ would omit future measurement noise.

Use when: incomplete observations have a justified observation model; missingness need not be harmless.

Fitting it

For $0 < k < d$ , order the sample covariance eigenvalues $λ_{1} \geq \dots \geq λ_{d}$ . At a regular PPCA likelihood solution,

σ^{2} = \frac{\sum _{j = k + 1}^{d} λ _{j}}{d - k}, W = U_{k} (Λ_{k} - σ^{2} I)^{1/2} R,

where $R$ is orthogonal. For our covariance, $σ^{2} = 3$ and $W = 2 (1, 1)^{⊤}$ . The principal subspace already matches PCA; positive noise does not prevent that agreement. Sending noise to zero separately can make the observation density singular. FA generally uses iterative likelihood fitting, including EM.

Why the probabilistic version is worth it

Likelihood supports held-out comparison, not automatic rank selection without an overfitting policy. Marginals support missing-feature prediction; the generative model supports sampling and mixtures of local factor models. VAEs extend the decoder and inference family, not the exact Gaussian conditioning identity.

What an interviewer expects you to say

Rescale $x_{1}$ by ten. Which parameters must change?

Answer

With $D = diag (10, 1)$ , use $D W$ , $D μ$ , $D Ψ D^{⊤}$ , and $D C D^{⊤}$ . An unrestricted diagonal-noise family remains compatible; fixed penalties or constraints may not. Isotropic noise generally stops being isotropic.

Common confusions

For orthogonal $R$ , $W$ and $W R$ have identical observed covariance. Factor axes are not uniquely identified causes; even one factor permits a sign reversal.