SVD and PCA

Summary

SVD identifies input directions, output directions, and nonnegative scaling factors for any real matrix. PCA applies that decomposition to centered observations, retaining directions that minimize Euclidean squared reconstruction error. Large retained variance does not guarantee retained label information.

The decomposition

$A \in R^{m \times n}$ has $A = U Σ V^{⊤}$ . Full factors have orthogonal $U : m \times m$ , orthogonal $V : n \times n$ , and rectangular diagonal $Σ : m \times n$ .

The first $r$ columns of $V$ span the row space; remaining columns span the nullspace.
The first $r$ columns of $U$ span the column space; remaining columns span the left nullspace.
Singular values satisfy $σ_{1} \geq σ_{2} \geq \dots \geq 0$ ; rank $r$ counts strictly positive values in exact arithmetic.

The orthogonal factors can include reflections, not just rotations. For an invertible square map, the spectral condition number is $σ_{1} / σ_{n}$ . Null directions prevent unrestricted inversion; $σ_{1} / σ_{r}$ describes conditioning only on the retained nonzero subspace.

Truncated SVD and low-rank approximation

For rank at most $k$ , a best approximation in Frobenius or spectral norm is

A_{k} = U_{k} Σ_{k} V_{k}^{⊤}

Keep the leading $k$ singular values and their corresponding directions. Orthogonal changes of basis preserve these norms, reducing the error calculation to discarded diagonal entries:

∥ A - A_{k} ∥_{F}^{2} = i > k \sum σ_{i}^{2}, ∥ A - A_{k} ∥_{2} = σ_{k + 1} .

Take omitted singular values as zero when none remain. This is the Eckart–Young result; a low-rank model constraint does not automatically prescribe truncating an unrelated matrix.

PCA as SVD

For $n > 1$ observations in rows, fit a training mean $μ$ and center $X = X - 1 μ^{⊤}$ .

Decompose $X = U Σ V^{⊤}$ .
Use leading columns $V_{k}$ as feature-space principal directions.
Compute scores $Z = X V_{k}$ and reconstruct $X = Z V_{k}^{⊤} + 1 μ^{⊤}$ .
Component sample variance is $σ_{i}^{2} / (n - 1)$ .

The covariance is $X^{⊤} X / (n - 1)$ ; its eigenvectors are these same principal directions. Direct SVD avoids explicitly forming a Gram matrix, which squares the nonzero singular-value condition ratio.

For an exact numerical companion to the schematic, use rows $(- 2, - 1), (- 2, 1), (2, - 1), (2, 1)$ . They are centered, and $X^{⊤} X = diag (16, 4)$ , giving singular values $4, 2$ and $V = I$ . Rank-one scores are $(- 2, - 2, 2, 2)^{⊤}$ ; reconstruct each second coordinate as zero. Four unit residuals give squared error $4$ , while retained variance is $16/ (16 + 4) = 80%$ .

Keep the coordinate along PC1; discard the perpendicular reconstruction error

Derivation

Read it this way: Follow the highlighted sample to scalar score z=85. Reconstruct the diamond at 85 v1. The perpendicular -20 v2 is discarded, giving squared error 400. These are illustrative coordinates, not measured data.

Check: Which object is the one-dimensional coordinate? How does it reconstruct the point on the PC1 line?

Read it this way: Projecting (3,2) onto the horizontal first principal direction retains (3,0). The two-unit residual is reconstruction error, not another retained coordinate.
Does the perpendicular displacement remain in the rank-one reconstruction?

Change the Requirement

Retain Ninety Percent of Variance

New task: Choose the smallest rank retaining at least $90%$ of total squared singular-value mass.

Picture and change: Accumulate $σ_{i}^{2}$ in descending order; change scores and reconstruction dimensions together.

Check: The fixture’s first component retains only $80%$ ; rank two retains $100%$ and reconstructs exactly.

Use when: A declared reconstruction requirement determines storage or representation size.

Standardize the Features

New task: Give both features equal sample variance before fitting PCA.

Picture and change: Divide centered columns by their training sample standard deviations, changing the feature-space metric.

Check: Fixture covariance becomes $I$ ; every unit direction is a possible first component, retaining $50%$ .

Use when: Units or domain priorities justify equalized feature weighting. Carry the fitted mean, scales, and directions into inference; refitting on test data changes the transformation.

Common pitfalls

Different objective. Uncentered SVD includes the mean’s contribution; its leading direction need not equal the mean direction.
Automatic scaling. Standardization changes the objective and is undefined for zero-variance columns without a policy.
Whitening confusion. PCA scores are uncorrelated, not generally unit variance. Dividing by positive component standard deviations whitens them; zero singular values need exclusion or regularization.
Direction ambiguity. Singular-vector signs and tied-eigenvalue bases can vary without changing the reconstructed subspace.
Data geometry. Centering sparse matrices can destroy sparsity; categorical data need a justified metric.

Reconstruct

What if the label is the sign of the second feature?

The rank-one fixture discards that feature completely despite retaining eighty percent of variance. Unsupervised reconstruction quality is not a guarantee of predictive sufficiency.

Matrices as linear maps. The geometry.
Eigenvalues and the spectral theorem. For symmetric matrices.