Eigenvalues and the spectral theorem

Summary

A real symmetric matrix acts by independent signed scaling along an orthonormal eigenbasis. An eigenvector is a nonzero vector $v$ satisfying $A v = λ v$ ; the scalar $λ$ is its eigenvalue. The spectral theorem supplies a complete orthonormal basis, not merely one unchanged direction. Use this decomposition when a symmetric linear map must be applied, inverted, or understood through its principal directions.

Eigenvalues, eigenvectors, characteristic polynomial

For square $A \in R^{n \times n}$ , $(A - λ I) v = 0$ has a nonzero solution exactly when its determinant vanishes. The characteristic polynomial has $n$ complex roots counting multiplicity. Their sum is the trace; their product is the determinant. Symmetry guarantees real eigenvalues and an orthonormal eigenbasis, including when eigenvalues repeat.

The spectral theorem (symmetric case)

If $A = A^{⊤} \in R^{n \times n}$ :

A = Q Λ Q^{⊤} = i = 1 \sum n λ_{i} q_{i} q_{i}^{⊤}

where $Q = [q_{1} ∣ \dots ∣ q_{n}]$ is orthogonal ( $Q^{⊤} Q = I$ ) and $Λ = diag (λ_{1}, \dots, λ_{n})$ .

First express the input as $z = Q^{⊤} v$ ; then scale with $Λ$ and return with $Q$ . Negative eigenvalues reverse their axis; zero eigenvalues erase its component.

Take $A = [2112]$ and $v = (2, 1)^{⊤}$ . The diagram uses $q_{1} = (1, 1)^{⊤} / 2$ , $q_{2} = (- 1, 1)^{⊤} / 2$ , and eigenvalues $3, 1$ . Thus $z = (3, - 1)^{⊤} / 2$ , $Λ z = (9, - 1)^{⊤} / 2$ , and $A v = (5, 4)^{⊤}$ .

Why does an entire basis exist? The continuous Rayleigh quotient reaches a maximum on the unit sphere. Its constrained stationary condition gives $A q = λ q$ . Symmetry makes the orthogonal complement invariant: $q^{⊤} A w = (A q)^{⊤} w = 0$ whenever $q^{⊤} w = 0$ . Repeat on that smaller symmetric subspace to construct the basis.

Learning objective

Change coordinates, scale eigenvector components, then change coordinates back

Derivation

Read it this way: Q^T changes coordinates without changing length. Λ triples the first coordinate and preserves the second. Q returns the result to the original axes: Av = (5, 4).

Check: Which step changes only coordinates? Which step actually scales the eigenvector components?

Read it this way: For this diagonal example, (3,1) becomes (6,3) under eigenvalues 2 and 3. A general symmetric matrix first uses its orthonormal eigenbasis.
Does scaling the first eigen-coordinate require mixing it with the second?

Original coordinate construction checked against MIT OpenCourseWare 18.06 and Sheldon Axler's open-access Linear Algebra Done Right.

Connection to SVD

For symmetric positive-semidefinite $A$ , one SVD uses $U = V = Q$ and $Σ = Λ$ . For symmetric indefinite $A$ , singular values are $∣ λ_{i} ∣$ , not the signed eigenvalues. General rectangular matrices have an SVD but no square-matrix eigendecomposition of the same form.

Change the Requirement

Apply the Map Repeatedly

New task: Compute $A^{2} v$ without two dense applications after the eigenbasis is available.

Picture and change: Replace each $λ_{i}$ by $λ_{i}^{2}$ , preserving the coordinate basis.

Check: $Q diag (9, 1) Q^{⊤} v = (14, 13)^{⊤}$ , matching $A (5, 4)^{⊤}$ .

Use when: Many powers or repeated queries reuse the same decomposition. Dense decomposition costs $O (n^{3})$ ; one dense basis transform costs $O (n^{2})$ .

Solve Instead of Apply

New task: Recover $x$ satisfying $A x = v$ .

Picture and change: Divide each eigen-coordinate by its nonzero eigenvalue.

Check: $x = Q diag (1/3, 1) Q^{⊤} (2, 1)^{⊤} = (1, 0)^{⊤}$ .

Use when: Every eigenvalue is nonzero. A zero eigenvalue makes some right-hand sides impossible and others nonunique. Adding a ridge changes the system; it is not an exact inverse of the original map.

Where eigenvalues show up in ML

Object	What its eigenvalues tell you
Covariance matrix	Variances along principal axes (PCA)
Positive-definite Hessian	Local curvature; spectral condition number $λ_{m a x} / λ_{m i n}$
Graph Laplacian	Connectivity, spectral clustering, GNN smoothness
Reversible Markov chain	Mixing depends on nontrivial eigenvalue magnitudes and the chain’s assumptions
General attention score matrix	May be asymmetric; singular values describe rank and norm structure
Recurrent weight matrix	Long-run dynamics; finite-step gradient norms also depend on singular values and nonnormality

Common pitfalls

Dropping symmetry. The matrix $[1011]$ has only one independent eigenvector. Its repeated eigenvalue does not provide a basis.
Dropping positivity. $diag (- 2, 1)$ has eigenvalues $- 2, 1$ but singular values $2, 1$ .
Ignoring scale. Dense decompositions need cubic arithmetic and quadratic storage. Lanczos targets symmetric eigenproblems; Arnoldi handles more general operators. Approximate methods require residual and convergence checks.

Reconstruct

Why may repeated eigenvalues still have orthogonal eigenvectors for a symmetric matrix?

Reasoned answer

Inside a repeated-eigenvalue eigenspace, every linear combination remains an eigenvector with that eigenvalue. Choose an orthonormal basis inside that subspace. Symmetry, not distinctness of all eigenvalues, guarantees a complete orthonormal collection.

SVD and PCA. Generalization to all matrices.
Positive definite matrices. The cone of PSD matrices.