SVM and the kernel trick

An SVM chooses a decision boundary by balancing its norm against small signed margins.

Summary

Let negative examples be $x = - 2, - 1$ and positive examples be $x = 1, 2$ . Which linear score $f (x) = w x + b$ separates them with the widest normalized margin? Use labels $y \in {- 1, 1}$ and predict the sign of the score, with an explicit tie policy at zero. The figure illustrates a soft-margin fit; its three highlighted points are not this four-point dataset.

Learning objective

Only nonzero dual weights contribute to the SVM decision sum

Derivation

compare each point with its class-margin rail. The three rings identify positive dual coefficients in this schematic fit. Strictly outside the margin, coefficients are zero. A point on a rail can also have zero coefficient in a degenerate solution. Soft margin permits positive coefficients inside the rails. Sources: Burges's tutorial and the SVM formulation.

Check: Which displayed points actually contribute to the decision sum? Why is lying on a margin alone not a universal proof of nonzero alpha?

Read it this way: Only support vectors with nonzero alpha enter the weighted kernel decision. Other training examples still exist but add zero here.
Does a zero dual contribution delete the corresponding training observation?

The hard-margin SVM (separable case)

Normalize the nearest signed scores to at least one:

w, b min \frac{1}{2} ∥ w ∥^{2} subject to y_{i} (w^{⊤} x_{i} + b) \geq 1.

The examples at $- 1$ and $1$ require $w - b \geq 1$ and $w + b \geq 1$ . Adding gives $w \geq 1$ . Thus $w = 1, b = 0$ achieves the minimum objective $1/2$ . Signed margins are $2, 1, 1, 2$ ; the rails are at $- 1$ and $1$ , with width $2/∥ w ∥ = 2$ . Without score normalization, multiplying both parameters changes no boundary and makes raw score gaps meaningless.

The soft-margin SVM (non-separable)

Change the requirement: price violations instead of forbidding them. For $C > 0$ , minimize

\frac{1}{2} ∥ w ∥^{2} + C i \sum max (0, 1 - y_{i} f (x_{i})) .

The separating fit costs $1/2$ ; the constant-zero score costs $4 C$ . At $C = 0.1$ , zero beats that separator: $0.4 < 0.5$ . Neither comparison proves that zero is optimal. The optimum is $w = 0.5, b = 0$ , with objective $0.125 + 0.1 = 0.225$ . Coefficients $(0.075, 0.1, 0.1, 0.075)$ satisfy the dual constraints and achieve the same objective. All four points now support the fit; retaining the hard-margin coefficients would violate their new upper bound. Slack is $ξ_{i} = max (0, 1 - y_{i} f (x_{i}))$ , not necessarily a misclassification. A correctly classified score $y_{i} f (x_{i}) = 0.4$ still pays $0.6$ .

The dual formulation and the kernel trick

The soft-margin dual is

α max i \sum α_{i} - \frac{1}{2} i, j \sum α_{i} α_{j} y_{i} y_{j} K (x_{i}, x_{j}), 0 \leq α_{i} \leq C, i \sum α_{i} y_{i} = 0.

For hard margin, remove the upper bound on $α_{i}$ . In the four-point example, coefficients $(0, 1/2, 1/2, 0)$ reconstruct $w = \sum_{i} α_{i} y_{i} x_{i} = 1$ . Their dual objective is $1 - 1/2 = 1/2$ , matching the primal and certifying optimality.

Change the requirement: use an RBF kernel on only $x = - 1, 1$ , with labels $- 1, 1$ and $C = 3$ . Set $K (x, z) = exp (- γ (x - z)^{2})$ and $γ = lo g (2) /4$ . The cross-similarity is $1/2$ ; diagonal entries are one. The equality constraint makes both coefficients $a$ . Their dual is $2 a - a^{2} /2$ , maximized at $a = 2$ within $[0, 3]$ . With $b = 0$ , both signed scores equal $2 (1 - 1/2) = 1$ . Doubling $γ$ makes cross-similarity $1/4$ and changes the optimum to $a = 4/3$ . Do not reuse the old coefficients after changing kernel geometry. Use symmetric positive-semidefinite kernels; an arbitrary similarity need not yield a convex dual.

Support vectors

Prediction uses $f (x) = \sum_{i : α_{i} > 0} α_{i} y_{i} K (x_{i}, x) + b$ . Zero coefficients remove terms from this fitted score, not a guarantee about arbitrary retraining or preprocessing changes. For a soft-margin optimum, $0 < α_{i} < C$ implies a point lies on its rail. Points with $α_{i} = C$ can lie on or inside it.

Reconstruct: does a large positive score mean a calibrated probability? No: margin optimization supplies no probability normalization or calibration guarantee.

When to use SVMs in 2026

Compare a scaled linear SVM, an RBF SVM and task-appropriate alternatives under the same validation protocol. A dense Gram matrix stores $n^{2}$ entries; solver cost depends on the implementation and data. Linear solvers need not materialize that matrix. Kernel prediction evaluates one term per retained support vector, so measure deployment cost too.

Common pitfalls

Scale features using training-only statistics and tune interacting parameters inside validation folds. For $K (x, z) = exp (- γ ∥ x - z ∥^{2})$ , changing units changes similarities unless $γ$ changes appropriately. Hinge loss is not log loss, and sigmoid similarities are not PSD for arbitrary parameters. The primal is strictly convex in $w$ , not jointly strictly convex in the unregularized intercept. For example, opposite labels at the same zero input admit every $b \in [- 1, 1]$ when $w = 0$ .

Kernel methods owns feature-inner-product equivalence.
Calibration separates scores from estimated probabilities.
Linear regression provides a squared-error contrast.