L1 versus L2: derive the zero coefficient

Compare the exact fitted coefficients produced by absolute and squared penalties under one explicitly defined quadratic objective.

Problem

Context. Fit $a = (2, 0.3)$ using data loss $L_{d a t a} (w) = \frac{1}{2} ∥ w - a ∥_{2}^{2}$ and coefficient $λ = 0.5$ . Compare L1 penalty $λ \sum_{i} ∣ w_{i} ∣$ with half-scaled L2 penalty $\frac{λ}{2} \sum_{i} w_{i}^{2}$ .

Task. Derive the minimizing coefficients for each penalized objective and identify which coefficients become exactly zero. Explain how changing the penalty convention or feature scaling changes the comparison.

Constraints. Coordinates are independent, feature units are comparable, and no intercept appears in this fixture. Use subgradients at the absolute-value corner; distinguish objective sparsity from a solver merely returning a small number.

Fix the objective

Use L1 penalty $λ \sum_{i} ∣ w_{i} ∣$ and L2 penalty $\frac{λ}{2} \sum_{i} w_{i}^{2}$ . An unhalved L2 convention, $λ \sum_{i} w_{i}^{2}$ , doubles its penalty derivative for the same numerical coefficient. Changing summed loss to mean loss also changes relative penalty strength unless rescaled. State intercept treatment and feature scaling in a real fit.

Solve the corner and the smooth condition

For one L1 coordinate, minimize $\frac{1}{2} (w - a)^{2} + λ ∣ w ∣$ .

If $w > 0$ , stationarity gives $w - a + λ = 0$ , valid for $a > λ$ .
If $w < 0$ , it gives $w - a - λ = 0$ , valid for $a < - λ$ .
At zero, $0 \in - a + λ [- 1, 1]$ holds exactly when $∣ a ∣ \leq λ$ .

Together these give $w = sign (a) max (∣ a ∣ - λ, 0)$ . For L2, $w - a + λ w = 0$ gives $w = a / (1 + λ)$ .

Objective	First coordinate	Second coordinate
Unpenalized	$2$	$0.3$
L1	$2 - 0.5 = 1.5$	$0$ , since $0.3 \leq 0.5$
Half-scaled L2	$2/1.5 = 4/3$	$0.3/1.5 = 0.2$

Learning objective

An L1 corner can touch the loss at an exactly zero coefficient

Derivation

expand a loss contour from the star until it touches the allowed region. A diamond corner can zero a coefficient; a smooth-circle contact can retain both. This schematic illustrates a tendency, not the numerical coordinates of our separate analytic fixture. Equivalent constrained budgets depend on the fitted optimum; equal numeric budgets across norms are not equivalent penalties. Sources: Tibshirani's lasso paper and The Elements of Statistical Learning.

Check: What feature of the L1 boundary makes a zero coefficient possible at first contact?

Read it this way: For squared distance from (2,0.4), the unit L1 constraint selects (1,0). The unit L2 constraint selects a nonzero second coordinate.
Which boundary has a corner on a coordinate axis?

Change the requirements

New task: reduce L1 strength to $0.2$ while keeping the fitting surface fixed. Move the threshold inward. Check: $(2 - 0.2, 0.3 - 0.2) = (1.8, 0.1)$ ; the removed coefficient returns. Use when: validation or a feature-count requirement favors less sparsity.

New task: combine L1 strength $λ_{1} = 0.5$ with L2 strength $λ_{2} = 1$ . Threshold, then shrink: $w = soft (a, λ_{1}) / (1 + λ_{2})$ , giving $(0.75, 0)$ . Use when: sparse selection and correlated-feature stability both matter. Boundary: this reference has independent coordinates; correlated designs require solving the coupled objective.

Failure: for $\frac{1}{2} w^{2} + ∣ w ∣$ , step size one sends $w = 0.25$ to $- 1$ , then alternates between $1$ and $- 1$ . The optimum is zero; fixed-step subgradient updates need not reach it. Numerical jitter is not an exact-sparsity method.

Reconstruct: multiply a feature by $100$ while preserving its prediction contribution. Is its coefficient penalty unchanged?

Answer

No. Its coefficient becomes one hundredth as large. The same contribution receives one hundredth the L1 penalty and one ten-thousandth the L2 penalty. Standardize features or justify scale-specific penalties.

Priors and solver choices

Independent Laplace and Gaussian priors yield L1 and L2 MAP penalties under matching likelihood units and scale. MAP is a point estimate, not the whole posterior. Intercept exclusions and rescaled features alter that interpretation.

Proximal gradient methods apply soft thresholding after the smooth-loss step; their threshold includes step size. ISTA, FISTA, coordinate descent, and specialized solvers handle nonsmoothness explicitly. The separable closed form does not extend unchanged to an arbitrary correlated design. See Parikh and Boyd’s Proximal Algorithms.

Keep optimizer effects separate

For frozen coordinate preconditioners, a coupled L2 gradient is preconditioned; AdamW’s shrinkage is separate. The decay trace computes that difference. Neither procedure is universally preferable for every statistical objective.

Optimizer noise, architecture, and early stopping can bias fitted solutions without an explicit penalty. Their generalization effects and relative importance depend on workload and training protocol.

Common follow-up

When should decay be disabled? Compare matched validation outcomes, parameter groups, schedules, and deployment constraints, including a zero-decay control. Model size alone does not prescribe a universal coefficient.

Related: regularization and Adam and AdamW.

L1 versus L2: derive the zero coefficient

Practice before you read