Logistic regression

Summary

Logistic regression models binary outcomes with affine log-odds, not an affine probability. Fit probabilities using a Bernoulli likelihood; choose actions separately using deployment costs.

Task: independent observations have $x = [0, 1]$ , $y = [0, 1]$ . Evaluate candidate intercept $b = 0$ and slope $w = lo g 2$ using summed negative log-likelihood. Find both probabilities, the likelihood, and the intercept/slope gradient. These coefficients are a checked candidate, not an asserted finite optimum for separable data.

The maximum-likelihood lesson supplies the fitting principle; here the conditional observation model is Bernoulli.

The model

For binary $y \in {0, 1}$ :

p (y = 1 ∣ x; w, b) = σ (w^{⊤} x + b) = \frac{1}{1 + e ^{- (w^{⊤} x + b)}} .

The log-odds (logit) is linear in $x$ :

lo g \frac{p ( y = 1 ∣ x )}{p ( y = 0 ∣ x )} = w^{⊤} x + b .

Odds are $p / (1 - p) = e^{z}$ , with $z = w^{⊤} x + b$ . Equal score increments multiply odds equally, but do not add equal probability increments.

Learning objective

Equal logit steps multiply odds rather than add fixed probability

Derivation

Read it this way: each unit score step multiplies odds by e. The five probability markers compress toward zero and one. The zero-score line marks p=0.5, appropriate for equal error costs under the stated probability model. Unequal costs can move the action threshold without refitting this sigmoid. Sources: An Introduction to Statistical Learning and Stanford CS229.

Check: Why does z increasing from 0 to 1 multiply odds by 2.72 rather than add fixed probability?

Read it this way: Moving z from 0 to 1 multiplies odds by e. The probability change is nonlinear and becomes smaller near the extremes.
Does another unit logit step add the same probability near the upper end?

Training

Negative log-likelihood (binary cross-entropy):

L (w, b) = - i = 1 \sum n [y_{i} lo g p_{i} + (1 - y_{i}) lo g (1 - p_{i})] .

Our scores $[0, lo g 2]$ give probabilities $[1/2, 2/3]$ . The observed-label likelihood is $(1 - 1/2) (2/3) = 1/3$ , so $L = lo g 3 \approx 1.098612$ . Residuals $p - y = [1/2, - 1/3]$ give

\frac{\partial L}{\partial b} = \frac{1}{2} - \frac{1}{3} = \frac{1}{6}, \frac{\partial L}{\partial w} = 0 \cdot \frac{1}{2} + 1 \cdot (- \frac{1}{3}) = - \frac{1}{3} .

The negative gradient locally decreases the intercept and increases the slope. This is Bernoulli cross-entropy, not a Gaussian squared-error objective.

With augmented design matrix $D$ containing the intercept column, $\nabla L = D^{⊤} (p - y)$ . Its Hessian is $D^{⊤} diag (p_{i} (1 - p_{i})) D$ , positive semidefinite for fixed features. Convexity does not ensure a finite, identifiable minimizer. Evaluate each term stably as $softplus (z) - y z$ , with $softplus (z) = max (z, 0) + lo g (1 + e^{- ∣ z ∣})$ .

Change decision costs

New task: false positives cost $1$ ; false negatives cost $4$ ; correct actions cost zero. Keep the probabilities and compare positive-action cost $1 - p$ with negative-action cost $4 p$ . Check: choose positive for $p > 1/5$ ; at $p = 0.3$ , costs are $0.7$ versus $1.2$ . Use when: these probabilities describe the deployment population and errors have unequal costs.

Change the feature basis

New task: represent XOR at $(0, 0), (0, 1), (1, 0), (1, 1)$ . Add interaction $x_{1} x_{2}$ ; candidate score $- 1 + 2 x_{1} + 2 x_{2} - 4 x_{1} x_{2}$ gives $[- 1, 1, 1, - 1]$ . Check: threshold $0.5$ recovers all four classes. Use when: additive log-odds in original features misses a known interaction. The model remains linear in expanded coefficients; these are fixture coefficients, not an MLE claim.

Common pitfalls

Failure: for $x = [- 1, 1]$ , $y = [0, 1]$ , and $b = 0$ , increasing $w$ drives loss toward zero. No finite $w$ attains it: complete separation defeats the implied finite optimum. A sigmoid does not establish calibration under finite samples, misspecification, regularization, or deployment shift.

Reconstruct: a coefficient $lo g 2$ doubles odds. Starting at $p = 0.2$ , what follows one feature unit, holding others fixed?

Answer

Odds move from $1/4$ to $1/2$ , giving $p = (1/2) / (1 + 1/2) = 1/3$ , not $0.4$ .

Multinomial / softmax regression

For mutually exclusive classes, normalize affine class scores with softmax and fit categorical cross-entropy. Multilabel outcomes instead need an appropriate joint or separate-output model; not every classifier uses softmax.

Properties

For an additive coefficient, $w_{j}$ is a conditional log-odds change holding other features fixed; $e^{w_{j}}$ is its odds ratio. Interactions alter that interpretation, and it is not automatically a causal effect. At probability threshold $t$ , the boundary is $w^{⊤} x + b = lo g (t / (1 - t))$ .

When to use vs. alternatives

Compare held-out probability quality, operating costs, latency, feature construction, and maintainability against relevant alternatives. Validate calibration separately; a model family is not a calibration certificate.

L-BFGS uses quasi-Newton updates; SGD/Adam use gradient updates; Newton-Raphson/IRLS uses curvature through weighted least squares. Solver choice depends on sparsity, dimensions, penalties, and resource limits.

An added $λ ∥ w ∥^{2} /2$ is one declared L2 convention, not an automatic implementation default. State whether the intercept is penalized and scale features using training-only statistics. Control feature engineering and leakage when comparing trees or nonlinear models. Use the separate derivation for the Bernoulli gradient cancellation.