Skip to content
mlmentorship

Derive logistic regression from MLE

Standard math-screen question. The senior signal is whether you can derive it cleanly and connect MLE to cross-entropy.

Published · 7 min read ·Core ·Advanced

Math / derivation · active recall

Practice before you read

15 minutes. State assumptions, derive cleanly, check dimensions or limiting cases, then interpret the result.

Back to my workbook How practice works

Math / derivation · closed-book attempt

Derive logistic regression from MLE

State assumptions, derive cleanly, check dimensions or limiting cases, then interpret the result.

Problem

Context. Independent examples have feature vectors x and binary labels y in {0, 1}. The conditional model is Bernoulli with probability sigmoid(w.T @ x); an intercept may be included as a constant feature. Task. Form the joint likelihood, derive the negative log-likelihood and weight gradient, and connect the objective to binary cross-entropy. Explain the sign of a single example's update for each label. Constraints. Begin with the unregularized summed loss and state any change to averaging or regularization. Separate the assumed probability model from empirical calibration; algebraic equivalence alone does not establish deployment quality.

15:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: ML breadth and math-screen rounds.

Derive the binary logistic-regression training objective and its gradient from an explicit probabilistic model.

Problem

Context. Independent examples have feature vectors x and binary labels y in {0, 1}. The conditional model is Bernoulli with probability sigmoid(w.T @ x); an intercept may be included as a constant feature.

Task. Form the joint likelihood, derive the negative log-likelihood and weight gradient, and connect the objective to binary cross-entropy. Explain the sign of a single example’s update for each label.

Constraints. Begin with the unregularized summed loss and state any change to averaging or regularization. Separate the assumed probability model from empirical calibration; algebraic equivalence alone does not establish deployment quality.

The setup

Binary classification: y in {0, 1}. We model P(y=1 | x) = sigma(w^T x) where sigma(z) = 1 / (1 + exp(-z)) is the sigmoid.

The derivation

For a single example (x, y), the likelihood under our model:

P(y | x; w) = sigma(w^T x)^y * (1 - sigma(w^T x))^(1-y)

Equivalent to a Bernoulli likelihood with parameter sigma(w^T x).

For N i.i.d. examples, the joint likelihood is the product:

L(w) = prod_i P(y_i | x_i; w)

The log-likelihood:

log L(w) = sum_i [ y_i * log sigma(w^T x_i) + (1 - y_i) * log (1 - sigma(w^T x_i)) ]

MLE picks w to maximize this, equivalently minimizes the negative log-likelihood:

NLL(w) = -sum_i [ y_i * log sigma(w^T x_i) + (1 - y_i) * log (1 - sigma(w^T x_i)) ]

This is exactly binary cross-entropy between y_i (the true label) and sigma(w^T x_i) (the predicted probability). Logistic regression’s standard loss is the MLE under a Bernoulli noise model.

The gradient

d NLL / d w = sum_i (sigma(w^T x_i) - y_i) * x_i

Notice: the gradient is (predicted - true) * input. Same form as linear regression’s gradient under MSE, except the prediction is now passed through a sigmoid. This is not a coincidence; both are MLE under different exponential-family noise models (Bernoulli for logistic, Gaussian for linear).

Learning objective

The probability factors cancel; the input still scales the weight gradient

The sigmoid factors cancel, but the input staysFor a positive label, the two logit-gradient factors multiply to -0.8. The weight still needs the input factor. At x=3, its gradient is -2.4.Positive label: p = 0.2-50.16x-0.8The weight still receives x = 3-0.83x-2.4
Derivation
Chain-rule derivation of the logistic-regression gradient Starting with logit z equal to w transpose x and probability p equal to sigmoid z, the Bernoulli negative log-likelihood is differentiated by the chain rule. Its probability derivative, negative y over p plus one minus y over one minus p, multiplies the sigmoid derivative p times one minus p. The denominators cancel to negative y times one minus p plus one minus y times p, which collects to p minus y. Multiplying by dz over dw equal to x gives dL over dw equal to p minus y times x. For label one the negative residual raises the logit under gradient descent; for label zero the positive residual lowers it. ONE EXAMPLE: z = w^T x, p = sigmoid(z) BERNOULLI NEGATIVE LOG-LIKELIHOOD L = -[y log(p) + (1-y) log(1-p)] 1 - APPLY THE CHAIN RULE LOSS CONTRIBUTION dL/dp = -y/p + (1-y)/(1-p) x SIGMOID dp/dz = p(1-p) 2 - MULTIPLY; THE RECIPROCALS CANCEL dL/dz = [-y/p + (1-y)/(1-p)] p(1-p) = -y(1-p) + (1-y)p = -y + yp + p - yp = p - y SINCE dz/dw = x dL/dw = (p - y)x y=1: p-1 < 0, raise z y=0: p > 0, lower z
Read it this way: follow the two chain-rule factors. Differentiating the log loss introduces 1/p and 1/(1-p); differentiating the sigmoid supplies p(1-p). Those factors cancel, leaving the probability residual p-y. The remaining derivative of w^T x contributes x, so each example updates the weights by (p-y)x. Original derivation checked against the Stanford CS229 notes.
Check: Which factors cancel? Why is the weight gradient (p-y)x rather than just p-y?
Read it this way: For a positive label, the two logit-gradient factors multiply to -0.8. The weight still needs the input factor. At x=3, its gradient is -2.4.
If the input is zero, can the weight receive this nonzero logit gradient?

The L6 connections

“…two things worth noting:

The sigmoid + binary cross-entropy gradient simplifies to (p - y). The same simplification holds for softmax + categorical cross-entropy. This isn’t algebraic coincidence; it’s a property of generalized linear models under the canonical link function. The numerical stability and ease of implementation come from this simplification.

MLE assumes the model is correct. If the true relationship is not log-linear in the features, MLE gives the best parameters under the wrong model. Diagnostic: if predictions don’t fit the data well, the issue is model misspecification, not optimization.

Regularization fits naturally: L2 regularization is MAP estimation with a Gaussian prior on w; L1 is MAP with a Laplace prior. The Bayesian framing makes regularization a derivation rather than an ad-hoc add-on.”

Tells that get you a strong-hire vote

  • You derive cleanly without skipping steps.
  • You identify NLL with binary cross-entropy explicitly.
  • You simplify the gradient to (p - y) * x.
  • You connect to GLMs and the canonical link function.
  • You discuss regularization as MAP with a prior.

Tells that get you down-leveled

  • Stating the formula without derivation.
  • Confusing logistic regression with linear regression.
  • Not knowing the gradient form.
  • Treating cross-entropy as separate from MLE.

Common follow-up

“Why use sigmoid and not another function that maps to (0, 1)?”

The L6 answer:

“The logit is the canonical link for the Bernoulli distribution, and sigmoid is its inverse: it maps the natural parameter (the log-odds) to a probability. This choice gives the gradient simplification we just showed, gives a convex loss in the parameters, and corresponds to maximum-entropy modeling subject to the constraint of matching feature expectations under the data. Other inverse links (e.g., probit, which uses the Gaussian CDF) work but don’t have the same algebraic and computational properties. Probit is occasionally preferred in econometrics for theoretical reasons; logit dominates in ML for the practical reasons above.”

Related: cross-entropy and softmax, regularization, and Bayesian versus frequentist.

Found an error or stale claim? Report a correction or send a private note.