How do you choose a loss function?

Choose a loss by identifying the quantity to predict and separating that estimate from the later deployment decision.

Problem

Context. Three equally weighted outcomes, 0, 0, and 9, represent the intended population for a constant predictor.

Task. Find the best constant under squared and absolute error, compare their objective values, and explain the implied prediction targets. Then distinguish changing a prediction loss from changing the costs of a downstream action.

Constraints. Use the same stipulated population and weights for both fits. Neither loss is universally better. A changed decision threshold does not always require retraining the probability model.

Predict one constant $a$ for three equally weighted outcomes: $0, 0, 9$ . All three represent the intended population. Find the best constant under squared error and absolute error. Then ask whether changing action costs requires a different prediction model.

Estimate a quantity, then choose an action

The same observations support different prediction targets

A probability of 0.8 does not pick an action without its costs

Derivation

Read it this way: lower bars win within each comparison, not across different losses. Squared distances favor 3; absolute distances favor 0. The separate action example chooses negative despite an unchanged positive probability of 0.8.

Check: Why can probability 0.8 favor a negative action under these costs?

Read it this way: False-positive cost is 10; false-negative cost is 1. Expected positive-action cost is 0.2x10=2; expected negative-action cost is 0.8x1=0.8.
Would the same score choose the same action if false-positive cost fell to one?

References: Gneiting and Raftery on proper scoring rules, Huber on robust location, and scikit-learn's threshold guidance.

Calculate both targets

For $n$ observations, define $MSE (a) = n^{- 1} \sum_{i} (a - y_{i})^{2}$ and $MAE (a) = n^{- 1} \sum_{i} ∣ a - y_{i} ∣$ . Here,

MSE (a) = \frac{a ^{2} + a ^{2} + ( a - 9 ) ^{2}}{3}, \frac{d MSE}{d a} = 2 a - 6.

Squared error selects $a = 3$ , the mean. Absolute error selects $a = 0$ , the median.

Constant	MSE	MAE
$0$	$27$	$3$
$3$	$18$	$4$

If nine is a genuine costly outcome, discarding it as an outlier changes the task. At each input, squared and absolute loss similarly target conditional means and medians when their risks exist.

Change the requirement

1. Limit contamination influence

New task: nine may be corrupted, and the required output is a robust location.

Picture and change: Huber loss bounds each residual’s gradient. For residual $r = a - y$ and scale $δ > 0$ ,

H_{δ} (r) = {r^{2} /2, δ (∣ r ∣ - δ /2), ∣ r ∣ \leq δ, ∣ r ∣ > δ .

Check: with $δ = 1$ at $a = 0$ , the nine observation contributes gradient $- 1$ , not squared-error gradient $- 18$ . The mean Huber loss here is minimized at $a = 0.5$ because $2 a - 1 = 0$ .

Use when: contamination may cause large residuals, rather than a genuine tail needed for mean estimation.

2. Change action costs

New task: false positives cost $9$ , false negatives cost $1$ , and the calibrated positive probability is $p = 0.8$ . Correct decisions have zero cost.

Picture and change: compare the expected costs of both actions before retraining. Choose positive when

C_{FP} (1 - p) < C_{FN} p ⟺ p > \frac{C _{FP}}{C _{FP} + C _{FN}} .

Check: the threshold is $0.9$ . Positive action costs $1.8$ in expectation; negative costs $0.8$ . Choose negative.

Use when: probabilities remain suitable but the cost of acting changes. Capacity and other constraints may require a different decision rule.

Do not treat every score as a probability

Weighted or focal objectives can change score meaning. Check calibration before applying a threshold derived for true probabilities. Focal modulation emphasizes difficult examples; it does not specify which error costs more.

Try increasing the precision requirement

The model already ranks relevant cases well. What changes first?

Check your answer

Test thresholds on representative validation data, separate from final evaluation. Report recall and accepted volume. Retrain only if the resulting operating choices fail the requirement.

Likelihood and output meaning

Negative log likelihood is $- lo g p_{θ} (y ∣ x)$ . Fixed-variance Gaussian likelihood gives squared error up to constants and scale. Fixed-scale Laplace likelihood gives absolute error. Learned scales require their log terms too. These likelihood assumptions are not required merely to target means or medians.

Categorical cross-entropy is $- \sum_{c} y_{c} lo g p_{c}$ for one-hot targets. Squared error on probability vectors can also be a proper Brier score. Gradient problems depend on the output link and parameterization, not just the loss name.

Focal loss multiplies log loss by $(1 - p_{t})^{γ}$ for the true-class probability $p_{t}$ . Optional class weights are separate from that modulation; see the original focal-loss paper.

Common follow-up

For a higher precision target, thresholding may suffice. Other options include cost-sensitive retraining, a ranking surrogate, or high-recall retrieval followed by a precision-focused filter. Evaluate representative held-out precision, recall, volume and segment behavior.

Combined objectives

Ranking losses target order; contrastive and triplet losses shape embedding relationships. Multi-task weights change the combined objective. Uncertainty weighting and GradNorm use different weight-selection rules. Auxiliary reconstruction or load-balancing losses also change the optimized objective, even when the main term stays unchanged. Next-token, preference and reward-based training objectives need downstream behavioral evaluation rather than assuming equal loss reductions mean equal improvements.

How do you choose a loss function?

Practice before you read