Semantic segmentation

Summary

Semantic segmentation predicts a semantic class at each evaluated pixel, rather than a box or a separate object identity. An encoder can recognize an object while discarding its precise boundary. Ask which spatial evidence survives and which denominator reveals a small-class failure.

Output logits have shape $B \times C \times H \times W$ : batch, classes, height, and width. Align predictions with the target coordinates and exclude declared ignore labels. Resize categorical targets by nearest-neighbor unless a separate soft-label policy applies.

Architectures

FCN: replaces dense heads with convolutional prediction.
U-Net: combines coarse semantics with retained higher-resolution encoder features.
DeepLab: uses dilation and multiscale context with a declared output stride.
Transformer decoders: combine global context with learned masks or pixel queries.

If two boundaries produce the same coarse representation, its decoder alone cannot identify which boundary occurred. For example, averaging $(1, 0)$ and $(0, 1)$ yields the same value $1/2$ . Interpolation restores size, not that erased distinction. A learned prior may infer plausible detail; a skip supplies additional evidence without guaranteeing exact recovery.

Learning objective

Enlarging a coarse grid cannot recover an edge location that downsampling lost

Derivation

the fine grid retains the edge, while the coarse grid leaves an uncertain cell. Enlarging that cell does not restore distinctions already erased. The dashed skip supplies fine evidence that can distinguish candidate boundaries; success still depends on those features and the decoder. The illustrated sharp edge is an outcome, not a reconstruction guarantee. Original synthesis informed by Long et al. (2015) and Ronneberger et al. (2015).

Check: Why does enlarging the coarse grid alone not determine the original one-pixel edge location?

Read it this way: Two high-resolution patterns can average to the same coarse value. Enlarging that value cannot identify which original boundary was present.
Which original pattern can be recovered from the coarse value alone?

Losses

Mean pixel cross-entropy averages only evaluated pixels; an empty valid set needs an explicit skip/error policy. Weighted or focal losses change contributions; Dice emphasizes overlap; boundary terms emphasize localization. None removes annotation uncertainty or automatically supplies calibrated probabilities.

Metrics

Intersection over Union for class $c$ is

$IoU_{c} = \frac{T P _{c}}{T P _{c} + F P _{c} + F N _{c}} .$

For 95 background and five object pixels, predicting background everywhere gives:

Class	TP	FP	FN	IoU
Object	0	0	5	0
Background	95	5	0	0.95

Accuracy is $0.95$ , while two-class macro mIoU is $(0 + 0.95) /2 = 0.475$ . The metric correctly measures pixel agreement, not rare-class success. Zero IoU denominators require a stated absent-class policy; ignored pixels are not background.

Make rare positives more important

New task: prioritize the small foreground class.

Picture and change: weight positive cross-entropy terms by $19$ , background terms by $1$ .

Check: weighted class masses become $19 (5) = 95$ and $1 (95) = 95$ . A constant-score model’s optimum changes from foreground probability $0.05$ to weighted score $0.5$ . Report class-specific results and validate calibration rather than interpreting the weighted score as the original probability.

Use when: missed foreground carries different task costs; the all-background object’s IoU remains zero.

Fit inference into smaller tiles

New task: full-resolution activations exceed memory.

Picture and change: a $51 2^{2}$ tile uses 262,144 pixels versus 921,600 for a $96 0^{2}$ image. Along each axis, origins $0$ and $448$ produce a 64-pixel overlap.

Check: global coordinate $500$ is local coordinate $52$ in the second tile. Align and blend overlapping logits before class selection; do not average discrete class IDs. Context, padding, and tile boundaries may still differ from full-image inference.

Use when: activation memory requires tiling, with measured overlap, latency, and boundary checks.

In an interview

Define the spatial transform, ignore mask, label set, and evaluation unit before choosing an architecture. Report per-class IoU, the declared class average, boundary quality, and relevant object/task outcomes. Memory savings require accounting for feature channels, temporary workspaces, tiling overlap, and batch size, not pixels alone.

Common confusions

Hard binary Dice is $2 T P / (2 T P + F P + F N)$ . One soft convention uses $(2 \sum_{u} p_{u} y_{u} + ϵ) / (\sum_{u} p_{u} + \sum_{u} y_{u} + ϵ)$ . Declare $ϵ$ , valid pixels, per-image versus pooled reduction, and the empty-mask policy. Other denominator conventions define different objectives.

Boundary penalties can emphasize uncertain or inconsistent annotations. Confidence thresholds and abstention need validation on the deployment population; higher mask precision is not automatically higher task value.

Can semantic labels distinguish two touching objects of the same class?

Answer

Not necessarily: both objects can share the same semantic label throughout their union. Instance segmentation supplies separate identities; detection supplies boxes rather than full pixel assignments.

Related: CNN architecture, ResNet, and object detection.