Skip to content
mlmentorship

Semantic segmentation

Assign a class to every pixel: encoder-decoder architectures, losses, IoU, class imbalance, boundaries, and deployment constraints.

Published · 5 min read ·Specialist ·Intermediate

Visual quick review

Visual first · depth when needed

Trace why decoder upsampling can restore output size but not an erased object boundary, and explain how a same-scale encoder skip supplies the fine spatial evidence needed to place that boundary.

Preparing the visual…

Summary

Semantic segmentation is the dense version of classification: a class label for every pixel, which is what scene parsing, medical imaging, and autonomous perception actually need. Two things make it hard. A network downsamples to build semantic context, so the central challenge is recovering spatial precision at the boundaries it blurred. And the obvious metric lies: pixel accuracy can look excellent while the model quietly predicts background everywhere. It differs from object detection, which predicts boxes, and instance segmentation, which separates individual objects of the same class.

Architectures

  • FCN: replaces dense heads with convolutional prediction.
  • U-Net: encoder-decoder with skip connections that restore spatial detail.
  • DeepLab: dilated convolution for multi-scale context without losing resolution.
  • Transformer decoders: combine global context with learned masks or pixel queries.

The recurring trade-off is semantic context versus precise boundaries.

Learning objective: trace why upsampling restores output size but cannot by itself recover an erased object boundary, then identify what a same-scale encoder skip contributes.

Learning objective

Why does a decoder need fine features, not only more pixels?

Downsampling loses exact boundary location, while an encoder skip supplies fine evidence to the decoder A fine eight-column input feature grid contains an object edge between columns five and six. Encoding and downsampling produce a coarse four-column semantic grid where the edge occupies an uncertain cell. Upsampling that coarse grid creates a larger mask but leaves a wide dashed uncertain boundary. A dashed skip path carries same-scale fine encoder features around the bottleneck; fusing those features with coarse semantics places a sharp solid boundary between columns five and six. SPATIAL SIZE IS NOT THE SAME AS SPATIAL EVIDENCE 1 · FINE ENCODER FEATURES high resolution · exact edge retained edge at one pixel boundary encode + downsample 2 · COARSE SEMANTICS object recognized · edge location compressed object object edge? wide cell background upsample alone 3 · LARGER GRID, SAME UNCERTAINTY boundary could lie here SKIP fine edge evidence 4 · FUSE SEMANTICS + SAME-SCALE DETAIL
Read it this way: start at the fine encoder grid: it retains where the object ends. Downsampling builds stronger context but compresses that edge into a coarse cell. Upsampling can produce an eight-cell mask again, yet the dashed two-cell region shows that a larger canvas does not recreate the discarded location. The dashed skip carries same-scale fine evidence around the bottleneck; fusing it with coarse semantics lets the decoder place the solid boundary. Skips help localization, but they do not guarantee perfect reconstruction. Original synthesis informed by Long et al. (2015) and Ronneberger et al. (2015).

Losses

Pixelwise cross-entropy is the baseline. Class-weighted or focal losses handle imbalance. Dice loss rewards overlap directly and is common when the positive region is small. Boundary losses emphasize shape but are sensitive to annotation noise.

Metrics

Intersection over Union for class is

Mean IoU averages across classes. Report per-class IoU and boundary quality when rare or safety-critical classes matter, because pixel accuracy can look excellent by predicting background.

In an interview

  1. Separate semantic from instance segmentation.
  2. Choose an encoder-decoder and explain how resolution is recovered.
  3. Discuss imbalance, annotation quality, and augmentation.
  4. Use mIoU plus critical-class and boundary metrics.
  5. Cover tiling, latency, memory, and confidence handling at deployment.

Common confusions

  • “Accuracy is enough.” Background dominance hides the failures that matter.
  • “Upsampling recovers lost detail.” Skip connections or high-resolution features carry the information; interpolation alone cannot.
  • “More precise masks are always better.” Annotation boundaries are often uncertain, and task value may depend on object-level outcomes.

Related: CNN architecture, ResNet, and object detection.

Found an error or stale claim? Report a correction or send a private note.