Skip to content
mlmentorship

Object detection: Faster R-CNN, YOLO, DETR

Localize and classify objects in an image. The three main architectural families: two-stage proposal-based, one-stage grid-based, and transformer-based.

Published · 6 min read ·Specialist ·Intermediate

Visual quick review

Visual first · depth when needed

Contrast how proposal- and grid-based detectors collapse redundant candidates with NMS against how DETR learns distinct prediction slots through one-to-one matching and emits its set without NMS.

Preparing the visual…

Summary

Object detection outputs, for each input image, a set of (bounding box, class label, confidence) tuples for the objects present. The three architectural families: two-stage (Faster R-CNN family. First propose regions, then classify), one-stage (YOLO, RetinaNet. Predict box and class in one pass at every grid cell), and set-prediction (DETR. Transformer encoder-decoder predicts a fixed-size set of boxes).

Detection is the core production CV task: autonomous driving, retail analytics, surveillance, medical imaging, robotics. The architectural choice determines latency-accuracy tradeoffs and what’s possible in your inference budget.

Two-stage: Faster R-CNN (Ren et al., 2015)

Architecture:

  1. Backbone (ResNet, ConvNeXt) extracts feature map.
  2. Region Proposal Network (RPN) slides a small network over the feature map, predicts (objectness, box-refinement) at each anchor (preset boxes of various scales / aspect ratios).
  3. RoI Pooling / RoIAlign: extract fixed-size feature for each top-K proposed region.
  4. Classification + bbox regression head: per region.

Strengths: high accuracy; standard for COCO leaderboards; basis for Mask R-CNN (adds segmentation head).

Weaknesses: slow (two stages, hundreds of region computations per image); 5–30 FPS on a GPU.

Used when: accuracy matters more than latency; offline analytics; medical imaging.

One-stage: YOLO (Redmon et al., 2015 onward), RetinaNet (Lin 2017)

Predict boxes and classes directly at every grid cell of the feature map, in a single forward pass:

  • YOLO v1: grid, each cell predicts B bounding boxes + class.
  • YOLO v3: multi-scale predictions across three feature-map levels.
  • YOLO v5/v7/v8/v10/v12: ongoing engineering improvements (anchor-free, attention, distillation).
  • RetinaNet: Focal Loss to handle the extreme class imbalance between foreground and background anchors.

Strengths: real-time (60–300+ FPS); simpler to deploy.

Weaknesses: historically lower accuracy than two-stage; gap closed by ~2020.

Used when: real-time required; embedded / edge deployment; autonomous driving.

Set prediction: DETR (Carion et al., 2020)

A transformer encoder-decoder that outputs a fixed set of predictions (e.g., ):

  1. CNN backbone extracts features.
  2. Flatten to a sequence; transformer encoder processes.
  3. Transformer decoder takes learned object queries; cross-attends to encoder features.
  4. Each query produces (box, class). Including a “no object” class for unused queries.
  5. Loss: bipartite matching (Hungarian) between predictions and ground-truth boxes.

Strengths: no anchors, no NMS; cleaner formulation; scales with transformer pretraining.

Weaknesses: slow training convergence; lower throughput than YOLO. Deformable DETR, DINO (DETR with denoising), Co-DETR addressed convergence and accuracy.

Used when: research / leaderboard work; integration with vision-language pretraining (Grounding DINO, OWL-ViT).

Learning objective

Where does each detector family resolve duplicate predictions?

Classic proposal and dense detectors resolve duplicates after prediction Faster R-CNN proposals and the dense predictions used by RetinaNet and most YOLO versions can produce three overlapping boxes for one object. At inference, non-maximum suppression keeps the highest-scoring box and removes the two lower-scoring duplicates. CLASSIC PROPOSAL / DENSE PIPELINE many scored candidates for one object 0.94 · highest overlap is expected, not three objects INFERENCE · NMS keeps 0.94 one kept box
DETR learns distinct prediction slots before direct inference Four learned object-query slots predict a dog box, a car box, and two no-object values in parallel. During training only, Hungarian bipartite matching assigns at most one query to each ground-truth object. At inference, the slots directly emit the final set and there is no NMS gate. DETR · SET-PREDICTION PIPELINE fixed learned query slots predict in parallel Q1 Q2 Q3 Q4 dog box car box no object no object TRAINING ONLY · one-to-one Hungarian match INFERENCE · direct final set dog + car · no NMS gate
Read it this way: classic proposal and dense detectors may score several boxes for the same object, so NMS removes lower-scoring overlaps at inference. DETR moves uniqueness into training: Hungarian matching gives each ground-truth object at most one query slot, unused slots learn “no object,” and inference emits the resulting set directly.

Key shared components

Bounding box parameterization

Either or . Critical for loss design.

IoU loss

Intersection over Union directly measures box overlap. Variants: GIoU, DIoU, CIoU. Handle non-overlapping boxes better than naive L1 / L2 on coordinates.

Non-Maximum Suppression (NMS)

Post-processing for anchor-based detectors: remove duplicate boxes for the same object by keeping highest-confidence and suppressing others with IoU > threshold (typically 0.5). DETR-family removes the need for NMS by design.

Anchors

Predefined boxes of various scales / aspect ratios. The detector predicts offsets from these. Anchor-free methods (FCOS, CenterNet, YOLOv8) predict box centers directly.

Datasets and metrics

  • PASCAL VOC (legacy): 20 classes; mAP@IoU=0.5.
  • COCO: 80 classes; metric mAP averaged over IoU thresholds 0.5 → 0.95 (the dominant 2026 benchmark).
  • Open Images, LVIS: large-vocabulary detection.

The standard evaluation: average precision per class, then mean across classes (mAP).

Production considerations

  • Latency: YOLO-class models for real-time; Faster R-CNN for accuracy.
  • Class imbalance: many backgrounds, few foregrounds. Focal loss (RetinaNet) addresses this.
  • Small objects: hardest case; multi-scale features (FPN. Feature Pyramid Network) help.
  • Open vocabulary: align detector classes with text embeddings (CLIP-style) for zero-shot detection (OWL-ViT, Grounding DINO).

Common pitfalls

  • Training on small datasets without strong augmentation. Detection is data-hungry; mosaic, mixup, copy-paste augmentations standard.
  • Confusing IoU thresholds. mAP@0.5 ≠ mAP@0.5:0.95; specify which.
  • Forgetting NMS / NMS thresholds. Setting NMS too aggressive merges distinct objects; too loose duplicates them.
  • Using the wrong evaluation tool. COCO eval and VOC eval differ; use the one for your reporting benchmark.
  • Treating YOLO as a single algorithm. Many YOLO versions exist with very different performance; cite the specific version.

Found an error or stale claim? Report a correction or send a private note.