CNN architecture

Summary

A convolution reuses one local linear operator across spatial positions; a CNN composes these operators with nonlinearities and other layers. Weight sharing ties parameters, while stride, padding, dilation, and channel counts determine shapes and information loss. The architectural bias does not by itself guarantee invariance, semantic hierarchy, or superior sample efficiency.

The building block: convolutional layer

Use channels-last notation here, with input $H \times W \times C_{in}$ . An ungrouped layer has $k_{H} k_{W} C_{in} C_{o u t}$ weights and optionally $C_{o u t}$ biases. Along either spatial axis,

H_{o u t} = ⌊ \frac{H + 2 p - d ( k - 1 ) - 1}{s} ⌋ + 1.

Here $p, d, s$ are padding, dilation, and stride; require valid positive output dimensions.

For unit stride and dilation, omitting boundary indexing, the common cross-correlation implementation computes:

y_{ij, c_{out}} = c_{in} \sum u, v \sum W_{u, v, c_{in}, c_{out}} \cdot x_{i + u, j + v, c_{in}} + b_{c_{out}} .

For a numerical fixture, use input $32 \times 32 \times 3$ , sixteen $3 \times 3$ filters, stride one, padding one, and dilation one. Output is $32 \times 32 \times 16$ ; parameters total $3 (3) (3) (16) + 16 = 448$ . There are $32 (32) (16) (27) = 442368$ multiply-accumulates, excluding biases. Counting one multiply and one add per MAC gives $884736$ FLOPs under that convention. The diagram uses different illustrative channel counts: thirty-two, then sixty-four.

Learning objective

Pooling collapses space while keeping one output per channel

Derivation

Read it this way: follow the stated grid and channel dimensions from left to right. These illustrated layers preserve, then downsample, the grid while changing channel width. Global average pooling produces one value per channel. The spatial choices determine which locations remain distinguishable.

Check: Why does spatial pooling produce 1x1x64 rather than one scalar for the entire image?

Read it this way: Global average pooling reduces every spatial channel map to one number. It does not merge all channels into one scalar.
How many outputs remain when only the spatial axes are averaged?

Standard CNN ingredients

Spatial convolution: shared local feature computation.
ReLU/GELU: nonlinear composition rather than one collapsed affine map.
Normalization: an implementation choice, not a prerequisite for every trainable CNN.
Pooling/strided convolution: different downsampling operators with distinct aliasing and information effects.
Global average pooling: one spatial mean per channel, not universal input-transformation invariance.
$1 \times 1$ convolution: channel mixing at each location.

Receptive field

Track receptive-field extent $r$ and input-coordinate spacing $j$ , starting from $r = j = 1$ . One layer gives $r^{'} = r + (k - 1) d j$ and $j^{'} = j s$ per axis. Two stride-one $3 \times 3$ layers therefore span five input positions, not an exponential extent. Boundary handling and strided sampling qualify translation equivariance; arbitrary one-pixel shifts need not produce one-cell output shifts.

Change the Requirement

Preserve Fine Spatial Evidence

New task: Retain small-object or boundary information that downsampling removes.

Picture and change: Consider higher-resolution paths, suitable dilation, or multiscale features rather than merely widening channels.

Check: Two $2 \times 2$ patches with one active pixel in different locations have the same average, $1/4$ . More channels after that scalar average cannot reconstruct which location was active.

Use when: Localization matters; measure compute, aliasing, and dense-task performance together.

Reduce Channel Work

New task: Replace full convolution with depthwise spatial filtering followed by pointwise mixing.

Picture and change: For the fixture, weight counts become $3 (3) (3) + 3 (16) = 75$ , ignoring biases.

Check: Arithmetic becomes $32 (32) (75) = 76800$ MACs, but the restricted factorization cannot represent every full convolution.

Use when: Measured hardware cost binds; fewer MACs do not guarantee lower latency.

Architectural eras

Era	Architecture	Key idea
2012	AlexNet	First major win; ReLU + dropout + GPU
2014	VGG	All $3 \times 3$ convs, very deep
2014	GoogLeNet / Inception	Multi-scale modules, dim reduction
2015	ResNet	Residual connections enable 50+ layers
2016	DenseNet	Dense feature reuse
2017	MobileNet	Depthwise separable convs for efficiency
2019	EfficientNet	Compound scaling depth × width × resolution
2020	ViT	Transformers replace CNN backbones
2022	ConvNeXt	Modernized ResNet matching ViT performance

The table names historical design examples, not a measured ranking or deployment census.

ConvNeXt and the modern CNN

ConvNeXt (Liu et al., 2022) modernized ResNet-50 by adopting transformer-era design choices:

LayerNorm instead of BatchNorm.
GELU instead of ReLU.
Larger kernels ( $7 \times 7$ depthwise).
Inverted bottleneck (channels-up then down).

Compare complete training recipes, workloads, quality targets, and hardware before attributing a result solely to convolution versus attention.

Common pitfalls

Shape assumptions. A $1 \times 1$ valid convolution need not shrink the grid; stride and padding must be declared.
Axis confusion. Distinguish the chosen tensor layout from the library’s default and physical memory layout.
Invariance claims. Pooling averages feature positions; it does not make every input transformation irrelevant.
Theoretical versus effective context. A receptive-field bound lists possible dependencies, not equal influence from every input pixel.

Residual connections. What made deep CNNs trainable.
BatchNorm vs LayerNorm. Normalization for vision.
Vision transformers. Alternative paradigm.