Dropout

Summary

Inverted dropout randomly masks activations and scales survivors to preserve each activation’s conditional expectation. Standard evaluation replaces that random operation with identity; it does not exactly average an arbitrary nonlinear network. Mask correlation and placement are part of the regularizer, not merely implementation details.

The mechanism (inverted dropout)

For fixed finite $x \in R^{d}$ and $0 \leq p < 1$ :

Sample a binary mask $m \in {0, 1}^{d}$ with $m_{i} \sim Bernoulli (1 - p)$ independently.
Compute $y = m ⊙ x \cdot \frac{1}{1 - p}$ .

Then $E [y_{i} ∣ x] = x_{i}$ and $V a r (y_{i} ∣ x) = p x_{i}^{2} / (1 - p)$ . For the pictured $x = (1, 2, 3, 4)$ and $p = 0.5$ , mask A produces $(2, 0, 6, 0)$ and B produces $(0, 4, 0, 8)$ . Their coordinate average is x; independently sampled masks need not arrive as complementary pairs. For coordinate two, the random output is zero or four with equal probability: mean two, variance four.

Worked expectation

Scaling preserves expected activation, not every realized training pass

Derivation

Training: mask, then scale survivors by 2

For p = 0.5, the keep probability is 1 - p = 0.5, so 1 / (1 - p) = 2.

Vector	1	2	3	4
Input x	1	2	3	4
Mask A	keep	drop	keep	drop
Output A	2	0	6	0
Mask B	drop	keep	drop	keep
Output B	0	4	0	8

Expectation: average restores x

The paired masks are only an illustration; independently sampled masks have the same expectation.

Vector	1	2	3	4
Mean(A, B)	1	2	3	4
Inference y	1	2	3	4

E[m_ix_i / (1 - p)] = (1 - p)x_i / (1 - p) = x_i

Read it this way: a survivor doubles because it survives with probability one-half. The two illustrative masks average back to x. This establishes an activation expectation, not an exact nonlinear model average.

Check: Does inverted dropout guarantee that two real training passes average exactly to the unmasked input?

Read it this way: With keep probability 1/2, activation 4 becomes either 0 or 8. Their average is 4; neither individual pass preserves the input.
Does one pass return the mean of the two branches?

For the counterexample $x = 1$ , $p = 0.5$ , and $f (y) = y^{2}$ , $E [f (d r o p o u t (x))] = 2$ but $f (E [d r o p o u t (x)]) = 1$ . Preserving E[y] therefore does not prove an exact ensemble prediction after subsequent nonlinear computation.

Change Drop Probability

New task: Reduce p from one-half to one-quarter for coordinate value two.

Picture and change: The retained value becomes $8/3$ with probability $3/4$ , rather than four with probability one-half.

Check: Mean remains two while variance falls from four to $4/3$ . One retained output is not the expectation.

Use when: Held-out behavior supports changing stochastic regularization strength; no architecture-independent rate is optimal.

New task: Drop whole channels in a two-channel $2 \times 2$ activation.

Picture and change: Broadcast mask [1,0] over spatial positions at p one-half. Channels constant at one and three become constant maps two and zero.

Check: Independent element masking could keep just one pixel in the second channel; this channel-shared mask cannot. Marginal expectations still match, but correlations between masked positions differ.

Use when: The intended regularizer removes an entire correlated feature map.

PyTorch nn.Dropout(p) follows its module’s train/eval mode; standard evaluation returns x without inverted scaling.

Disabling gradient recording with no_grad alone does not change that mode. Evaluation mode alone does not disable gradients.

Where to apply

Architecture	Where dropout typically goes	Typical $p$
CNN	Element or channel masking at chosen sites	Tune against a matched baseline
Transformer	Selected attention, FFN, or residual sites	Zero is also a candidate
RNN	Inter-layer or explicitly time-shared masks	Match the recurrent implementation
Embeddings	Selected input features	Preserve task semantics

Variants

Spatial dropout: channel-shared masking.
DropConnect: weight masking.
Variational recurrent dropout: masks shared across timesteps under its specified model.
Stochastic depth: residual-branch masking, with its own scaling convention.

Relation to other regularization

Noise-induced penalties can be derived for restricted models, but dropout is not generally identical to weight decay. Normalization, augmentation, and masking can interact rather than contributing independent benefits.

Common pitfalls

Mode confusion: intentionally sampling dropout at inference differs from standard evaluation and does not automatically yield calibrated uncertainty.
Unsupported causality: a smaller train/validation gap alone does not prove improved deployment quality.
Wrong mask units: element, channel, timestep-shared, weight, and branch masks induce different dependencies.