The neural-network training recipe

Summary

A training recipe orders evidence checks before expensive changes: inspect, build a baseline, fit, regularize, tune and refine. Passing a fitting check does not establish generalization.

Consider 32 distinct separable inputs, eight per class, with four classes and no regularization. Zero logits predict each class with probability $1/4$, giving mean cross-entropy $\log 4\approx 1.386294$. Suppose gradients are nonzero but parameters never change. Should you search architectures, or inspect the update path?

Return to the earliest failed check

References: Karpathy's recipe, Deep Learning practical methodology, and Bergstra and Bengio.

Check the update, then fit the known batch

With no update, the same fixed inputs retain the same predictions and loss. Inspect optimizer ownership, learning rate and step(), not only backward().

This complete CPU fixture separates gradient creation from a parameter update. Its 500-update budget and $0.02$ loss threshold are diagnostic choices, not convergence guarantees for other datasets.

import torch
from torch import nn

torch.manual_seed(0)
scales = torch.linspace(0.9, 1.1, 8).repeat(4)
inputs = torch.eye(4).repeat_interleave(8, dim=0) * scales[:, None]
targets = torch.arange(4).repeat_interleave(8)
model = nn.Linear(4, 4)
with torch.no_grad():
	model.weight.zero_()
	model.bias.zero_()
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
criterion = nn.CrossEntropyLoss()
model.train()

before = model.weight.detach().clone()
optimizer.zero_grad(set_to_none=True)
initial_loss = criterion(model(inputs), targets)
initial_loss.backward()
assert model.weight.grad.abs().sum().item() > 0
assert torch.equal(before, model.weight)
optimizer.step()
assert not torch.equal(before, model.weight)

for update in range(499):
	optimizer.zero_grad(set_to_none=True)
	loss = criterion(model(inputs), targets)
	loss.backward()
	optimizer.step()

model.eval()
with torch.no_grad():
	final_loss = criterion(model(inputs), targets)
assert final_loss.item() < 0.02
print(initial_loss.item(), final_loss.item())

A true-class probability of $0.99$ gives $-\log 0.99\approx 0.0100503$, below that illustrative threshold. Neither number establishes held-out performance.

Follow the evidence gates

Change the prior or search purpose

1. Initialize a nonuniform prior

New task: class proportions become [0.7, 0.1, 0.1, 0.1].

Picture and change: use zero final-layer weights and biases $b_k=\log {\pi }_k$, for positive priors. Softmax then predicts the declared proportions.

Check: mean loss at those frequencies is $-{\sum }_k{\pi }_k\log {\pi }_k\approx 0.940448$, not $\log 4$. That lower initial loss is not a bug.

Use when: a deliberate marginal baseline replaces uniform predictions.

2. Spend 12 trials on search

New task: the verified pipeline has 12 tuning trials left.

Picture and change: sample learning rate and weight decay jointly over declared log ranges. For causal diagnosis, instead vary one factor under a matched comparison.

Check: a three-rate by four-decay grid tests only three rates. Twelve continuous random draws can explore twelve, but do not isolate a causal learning-rate effect.

Use when: broad search coverage, rather than an explanatory intervention, is the goal.

A tiny-batch failure is not always a bug

Two identical inputs with contradictory binary labels cannot both be fitted by a deterministic predictor. Their minimum mean cross-entropy is $\log 2$, reached by predicting one half. Inspect target consistency, capacity and the attainable loss before demanding zero.

Transfer to a binary baseline

For a 1% positive prior, must initial loss equal $\log 2$?