Skip to content
mlmentorship

Microannealing and midtraining

Compare matched late-checkpoint continuations to test a data-mixture change, without confusing domain loss, task gains or full-training evidence.

Published · 7 min read ·Core ·Intermediate
Reading depth

Summary

Microannealing tests a data-mixture change through short continuations from the same mostly trained state. It measures a local intervention, not every possible full-training recipe. Midtraining is broader: it includes changes to data, curriculum or training between larger training stages.

Use a hypothetical 1000-token continuation to compare base data with a 70% base, 30% candidate mixture. The stated decision rule requires task score at least as high as control. All numbers below are invented teaching values, not Marin measurements or recommended training budgets.

Hypothetical controlled comparison

A better domain loss cannot override a failed preservation check

A domain gain cannot waive a preservation failureThe probe improves domain loss but fails the declared preservation check. Promotion requires both the gain and preservation conditions.Every required gate must pass on thesame route+domain gainxpreserveThe probe stays unpromoted after itspreservation failure.
Derivation
A controlled microannealing fork with conflicting evaluation signals Hypothetical teaching data, not Marin measurements. One saved checkpoint forks into matched 1000-token cooldowns. Control uses 1000 base tokens and obtains domain loss 2.0 and task score 0.60. Candidate uses 700 base and 300 candidate tokens and obtains loss 1.8 and task score 0.58. The candidate fails the prespecified task rule despite its lower domain loss. ONE LATE CHECKPOINT · TWO MATCHED COOLDOWNS mostly trained checkpoint same weights and optimizer state CONTROL 100% normal mix reference cooldown CANDIDATE EXAMPLE 70% base + 30% HQ one changed factor LOCKED IN BOTH RUNS same LR decay · 1000 tokens each same evaluation suite HYPOTHETICAL RESULTS · CONTROL → CANDIDATE MATCHED-DOMAIN LOSS 2.0 → 1.8 DOWNSTREAM TASKS 0.60 → 0.58 reject under the task rule
both arms receive the same continuation budget. Compare both evaluation rows with control. Lower domain loss does not rescue a failed task criterion. These values are hypothetical.
Check: Why is the candidate rejected when its matched-domain loss improves from 2.0 to 1.8?
Read it this way: The probe improves domain loss but fails the declared preservation check. Promotion requires both the gain and preservation conditions.
Can improved domain loss waive the preservation failure?

The procedure

Freeze the comparison contract before running either arm:

  • Starting state: model, optimizer, scheduler, random state and checkpoint digests.
  • Data: versions, deduplication rules, mixture sampling, token counts and evaluation exclusion.
  • Continuation: schedule, batch, precision, execution conditions and measured compute.
  • Evaluation: suite versions, repetitions, metrics and decision criteria.

Change the mixture while holding the continuation design fixed. Equal tokens do not automatically imply equal compute; record both. Include failed runs under a stated rule, not only successful checkpoints. For schedule context, see WSD and WSD-S.

Read all three hypothetical arms

ArmBase / HQ / instruction tokensDomain lossTask score
Control
Candidate
Mixed candidate

The candidate fails the task rule. The mixed candidate is promising, but needs independent confirmation after selection. Comparing either cooldown with the uncooled starting checkpoint would confound mixture and continuation effects.

Change the experimental question

1. Change the substitution ratio

New task: choose between 15% and 30% candidate data.

Picture and change: start both arms from the same state with the same total continuation budget. Specify the ratio comparison before evaluation and reserve confirmation evidence.

Check: 1000 tokens contain 150 versus 300 candidate tokens. Increasing total tokens would introduce another changed factor.

Use when: the source appears useful but its substitution rate is uncertain.

2. Choose a from-scratch recipe

New task: decide whether the candidate belongs in pretraining from initialization.

Picture and change: use the late probe as limited evidence, then design an appropriately scaled full-training comparison. Checkpoint and horizon interactions may change the result.

Check: a cooldown win from one late state does not establish a win from random initialization.

Use when: the actual decision exceeds the probe’s starting state or continuation regime.

Matched budgets do not prevent leakage

A candidate can improve an evaluation contaminated by its added source. Deduplicate and check evaluation overlap; equal token counts do not repair leakage.

Try explaining the mixed candidate

Does HQ plus instruction data winning prove that instruction format caused the gain?

Check your answer

No. Content, coverage and format changed together. Use matched alternatives that distinguish those explanations, then confirm on independent evidence.

What the experiments actually showed (Marin)

The Marin 8B retrospective reports three local findings:

  1. Oversampling HQ sources improved HQ-domain loss but degraded general task performance.
  2. Within its experiment budget, 70% base / 15% FLAN / 15% HQ gave its best microannealing results.
  3. A 70% base / 30% FLAN mix underperformed the base-only control.

The authors propose lost few-shot-friendly structure as an explanation, not an isolated causal result. The retrospective points to issue 784 and issue 820. Its ratios are experiment-specific, not universal recommendations.

Common pitfalls

Probe cost, useful horizon and predictive transfer depend on model, checkpoint, source and evaluation. No fixed token budget or fraction guarantees a reliable signal. Repeated searches on one evaluation suite also create selection bias; reserve confirmation evidence.

When to use it vs. full ablations

SituationUse
Deciding whether to include a candidate dataset in your final cooldown mixMicroannealing
Comparing two pretraining recipes from scratchFull ablation at small scale
Tuning the mixing ratio of a known-good datasetMicroannealing
Comparing optimizers, architectures, or LR schedulesFull ablation, microannealing is not a substitute

Changing architecture or optimizer introduces a different experimental question. The label microannealing does not make such comparisons controlled.

What an interviewer expects you to say

Specify the intervention, matched control, decision metric and scope of the conclusion. Explain why the tested checkpoint and horizon are relevant to the actual data decision.

Further reading

Found an error or stale claim? Report a correction or send a private note.