Skip to content
mlmentorship

Cross-validation strategies

Hold-out, k-fold, stratified, grouped, and time-series CV. And when each one is and isn't appropriate.

Published · 7 min read ·Core ·Intermediate
Reading depth

Summary

Cross-validation scores a training procedure on held-out observations under a declared deployment boundary. Decide whether the target is a new row, an unseen entity, or a future observation before choosing folds. The preprocessing, selection, fitting, scoring, and aggregation must all respect that boundary.

For a hypothetical new-patient task, patients A, B, and C each supply three visits. Hold out each entire patient, fitting on the other two; suppose the respective held-out errors are zero, one, and two. Equal-patient mean error is , matching because sizes are equal. Putting A1 into fitting and A2 into scoring fails this particular unseen-patient claim, even with balanced labels.

Learning objective

Choose splits that preserve users, time, and the evaluation claim

Choose the split that matches the deployment claimRandom rows can leak user identity or future information. Group and time boundaries protect different dependencies; neither is interchangeable with random splitting.The held-out boundary must respectuser identityRandom rows: A crosses train and testAtrainAtestAtrainGroup split: all three A rows staytogetherAtrainAtrainAtrain
Derivation
Three cross-validation splitting constraints Three panels compare validation rules. For independent classification rows, two folds each contain four negative examples and two positive examples, preserving class proportions. For grouped rows, all examples from groups A and B remain in training while every example from group C is held out for validation. For time-ordered rows, two successive folds use an expanding prefix for training and only later observations for validation. Every region is labelled train or validate, so meaning does not depend on color. CHOOSE THE BOUNDARY PRODUCTION WILL ENFORCE I.I.D. CLASSIFICATION preserve label ratio fold 1 TRAIN N N P N N P VALIDATE N N P fold 2 TRAIN P N N P N N VALIDATE P N N REPEATED ENTITIES keep each group intact TRAIN A1 A2 A3 B1 B2 B3 group A group B VALIDATE C1 C2 C3 whole group C ORDERED EVENTS validate only on the future fold 1 TRAIN t1 - t3 VALIDATE t4 fold 2 TRAIN t1 - t4 VAL t5 time moves left to right; training history expands →
first identify what will be unseen. Stratification can preserve class proportions. Unseen-entity evaluation keeps each entity intact. Forecast evaluation respects information available before its cutoff. These constraints can need composition. The original diagram follows scikit-learn's cross-validation guide.
Check: What must stay separated when the same users generate events over time?
Read it this way: Random rows can leak user identity or future information. Group and time boundaries protect different dependencies; neither is interchangeable with random splitting.
Which assignment keeps user A from appearing on both sides?

Standard schemes

Single hold-out

Fit on one subset and score once on another. Use when the held-out population is adequate and repeated training costs more than its expected diagnostic value. One split does not by itself quantify selection or sampling uncertainty.

k-fold

Fit models, holding out one fold each time. For equal-sized folds and the same loss definition, average their scores; unequal folds need estimand-appropriate weights. For equal-user risk, first average within users instead of letting frequent users silently dominate. Fold training sets overlap, so their scores are dependent. The fixture’s fold sample standard deviation is , descriptive variability rather than a justified confidence interval.

Stratified k-fold

Preserve label proportions when compatible with the deployment boundary and available independent units. Check that each fold supports the requested metric. Class balance does not repair group overlap, future-derived features, or preprocessing leakage.

Group / GroupKFold

Assign every observation of an entity to the same fold when evaluating unseen entities. User, patient, or document grouping prevents that identity from crossing the boundary, not every possible leakage mechanism.

Time-series / TimeSeriesSplit

For forecasting, train on information available before the validation cutoff and score later observations. Check label availability, overlapping outcome windows, and necessary gaps or purging. A timestamp alone does not establish that forecasting is the intended evaluation task.

New task: Predict later visits for already known patients.

Picture and change: Replace unseen-patient exclusion with valid temporal cutoffs while retaining permitted identity history.

Check: Training through day ten may score day eleven, not day nine using day-ten information.

Use when: Future behavior, rather than unseen identity, is the claim.

Nested CV

Keep configuration selection inside each outer training subset; reserve its outer validation subset for scoring the selection procedure. Fit preprocessing, feature selection, and calibration only within their permitted training subsets too.

New task: Compare two configurations using two inner folds inside each of three outer folds.

Picture and change: Run the inner search, select, then refit on the outer training subset.

Check: fits, versus three without tuning. A later full-data refit adds one more.

Use when: Selection needs independent outer scoring and the compute budget supports it. The estimate still reflects its sampling design and training sizes, not an unconditional unbiasedness guarantee.

When NOT to cross-validate

  • Adapted test set. Repeatedly changing the procedure after seeing test results removes its untouched-test interpretation.
  • Full-data fitting. Feature selection, normalization, imputation, and target-derived transformations belong inside their training boundaries.
  • Inappropriate deployment model. Repeated random splits cannot establish a future-only or unseen-entity claim they do not enforce.

Common pitfalls

  • Wrong aggregation. Equal-fold, equal-row, and equal-entity averages need not match when sizes differ.
  • False interval. Mean plus/minus fold standard deviation is not automatically a confidence interval.
  • Wrong information time. Event timestamps do not prove that their features and labels were available at training time.

Reconstruct

What does the three-patient result fail to establish?

It does not provide a causal policy effect, a calibrated confidence interval, or a universal deployment guarantee. Its interpretation remains conditional on the hypothetical held-out predictions, chosen population, loss, and sampling design.

Found an error or stale claim? Report a correction or send a private note.