Skip to content
mlmentorship

Cross-validation strategies

Hold-out, k-fold, stratified, grouped, and time-series CV. And when each one is and isn't appropriate.

Published · 6 min read ·Core ·Intermediate

Visual quick review

Visual first · depth when needed

Choose stratified, grouped, or time-series cross-validation by identifying which class balance, entity boundary, or temporal direction the split must preserve.

Preparing the visual…

Summary

Cross-validation estimates a model’s generalization error by repeatedly partitioning the training data into a fitting set and a validation set, training on the first, scoring on the second, and averaging the scores. The right partitioning scheme depends on the data’s structure (i.i.d. vs. grouped vs. temporal).

Pick the wrong CV scheme and your validation score is optimistically biased. The model looks great in CV and falls apart in production. The classic failures are (a) k-fold on grouped data leaking the same group into both folds, and (b) random splits on time-series leaking the future into the past.

Learning objective

Choose the split that preserves the dependency in your data.

Three cross-validation splitting constraints Three panels compare validation rules. For independent classification rows, two folds each contain four negative examples and two positive examples, preserving class proportions. For grouped rows, all examples from groups A and B remain in training while every example from group C is held out for validation. For time-ordered rows, two successive folds use an expanding prefix for training and only later observations for validation. Every region is labelled train or validate, so meaning does not depend on color. CHOOSE THE BOUNDARY PRODUCTION WILL ENFORCE I.I.D. CLASSIFICATION preserve label ratio fold 1 TRAIN N N P N N P VALIDATE N N P fold 2 TRAIN P N N P N N VALIDATE P N N REPEATED ENTITIES keep each group intact TRAIN A1 A2 A3 B1 B2 B3 group A group B VALIDATE C1 C2 C3 whole group C ORDERED EVENTS validate only on the future fold 1 TRAIN t1 - t3 VALIDATE t4 fold 2 TRAIN t1 - t4 VAL t5 time moves left to right; training history expands →
Read it this way: first ask what a future prediction must generalize beyond. With exchangeable classification examples, preserve the class ratio. With repeated entities, move the whole entity together. With forecasts, train only on the past and validate on the future. The split should imitate the unit that will actually be unseen in production. Original diagram; semantics checked against scikit-learn's cross-validation guide.

Standard schemes

Single hold-out

Split once into train and val (e.g., 80/20). Cheap; high variance in the score.

Use when: large dataset (millions of examples), or when a single CV iteration is too expensive (LLM fine-tuning).

k-fold

Partition data into folds. Train models, each holding out one fold. Average the scores.

  • or are standard.
  • Average and standard deviation across folds give a confidence interval on generalization error.
  • Each example is used for training times and validation once.

Use when: i.i.d. data, moderate size, and training is cheap relative to the value of a robust score.

Stratified k-fold

k-fold where each fold preserves the class distribution of the full dataset. Essential for imbalanced classification.

Use when: classification with skewed class frequencies. Always.

Group / GroupKFold

Each example has a group identifier (user ID, patient ID, document ID). All examples from the same group go to the same fold. Prevents leakage from one group leaking labels into another.

Use when: multiple examples come from the same entity. Examples: user-level recommendation models, patient-level medical models, document-level NLP tasks where one document has many sentences.

Time-series / TimeSeriesSplit

Folds are chronological. Validation always comes after training in time. Earlier folds are smaller; later folds use more history. Never randomize.

Use when: any data with temporal ordering and predictions are forecasts. Examples: demand forecasting, recsys with time-evolving interests, fraud detection.

Nested CV

Outer loop: estimate generalization. Inner loop: tune hyperparameters within each outer fold.

Use when: hyperparameter tuning matters and you need an unbiased estimate of generalization. Standard in academic ML; rare in industry due to cost.

When NOT to cross-validate

  • Test set evaluation. Test set is held out once and scored once at the end. Repeating on test set leaks information.
  • Feature selection on full data. Selecting features on the entire dataset before CV is leakage. Move feature selection inside the CV loop.
  • Hyperparameter search on full data. Same. Must be inside the loop or in nested CV.
  • Hidden time leakage. Even a “random” k-fold on time-stamped data can leak if features include future-derived signals.

Common pitfalls

  • Random k-fold on time series. Validation contains points from the same week as training → trivially memorizable. Use chronological splits.
  • Random k-fold on user-grouped data. Two reviews from the same user end up in different folds; the model learns user-specific patterns and “generalizes” via user identity. Use GroupKFold.
  • Stratifying by the target on regression. Stratification needs discrete bins; for regression, stratify by quantile bins of the target if needed.
  • Reading too much into one fold’s score. Single-fold scores are noisy; report mean ± std across folds.
  • Tuning on the test set. Number-one source of fake research results.

Found an error or stale claim? Report a correction or send a private note.