Skip to content
mlmentorship

When would you not use cross-validation?

Cross-validation is a tool, not a default. The senior answer names the cases where it's wrong, expensive, or misleading.

Published · 6 min read ·Core ·Mixed

ML breadth · active recall

Practice before you read

8 minutes. Explain the mechanism, why it works, when it fails, and one alternative.

Back to my workbook How practice works

ML breadth · closed-book attempt

When would you not use cross-validation?

Explain the mechanism, why it works, when it fails, and one alternative.

Problem

Context. A model may face future observations, repeated observations from the same entity, distribution shift, or expensive repeated training. You need an estimate relevant to deployment, not merely a score from conveniently available rows. Task. Explain when ordinary random folds are unsuitable and select a replacement for each failure mode. Address evaluation population, temporal or group boundaries, training cost, and model-selection bias. Constraints. Keep preprocessing and tuning inside the appropriate training boundary. State what the chosen split estimates; neither more folds nor a structure-aware split automatically repairs deployment distribution shift.

08:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: ML breadth at every level.

Decide when ordinary cross-validation would answer the wrong deployment question and propose a more appropriate evaluation.

Problem

Context. A model may face future observations, repeated observations from the same entity, distribution shift, or expensive repeated training. You need an estimate relevant to deployment, not merely a score from conveniently available rows.

Task. Explain when ordinary random folds are unsuitable and select a replacement for each failure mode. Address evaluation population, temporal or group boundaries, training cost, and model-selection bias.

Constraints. Keep preprocessing and tuning inside the appropriate training boundary. State what the chosen split estimates; neither more folds nor a structure-aware split automatically repairs deployment distribution shift.

Split by the dependency that deployment must respect, not random rows

A split is valid only for the dependency the claim must respectRandom rows test a different claim from unseen users or future deployment. Select the boundary before interpreting the held-out score.New-user and future-time claims needdifferent boundariesGroup split: A trains, new user BtestsAtrainAtrainBtestBtestTemporal split: past train, futuretest1train2train3test4test
Derivation
A valid fold boundary follows the dependency and the deployment questionA time example trains Monday and Tuesday and tests Wednesday and Thursday; random training on Thursday while testing Tuesday leaks future information. A new-entity example trains A1/A2 and tests B1/B2; splitting A1/A2 across train/test leaks the entity. Model selection remains inside an untouched outer evaluation boundary. These repairs do not solve distribution shift.Two dependency examples; deployment target stillmattersTIME: training must precede evaluationMontrainTuetrainWedtestThutestRandom train Thu / test Tue crosses future ->past.GROUPS: evaluate unseen entitiesTrain: A1,A2entity A onlyTest: B1,B2entity B onlyRandom A1 train / A2 test leaks entity A across.Model selection stays inside training:inner tuning -> freeze -> untouchedouter testnever choose the model on its finalgradeCorrect boundaries do not repair deploymentshift.
The corrected boundaries protect time direction and unseen entities. Model selection must also stay inside the evaluation boundary. Dependency-safe folds still do not guarantee deployment representativeness.
Check: Which dependency makes the random split invalid? Why does repairing that boundary not by itself fix deployment distribution shift?
Read it this way: Random rows test a different claim from unseen users or future deployment. Select the boundary before interpreting the held-out score.
Which boundary matches future deployment rather than unseen users?

What an L4 answer sounds like

“Cross-validation is the standard way to evaluate a model. I’d always use 5-fold or 10-fold CV unless the dataset is really large.”

This is fine for textbook ML. It misses several real-world cases where CV is misleading or wasteful.

What an L5 answer sounds like

“I’d not use cross-validation when:

  1. The data has structure CV breaks. Time series (random folds leak future info), grouped data (random folds leak across groups, e.g., the same patient in train and test), spatial data (nearby points are correlated). For these I’d use time-aware splits, group splits, or spatial blocking.

  2. The training cost is too high. For deep networks with day-long training, 10x the cost is impractical. A single hold-out validation set with a representative distribution is the right call.

  3. A different evaluation answers the deployment question. A production sample, randomized experiment, or expert-labeled set may target a more relevant population or outcome. Check selection, maturation, and assignment assumptions; none automatically supersedes every CV estimate.

  4. One split has adequate precision for the decision. There is no universal row-count cutoff. Dependence, rare outcomes, slice coverage, candidate differences, and training cost determine whether repeated fitting adds useful information.”

This is L5. You’ve named the cases concretely and given the right alternative for each.

What an L6 answer sounds like

“…a few subtler points:

CV doesn’t fix distribution shift. If your test data has a different distribution than your training data (which it usually does in production), CV on training data tells you about training-distribution generalization, not deployment performance.

Model selection reuses evidence. Searching many candidates can select a lucky CV score. Repeated folds do not create an untouched confirmation set. Compare candidates on matched splits and use uncertainty methods that respect shared training data and entity/time dependence.

Separate selection from final evaluation. CV can select hyperparameters, but the selected CV score is optimistic as an independent performance estimate. Nested CV evaluates the selection procedure in outer folds; a genuinely untouched, relevant test set is another option. Keep preprocessing and tuning inside each training boundary.”

Tells that get you a strong-hire vote

  • You name specific structures that break CV (time, grouping, spatial).
  • You discuss cost as a legitimate reason to skip.
  • You mention distribution shift as a confound CV doesn’t address.
  • You bring up nested CV (or the alternative: separate test set) for honest hyperparameter selection.

Tells that get you down-leveled

  • Treating CV as universally applicable.
  • Not knowing what time-series CV looks like.
  • “Use 5-fold for everything.”
  • Confusing CV with bootstrap or with a held-out test set.

Common follow-up

“How would you cross-validate a time series?”

The L6 answer:

“Walk-forward (also called expanding-window or sliding-window) CV: train on data up to time t, test on (t, t+h]. Slide forward, repeat. Never sample test points from before training points. The fold count and window size are tuned to the autocorrelation length and the prediction horizon.”

Related: Walk me through the bias-variance tradeoff, A/B testing for ML systems.

Found an error or stale claim? Report a correction or send a private note.