Keep model selection out of the folds that judge it.

Trying more hyperparameters makes the best validation score look better by chance. Nested cross-validation separates tuning from evaluation so the outer folds estimate the whole selection procedure.

Run the folds

Inner selection, outer evaluation

5 outer tests × 4-fold inner searches
Reported score—

Mean score under the selected protocol.

True procedure score0.78

Synthetic held-out reference.

Selection optimism—

Inflation from choosing the luckiest candidate.

Outer uncertainty—

Standard error across completed folds.

Evaluation must include the act of choosing.

Inner folds select

Hyperparameters, features, prompts, thresholds, and preprocessing are chosen only with inner-training data.

Outer folds estimate

Each untouched outer fold evaluates the full selection pipeline once. Its labels cannot influence the winning configuration.

A final test still matters

Repeatedly inspecting nested-CV results can itself become tuning. Keep a final test set or external evaluation untouched for the last claim.

Primary reading

Nested versus non-nested cross-validation scikit-learnBias in error estimation when using cross-validation for model selection Varma & SimonCross-validation guide scikit-learn