CV Filter Leakage: 5000 Random Predictors Report 0% Error

Predictors (N)
1,000
Labels: 50 (fixed, random)  |  Folds: 5  |  Kept after filter: 100
Wrong — Label Leakage
All 1,000 predictors + ALL 50 labels LABELS EXPOSED
↓
Filter to top 100 by correlation with ALL 50 labels
↓
Cross-validate on top 100
(test labels already seen during filter)
Labels seen before test fold? YES
CV Error ≈ 3%
A LIE — optimism bias from leakage
Correct — Within-Fold Filtering
1,000 predictors · Split into 5 folds
Each fold: 40 train labels + 10 test labels held out
↓
For each fold: Filter on training labels only
40 labels · test fold never touched
↓
Evaluate on held-out fold
Labels are genuinely unseen
Labels seen before test fold? NO
CV Error ≈ 50%
Honest — random predictors are worthless
Wrong pipeline:
E[reported error] → 0% as N → ∞
With N random predictors filtered by ALL labels,
the best-of-N by chance looks perfect.
Correct pipeline:
Within-fold error → true error ≈ 50%
Random predictors carry zero signal;
honest CV reveals the null result.