Cross-Validation & Hyperparameter Tuning by Hand: Train/Validation/Test Splits, k-Fold Scores, Grid Search, Overfitting Curves & Data Leakage
Choose model settings without fooling yourself. You will split data into train, validation and test sets, assign k folds, score a simple model fold by fold, run a small grid search over settings, read under- and overfitting curves, spot leakage and time-ordering mistakes and report an honest final score, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course shows how to choose model settings without fooling yourself, using a toy bike-rental dataset. You split 20 days into train, validation and test sets with MOD on a shuffle number, deal the pool into plain and stratified folds and build a k-nearest-neighbour prediction for a held-out day. You score every fold with its mean absolute error, average the folds and measure their spread. A grid search over k with standard errors leads to the one-standard-error rule. You read under- and overfitting from training and validation curves, catch three kinds of leakage, check a forecast in time order and score the locked test set once.
Lessons
- Three Piles of Data: Train, Validation & Test Splits from a Shuffle Number Pro
- k-Fold Cross-Validation: Assigning Folds with MOD and Stratifying by Class Pro
- A Toy Model per Fold: k-Nearest-Neighbour Predictions for a Held-Out Day Pro
- Fold Scores: Mean Absolute Error per Fold, the CV Estimate and Its Spread Pro
- Grid Search: A Setting × Fold Table, the Best k and the One-Standard-Error Rule Pro
- Under- and Overfitting: Training vs Validation Error by Model Complexity Pro
- Data Leakage: Scaling Before Splitting, Duplicates Across Folds and Rolling-Origin Checks Pro
- The Final Report: Choose by CV, Retrain on All Pool Days and Score the Test Set Once Pro