ML Model Monitoring: Data Drift, PSI, KS Test & Performance Decay
Monitor a (fictional) credit-scoring model hands-on: bin baseline and current data, calculate the PSI, the Kolmogorov–Smirnov statistic, chi-square and Jensen–Shannon drift, track prediction drift and performance decay once labels arrive, build a traffic-light alert dashboard and decide when retraining pays, every cell checked live.
The first lesson is free — no account needed.
About this course
A machine learning model monitoring course that watches one fictional credit-scoring model at Corvel Lending on synthetic data in a spreadsheet, for data scientists, ML engineers, model risk and validation staff, and credit risk analysts. You bin baseline and current data with COUNTIFS and FREQUENCY, calculate the Population Stability Index per feature and classify it with the 0.1 / 0.25 rule of thumb, find the Kolmogorov–Smirnov statistic as the largest gap between two cumulative distributions in a chart, test categorical drift with chi-square and the Jensen–Shannon divergence, track prediction drift month by month, measure precision, recall and balanced accuracy once labels arrive, turn eight metrics into a traffic-light alert dashboard, and weigh the cost of drift against the cost of retraining. Synthetic data, educational only, not financial advice. The first lab is free.
Lessons
- Baseline vs Current: Binning a Feature with COUNTIFS & FREQUENCYFree
- Population Stability Index (PSI) per Feature, with Thresholds Pro
- Kolmogorov–Smirnov: Cumulative Distributions & the Largest Gap (with a Chart) Pro
- Categorical Drift: Chi-Square Test & Jensen–Shannon Divergence Pro
- Prediction Drift: The Score Distribution Month by Month (with a Chart) Pro
- Performance Decay Once Labels Arrive: Precision, Recall & an AUC Proxy Pro
- Alert Rules & a Monitoring Dashboard with Traffic Lights Pro
- The Retraining Decision: Cost of Drift vs Cost of Retraining Pro