Data Pipeline Metrics: Freshness, Throughput, Failures & Cost
Run durations and p95, rows per second and GB per hour, freshness SLAs, failure and retry rates, backlog catch-up, storage growth and partition sizing, cost per TB and a data quality dashboard: 8 labs on one fictional company's ETL logs.
The first lesson is free — no account needed.
About this course
A data pipeline metrics course that turns one fictional company's ETL logs into the numbers a data platform team reports, for data engineers, analytics engineers and their managers. You convert start and end timestamps into minutes and compare the median with the p95 runtime against an SLO, measure throughput in rows per second and GB per hour and the time for a 1.8 TB backfill, check table freshness against SLAs, calculate first-attempt failure, retry recovery and final success rates, size the backlog and catch-up time after a five-hour outage with a recovery chart, project storage growth and partition file sizes, price every job per run and per TB processed and build a data quality dashboard with null rates, duplicates and row-count reconciliation. It works for any orchestrator. The first lab is free.
Lessons
- Run Durations: Minutes from Timestamps, Median & p95Free
- Throughput: Rows per Second, GB per Hour & Backfill Time Pro
- Data Freshness & SLA Breaches: Now − Last Load Pro
- Failure Rate & Retries: First Attempt, Recovery & Final Success Pro
- Backlog & Catch-Up Time after an Outage Pro
- Storage Growth & Partition Sizing Pro
- Compute Cost per Run & per TB Processed Pro
- Data Quality Dashboard: Null Rates, Duplicates & Row-Count Reconciliation Pro