Evaluating AI & Search: Ranking and LLM Evaluation Metrics
Measure search, RAG retrieval and LLM quality hands-on: precision and recall@k, MRR, NDCG, MAP, ROC curves and AUC, calibration error, LLM-judge agreement with Cohen's kappa and A/B win rates with confidence intervals, every cell checked live.
The first lesson is free — no account needed.
About this course
An AI evaluation course that measures one support assistant's search, retrieval and answers in a spreadsheet, for ML engineers, data scientists and product teams who ship search, RAG or LLM features. You score queries with precision@k, recall@k and the hit rate, find the mean reciprocal rank with MATCH, compute NDCG@k with graded relevance and the ideal ranking, turn precision at every relevant rank into average precision and MAP, sweep a classifier's threshold to draw a ROC curve and calculate the AUC by trapezoids, check calibration with the expected calibration error, test an LLM-as-a-judge against a human grader with Cohen's kappa and compare two models on win rate with ties and a 95% confidence interval. The first lab is free.
Lessons
- Search Quality: Precision@k, Recall@k & Hit RateFree
- Mean Reciprocal Rank: How Soon Is the First Good Answer? Pro
- NDCG@k: Graded Relevance and the Ideal Ranking Pro
- Average Precision & MAP: Rewarding Good Order Pro
- Thresholds, the ROC Curve & AUC by Trapezoids Pro
- Calibration: Does 90% Confident Mean 90% Correct? Pro
- LLM-as-a-Judge: Percent Agreement & Cohen's Kappa Pro
- A/B Testing Two Models: Win Rate, Ties & a Confidence Interval Pro