Data Labeling for AI: Inter-Annotator Agreement, Label Quality & Labeling Cost
Measure and budget the labels behind an AI model in a live grid: class balance and weights, percent agreement vs Cohen's kappa, Fleiss' kappa, majority vote and confidence, gold-item accuracy, label-noise effects, labelling cost per item and active learning savings, every cell checked as you go.
The first lesson is free — no account needed.
About this course
A data labeling course that measures and budgets the labels behind one fictional AI project, Quillfeather AI's toxicity classifier, in a spreadsheet, for ML engineers, data scientists, annotation and data operations leads and AI product managers. You measure class balance and balanced class weights, see why two annotator pairs with the same 95.5% agreement get very different Cohen's kappa, calculate Fleiss' kappa for four annotators, aggregate labels by majority vote and flag low-confidence items for expert review, score annotators on hidden gold items, simulate how label noise in a test set hides the gap between two models, turn throughput into annotator and reviewer hours and a cost per item, and price the labels active learning saves against random sampling in a chart. Fictional project, synthetic data. The first lab is free.
Lessons
- Label Distribution, Class Balance & Class WeightsFree
- Percent Agreement vs Cohen's Kappa: the Rare-Label Trap Pro
- Fleiss' Kappa: Agreement with More than Two Annotators Pro
- Majority Vote, Label Confidence & Expert Review Pro
- Gold-Standard (Honeypot) Accuracy & Annotator Flags Pro
- Label Noise: What Wrong Test Labels Do to Measured Accuracy (with a Chart) Pro
- Labelling Cost & Throughput: Hours, Review Overhead & Cost per Item Pro
- Active Learning vs Random Sampling: Labels to Target & Budget Saved (with a Chart) Pro