Scoring Generated Text by Hand: N-Gram Overlap, Clipped Precision, BLEU & Brevity Penalty, ROUGE Recall, Token F1, Exact Match & Agreement with Human Ratings
Find out what automatic text scores really measure. You will split toy outputs into tokens, count matching n-grams with clipping, combine them into a BLEU score with a brevity penalty, compute ROUGE recall and F1 against several references, score answers with exact match and token F1, see how paraphrases and short answers fool the metrics and test whether the scores agree with human ratings, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course shows what automatic text scores really count, one cell at a time. You put toy outputs and references into token rows, count unigrams with COUNTIF and build bigrams and trigrams with &. You clip repeated n-grams against several references and combine four precisions into BLEU with a geometric mean, add-one smoothing and the brevity penalty, at sentence and corpus level. You calculate ROUGE-1, ROUGE-2 and ROUGE-L with β, score short answers with exact match and token F1, test paraphrases, very short outputs and character bigrams, and finally check rank and Pearson agreement with two raters.
Lessons
- Tokens as Rows: Unigram Counts, Matches, Precision & Recall Pro
- N-Gram Precision with Clipping: Bigrams, Trigrams & Several References Pro
- BLEU by Hand: Geometric Mean, Add-One Smoothing, Brevity Penalty & Corpus Level Pro
- ROUGE-1 & ROUGE-2: Recall, Precision & F1 against Several References Pro
- ROUGE-L: Longest Common Subsequence, Recall, Precision & the F-Measure with β Pro
- Question-Answer Scores: Exact Match, Token F1 & Partial Credit Pro
- Length & Paraphrase Effects: Short Outputs, Same Meaning in Other Words & Character N-Grams Pro
- Capstone: Do the Metrics Agree with People? Rank Correlation, Pearson & a System Comparison Pro