How Text Prediction Worked Before Transformers: n-Gram Counts, Smoothing, Backoff, Perplexity, Autocomplete & Text Generation by Hand
Build a small language model from word counts. You will count unigrams and bigrams in a tiny corpus, turn the counts into probabilities, smooth them so unseen words still get a chance, combine models with backoff and interpolation, score sentences with perplexity, rank autocomplete suggestions and generate text from given random numbers, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course builds a small language model from nothing but word counts. You count tokens and unigrams in a tiny original corpus, pair every word with the next one and fill a next-word table with COUNTIFS. You turn counts into probabilities, multiply them along a sentence and see why one unseen pair makes it zero. Add-one and add-k smoothing fix that, interpolation and backoff mix bigram and unigram models, and you choose λ on a held-out sentence. You compare two models with log-probabilities and perplexity, rank top-3 autocomplete suggestions with tie-breaking, and generate a new sentence from cumulative probabilities and given random numbers.
Lessons
- Corpus & Tokens: Counting Words, Vocabulary Size & Unigram Probabilities Pro
- Bigrams: Pairs of Neighbouring Words & the Next-Word Table Pro
- Probabilities: P(next | word) from Counts & the Probability of a Sentence Pro
- Smoothing: Add-One & Add-k So Unseen Words Still Get a Chance Pro
- Backoff & Interpolation: Mixing Bigram and Unigram, Choosing λ on Held-Out Text Pro
- Sentence Scores & Perplexity: Comparing Two Models on the Same Test Text Pro
- Autocomplete: Top-3 Suggestions, Breaking Ties & Falling Back for Unseen Contexts Pro
- Generation: Cumulative Probabilities, Random Numbers & a Sentence the Corpus Never Had Pro