Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency
This lesson is part of the course
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Pro
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Pro
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Pro
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Pro
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Pro
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Pro
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Pro
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Pro