How Tokenizers Work by Hand: Characters & Pairs, Byte-Pair Merges, Vocabulary Size, Tokens per Word, Sequence Length & Embedding-Table Costs
See how text becomes tokens. You will split a toy corpus into characters, count adjacent pairs, perform byte-pair merges step by step, watch the vocabulary grow, tokenize new words with the learned merges, measure tokens per word and compression, and weigh vocabulary size against sequence length and embedding-table size, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course shows how a tokenizer turns text into tokens, one cell at a time. You split a toy corpus of six words into characters, count neighbouring pairs weighted by word frequency and find the most frequent pair. You perform the first byte-pair merge with IF rules on neighbouring cells, close the gaps, keep a merge log and watch the vocabulary grow while the corpus shrinks. Then you tokenize new words with the learned merges, mark unknown characters, measure tokens per word and characters per token, weigh vocabulary size against embedding parameters and sequence length, and budget tokens and fictional costs for five toy documents.
Lessons
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Pro
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Pro
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Pro
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Pro
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Pro
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Pro
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Pro
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Pro