How Language Models Work by Hand: Tokens, Embeddings, Attention Weights, Softmax & Sampling with Temperature
Open up a tiny language model in a spreadsheet: token ids, embedding lookups, positional signals, attention scores, softmax and next-token sampling with temperature, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course opens up a toy language model cell by cell. You split a prompt into tokens and look up their ids, read 4-dimensional embedding vectors with INDEX and compare tokens with dot products, and add sine and cosine position signals. With small weight tables and SUMPRODUCT you compute queries, keys and values, score every token against every earlier one with q · k ÷ √d, apply a causal mask and softmax, and mix the values into the attention output. Finally you turn logits into next-token probabilities with temperature, compare greedy, top-k and sampled choices, and see why long prompts cost more tokens, more scores and more memory.
Lessons
- Tokens: Splitting a Prompt, Vocabulary Ids & Token Budgets Pro
- Embeddings: Looking Up Token Vectors & Dot Products Pro
- Positional Signals: Sine & Cosine Waves Added to Embeddings Pro
- Queries, Keys & Values: Weight Tables and SUMPRODUCT Pro
- Attention Scores: Scaling by √d, the Causal Mask & Softmax per Row Pro
- Weighted Values: The Attention Output & Which Token Looks Where Pro
- Next-Token Probabilities: Temperature, Greedy Choice, Top-k & Sampling Pro
- Context Length: Re-Sent History, Turn Costs & Why Long Prompts Weigh More Pro