GPU Inference Capacity Planning: Tokens per Second, Batching, Memory & Cost
Model memory for FP16/INT8/INT4, KV-cache per token, throughput vs batch size, time to first token, GPUs for the peak with Little's law, cost per 1M tokens, self-hosting vs API break-even and a six-month GPU plan: 8 graded labs on one fictional AI start-up.
The first lesson is free — no account needed.
About this course
A GPU inference capacity planning course that sizes the serving fleet of one fictional AI start-up, Kestrel Labs, in a spreadsheet, for ML and platform engineers, AI product managers, and FinOps and finance staff. You work out the GPU memory of model variants at FP16, INT8 and INT4 and whether they fit on one GPU, calculate KV-cache memory per token and the concurrent sequences that fit at each context length, read a throughput curve for tokens per second per GPU and per user, estimate time to first token and total response time, size the fleet for the peak with Little's law and ROUNDUP by memory and by throughput, turn GPU-hour prices and utilisation into the cost per 1M tokens, find the break-even volume between self-hosting and an API, and build a six-month GPU plan with headroom and budget. Illustrative GPUs and numbers, with no vendor affiliation. The first lab is free.
Lessons
- Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4)Free
- KV-Cache Memory per Token & Max Concurrent Sequences Pro
- Throughput vs Batch Size: Tokens per Second per GPU and per User (Chart) Pro
- Latency: Time to First Token, Time per Token & Total Response Time Pro
- GPUs for the Peak: Little's Law, Throughput & ROUNDUP Pro
- Utilisation & Cost per 1M Tokens from the GPU Hourly Price Pro
- Self-Hosting vs API: Monthly Cost & the Break-Even Volume (Chart) Pro
- Capacity Plan: Six Months of Growth, Headroom, GPUs & Budget Pro