Latency: Time to First Token, Time per Token & Total Response Time
This lesson is part of the subscription
The first lesson of every course is free. A subscription unlocks all lessons, labs, exams and certificates.
Or start the free course: Excel for Finance
Lessons in this course
- Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4)Free
- KV-Cache Memory per Token & Max Concurrent Sequences Pro
- Throughput vs Batch Size: Tokens per Second per GPU and per User (Chart) Pro
- Latency: Time to First Token, Time per Token & Total Response Time Pro
- GPUs for the Peak: Little's Law, Throughput & ROUNDUP Pro
- Utilisation & Cost per 1M Tokens from the GPU Hourly Price Pro
- Self-Hosting vs API: Monthly Cost & the Break-Even Volume (Chart) Pro
- Capacity Plan: Six Months of Growth, Headroom, GPUs & Budget Pro