How Neural Networks Learn: Gradient Descent, Momentum, RMSProp, Adam, Learning-Rate Schedules & Warm-Up by Hand
Watch optimisers work step by step on small loss functions: plain gradient descent, momentum, adaptive step sizes, bias correction, learning-rate schedules with warm-up and weight decay, with every cell checked as you go.
One-time payment · lifetime access · lessons, labs, exam and certificate included · no subscription.
About this course
This course shows how neural networks actually learn, one cell at a time. You compute a squared loss and its gradient by formula and by finite differences, run plain gradient descent and see small learning rates crawl and large ones diverge. You add a momentum velocity on a stretched two-weight bowl, scale steps per weight with RMSProp, and build Adam's two running averages with bias correction. You model mini-batch noise and smoothing, build step-decay, cosine and warm-up schedules with weight decay, and finish with an optimiser race: loss after N steps, time to target and a chart of the loss curves.
Lessons
- Loss & Gradient: Which Way Is Downhill? Pro
- Plain Gradient Descent: Step Tables, Learning Rates & Divergence Pro
- Momentum: A Velocity Term, Overshoot & Faster Progress on a Stretched Bowl Pro
- RMSProp: Running Averages of Squared Gradients & Per-Weight Step Sizes Pro
- Adam: First & Second Moments, Bias Correction and the Adam Step Pro
- Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect Pro
- Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay Pro
- The Optimiser Race: Loss after N Steps, Time to Target & Loss Curves Pro