Gradient Descent, Polyak Momentum & Adam Optimization Geometry
Understand ravine oscillation damping, exponential moving averages of moments, and decoupled weight decay (AdamW).
First-order optimization minimizes an objective L(theta) by iteratively stepping opposite the local gradient nabla L(theta). Momentum accumulates an exponentially decaying velocity v_t, while Adam normalizes the first moment m_t by the element-wise root mean square of the second moment sqrt(v_t) + epsilon.
On an elongated elliptical canyon (ill-conditioned Hessian with high condition number kappa = lambda_max / lambda_min), raw gradient descent zig-zags violently across the steep canyon walls while inching slowly along the valley floor. Momentum cancels out the opposing wall oscillations and accelerates along the consistent valley direction; Adam rescales each coordinate by its historical gradient variance.
Gradient Descent & Adam Geometry (GradientDescentTrajectoryScene)
Step-by-Step Symbolic LaTeX Derivation
Follow each algebraic transformation from first principles to computational implementation.
First & Second Moment Bias Correction in Adam
Because m_0 = v_0 = 0, raw moments are biased toward zero during early steps; dividing by (1 - beta^t) restores unbiased expectation.
Why is L2 regularization NOT equivalent to decoupled weight decay (AdamW) in adaptive optimizers?
Why do LLMs require a linear learning rate warmup for the first 1-2% of steps?
At step 0, the second-moment estimate v_t has seen very few samples and has high variance; a warmup prevents huge destabilizing parameter jumps before v_t stabilizes.