Backpropagation & Reverse-Mode Automatic Differentiation on Computational Graphs
Trace upstream adjoint gradients backward through DAG nodes in O(1) passes regardless of parameter count.
Backpropagation is reverse-mode automatic differentiation applied to a scalar loss L(theta): starting from dL/dL = 1 at the output node, each node u receives upstream gradient dL/du and propagates dL/dv = sum_u (dL/du * du/dv) backward to its parent nodes v via vector-Jacobian products (VJPs).
Forward-mode differentiation asks how perturbing 1 input weight ripples forward to all outputs (requiring N passes for N million weights). Reverse-mode differentiation flips the question: starting from the single scalar loss L at the end, it propagates sensitivity backward through every intermediate gate in a single backward sweep.
Backpropagation & AutoDiff (BackpropComputationalGraphScene)
Step-by-Step Symbolic LaTeX Derivation
Follow each algebraic transformation from first principles to computational implementation.
Local Gate Rules (Add, Multiply, Residual)
Because multiplication gates multiply upstream gradients by the forward activation x, we must cache forward activations during training.
Matrix-Multiply (Linear Layer) Adjoint Identities
Every linear layer backward pass requires exactly two GEMMs: X^T @ dY for weight gradients and dY @ W^T for input gradients.
Why does reverse-mode automatic differentiation require caching forward-pass activations in GPU memory?
Why is reverse-mode AutoDiff O(1) sweeps for neural networks while forward-mode is O(P)?
Neural networks map millions of parameters R^P to a single scalar loss R^1. Reverse-mode computes the full gradient with respect to all P parameters in 1 backward pass.