Spectral Theorem, Singular Value Decomposition (SVD), PCA, LoRA low-rank adaptation, and Jacobian/Hessian identities.
A = U \Sigma V^\top = \sum_{i=1}^{r} \sigma_i u_i v_i^\top
Every m x n matrix A decomposes into orthogonal U (m x m), diagonal singular values Sigma >= 0, and orthogonal V (n x n).
A_k = \arg\min_{\text{rank}(B)=k} \|A - B\|_F = \sum_{i=1}^{k} \sigma_i u_i v_i^\top
Truncating SVD to top-k singular values minimizes Frobenius and spectral reconstruction error.
W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k)
Freezes pretrained W_0 and trains rank-r matrices A and B, cutting trainable parameters by >99%.
For centered data matrix $X \in \mathbb{R}^{n \times d}$, the sample covariance matrix is $C = \frac{1}{n-1}X^\top X$. Computing PCA via the thin SVD $X = U \Sigma V^\top$ avoids forming $X^\top X$ explicitly, squaring the condition number $\kappa(X^\top X) = \kappa(X)^2$.
| Decomposition | Formula | Matrix Requirement | Primary ML Use Case |
|---|---|---|---|
| Spectral (Eigen) | A = Q Lambda Q^T | Real symmetric (A = A^T) | Covariance matrices, Graph Laplacians |
| SVD | A = U Sigma V^T | Any m x n rectangular matrix | PCA, Moore-Penrose Pseudoinverse, Recommender Systems |
| Cholesky | A = L L^T | Symmetric Positive Definite (SPD) | Gaussian Processes, Multivariate Normal sampling |
| QR Factorization | A = Q R | Full column rank m >= n | Numerically stable Ordinary Least Squares (OLS) |
The non-zero singular values sigma_i of X are the exact square roots of the non-zero eigenvalues lambda_i of X^T X (sigma_i = sqrt(lambda_i)), and the right singular vectors V are the eigenvectors of X^T X.