LA 101
M09 · L02
Module 9: Numerical Linear Algebra

Iterative Methods

When a matrix is too large to factor directly, iterative methods build a sequence of approximations converging to the true solution — exploiting sparsity for O(n) cost per step instead of O(n³) for full factorization.

01 / 11
LA 101
M09 · L02
Motivation

Why Iterate?

O(n³)
Direct cost
O(nnz)
Iterative / step
10⁶
PDE unknowns
Key insight
Large-scale problems are sparse — each row has O(1) nonzeros. Iterative methods only need matrix-vector products Av, never the full factorization. They trade exactness for scalability.
02 / 11
LA 101
M09 · L02
Jacobi Iteration

Simultaneous Updates

Jacobi Update
x_i^{(k+1)}=\frac{1}{a_{ii}}\!\left(b_i-\sum_{j\neq i}a_{ij}x_j^{(k)}\right)
Convergence
Converges if spectral radius ρ(T_J) < 1. Diagonal dominance (|a_ii| > Σ|a_ij|) guarantees this. Trivially parallelizable — all components updated independently.
03 / 11
LA 101
M09 · L02
Gauss-Seidel

Immediate Reuse

Gauss-Seidel Update
x_i^{(k+1)}=\frac{1}{a_{ii}}\!\left(b_i-\sum_{j<i}a_{ij}x_j^{(k+1)}-\sum_{j>i}a_{ij}x_j^{(k)}\right)

Use freshly updated values x₁…xᵢ₋₁ immediately. For SPD matrices, always converges. Often ~2× faster than Jacobi. Sequential nature limits parallelism.

04 / 11
LA 101
M09 · L02
SOR Acceleration

Over-Relaxation

  • ω = 1: reduces to Gauss-Seidel
  • ω ∈ (1, 2): over-relaxes → faster convergence
  • ω ∈ (0, 1): under-relaxes → stabilizes divergent cases
  • Optimal ω* for Poisson: 2/(1 + sin(π/n))
  • Speedup: O(n²) iterations → O(n) with optimal ω
05 / 11
LA 101
M09 · L02
Convergence Conditions

When Do They Converge?

Necessary & sufficient
ρ(T) < 1 — spectral radius of iteration matrix must be less than 1.
Sufficient conditions
Strict diagonal dominance → Jacobi and G-S converge. SPD matrix + ω ∈ (0,2) → SOR converges. Comparison theorem: ρ(T_GS) = ρ(T_J)² for M-matrices.
06 / 11
LA 101
M09 · L02
Conjugate Gradient

Optimal Krylov Method

CG Convergence Bound
\frac{\|e_k\|_A}{\|e_0\|_A}\leq 2\!\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^k

CG minimizes A-norm error over Krylov subspace 𝒦ₖ(A,b). Converges in ≤ n steps exactly. Requires A symmetric positive definite. Rate governed by √κ(A).

07 / 11
LA 101
M09 · L02
Preconditioning

Reducing Condition Number

  • Diagonal (Jacobi): M = D, cheap, often insufficient
  • ILU / IC: incomplete factorization, O(nnz) per apply
  • SSOR: reduces κ from O(n²) to O(n) for Laplacians
  • Algebraic Multigrid: nearly O(n) for elliptic PDEs
  • Goal: cluster eigenvalues of M⁻¹A near 1
08 / 11
LA 101
Knowledge Check

Check what stuck

Four questions from this lesson. Answer to see why — the explanation appears whether you were right or wrong. Nothing is scored or saved.

Question 1 of 0
Score 0/0

09 / 11
LA 101
M09 · L02
Direct vs. Iterative

Choosing the Right Solver

  • n < 10⁴: direct methods — robust, no tuning needed
  • n ~ 10⁵–10⁶, 2D PDE: preconditioned CG or GMRES
  • n > 10⁶ or 3D PDE: multigrid only viable option
  • Many right-hand sides: factor once, solve each in O(n)
  • Matrix-free: only iterative methods possible
10 / 11
LA 101
M09 · L02
Module 9 · Lesson 2 Complete

Iteration to Convergence

Jacobi: simultaneous updates, parallelizable. Gauss-Seidel: immediate reuse, 2× faster. SOR: relaxation ω turns O(n²) into O(n). CG: optimal for SPD, converges at rate ((√κ−1)/(√κ+1))^k. Preconditioning reduces effective κ. Direct methods for small n; iterative for large sparse systems.

Module 9: Numerical Linear Algebra
Floating Point · Iterative Methods · Sparse Matrices
Next lesson All lessons Read article
11 / 11