LA 101
M09 · L03
Module 9: Numerical Linear Algebra

Sparse Matrices

When a matrix is overwhelmingly zeros, sparse storage formats reduce memory from O(n²) to O(nnz) — enabling computations on graphs, meshes, and networks that would otherwise be impossible.

01 / 11
LA 101
M09 · L03
What Is Sparse?

Mostly Zeros

10¹²
Dense entries
3×10⁸
Nonzeros (nnz)
0.03%
Density
Example: social network (n = 10⁶ users)
Dense adjacency needs 8 TB. Sparse storage with nnz ≈ 3×10⁸ needs ~2.4 GB — a 3000× saving. FEM meshes and Markov chains show the same pattern.
02 / 11
LA 101
M09 · L03
Storage Format 1

COO — Coordinate

Three arrays
row[k], col[k], val[k] — one entry per nonzero. No ordering required. Ideal for assembly: FEM generates (row, col, val) triples naturally. Convert to CSR once before arithmetic.
  • Memory: 3 × nnz words
  • SpMV: simple but random row access
  • Best for: matrix construction
03 / 11
LA 101
M09 · L03
Storage Format 2

CSR — Compressed Row

SpMV Inner Loop
y_i = \sum_{j=\text{rp}[i]}^{\text{rp}[i+1]-1} \text{val}[j]\cdot x[\text{ci}[j]]

Three arrays: val[], col_ind[], row_ptr[]. Sequential row access → cache-friendly. The workhorse for iterative solvers (CG, GMRES). Memory: nnz + nnz + (m+1) words.

04 / 11
LA 101
M09 · L03
Storage Format 3

CSC — Compressed Column

  • Mirrors CSR but column-major: col_ptr[], row_ind[], val[]
  • Native in MATLAB — all sparse() matrices use CSC internally
  • Direct solvers (Cholesky, LU) prefer column access
  • Aᵀv is cheap in CSC = Av in CSR
  • Row access expensive — scan entire matrix
05 / 11
LA 101
M09 · L03
Sparse Operations

SpMV and Fill-In

SpMV arithmetic intensity
~2 flops per nonzero, memory reads ≈ 3×nnz + n words → memory-bandwidth bound. Reordering (Cuthill-McKee) improves cache locality.
Fill-in during factorization
New nonzeros appear in L and U. 2D PDE + nested dissection: nnz(L) = O(n log n). Naive ordering: O(n^{3/2}). Minimum degree and nested dissection reorderings minimize fill-in.
06 / 11
LA 101
M09 · L03
Applications

Graphs and Networks

  • PageRank: power iteration on sparse column-stochastic matrix
  • Spectral clustering: eigenvectors of sparse Laplacian L = D − A
  • GNNs: message passing = sparse à H W per layer
  • Shortest paths: Bellman-Ford as SpMV in (min, +) semiring
  • Triangle counting: SpGEMM diagonal of A³
07 / 11
LA 101
M09 · L03
Applications

FEM and Stiffness Matrices

FEM on a 2D mesh with n nodes generates a sparse symmetric positive definite stiffness matrix K with nnz ≈ 7n. Basis functions have local support — each equation couples only neighboring nodes. Assembly: COO triples per element → convert to CSR → iterative solve (CG + multigrid).

7n
nnz (2D mesh)
SPD
Matrix type
O(n)
PCG cost
08 / 11
LA 101
Knowledge Check

Check what stuck

Four questions from this lesson. Answer to see why — the explanation appears whether you were right or wrong. Nothing is scored or saved.

Question 1 of 0
Score 0/0

09 / 11
LA 101
M09 · L03
Direct vs. Iterative

Choosing the Sparse Solver

  • Direct (UMFPACK, CHOLMOD): robust, optimal for multiple RHS, limited by fill-in in 3D
  • Iterative (CG, GMRES): O(nnz) per step, scales to n > 10⁷, needs preconditioner
  • ILU preconditioner: sparse approx. factorization → fewer CG iterations
  • Multigrid: O(n) total for elliptic PDEs — gold standard in 3D
  • Matrix-free: only iterative possible if A defined implicitly
10 / 11
LA 101
M09 · L03
Module 9 · Lesson 3 Complete

Sparse by Design

COO for assembly → CSR for arithmetic → CSC for column operations. SpMV is the critical primitive — memory-bandwidth bound. Fill-in during factorization requires reordering. Graphs, FEM, and Markov chains all generate naturally sparse matrices. Direct solvers for robustness and multiple RHS; iterative + preconditioner for large 3D problems.

Module 9: Numerical Linear Algebra
Floating Point · Iterative Methods · Sparse Matrices
Next lesson All lessons Read article
11 / 11