Convolution as Matrix Multiplication
Convolution is the engine of signal processing. But beneath the sum formula lies a deeper truth: convolution is a linear operation, and every linear operation is a matrix. Uncovering that matrix reveals the entire algebraic structure of filtering.
Every Linear Operation Has a Matrix
Convolution maps input vectors to output vectors. It satisfies additivity and homogeneity — making it linear. And the fundamental theorem of linear algebra says: every linear map from ℝⁿ to ℝᵐ is multiplication by some m×n matrix. The impulse response h completely determines the matrix.
The Toeplitz Matrix
Linear convolution y = x * h corresponds to multiplying by a Toeplitz matrix H, where every diagonal holds the same filter coefficient. Each row is a shifted copy of the filter h.
Structure of Toeplitz Matrices
Toeplitz matrices are banded: only M diagonals are non-zero for a length-M filter. Direct matrix-vector multiplication costs O(N²) for length-N signals. But the constant-diagonal structure is a clue that something faster must be possible.
Circular Convolution
Replace zero-padding with periodic wrap-around: the signal repeats every N samples. This gives circular convolution — simpler algebra, and the key to fast computation. Circular convolution can be computed exactly using the DFT.
Circulant Matrices
Circular convolution corresponds to a circulant matrix: each column is a cyclic shift of the column to its left, so C_ij = h[(i−j) mod N]. The first column determines the entire matrix. All N×N circulant matrices share the same eigenvectors — the DFT basis vectors.
- Column 0: [h₀, h₁, h₂, ..., h_{N−1}]ᵀ
- Column 1: [h_{N−1}, h₀, h₁, ..., h_{N−2}]ᵀ
- Column 2: [h_{N−2}, h_{N−1}, h₀, ..., h_{N−3}]ᵀ
- Eigenvectors: complex exponentials e^(j2πkn/N)
DFT Diagonalizes Circulants
Because DFT basis vectors are the eigenvectors of every circulant matrix, the DFT matrix F simultaneously diagonalizes all of them. The eigenvalues are the DFT of the filter's first column — equivalently, the filter's frequency response. This is why convolution = pointwise multiplication in frequency.
The Diagonalization Formula
Every N×N circulant matrix C can be written as:
F is the DFT matrix; Λ is diagonal with the DFT of h on its diagonal; F⁻¹ = F* (DFT is unitary). This is the algebraic foundation of all FFT-based filtering.
FFT-Based Fast Convolution
Apply F, multiply pointwise by H[k], then apply F⁻¹. Using the FFT, each transform costs O(N log N):
At N = 1,000,000 this is ~50,000× faster than direct multiplication. The ⊙ symbol denotes element-wise (pointwise) multiplication.
Key Takeaways
- Convolution is linear, so it is equivalent to multiplication by a Toeplitz matrix
- Circular convolution gives a circulant matrix with cyclic shift structure
- All circulant matrices share the DFT basis vectors as eigenvectors
- DFT diagonalizes circulants: C = F⁻¹ΛF — eigenvalues are the filter's frequency response
- FFT reduces convolution from O(N²) to O(N log N) via pointwise multiplication
M10-L2: Filter Design with Linear Algebra
Now that we know filters are matrices, we can design filters by specifying their eigenvalues — their frequency response. Least-squares filter design, Wiener filters, and optimal filtering all emerge naturally from the linear algebra framework we've built.