PCA and Dimensionality Reduction

Data as Points in High-Dimensional Space

Suppose we have N observations, each described by d features — pixel intensities, sensor readings, stock prices, frequency-domain coefficients. Each observation is a point in ℝᵈ, and the full dataset is a cloud of N points in that space. The key question of dimensionality reduction is: does this cloud actually live near a lower-dimensional subspace?

Almost always, the answer is yes. Natural data is highly redundant — neighboring pixels are correlated, sensor readings co-vary, financial instruments move together. The intrinsic dimensionality of the data is far smaller than d. PCA finds the best low-dimensional linear subspace to represent the data, where "best" means minimum reconstruction error, which is equivalent to maximum preserved variance.

The Covariance Matrix

Let the data matrix X ∈ ℝ^{N×d} have each row as one observation. The first step is to center the data by subtracting the mean of each feature, producing a zero-mean matrix X̃. The sample covariance matrix is then:

The covariance matrix encodes the entire second-order statistical structure of the data. It is the central object in PCA, and its eigendecomposition reveals the geometry of the data cloud.

Eigendecomposition of the Covariance Matrix

Since C is real, symmetric, and positive semidefinite, the spectral theorem guarantees a complete orthonormal eigenbasis. We decompose:

This decomposition is both geometrically and statistically meaningful. Geometrically, it reveals the principal axes of the data ellipsoid. Statistically, it decomposes the total variance (tr C = Σλₖ) into d independent components, each capturing a fraction λₖ / Σλⱼ of the total.

Connection to SVD

The PCA eigenvectors are the right singular vectors of X̃, and the eigenvalues satisfy λₖ = σₖ² / (N−1), where σₖ are the singular values of X̃. In practice, computing PCA via the SVD of X̃ is numerically more stable than forming C = X̃ᵀX̃ / (N−1) and eigendecomposing it — forming the outer product squares the condition number of the problem.

Principal Components and Projections

To project a data point x̃ (centered) onto the top k principal components, multiply by the transpose of the d×k matrix Qₖ = [q₁ | q₂ | … | qₖ], whose columns are the first k eigenvectors:

PCA is therefore the optimal linear encoder-decoder: among all rank-k linear projections, the PCA projection minimizes the expected reconstruction error. This is the Eckart-Young theorem applied to the data matrix.

Choosing the Number of Components

How many principal components should we keep? Two complementary tools guide this choice.

Explained Variance Ratio

The fraction of total variance explained by the top k components is:

The Scree Plot

A scree plot graphs the eigenvalues λ₁ ≥ λ₂ ≥ … ≥ λd in decreasing order. In many datasets, the plot exhibits a sharp "elbow" — a point beyond which the eigenvalues drop off rapidly and flatten to a near-constant noise floor. The elbow suggests the natural intrinsic dimension of the data. The choice of k at the elbow balances signal preservation against compression efficiency.

PCA for Compression and Visualization

PCA is widely used for two immediate tasks: compressing high-dimensional data and visualizing it in 2D or 3D.

Image Compression

An image of N × d pixels can be treated as N rows (patches or rows of pixels) in ℝᵈ. PCA on this matrix produces a basis of "eigenfaces" (for face images) or "eigenpatches" (for textures). Representing each row with k ≪ d coefficients and transmitting only those coefficients, plus the k basis vectors, dramatically reduces storage while preserving perceptual quality.

Visualization

Setting k = 2 or k = 3 projects the data onto the plane or volume of maximum variance. Clusters that were invisible in the original high-dimensional space often separate cleanly in the PCA projection — making PCA the first step in any exploratory data analysis or clustering pipeline.

PCA in Signal Processing: Noise Reduction and Feature Extraction

In signal processing, PCA connects to the Karhunen-Loève Transform (KLT) — the optimal linear transform for decorrelating a random process. For a stationary process, the covariance matrix is Toeplitz, and the KLT eigenvectors approach sinusoids as d → ∞, recovering the DFT as a special case. PCA is thus a data-driven generalization of the Fourier transform for non-stationary signals.

Noise Reduction via Rank Truncation

Suppose observations are contaminated: x̃ = s + n, where s is a low-rank signal (intrinsic dimension k) and n is white noise with variance σ². The covariance of x̃ is C = C_s + σ²I. The top k eigenvalues of C exceed σ² (signal subspace), while the remaining d−k eigenvalues equal σ² (noise subspace). Projecting onto the signal subspace and reconstructing — keeping only the top k components — removes the noise in the orthogonal complement:

Feature Extraction

Before training classifiers or regression models on high-dimensional data, PCA reduces the input to its most informative k dimensions, removing redundancy and noise simultaneously. The resulting features are uncorrelated (diagonalized covariance), which benefits methods that assume feature independence or require inverting the feature covariance. For very high-dimensional data (d ≫ N), PCA also addresses the "curse of dimensionality" — concentrating the available data into a low-dimensional space where distance metrics and density estimates are meaningful.