Dimensionality Reduction

The Curse of Dimensionality

High-dimensional data is deceptive. A dataset with 1,000 features sounds rich with information, but as dimensionality grows, the volume of the space increases so rapidly that data becomes sparse almost everywhere. Distance metrics lose meaning — in very high dimensions, the distance between the nearest and farthest neighbor of any point converges, making nearest-neighbor search and clustering unreliable. This is the curse of dimensionality.

Dimensionality reduction addresses this by projecting data from a high-dimensional space into a lower-dimensional representation that retains as much structure as possible. The goals are practical: faster training, better generalization, reduced memory use, and — crucially — the ability to visualize data that no human eye could otherwise see.

There are two broad strategies: linear methods that find low-dimensional projections via linear transformations, and nonlinear methods that learn curved manifolds through the data. PCA is the canonical linear technique; t-SNE and UMAP are the dominant nonlinear ones.

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in the data and projects the data onto those directions. Each direction is a principal component — an orthogonal axis in the original feature space. The first principal component captures the most variance, the second (orthogonal to the first) captures the next most, and so on.

The principal components are the eigenvectors of the data’s covariance matrix, ordered by their eigenvalues. The eigenvalue of each component tells you how much variance it captures. To reduce dimensionality, you keep only the top k components and project the data onto that k-dimensional subspace.

PCA is computed via the singular value decomposition (SVD) of the centered data matrix, which is numerically more stable than explicitly forming the covariance matrix. The result is exact and deterministic.

PCA is a preprocessing powerhouse. It removes correlated features, reduces noise (components with tiny eigenvalues often capture noise rather than signal), and speeds up downstream algorithms. Its main limitation is linearity: it cannot capture curved, nonlinear structure in the data.

t-SNE: Nonlinear Visualization

t-SNE (t-Distributed Stochastic Neighbor Embedding) takes a fundamentally different approach. It models the similarity between points as probabilities — pairs of similar points in the high-dimensional space should remain close in the low-dimensional embedding; dissimilar points should be pushed apart. It then minimizes the mismatch between the high-dimensional and low-dimensional probability distributions.

In the high-dimensional space, t-SNE uses a Gaussian kernel to define a probability distribution P over pairs of points based on their distances. In the low-dimensional embedding, it uses a heavier-tailed Student-t distribution with one degree of freedom (the Cauchy distribution). The heavy tail allows moderately distant points to be placed further apart in the embedding, relieving the crowding problem that affects Gaussian-based methods.

UMAP: Faster and More Faithful

UMAP (Uniform Manifold Approximation and Projection) is a newer technique based on Riemannian geometry and algebraic topology. Like t-SNE, it preserves local structure, but it is significantly faster, scales to larger datasets, and (unlike t-SNE) better preserves global structure — the relative positions of clusters are more meaningful in a UMAP plot than in a t-SNE plot.

UMAP constructs a fuzzy topological representation of the high-dimensional data and then optimizes a low-dimensional embedding to match it. The key hyperparameters are n_neighbors (controls local vs. global structure, analogous to perplexity in t-SNE) and min_dist (how tightly points are packed together in the embedding).

UMAP is the current default for high-dimensional visualization. It runs in near-linear time (compared to O(n² log n) for t-SNE), can embed new points after training (t-SNE cannot), and produces embeddings that generalize better across multiple runs.

Choosing Your Method

PCA is the default for preprocessing and linear structure exploration. Use it first, always. It is fast, deterministic, interpretable, and scales to millions of points. Apply it before t-SNE or UMAP to reduce dimensionality from thousands to ~50 features before the expensive nonlinear step.

t-SNE excels at revealing tight, well-separated clusters in small-to-medium datasets (up to ~100,000 points). It is the standard tool in genomics and single-cell biology, where discovering subtypes in high-dimensional cell profiles is critical. Its weakness is speed and the inability to embed new points.

UMAP is the preferred choice for larger datasets, when global structure matters, or when you need to embed new data after training. It is faster, more scalable, and increasingly the standard for NLP embeddings, recommendation system analysis, and image feature visualization.