Two Paradigms, One Signal
Classical DSP is built on mathematics: Fourier analysis, filter theory, estimation theory, and information theory. Every algorithm is derived from first principles, its behavior fully predictable and analytically understood. Machine learning, by contrast, is built on data: a model learns internal representations by optimizing a loss function over a large training set, with no guarantee that the learned weights correspond to any human-interpretable quantity. These two paradigms have coexisted for decades, but the last ten years have seen them converge.
Neural networks now appear inside nearly every layer of the modern signal-processing stack — denoising microphone audio, classifying modulation schemes, detecting anomalies in vibration signals, and equalizing wireless channels. This lesson surveys where ML amplifies DSP, where classical DSP remains superior, and the emerging hybrid architecture that combines the analytical guarantees of signal processing theory with the representational power of neural networks.
Classical DSP trades model assumptions for interpretability: if you assume the signal is bandlimited, stationary, or follows a known distribution, you can derive the optimal processor analytically. ML trades assumptions for flexibility: by learning from data, a model can adapt to signals whose statistics are unknown or non-stationary — at the cost of interpretability and the need for labeled training data.
Neural Signal Denoising
Wiener filtering is the classical solution to signal denoising: given a signal corrupted by additive noise, design a frequency-domain filter that minimizes the mean-squared error between the clean and estimated signal. Wiener filtering is optimal when the power spectral densities of signal and noise are known. When they are not — or when the noise is non-stationary or non-Gaussian — the Wiener filter degrades.
Deep neural networks learned from data can overcome this limitation. The most successful architecture for 1-D signal denoising is the dilated causal convolutional network (related to WaveNet). Rather than learning in the frequency domain, these networks process the raw time-domain waveform through a stack of 1-D convolutions with exponentially increasing dilation factors (1, 2, 4, 8, …, 512), capturing structure across multiple timescales simultaneously. The receptive field grows exponentially with depth while the parameter count grows only linearly.
In practice, deep denoising networks outperform classical Wiener filtering when the noise is non-stationary (e.g., background chatter in a voice recorder) or when the signal has complex structure difficult to model analytically (e.g., music or polyphonic speech). The trade-off: the model requires thousands of paired noisy/clean training examples and tens of millions of multiply-accumulate operations per second of audio — far more compute than a Wiener filter.
Automatic Modulation Classification
Automatic modulation classification (AMC) is the problem of identifying the modulation scheme of a received signal — distinguishing AM from FM, or BPSK from QPSK from 16-QAM — without prior knowledge of what was transmitted. Classical AMC algorithms extract hand-crafted features from the signal (instantaneous amplitude, phase, and frequency statistics; higher-order cumulants) and feed them into a likelihood ratio test or decision tree. These feature extractors require domain expertise and must be redesigned for each new modulation type.
Convolutional neural networks (CNNs) applied directly to the complex baseband waveform learn to extract modulation-discriminating features automatically. The seminal RadioML dataset demonstrated that a relatively shallow CNN — four convolutional layers operating on in-phase/quadrature (IQ) sample pairs — achieves accuracy competitive with expert feature engineering across over 20 modulation classes and a wide range of SNR values. At high SNR (>10 dB), deep models exceed the accuracy of any known hand-crafted feature set.
The key insight is that raw IQ samples already carry all the information needed for modulation classification — the same information that classical methods laboriously extract as hand-crafted features. A CNN, given enough data, learns an internal representation that is strictly richer than any fixed feature set. The penalty is opacity: it is difficult to explain why the network classifies a particular segment as QPSK rather than 8-PSK, which matters in regulated spectrum environments where the classifier's decision must be auditable.
Learned Channel Equalization
In the previous lesson, OFDM channel equalization was reduced to a single complex division per subcarrier — trivially simple because OFDM converts the frequency-selective channel into parallel flat subchannels. In single-carrier or non-OFDM systems, the channel still looks like an FIR filter, and equalization requires inverting it, typically with a decision-feedback equalizer (DFE) or MMSE-LE filter. When the channel is unknown or rapidly time-varying, the filter must be estimated and updated continuously.
Recurrent neural networks (RNNs) and, more recently, transformer-based sequence models can learn to equalize channels directly from received data, without explicit channel estimation or filter design. The model is trained on pairs of (received signal, transmitted bits) and learns an end-to-end mapping that implicitly performs channel inversion. This approach is particularly attractive in channels with memory (e.g., underwater acoustic channels with delay spreads of hundreds of milliseconds) where the optimal classical equalizer would require a very long filter.
Benchmark results on standardized channel models (ITU vehicular, underwater acoustic, satellite) show that neural equalizers match or slightly exceed MMSE performance in high-mobility scenarios where the channel changes faster than pilot-based estimation can track. However, neural equalizers fail catastrophically on out-of-distribution channels — a channel type not seen during training — while the classical MMSE equalizer degrades gracefully. This brittleness is a fundamental limitation of data-driven methods in safety-critical applications.
DSP-Aware Neural Architectures
The most productive recent research direction is not replacing DSP with ML but embedding DSP structures inside neural architectures. By constraining the architecture to match known signal-processing priors, the model gains inductive bias that reduces data requirements and improves generalization. This hybrid approach is called model-based deep learning or algorithm unrolling.
Algorithm unrolling converts an iterative classical algorithm (such as the ISTA sparse recovery algorithm or the turbo decoder) into a finite-depth neural network, where each layer corresponds to one iteration. The fixed algorithmic parameters (step sizes, thresholds, damping factors) are replaced by learned parameters optimized end-to-end on training data. The resulting network is interpretable (each layer has a clear signal-processing meaning), data-efficient (it inherits the algorithm's structure), and trainable (gradients flow through the unrolled iterations via backpropagation).
The ISTA algorithm for sparse signal recovery iterates: x⊃ ← Sλ(Wx⊃ + b · y), where Sλ is a soft-threshold function, W is a matrix derived from the sensing matrix, and b is a step-size scalar. LISTA replaces W and b with learned parameters and unrolls K iterations into a K-layer network. A 10-layer LISTA network achieves the same reconstruction accuracy as 1000 iterations of classical ISTA — a 100× speedup — because the learned parameters exploit training data statistics that classical ISTA ignores.
Other examples of DSP-aware architectures include: learned filter banks (replace fixed STFT windows with learned 1-D convolutional kernels initialized to analysis filters, then fine-tuned end-to-end for a downstream task); deep unfolded OFDM receivers (replace pilot-based channel estimation and ZF equalization with a learned iterative refinement network); and neural beamformers (replace the MVDR spatial filter with a complex-valued neural network that learns the array manifold from training microphone recordings).
The feature that learned filter banks are replacing has a name worth knowing, because it dominated speech recognition for decades: the Mel-Frequency Cepstral Coefficients (MFCCs). The MFCC front end is a fixed DSP pipeline — take the short-time magnitude spectrum, warp it onto the perceptual mel scale, and take the logarithm of each mel band's energy (both steps introduced in Module 9, Lesson 3) — capped by one last transform: a Discrete Cosine Transform (DCT) applied across the log-mel energies. The DCT does two jobs at once. It decorrelates the mel bands, so a classifier with a diagonal covariance model suffices, and it compacts most of the energy into the first dozen or so coefficients — a speech front end typically keeps about 13 and discards the rest, yielding a small, robust feature vector. The learned filter banks above are, in effect, this same pipeline with the mel filters and the DCT replaced by trainable convolutions: the model discovers its own front end instead of inheriting the mel–DCT one.
Where Classical DSP Still Wins
Despite impressive results, ML-based signal processing has clear failure modes that classical DSP does not. Understanding these boundaries is essential for any engineer deciding which approach to use.
| Criterion | Classical DSP | ML / Neural DSP |
|---|---|---|
| Interpretability | Fully analytical; behavior derivable from theory | Black-box; internal representations opaque |
| Generalization | Degrades gracefully on new conditions | Can fail catastrophically out-of-distribution |
| Data requirement | None (model-based) | Thousands to millions of labeled examples |
| Compute | Low (especially FFT-based) | High (inference may require GPU/NPU) |
| Latency | Predictable, deterministic | Variable; hard real-time challenging |
| Certification | Verifiable bounds on performance | Statistical guarantees only |
| Non-stationary signals | Requires model adaptation | Can learn non-stationary patterns |
| Unknown noise type | Performance degrades if model is wrong | Learns from examples regardless of type |
Classical DSP remains dominant in safety-critical and standards-based systems. A 5G base station cannot use a neural equalizer whose behavior is only probabilistically characterized — the 3GPP standard mandates specific link-budget calculations that require analytical performance bounds. FFT-based filter banks, Viterbi decoders, and Kalman filters dominate aerospace, medical instrumentation, and standards-compliant communications because they are certifiable.
ML excels in applications where labeled data is plentiful and interpretability is not required: consumer audio enhancement (noise cancellation in earbuds), predictive maintenance (vibration anomaly detection), and cognitive radio (spectrum sensing in unknown environments). The trend is toward hybrid architectures that apply ML to subproblems where the signal statistics are hard to model analytically, while retaining classical DSP for the analytically tractable stages of the processing chain.
DSP-101: A Closing Perspective
This final lesson of DSP-101 closes the arc from the Nyquist-Shannon sampling theorem in Module 1 to the intersection of machine learning and signal processing in Module 12. The journey has covered sampling, quantization, the DFT, FIR and IIR filter design, spectral analysis, multirate systems, and real-world DSP applications in embedded systems, audio, communications, and now machine learning.
The unifying thread throughout is the frequency-domain perspective: virtually every DSP operation — filtering, correlation, interpolation, OFDM modulation, Wiener filtering, even the receptive field of a dilated convolutional network — can be understood as a manipulation of the signal's spectral content. The engineer who can read a spectrum, understand what a filter's frequency response means, and reason about bandwidth, noise, and distortion has a lens that applies equally to 1930s radio and 2030s neural signal processors.
- Classical DSP is optimal when signal and noise statistics are known; ML excels when those statistics are unknown, non-stationary, or too complex to model analytically.
- Dilated causal convolutional networks process raw time-domain waveforms, growing their receptive field exponentially with depth while keeping parameter count linear — an architecture inspired by signal processing multirate theory.
- Automatic modulation classification with CNNs feeds raw IQ samples directly into the network, learning feature representations richer than any hand-crafted statistical feature set, at the cost of interpretability.
- Neural channel equalizers match MMSE performance in high-mobility channels but fail catastrophically on out-of-distribution channel types; classical MMSE degrades gracefully.
- Algorithm unrolling (e.g., LISTA) converts iterative signal processing algorithms into finite-depth neural networks with learned parameters, combining interpretability with data-driven optimization.
- Classical DSP remains mandatory in safety-critical, standards-compliant, and low-latency systems; ML is most productive where data is plentiful and interpretability is not a regulatory requirement.