Signal Detection in Noise

Is the Signal There?

Every sensor, every receiver, every measurement system faces the same fundamental question: is the signal I’m looking for actually present, or am I seeing pure noise? This is the signal detection problem, and it is one of the oldest and most important problems in engineering. A radar must decide whether an echo returns from a real target or from random thermal noise. A medical monitor must decide whether a heartbeat pattern is genuine or an artifact. A wireless receiver must decide whether a bit was transmitted as a 0 or a 1.

The challenge is that noise is unpredictable and can, by chance alone, mimic a signal. No matter how sophisticated your detector, you can never achieve perfect reliability — there will always be a trade-off between catching real signals and avoiding false alarms. The mathematical framework for navigating this trade-off is hypothesis testing, and it gives us the tools to make that trade-off explicit and optimal.

Core Idea

Signal detection is a binary decision problem: choose between “signal absent” (H₀) and “signal present” (H₁). The two error types — missing a real signal (missed detection) and falsely claiming one (false alarm) — trade off against each other. Choosing the detection threshold controls where you sit on that curve.

Hypothesis Testing: H₀ vs. H₁

We model the received observation y as one of two possibilities. Under the null hypothesis H₀, only noise is present. Under the alternative hypothesis H₁, a known signal s[n] is embedded in the noise:

H₀: Noise Only

y[n] = w[n], \quad w[n] \sim \mathcal{N}(0, \sigma^2)

The observation y[n] consists of additive Gaussian noise w[n] with variance σ².

H₁: Signal + Noise

y[n] = s[n] + w[n]

The observation contains the known signal s[n] plus the same noise w[n].

The detector processes y[n] and produces a test statistic T(y) — a single number that summarizes the evidence for signal presence. It then compares T(y) against a threshold γ: if T(y) ≥ γ, decide H₁; otherwise decide H₀.

The Two Error Types

Two types of errors are possible. A false alarm occurs when the detector declares H₁ but H₀ was true — the noise happened to look like a signal. The probability of false alarm P_FA measures how often this occurs. A missed detection occurs when the detector declares H₀ but H₁ was true — the signal was present but too weak (or obscured) to trigger the detector. The probability of detection P_D = 1 − P_miss measures how reliably we catch real signals.

P_D

Prob. of Detection — want this HIGH

P_FA

Prob. of False Alarm — want this LOW

Threshold — controls the trade-off

Raising the threshold γ reduces false alarms (harder to trigger a detection) but also reduces P_D (real signals are also less likely to exceed the higher bar). Lowering γ catches more real signals but at the cost of more false alarms. This is a fundamental and unavoidable trade-off.

The Neyman–Pearson Criterion

How do we choose the threshold γ optimally? The Neyman–Pearson (NP) criterion provides a principled answer: fix the maximum allowable false alarm probability P_FA = α, and choose the detector that maximizes P_D subject to that constraint. In other words: given a budget of false alarms, extract as many true detections as possible.

Neyman–Pearson Theorem

The optimal NP detector computes the likelihood ratio Λ(y) = p(y | H₁) / p(y | H₀) and compares it to a threshold determined by P_FA = α. No other detector with the same false alarm rate can achieve a higher detection rate.

Decide H₁ if Λ(y) ≥ γ | where γ is set so P(Λ ≥ γ | H₀) = α

For the case of a known signal s[n] in additive white Gaussian noise (AWGN), the likelihood ratio test simplifies beautifully: the optimal test statistic is just the matched filter output — the cross-correlation of the received signal with the known signal template. This is a profound result: correlation is not just a convenient tool, it is the mathematically optimal detector for a known signal in white Gaussian noise.

NP Test Statistic (AWGN)

T(\mathbf{y}) = \sum_{n=0}^{N-1} y[n]\, s[n]

T is the cross-correlation of the received signal y with the known signal s. This is the sufficient statistic for the optimal NP detector in AWGN.

ROC Curves: Visualizing the Trade-off

The Receiver Operating Characteristic (ROC) curve plots P_D vs. P_FA as the detection threshold γ sweeps from −∞ to +∞. Each point on the curve corresponds to a different threshold setting, and the curve traces out all achievable (P_FA, P_D) pairs for a given detector and SNR.

The ROC curve has several important properties. The diagonal line P_D = P_FA represents a random guesser — no better than chance. Any useful detector must lie above this diagonal. The top-left corner (P_FA = 0, P_D = 1) represents perfect detection. The area under the ROC curve (AUC) is a single number summarizing detector performance — a perfect detector has AUC = 1, a random guesser has AUC = 0.5.

ROC for AWGN Detection

P_D = Q\!\left(Q^{-1}(P_{FA}) - d\right), \quad d = \frac{\sqrt{\sum_n s^2[n]}}{\sigma}

For a known signal in AWGN with SNR d² = Σ s[n]² / σ², P_D is a function of P_FA and d. Q is the Q-function (tail probability of the standard normal).

As the SNR increases, the ROC curve bows further toward the top-left corner, meaning better detection at every false alarm rate. At very high SNR, almost any threshold setting gives near-perfect detection with near-zero false alarms. At very low SNR, even the optimal detector can barely outperform random guessing.

SNR Requirements for Reliable Detection

How much SNR is needed? A commonly cited design point is P_D = 0.9 and P_FA = 10²²³. For a single coherent measurement (matched filter), this requires an SNR of approximately 13–15 dB. In practice, systems use several strategies to operate at lower raw SNR.

Integration: averaging N independent measurements reduces the noise variance by N, so the effective SNR improves by a factor of N (10 log₁₀ N dB). A radar integrating 10 pulses coherently gains 10 dB of SNR. Diversity: using multiple independent channels (spatial, frequency, time) and combining them guards against deep fades where any single channel might be unusable. Spreading: spread-spectrum systems spread a signal over a wide bandwidth so each individual frequency bin has power below the noise floor, yet the full signal energy is recoverable via correlation, providing processing gain equal to the time-bandwidth product.

Processing Gain

Coherent integration of N independent samples raises effective SNR by a factor of N. GPS, for instance, uses 1 ms of spreading code (1023 chips) to achieve a processing gain of ~30 dB, which is why GPS signals work well below the thermal noise floor and are invisible to casual spectrum analysis.

Applications

Radar target detection: A radar transmits pulses and listens for returns. The detection problem is: did an echo return, or is this just receiver noise? A false alarm causes the radar to display a ghost target; a missed detection allows a real target to go unnoticed. Modern radar systems set P_FA at approximately 10⁻⁶ to 10⁻⁸ to keep ghost targets manageable while maintaining high P_D. Constant False Alarm Rate (CFAR) processing dynamically adjusts γ as local noise levels change.

Digital communications: A receiver decides whether each received sample corresponds to a transmitted 0 or 1. The detection threshold is the decision boundary between the two symbol hypotheses. The bit error rate (BER) is essentially 1 − P_D for binary signaling, and it improves monotonically with E_b/N₀ (energy per bit to noise spectral density ratio). The matched filter receiver is the standard implementation of the optimal NP detector for AWGN channels.

Medical diagnostics: A clinical test for a disease has a sensitivity (P_D) and a specificity (1 − P_FA). The ROC curve is the standard tool for evaluating and comparing diagnostic tests. A test is considered good if its AUC exceeds 0.9. Choosing the operating point (threshold) reflects the clinical cost of a false positive versus a false negative — a cancer screening test may tolerate more false alarms (unnecessary biopsies) to avoid missing real tumors.

Spectrum sensing in cognitive radio: A secondary radio must detect whether a licensed primary user is transmitting before occupying a spectrum band. The NP framework governs the design: set P_FA at a level that sufficiently protects the primary user, then maximize P_D to allow the secondary system to opportunistically access unused spectrum.

Next lesson: Matched Filtering — the optimal linear filter for detecting a known signal in noise, and how it connects to pulse compression in radar and correlator receivers in digital communications.

Key Takeaways

Signal detection is a binary hypothesis test: H₀ (noise only) vs. H₁ (signal + noise). The detector computes a test statistic and compares it to a threshold γ.
Two error types exist: false alarm (P_FA) and missed detection (1 − P_D). Raising the threshold lowers P_FA but also lowers P_D — they cannot both be minimized simultaneously.
The Neyman–Pearson criterion maximizes P_D subject to a fixed P_FA = α. The optimal detector is the likelihood ratio test.
For a known signal in AWGN, the NP-optimal test statistic is the cross-correlation (matched filter output) of the received signal with the known signal template.
The ROC curve plots P_D vs. P_FA across all thresholds. Higher SNR curves bow closer to the ideal top-left corner. AUC summarizes overall detector performance.
Coherent integration of N samples improves effective SNR by a factor of N (10 log₁₀ N dB), enabling reliable detection far below the single-sample SNR requirement.