Quantization & Bit Depth

The Second Step: Amplitude Goes Discrete

Sampling solves one half of the analog-to-digital conversion problem: it discretizes time, turning a continuous waveform into a sequence of values at regular intervals. But those values are still continuous — they can take any real number in some range. A real digital system can only store and process a finite set of numbers, so a second operation must follow: quantization, the mapping of continuous amplitude values to a finite set of discrete levels.

Quantization is unavoidable in any digital system that must store or process real-world signals. Every audio recording, every digital image, every sensor reading involves quantization. And unlike aliasing — which can be prevented entirely with the right anti-aliasing filter — quantization always introduces a small but irreducible error. Understanding that error, measuring it, and keeping it acceptably small is the art of choosing the right bit depth.

Levels, Step Size, and Bit Depth

A b-bit quantizer divides the input amplitude range [x_min, x_max] into L = 2^b equal intervals. The width of each interval is the quantization step size Δ:

The input is rounded to the nearest level. Every sample is approximated to within ±Δ/2 of its true value. This rounding error — the quantization noise — is what separates a perfect continuous representation from a practical digital one. The more bits you use, the smaller Δ becomes, and the closer the digital representation tracks the original signal.

To appreciate the scale: a 1-bit quantizer has only 2 levels — it can represent a signal as either “high” or “low.” An 8-bit quantizer has 256 levels. A 16-bit quantizer (used for CD audio) has 65,536 levels. A 24-bit studio recording system has over 16 million levels. The difference in fidelity is enormous.

Quantization Error: The Irreducible Noise Floor

The quantization error e[n] for sample n is the difference between the true sample x[n] and its quantized version x̂[n]:

When the signal uses many quantization levels — that is, when the number of bits is large enough that the signal traverses many steps — the quantization error approximates a uniform random variable on [−Δ/2, Δ/2]. This statistical model, called the quantization noise model, is remarkably accurate for well-designed systems and allows us to derive a simple formula for the resulting signal-to-noise ratio.

The variance (power) of the quantization noise under this model is Δ²/12. For a sinusoidal input signal with amplitude A (peak-to-peak range 2A matched to the quantizer range), the signal power is A²/2 and the quantizer step size is Δ = 2A/2^b = A⋅2^1−b. Substituting gives the classic SQNR formula.

SQNR: The 6 dB per Bit Rule

The Signal-to-Quantization-Noise Ratio (SQNR) measures how much larger the signal power is compared to the quantization noise power. For a full-scale sinusoidal input quantized with b bits:

The “6 dB per bit” rule is remarkably easy to remember and apply. A 16-bit ADC delivers approximately 6.02 × 16 + 1.76 = 98.1 dB of dynamic range for a full-scale sinusoidal input — wide enough to cover the entire range from the quietest audible whisper to a loud concert without distortion. A 24-bit system achieves about 146 dB, exceeding the dynamic range of the human auditory system.

Overload and Clipping

The SQNR analysis assumes the input stays within the quantizer’s designed range [x_min, x_max]. If the signal exceeds this range, the quantizer saturates (clips): all values above x_max are mapped to the top level, and all values below x_min to the bottom. Clipping introduces severe waveform distortion — the signal’s peaks are cut flat — creating strong harmonic distortion that sounds harsh and unpleasant in audio.

Unlike quantization noise, which is spread evenly across the spectrum at a low level, clipping distortion produces discrete harmonics that concentrate at multiples of the fundamental frequency. Even brief, occasional clipping — a single sample out of range — is often clearly audible. For this reason, ADC input stages include headroom: the quantizer range is designed somewhat larger than the expected maximum signal level, accepting a slight SQNR penalty in exchange for protection against occasional amplitude peaks.

Dithering: Turning Error into Noise

At low bit depths or low signal levels, quantization error ceases to behave like random noise and takes on a structured, periodic character. A slowly varying signal near a quantization boundary oscillates between two adjacent levels, generating a tonal artifact at the signal frequency — audible as a “buzzing” or “granular” distortion quite different from the gentle hiss of uncorrelated noise.

Dithering is the solution: add a small amount of random noise to the signal before quantization. The added noise randomizes the quantization error, breaking the correlation between the error and the signal. The result is that the structured tonal artifact is replaced by a flat, low-level noise floor — perceptually much more benign than the tonal distortion it replaces. The noise is usually chosen to be spectrally shaped (triangular probability density) to minimize its audible impact.

Common Bit Depths in Practice

Different applications impose different requirements on bit depth, trading off storage cost, computational complexity, and perceptual quality:

8-bit audio (SQNR ≈ 50 dB) was common in early digital systems and telephone codecs. The 256 available levels are perceptibly coarse for music — quantization granularity is audible — but adequate for speech intelligibility. Legacy formats like μ-law and A-law companding improve perceived quality by using logarithmic rather than uniform quantization, allocating more levels to the low-amplitude range where human hearing is most sensitive.

16-bit audio (SQNR ≈ 98 dB) is the standard for CD-quality audio and most consumer devices. With 65,536 levels spanning a 98 dB dynamic range, quantization noise is inaudible under normal listening conditions, especially with proper dithering. The 16-bit format has served consumer audio well for four decades.

24-bit audio (SQNR ≈ 146 dB) is the standard for studio recording and post-production. The vast headroom means engineers can set conservative input levels with ample protection against clipping, then use the full dynamic range in mixing and mastering. Modern DAWs process audio internally at 32-bit or 64-bit floating point to avoid accumulation of rounding errors across hundreds of processing stages.

12-bit ADCs are ubiquitous in microcontrollers and embedded systems, striking a balance between resolution and silicon cost for sensor measurements, motor control, and instrumentation. A 12-bit ADC delivers about 74 dB of SQNR — adequate for most industrial sensing applications where noise from the physical environment exceeds the quantization floor anyway.

Non-Uniform Quantization and Companding

Uniform quantization — equal-width steps across the entire amplitude range — is optimal when the signal has uniform amplitude distribution. But most real-world signals spend far more time near zero than near their maximum amplitude. Speech, in particular, has a roughly logarithmic amplitude distribution: the probability of a low-amplitude sample is much higher than a high-amplitude one.

For such signals, non-uniform quantization allocates more (smaller) steps to the low-amplitude region and fewer (larger) steps to rarely-occurring high amplitudes. This gives better average SQNR for the typical signal level at the cost of slightly worse SQNR for the largest signals. The perceptual result is much better: quiet sounds are reproduced faithfully, which matters most to human perception.

In practice, non-uniform quantization is implemented via companding: compress the signal amplitude logarithmically before uniform quantization, then expand it symmetrically after reconstruction. The two standard companding laws — μ-law (North America, Japan) and A-law (Europe) — defined the quality of telephone audio for decades and remain in use in VoIP codecs today.