Dither When Exporting to 16-Bit: Distortion, Noise Modulation and Noise Shaping, Measured

FL Studio’s mixer calculates with 32-bit floating-point numbers. When exporting to a 16-bit file, every sample has to land on one of 65,536 steps. One step, the LSB, corresponds to 1/32,768 of full scale or −90.3 dBFS. The quieter the signal, the larger the rounding error is in relation to it: for a sine at −60 dBFS it was 38 dB below the tone, at −96 dBFS above it. The FL Studio manual describes dither as a way to break up the predictability of these rounding errors and applies it only when exporting 16-bit WAV and MP3.

The measurement converts test signals from 32-bit float to 16-bit with ffmpeg: without dither, with rectangular dither (RPDF), with triangular dither (TPDF), with high-pass filtered TPDF and with three noise-shaping methods. Robert A. Wannamaker summarised the theory in his 1997 doctoral thesis: undithered quantization can produce audible distortion and noise modulation; RPDF dither makes the mean error independent of the signal, TPDF dither its power as well. Level and peaks of the master are covered by the measurements True Peak After Encoding and Clipper or Limiter Before Export; this measurement concerns the last step, the word length.

How the measurement was made

• Conversion: ffmpeg 7.1.5 with the aresample filter and the options osf=s16 and dither_method, input a 32-bit float WAV at 44.1 kHz, output 16-bit PCM, then read back. The default value of dither_method is 0, which means no dither.
• Routines: ffmpeg ran with the option -cpuflags 0 and thus without the routines optimised for the processor. On the ARM computer used for the measurement, those routines shifted RPDF and TPDF by half an LSB; more on this under context and limits.
• Staircase of constant values: 41 constant input values from 0 to 2 LSB in steps of 0.05 LSB, one second each. For each step, the mean error (output minus input) and the spread of the output as standard deviation were calculated over the middle 0.8 seconds. If the mean error depends on the input, the conversion distorts; if the spread depends on it, the noise varies with the signal.
• Sine: 1 kHz with a peak level of −60 and −96 dBFS, ten seconds long, analysed in the spectrum of the middle eight seconds: the fundamental against its target, all other content between 20 Hz and 20 kHz, and harmonics 2 to 19 against the surrounding noise. For comparison, the same analysis ran at frequencies exactly between the harmonics.
• Silence: ten seconds of digital zero. Noise is given as an RMS value relative to full scale, in total and in five frequency bands.
• Dithered twice: the dithered silence was stored losslessly as 32-bit float and converted to 16-bit a second time with the same method.

Without dither: the error depends on the signal

Without dither, the conversion rounds every value to the nearest step. The mean error therefore followed the input like a sawtooth between −0.5 and +0.5 LSB, and the spread stayed at zero. The same input always gave the same output; the error is a fixed function of the signal, and digital silence remained digital zero.

The sine shows what this means for quiet passages. At −60 dBFS, all other content together was 38.3 dB below the fundamental. Because the pattern of tone and rounding repeats regularly, it formed individual lines instead of even noise; the strongest was at 17.5 kHz, 51.7 dB below the fundamental. At −96 dBFS the sine reaches only just over half a step, and short pulses were all that remained of it: the fundamental lost 3.7 dB, the third harmonic at 3 kHz was only 0.9 dB below it, the fifth 2.7 dB, and the other content together had 2.9 dB more power than the fundamental.

RPDF and TPDF: distortion removed, even noise only with TPDF

With RPDF dither, the mean error was within 0.004 LSB of zero at every input value: the distortion had disappeared. The spread, however, followed the input, from zero at whole-number values to 0.5 LSB exactly between two steps. The noise therefore varies with the signal, and digital silence remained entirely free of noise. This matches the theory: rectangular dither makes only the mean of the error independent of the signal, not its power.

With TPDF dither, the spread was between 0.49 and 0.50 LSB at every input value. According to Wannamaker, of all dither types that remain in the signal and remove both distortion and noise modulation, triangular dither raises the error power the least: to a quarter of the square of one step, as an RMS value 0.5 LSB or −96.3 dBFS. In silence, 0.50 LSB and −96.4 dBFS were measured, spread evenly across all frequencies. The high-pass filtered variant had almost the same total noise, but 5.3 dB less below 4 kHz.

On the sine, no dither method left a harmonic that stood out from the noise: at the harmonics the level was at most 4.6 dB above the surrounding noise, at control frequencies between the harmonics up to 4.2 dB. The fundamental deviated from its target by no more than 0.05 dB, even at −96 dBFS. There the noise between 20 Hz and 20 kHz had more power than the tone, but it is spread across all frequencies, while the tone sits at a single one.

Noise shaping: less noise below 4 kHz, more noise in total

Noise shaping feeds the rounding error back into the conversion through a filter and thus shapes its spectrum. Wannamaker names the aim of placing most of the error power in high frequency bands where the ear is relatively insensitive. Compared with TPDF, the three ffmpeg methods lowered the noise below 4 kHz by 16.8 dB (Lipshitz), 16.0 dB (F-weighted) and 9.8 dB (Shibata). In return, the total noise rose by 12.3, 18.5 and 24.3 dB, almost entirely above 16 kHz: with Shibata it reached −73.8 dBFS between 16 and 20 kHz, 29.9 dB above TPDF. The mean error remained independent of the input, and the spread stayed between 2.01 and 2.08 LSB (Lipshitz), 4.11 and 4.29 LSB (F-weighted) and 7.95 and 8.31 LSB (Shibata).

According to its manual, FL Studio also uses a shaped dither that moves most of the noise above 10 kHz. The manual does not name the filter behind it; the ffmpeg values show the order of magnitude by which such methods can differ.

Dithered twice: every pass adds noise

According to the FL Studio manual, dither should be applied to a 16-bit file only once, during the final render, if it is used at all. The measurement shows what a second pass costs: when the dithered silence was dithered again, the noise rose by 3.0 dB each with TPDF, Lipshitz and Shibata, to −93.4, −81.1 and −69.1 dBFS. This applies to an independent second noise sequence. ffmpeg starts its noise sequence identically on every run; with both sequences exactly aligned, the noise rose by 6.0 dB (TPDF), 5.7 dB (Lipshitz) and 5.2 dB (Shibata).

What this means for the export

• When exporting at 24-bit or 32-bit float, as in the mastering tutorial for FL Studio, FL Studio applies no dither. A 24-bit step is 256 times smaller than a 16-bit step; TPDF noise would be at −144.5 dBFS there by calculation.
• For a 16-bit file, TPDF dither removes distortion and noise modulation with the least added noise, measured at −96.4 dBFS. Without dither, the third harmonic at −96 dBFS was only 0.9 dB below the fundamental; with RPDF the noise varied with the signal between zero and 0.5 LSB.
• Noise shaping trades less noise below 4 kHz for more noise in total. In the measurement this cost 12 to 24 dB, almost entirely above 16 kHz.
• Dither belongs in the chain only once: every further pass raised the noise by 3 dB, with an aligned noise sequence by up to 6 dB.
• When converting to 16-bit, ffmpeg applies no dither unless told to; aresample=osf=s16:dither_method=triangular switches on TPDF.

Context and limits

The measurement covered the conversion in ffmpeg, not FL Studio’s export; its shaped dither may give different values. The test signals were constant values, sine tones and silence; a listening test was not part of the measurement. How audible distortion at −96 dBFS or total noise of −72 dBFS is depends on the material, listening level and playback chain. With ffmpeg’s optimised routines, RPDF and TPDF on the ARM computer used for the measurement (aarch64) produced a mean error of −0.50 LSB instead of zero, a DC offset of −96.3 dBFS, and with RPDF digital silence carried 0.50 LSB of noise. With -cpuflags 0 the offset disappeared; the noise-shaping methods gave the same values in both cases.