Time-Stretching by 10 Percent With atempo and rubberband: Timing, Transients, Tone and Stereo, Measured

ffmpeg includes two filters for changing tempo while keeping the pitch unchanged. atempo is built into ffmpeg and uses WSOLA: it cuts audio into overlapping pieces and shifts each piece to match the preceding one as closely as possible. rubberband passes the audio to the Rubber Band Library, which works in the frequency domain and applies its own rules at transients.

The measurement used tempo factors of 0.9 and 1.1, which turn a track at 120 BPM into one at 108 or 132 BPM. It examines the effects on four aspects of the signal: timing, individual hits, tones and the stereo image. It does not compare which method sounds better.

How it was measured

• Methods: ffmpeg 7.1.5 with Rubber Band Library 3.3.0, 44.1 kHz, output as 32-bit float. atempo and rubberband at their defaults, plus rubberband with transients=mixed, transients=smooth, window=long and channels=together.
• Time: 20 short clicks of 2 ms of a 3 kHz tone, 0.5 s apart. Measured were the length of the output, the position of every click against its target and whether every click arrives.
• Hits: 12 single kicks and snares a little over a second apart, each moved 37 ms further against the grid so that it meets different parts of the windows. The evaluation used a 3 ms RMS envelope: the moment it reaches half the peak, and its level 10 and 20 ms before that.
• Tones: pure sine tones at 110, 440 and 1760 Hz and sawtooth tones at 55, 110, 220 and 440 Hz, 5 s each, stretched at tempo 0.9, 0.95, 1.05 and 1.1. For the sine, what counts is how far the level in 10 ms windows falls below its average. For the sawtooth, what counts is the energy between the harmonics, relative to the total energy and compared with the unprocessed sawtooth.
• Mix: 16 s of stereo made of kick, snare, hi-hats, bass, pad and lead. Measured were the correlation of the two channels, the level and the processing time, the shortest of three runs.
• Cross-checks: the first count of doubled hits found 10 to 13 onsets even in the unprocessed hit, because a 0.25 ms envelope follows the cycles of the kick; the count therefore uses 3 ms and always sets the original beside it. The first run with a single sawtooth at 110 Hz made atempo look clean, and the series over four fundamentals shows that this was the exception.

Time: rubberband hits it, atempo moves single hits

A negative value means earlier or shorter than required. All 20 clicks arrived with every method. rubberband delivered the length to the sample and put every hit within 0.62 ms of its place at its default and with smooth or together, within 0.75 ms with window=long and within 2.00 ms with mixed. That was not a given: the filter starts the library in real-time mode, and there, according to its documentation, no delay or latency compensation takes place at the start of the signal. What remained was an offset of 3.0 to 7.3 ms that changes sign with the direction of the stretch.

atempo started the whole output 15 to 20 ms early and made it 21 to 27 ms too short. More significant for timing is the error that varies from hit to hit: up to 10.84 ms at tempo 0.9 and up to 8.37 ms at 1.1. The method cuts audio into windows of 2048 samples, or 46.4 ms at 44.1 kHz, and places each window where it best matches the preceding one. Any hit within that window shifts with it.

Transients: rubberband builds a ramp in front of the hit

The levels before the hit are relative to its peak and are the highest value of twelve hits. In the original there is silence at that point. With rubberband the envelope rises out of the noise as early as 30 ms before the hit and reaches −12.3 to −22.8 dB 10 ms before it; with window=long, −17.1 dB is still present 20 ms before. The hit itself loses 0.5 to 2.0 dB of peak. The chart shows this ramp for both settings.

atempo has no ramp; 20 ms before the hit there is nothing. The values 10 ms before come from something else: for some hits a piece of the attack starts a few milliseconds before the actual onset, with a dip in between. The peak is kept, to within 0.24 dB.

Pure tones: rubberband’s default dips

The figures give how far the level of a steady tone falls below its average in a 10 ms window. At its default, rubberband detects transients even in a pure sine and resets the phases there, as the documentation describes for the crisp setting. The result is dips of up to 11.5 dB, and 10.4 dB at tempo 0.95 and 110 Hz. At 440 Hz the pitch swung between −7.0 and +11.5 cents; at 110 Hz no clean line could be read from the spectrum any more.

According to the documentation, transients=mixed resets the phases only outside the range of typical fundamentals, and that is exactly what the table shows: 440 Hz stays calm, 110 and 1760 Hz dip as with the default. transients=smooth never resets and keeps the tone within 0.4 dB except in one case, with 0.0 cents of deviation. atempo holds the level just as well but moves the pitch by a small fixed amount, +0.4 cents at 440 Hz for tempo 0.9 and −0.4 cents for 1.1.

Harmonic-rich tones: no method stays clean at every fundamental

The method rows give how far the energy between the harmonics lies above that of the original. A sawtooth has harmonics up into the treble and stands for bass, pads and leads. Across all four tempi the increase ranged from 1.4 to 40.3 dB with atempo, from 15.4 to 46.5 dB with the default of rubberband and from 4.2 to 26.1 dB with transients=smooth.

The outlier of atempo is the low tone. At 55 Hz and tempo 0.9 the interference lies 9.8 dB below the total energy, the level dips by 5.2 dB and the pitch wanders up to +20.4 cents. One period is 802 samples long there and the window 2048; it no longer finds a matching counterpart for every position. At tempo 1.1 the same value falls to +10.3 dB, and at 110 Hz atempo stayed within 2.0 dB of the original at all four tempi.

Stereo image, level and processing time

Before stretching, the correlation was 0.92. By default, rubberband processes the channels separately; according to the documentation, this produces a more diffuse stereo image and reduces mono compatibility. With smooth the correlation fell to 0.49, and with mixed to 0.75. channels=together keeps it above 0.91 in every combination.

This does not correct the level loss with smooth: the mix is still 2.6 to 3.0 dB quieter with together, so the loss occurs within the channels. You can compare the correlation of your own mix before and after stretching with the stereo correlation and mid/side meter.

On the Raspberry Pi 4, atempo ran 19 to 26 times faster than real time and rubberband 3.7 to 8.1 times. Both are enough for any file; in the time frame of a running project the difference would be noticeable.

What follows from it

• For drums and whole mixes where timing counts: rubberband=tempo=…:channels=together. Every hit landed within 0.48 ms of its place, and the stereo image stayed at 0.92. The price is the dips on steady tones and the ramp in front of every hit.
• For pads, vocals and held notes: rubberband=tempo=…:transients=smooth:channels=together. Pure tones stayed almost unchanged, the stereo image at 0.91, and the harmonics were damaged least, with an increase of 4 to 26 dB. The mix becomes 2.6 to 3.0 dB quieter and needs to be brought back to loudness afterwards.
• atempo where there are no fixed hits and no low tones. Tone, level and stereo image were kept best, but single hits moved by up to 10.84 ms, and a 55 Hz tone lost most of its cleanness at tempo 0.9.
• The tempo as a factor: 120 to 132 BPM is tempo 1.1, 120 to 108 BPM is tempo 0.9. The delay times and note lengths that belong to the new tempo are worked out by the BPM and delay time calculator.

Scope and limits

• The signals were synthetic, not recordings. Voice, acoustic instruments and reverb were not tested, and a listening test was not part of the measurement.
• ffmpeg offers no choice of engine for rubberband and therefore uses the older R2 engine. According to the documentation, the R3 engine gives better results for most material, especially complex mixes, vocals and heavy bass, but needs considerably more processing power. It cannot be reached through ffmpeg.
• The processing times apply to a Raspberry Pi 4 under changing load; on other hardware the absolute values change.
• The limits of the evaluation, 3 ms for the envelope and 10 ms windows for the level, co-determine how large a dip appears. Shorter windows show deeper dips.

Sources

• ffmpeg: Filters Documentation: the sections on atempo and rubberband; options and defaults are taken from the built-in help of ffmpeg 7.1.5.
• FFmpeg, libavfilter/af_atempo.c: WSOLA as the method and the window size from the sample rate divided by 24, rounded up to a power of two.
• FFmpeg, libavfilter/af_rubberband.c: the library started in real-time mode and without a choice of engine.
• Rubber Band Library: RubberBandStretcher: the R2 and R3 engines and which of them is the default, the missing latency compensation in real-time mode, the settings for transients, window and channels.