Speculative Listening speeds up full-duplex speech models without training

Chunk-interleaved full-duplex speech language models listen and speak in fixed time chunks, and each chunk must be processed before the next one arrives. The paper starts from an imbalance in that workload. Listen chunks leave compute idle while the model waits for audio. Speak and tool-call chunks need serial generation, which delays responses and tool calls.

The obvious fix is to use the idle compute to decode the next chunk early. That means decoding it before its audio exists, under an assumed input. The authors took silence as the assumed input and compared the resulting output with the output under real audio, on four models. They report two properties. First, the next audio chunk does not change the output in 95.3% of chunks. Second, 88% of the changes that do occur begin at the first output token.

Building on these two properties, they propose Speculative Listening, described as a training-free, lossless framework for input-time self-speculative decoding. During idle time the model drafts the next chunk under silence. For changes at the first token, it also drafts a second branch starting from the runner-up first token. All drafts go into a cross-chunk hypothesis pool, and they are verified when the real audio arrives.

Results come from the authors' eight-task benchmark, TickTock-Bench. Speculative Listening speeds up speak chunks by 2.6-2.8x and tool-call chunks by up to 23.4x. On average, it cuts the decoding delay before a response starts by 73%, to 74-241 ms, which the authors place close to the 200 ms gap between human turns. It also cuts the maximum lag behind real time by 94.4%.

Key facts

Taking silence as the assumed input, the next audio chunk left the model's output unchanged in 95.3% of chunks, observed on four models.
88% of the output changes that did occur began at the first output token, which is why a second draft branch from the runner-up first token is enough to cover most of them.
Speculative Listening is described as training-free and lossless: the model drafts the next chunk during idle listen time, keeps drafts in a cross-chunk hypothesis pool, and verifies them when audio arrives.
On the eight-task TickTock-Bench, speak chunks run 2.6-2.8x faster and tool-call chunks up to 23.4x faster.
On average the decoding delay before a response starts falls by 73%, to 74-241 ms, and the maximum lag behind real time falls by 94.4%.

Why it matters

Full-duplex speech models have to keep up with a live conversation, so every chunk has a hard deadline. The paper targets a specific waste: listen chunks leave compute idle, while the chunks that produce speech or tool calls are slow and serial. If drafting can be moved into the idle time, the delay before a reply starts shrinks. The authors report 74-241 ms on average, which they compare with the roughly 200 ms gap between human turns.

Who it affects

The work is aimed at chunk-interleaved full-duplex speech language models, the kind that listen and speak in fixed time chunks. Anyone building voice assistants on that design, including ones that call tools mid-conversation, is the audience, since tool-call chunks show the largest reported gain (up to 23.4x). The text names no specific models or products.

How to use it

The method is described as training-free, so by the authors' account it is applied at decoding time rather than by retraining the model. The mechanism is: draft the next chunk under assumed silence during idle time, add a second branch from the runner-up first token for first-token changes, store all drafts in a cross-chunk hypothesis pool, and verify them when the real audio arrives. The abstract does not say whether code or models are released.

How solid is it

This is a preprint listed on Hugging Face Papers, and the speed figures come from the authors' own eight-task benchmark, TickTock-Bench. The 95.3% and 88% observations come from four models, which the abstract does not name. The abstract does not give per-model or per-task breakdowns beyond the quoted ranges, does not say which model or task produces the 23.4x tool-call speedup, and does not compare against other speculative-decoding methods. The 73% and 94.4% reductions are stated as averages.

Risks and caveats

The approach rests on a bet: that silence is a good guess for the next chunk. It held in 95.3% of chunks on four models, but the other 4.7% of chunks need the verification step, and the authors found that most such changes start at the first token. The 2.6-2.8x and up-to-23.4x speedups are on a benchmark the authors built, so they may not carry over to other models or workloads. The abstract does not state the chunk length in milliseconds. The 'lossless' label is the authors' claim.

“a training-free, lossless framework for input-time self-speculative decoding”

— From the paper's abstract, describing Speculative Listening