Late entry cuts looped LM decoding latency, up to 8.4x on Huginn-3.5B

Looped language models get deeper reasoning from a small parameter count by running one shared layer stack T times for every token. The catch, as the paper's abstract puts it, is that recurrence depth turns directly into sequential decoding latency.
The usual fix is early-exit decoding, which stops after the first few passes. The abstract notes that this halts without completing the cache, so the cached key-value states of past tokens are truncated to shallow depth. The paper studies the opposite strategy, called late entry. New tokens skip the initial passes and enter directly into the final K < T passes, reading a deep context cache. Once lightweight fine-tuning resolves a hidden-state norm mismatch at the entry boundary, the deep cached context substitutes for the skipped token-local passes. On that basis the abstract says late entry outperforms matched early exit across model families and pass budgets.
To get real wall-clock gains without stalling to build deep caches, the method splits the work. Candidate blocks are verified through only the final K passes, which are the ones on the critical path. Tokens that have already been committed complete passes 1 to T alongside later steps, at zero extra serial calls. The finished T-pass "wake" states are projected inside the existing output GEMM. This is called Wake verification and is governed by a logit-margin tolerance tau between 0 and infinity. One setting covers three modes inside a single static CUDA graph: greedy-exact T-pass decoding at tau = 0 (matching K = T block decoding within bf16 reduction order), margin-gated verification at tau > 0, and non-rollback lossy late entry at tau = infinity.
On Ouro-2.6B (T = 4), K = 2 Wake reaches a 5.2x greedy-exact speedup at tau = 0, which is +26% over K = 4 block decoding on GSM8K. At tau = infinity it reaches 6.8x, with +16.9 points over trained early exit at matched speed.
The authors also scale test-time recurrence on Huginn-3.5B across T in {8, 16, 32, 64}. With K = T/4, rollbacks fall 21.5x with depth. Greedy-exact (tau = 0) speedups reach 6.1x at T = 32 and 8.4x at T = 64, the latter +153% over K = 64 block decoding. Speedups at tau = infinity range from 10.6x to 12.5x.
Key facts
- Late entry lets new tokens skip the early passes of a looped model and enter the final K < T passes over a deep context cache, instead of exiting early over a truncated shallow cache.
- On Ouro-2.6B (T = 4), K = 2 Wake gives a 5.2x greedy-exact speedup at tau = 0 (+26% over K = 4 block decoding on GSM8K) and 6.8x at tau = infinity.
- On Huginn-3.5B, greedy-exact speedups reach 6.1x at T = 32 and 8.4x at T = 64 (+153% over K = 64 block decoding); tau = infinity speedups run from 10.6x to 12.5x.
- Wake verification, controlled by a logit-margin tolerance tau, covers greedy-exact, margin-gated and non-rollback lossy decoding in one static CUDA graph.
- Lightweight fine-tuning to fix a hidden-state norm mismatch at the entry boundary is what lets late entry beat matched early exit, per the abstract.
Why it matters
Looped models trade parameters for compute: they iterate a shared stack T times per token, so more recurrence means more serial latency. That cost grows exactly where these models are meant to shine, at deeper test-time recurrence. The paper attacks the latency directly and reports that the gains grow with depth: on Huginn-3.5B the greedy-exact speedup rises from 6.1x at T = 32 to 8.4x at T = 64, and rollbacks at K = T/4 fall 21.5x with depth.
Who it affects
Mainly researchers and engineers working on looped or recurrent-depth language models and on fast inference for them. The two models tested are Ouro-2.6B and Huginn-3.5B. Standard fixed-depth transformers are not part of the reported results.
How to use it
Nothing is packaged for use yet: no code or model release is mentioned. For someone reimplementing the idea, the abstract describes the recipe: lightweight fine-tuning to fix the hidden-state norm mismatch at the entry boundary, verification of candidate blocks through the final K passes, background completion of passes 1 to T for committed tokens, and projection of the finished wake states inside the output GEMM. Tau selects the mode: 0 for greedy-exact, above 0 for margin-gated, infinity for lossy non-rollback.
How solid is it
This rests on an abstract alone. No authors or institutions are named. The abstract does not say what baseline the 5.2x, 6.8x, 6.1x, 8.4x and 10.6x to 12.5x speedups are measured against, beyond naming greedy-exact decoding. The hardware (GPU model) and batch size are not stated. The metric behind the +16.9 points is not named in the abstract; GSM8K is named only for the +26% comparison. The tau = 0 mode is said to match K = T block decoding within bf16 reduction order, which is the paper's own claim about exactness.
Risks and caveats
The largest speedups, 6.8x on Ouro-2.6B and 10.6x to 12.5x on Huginn-3.5B, come from the lossy tau = infinity mode, and no accuracy loss figure is given for it apart from the +16.9 points comparison against trained early exit. The greedy-exact numbers are the safer ones to compare. Late entry also depends on fine-tuning to fix the norm mismatch at the entry boundary, and the reported gains cover only two models.
“deep cached context substitutes for skipped token-local passes”
— Abstract of the paper "Late Entry in Looped Language Models"