Glonce

FLEET adds memory to LLM sampling, cutting compute 3x

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact68Novelty72Relevance100Surprise50Players22Credibility78
FLEET adds memory to LLM sampling, cutting compute 3x

Researchers describe FLEET, a method that changes how large language models generate text when multiple samples are drawn to improve accuracy. The common approach, temperature sampling, pulls several completions from a model's output distribution and aggregates them, but it does so without any memory of what has already been generated. The authors argue this is inherently suboptimal: because each new sample is drawn without awareness of prior generations and how they scored, an increasing share of the samples end up being semantically duplicate answers as more are drawn, so the accuracy gains taper off quickly. FLEET addresses this by giving the generation process a memory. It represents each generation as a sparse trajectory that passes through states where the token-level entropy exceeds a set threshold, and it uses these tracked trajectories to derive a per-token utility score. That score is then used to adjust the logits during subsequent generation, steering the model away from paths that have already been explored and evaluated. In benchmark evaluations, FLEET reaches the same accuracy as the repeated-sampling baseline while running three times faster, and it produces a larger accuracy gain specifically on complex coding tasks: on LiveCodeBench Pass@32, accuracy rises from 59.9% to 66.2% under the same computational budget. The paper also reports that in the greedy-decoding configuration it evaluated, FLEET behaves deterministically and needs only a single calibration pass to set its main hyperparameters, which the authors say means it requires only minimal changes to existing LLM generation pipelines.

Key facts

  • FLEET replaces memoryless temperature sampling with a mechanism that tracks sparse trajectories through high-entropy states and turns them into per-token utility scores that adjust the logits.
  • The authors say temperature sampling wastes compute because samples drawn independently increasingly overlap in meaning, giving diminishing returns as more samples are taken.
  • FLEET matches the accuracy of the repeated-sampling baseline while running 3x faster under the same computational budget.
  • On the LiveCodeBench Pass@32 coding benchmark, accuracy rises from 59.9% to 66.2% with FLEET under the same budget.
  • In the greedy-decoding setup evaluated, FLEET is deterministic and its main hyperparameters are set with a single calibration pass, requiring only minimal changes to existing pipelines.

Why it matters

Repeated sampling is a common way to squeeze more accuracy out of an LLM at inference time, but it is expensive and, per the authors, wasteful: without memory of earlier attempts, a model keeps regenerating answers that are effectively the same. FLEET's contribution is to give the sampling process memory by tracking high-entropy trajectories and folding what has already been tried back into the logits, so later generations are pushed toward paths that have not been explored yet.

Who it affects

The method targets anyone building or operating LLM pipelines that lean on multi-sample generation for accuracy, including systems tackling complex coding tasks, where the paper reports its largest measured gains.

How to use it

The paper describes FLEET as requiring only minimal modifications to existing LLM generation pipelines: in the greedy-decoding configuration it evaluated, the method is deterministic and its principal hyperparameters are derived from a single calibration pass rather than extensive tuning. The source gives no information on when or how FLEET might become available for adoption.

How solid is it

The account rests on the paper's own benchmark evaluations, including a controlled comparison against a repeated-sampling baseline and the LiveCodeBench Pass@32 result. The source text does not name the authors or their institution, does not specify which LLMs were used in the benchmarks, and does not state how many samples the baseline drew for the speedup comparison, which limits independent verification of the exact experimental setup.

Risks and caveats

The reported 3x speedup and accuracy gains are tied to a specific benchmark setup and an unspecified baseline configuration, so it is unclear how the results generalize beyond the tasks tested. The determinism and single-pass calibration are described only for the greedy-decoding configuration the authors evaluated, leaving open how the method behaves under other decoding settings.

“this memoryless approach is inherently suboptimal: because it lacks awareness of prior generations and their evaluations, it produces an increasing proportion of semantically duplicate answers as more samples are drawn, leading to diminishing returns”

— from the paper