Glonce

A*-Thought-V2 cuts LLM response length up to half, lifts accuracy

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact62Novelty72Relevance92Surprise52Players38Credibility85
A*-Thought-V2 cuts LLM response length up to half, lifts accuracy

Chain-of-thought (CoT) prompting improves how well large language models reason, but the extra tokens it generates add computation and context cost. The authors frame the existing options as a poor trade-off: hard pruning of reasoning steps throws away information, while continuous compression methods lack a principled way to decide what to compress. A*-Thought-V2 is a new framework that instead treats a chain of thought as a hidden-state trajectory and decides, step by step, whether to keep a reasoning step as explicit text or fold it into a latent token, using what the authors call an explicit-implicit interleaved latent architecture.

The method projects the representations of the question, each reasoning step, and the final solution into a 3D PCA space, then measures the angle between each local step-to-step transition and the global direction from question to solution. Steps whose direction aligns closely with that global path stay as explicit text; steps that deviate are compressed into continuous latent tokens. The size of these directional angles carries meaning of its own: small angles correspond to direct execution and answer formation, while large angles occur more often during checking, correction, and branch exploration. Tracking how the angles change over the course of a solution reveals distinct exploration, convergence, and refinement stages in the model's reasoning.

Training this architecture required two new techniques. Stepwise embedding forcing pools each redundant reasoning step into a single latent embedding, and label forcing supervises that latent token against a soft, multi-modal distribution over the vocabulary rather than a single hard label, which the authors say encourages richer step-level feature learning.

The authors tested A*-Thought-V2 on Qwen3.5-9B and Qwen3.6-27B across six in-domain and out-of-domain benchmarks, though the paper does not name the benchmarks or the existing methods it compares against. Relative to those methods, A*-Thought-V2 improves average accuracy by up to 2.6%, cuts response length by up to half, raises Accuracy per Computation Unit by 2.29 times, and reduces preprocessing time by 94.6% and training time by up to 80.3%. The paper does not report absolute accuracy figures, only the size of these improvements. A separate representation analysis found that the latent states the model produces occupy a compact region distinct from ordinary textual hidden states, and that positions holding a latent token show higher entropy, which the authors attribute to the broader, softer training targets encouraging richer feature learning at those steps.

Key facts

  • A*-Thought-V2 projects the question, each reasoning step, and the solution into a 3D PCA space, then uses the angle between each step's local transition and the global question-to-solution direction to decide whether that step stays explicit text or gets compressed into a continuous latent token.
  • Small alignment angles mark direct execution and answer formation, while large angles mark checking, correction, and branch exploration; how the angles change over a solution traces distinct exploration, convergence, and refinement stages.
  • Training combines stepwise embedding forcing, which pools a redundant reasoning step into one latent embedding, with label forcing, which supervises that embedding against a soft, multi-modal vocabulary distribution instead of a single hard label.
  • Tested on Qwen3.5-9B and Qwen3.6-27B across six in-domain and out-of-domain benchmarks, A*-Thought-V2 improves average accuracy by up to 2.6%, cuts response length by up to half, and raises Accuracy per Computation Unit by 2.29 times.
  • The method also cuts preprocessing time by 94.6% and training time by up to 80.3%; representation analysis shows the latent states form a compact region distinct from ordinary textual hidden states.

Why it matters

Chain-of-thought prompting is one of the main levers for getting better answers out of large language models, but every extra reasoning token adds compute and context cost. The paper frames the existing options as a poor trade-off: hard pruning of reasoning steps throws away information, while continuous compression methods lack a principled way to decide what to compress and what to keep. A*-Thought-V2 proposes a geometric criterion instead, using the angle between a reasoning step and the overall question-to-solution direction to decide whether that step deserves to stay explicit or can be folded into a latent token. That gives compression a rationale tied to whether a step is doing useful work, rather than an arbitrary length or position cutoff.

Who it affects

The direct audience is researchers and engineering teams that build or fine-tune reasoning-heavy language models, particularly teams running long chain-of-thought inference at scale, where a shorter response maps directly to lower latency and serving cost. The paper demonstrates the method on Qwen3.5-9B and Qwen3.6-27B, covering both a smaller and a considerably larger model. Because A*-Thought-V2 changes how the model is trained, not just how it is prompted, adopting it means retraining or fine-tuning a model rather than switching a setting on an existing deployment.

How to use it

There is no product or API here, only a training recipe described in enough detail to reproduce the mechanics: project question, step, and solution representations into a 3D PCA space; measure the angle each step's local transition makes with the global question-to-solution direction; keep aligned steps as text and compress the rest into continuous latent tokens through an explicit-implicit interleaved architecture. Two training techniques make that architecture learnable: stepwise embedding forcing, which pools a redundant step into one latent embedding, and label forcing, which supervises that embedding with a soft, multi-modal distribution over the vocabulary instead of a single hard label. The paper does not say whether code, model weights, or data are released, so for now this is a blueprint to reimplement rather than a tool to download.

How solid is it

The evidence comes from the paper's own experiments: two model sizes, Qwen3.5-9B and Qwen3.6-27B, across six in-domain and out-of-domain benchmarks, with consistent gains reported on accuracy, response length, and the derived Accuracy per Computation Unit metric, alongside large drops in preprocessing and training time. What limits independent judgment is what the text leaves out: it does not name the six benchmarks, does not name the existing methods it is compared against, and gives no absolute accuracy numbers, only the size of the improvement over an unspecified baseline. It also carries no author names or institutional affiliation, so there is no way from this text alone to check who did the work or verify it independently.

Risks and caveats

Every headline number in the paper is a best case: 'up to 2.6%' accuracy and 'up to half' shorter responses are stated as ceilings, not averages, while the Accuracy per Computation Unit gain of 2.29 times and the 94.6% preprocessing-time cut are given as flat figures with no named point of comparison. The representation analysis, that latent states occupy a distinct, compact region and show higher entropy than textual states, is the authors' own interpretation of their model's internals rather than an independently verified property, and its evidence is limited to the two Qwen model families tested.