SteerDistill uses a full-precision teacher to repair quantized LLM agents

SteerDistill uses a full-precision teacher to repair quantized LLM agents

Low-bit quantization makes LLM agents cheaper to run, but it can hurt how often they finish their tasks. A paper on the Hugging Face papers page, available as an abstract, looks at why. By examining failed rollouts, the authors find that quantized agents overlook task requirements or environmental conditions. As a result, the environment transitions they produce diverge from those of successful full-precision references.

The authors say these behavioral divergences are hard to correct with standard data. Offline data may miss the student's failure contexts, and the student's own rollouts may contain few correct responses in those contexts.

Their answer is SteerDistill, a quantization-aware distillation method. A full-precision teacher guides the quantized student to revise its failed responses, and the guidance is refined iteratively using feedback from the student's execution. The authors call this closed-loop teacher steering and say it produces locally verified behavioral corrections.

The corrections are then internalized. The student is trained, without any guidance in its input, to match the steered student's predictions at the turns that were corrected and the full-precision teacher's predictions at all other turns. The goal is unassisted inference: at deployment the quantized agent runs on its own.

The evaluation covers five agentic benchmarks and three models. Under W4A16, SteerDistill exceeds the strongest quantized baseline by 1.67 to 2.18 points in average score, and it stays within one point of BF16. On τ^2-Bench specifically, it outperforms QeRL by 1.84 percentage points in success rate while using 67% fewer training GPU hours. The authors also report that it produces shorter successful trajectories than the quantized baselines on that benchmark.

Key facts

  • SteerDistill is a quantization-aware distillation method in which a full-precision teacher guides a quantized student to revise failed responses, refining its guidance from feedback on the student's execution.
  • The authors' diagnosis from failed rollouts: quantized agents overlook task requirements or environmental conditions, so their environment transitions diverge from successful full-precision references.
  • Across five agentic benchmarks and three models, it beats the strongest quantized baseline by 1.67 to 2.18 points in average score under W4A16 and stays within one point of BF16.
  • On τ^2-Bench it outperforms QeRL by 1.84 percentage points in success rate with 67% fewer training GPU hours.
  • The final student is trained to run without guidance, so inference is unassisted.

Why it matters

Quantization is a standard way to cut the inference cost of LLM agents, but the authors note it can compromise task success. Agents act over many turns in an environment, so a small misreading of a task requirement can send a whole trajectory off course. This work targets that failure directly, correcting the specific turns where the quantized agent diverges from a full-precision reference, rather than only matching outputs in general. The reported result is a quantized agent within one point of BF16 in average score, at a lower training cost than the QeRL baseline on τ^2-Bench.

Who it affects

The work is aimed at people who want to deploy LLM agents at lower inference cost with low-bit weights and still keep task success close to full precision. It is also relevant to researchers working on quantization-aware training and distillation for multi-turn agentic settings. The abstract does not name the authors or their institutions.

How to use it

The abstract describes the recipe rather than a product. A full-precision teacher steers the quantized student on its failed responses, the guidance is refined from execution feedback, and the student is then trained without guidance to match the steered student's predictions at corrected turns and the teacher's at other turns. No code release, license or publication venue is mentioned in the source, so there is nothing to download or run from this page.

How solid is it

This is a research preprint summarized from its abstract alone. The headline numbers are margins: 1.67 to 2.18 points over the strongest quantized baseline in average score under W4A16, and 1.84 percentage points over QeRL in success rate on τ^2-Bench with 67% fewer training GPU hours. The source does not name the five benchmarks (other than τ^2-Bench) or the three models, and gives no per-benchmark or per-model scores or absolute baseline numbers. It also does not state whether the within-one-point gap to BF16 holds for every model and benchmark or for the average score.

Risks and caveats

Only the W4A16 setting is mentioned; other bit-width settings are not. The claim that successful trajectories are shorter is made for τ^2-Bench against the quantized baselines, and the abstract does not say how much shorter. The efficiency comparison with QeRL (67% fewer training GPU hours) is likewise reported for τ^2-Bench only. Without the full paper, the margins cannot be checked against the underlying scores.

“This closed-loop teacher steering produces locally verified behavioral corrections.”

— SteerDistill paper abstract, Hugging Face papers