Glonce

ResCom adds a low-rank fix for 4-bit LLM quantization error

researchThe Glonce editorsarXiv🔥58Glonce rating58Impact45Novelty55Relevance100Surprise25Players10Credibility75

Low-bit weight-activation quantization cuts the memory and compute needed to run large language models, but rounding the activations can still hurt accuracy even after calibration. The paper's diagnosis is specific: reconstruction from rounded inputs cannot tell apart activations that fall in the same quantization bin, so information about where inside the bin each value sat is lost.

The proposed fix is ResCom, a low-rank correction that uses the activation rounding residual (the gap between the original activation and its rounded value) to compensate for the resulting error in the layer's output. An output-aware calibration procedure picks the residual directions by how much they contribute to reconstructing the output. The correction is then computed in closed form, with no gradient training and no updates to the calibrated weights.

The headline result is on Llama-3-8B with 4-bit weights and activations. With rank-32 attention corrections, WikiText-2 perplexity falls from 7.708 to 7.147 (lower is better), MMLU improves by 2.89 percentage points, and five-shot GSM8K accuracy rises from 23.96 to 36.16 percent. The authors also report evaluations on models from 1B to 70B parameters and on held-out C4, to test the method beyond the Llama-3-8B setting; the abstract gives no figures for those.

On cost, the evaluated FP32 implementation adds 2.25 percent to 4K prefill time. A separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

Key facts

  • ResCom is a low-rank correction that uses the activation rounding residual to compensate layer-output error in low-bit weight-activation quantization.
  • Calibration is output-aware and closed-form, with no gradient training and no updates to the calibrated weights.
  • On Llama-3-8B with 4-bit weights and activations, rank-32 attention corrections cut WikiText-2 perplexity from 7.708 to 7.147 and lift MMLU by 2.89 percentage points.
  • Five-shot GSM8K accuracy on the same setup rises from 23.96 to 36.16 percent.
  • The evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation keeps a 27.7 percent peak-memory reduction relative to FP16.

Why it matters

Running models with 4-bit weights and activations saves memory and compute, but activation rounding is a known source of accuracy loss that calibration does not fully remove. The paper targets that gap directly, arguing that rounded inputs hide which activations shared a quantization bin. Its fix needs no gradient training and leaves the calibrated weights untouched, which keeps the procedure light. The largest reported gain is on a reasoning benchmark: five-shot GSM8K goes from 23.96 to 36.16 percent on Llama-3-8B.

Who it affects

The work is aimed at people who deploy large language models with low-bit weight-activation quantization and want to recover accuracy lost to activation rounding. The reported evidence is centred on Llama-3-8B, with further evaluations on models from 1B to 70B parameters.

How to use it

The abstract describes a recipe rather than a product: compute the activation rounding residual, select residual directions by their contribution to output reconstruction, and form a low-rank correction in closed form on top of an already calibrated model. The headline result uses rank-32 corrections on attention. No code release, license or publication venue is stated. Two implementations are reported: an FP32 one that adds 2.25 percent to 4K prefill time, and a separate compact-storage one that retains a 27.7 percent peak-memory reduction relative to FP16.

How solid is it

This is an arXiv abstract, so the numbers are the authors' own and come without a tables-level breakdown here. The concrete figures cover Llama-3-8B only. The 1B to 70B and held-out C4 evaluations are mentioned without figures. The baseline for the perplexity, MMLU and GSM8K comparisons is described only as calibrated 4-bit weight-activation quantization, and the baseline is not named further. Latency and memory numbers belong to different implementations, so they should not be read as one combined result.

Risks and caveats

The authors themselves frame the 1B to 70B and C4 runs as a test of the method beyond Llama-3-8B, and the abstract gives no results for them. The 2.25 percent prefill overhead applies to the FP32 implementation and the 27.7 percent memory saving to a separate compact-storage one, so the abstract does not show both benefits at once. No prefill baseline in absolute units and no hardware are given.

“Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin.”

— ResCom paper abstract, arXiv 2610.35955