Glonce

MACaQ quantizer from huawei-csl targets GGUF, vLLM and Diffusers models

toolsThe Glonce editorsHacker News🔥63Glonce rating63Impact55Novelty55Relevance95Surprise35Players40Credibility75
MACaQ quantizer from huawei-csl targets GGUF, vLLM and Diffusers models

A GitHub repository under the huawei-csl organization publishes MACaQ, a model-agnostic quantization tool. Its tagline is "High-quality GGUF for local inference. Model-agnostic quantization beyond text." The tool quantizes text, image, video and audio models and produces checkpoints that run with llama.cpp, vLLM, Diffusers and PyTorch. An accompanying paper, "MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations" (arXiv 2611.10649, 2026), is by Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli.

The README leads with two claims. First, MACaQ can quantize a 122B-parameter model in 64 minutes using just 32.5 GB of device memory. Second, users can get lower perplexity than MR-GPTQ and DuQuant++, or match their best Llama-3-8B perplexity with 11x less quantization time.

The repository is split by backend, and each backend has its own guide and should get its own Python 3.12 environment. Diffusers covers image, video and audio generation at W4A4 (4-bit weights and activations); it quantizes diffusion transformers, saves safetensors checkpoints, and image-to-video uses conditioning images during calibration. The llama.cpp backend handles text generation with 2-, 3- and 4-bit GGUF formats, includes its own runtime, and also supports CPU execution. The vLLM backend covers text generation and serving, in W4A4 or weight-only mode, with W4A4 as the default. The PyTorch backend handles speech generation at W4A4 and saves compact checkpoints that load without repeating calibration.

A variant called MACaQ-R refines the weight codes in an already quantized GGUF while preserving its native formats and block scales, so its GGUFs run with a standard llama.cpp installation. The README shows a refine command that takes a quantized MoE GGUF, the matching original model, an FP16 GGUF as teacher and a calibration text file, and writes a new GGUF. A Qwen3.6-35B-A3B MACaQ-R GGUF is offered for download on Hugging Face, with a chart of KL divergence to FP16 (lower is better). Another comparison sets MACaQ against published Qwen3-4B GGUFs by file size and KL divergence to FP16, and a Qwen3.8-27B section reports quality and speed against the original BF16 weights in vLLM, taken as 1x. The chart values are not in the text.

For Huawei Ascend hardware, MACaQ W4A4 dense and grouped matmul kernels are available in two CANN pull requests: ops-nn #12568 (PTO-ISA kernels via ascend_ops_nn) and ops-tilelang #44 (TileLang kernels with the CCE backend). The README says to follow their build instructions for compatible CANN, PyTorch and torch_npu versions, and that full-model execution requires integration with an Ascend inference runtime.

The project is licensed under the Apache License 2.0, and bundled third-party components keep their own licenses. The README also asks that anyone, human or AI agent, who uses MACaQ to quantize a model star the repository.

Key facts

  • MACaQ is a model-agnostic quantizer for text, image, video and audio models, with outputs for llama.cpp, vLLM, Diffusers and PyTorch.
  • The README says it can quantize a 122B-parameter model in 64 minutes using 32.5 GB of device memory.
  • The README claims lower perplexity than MR-GPTQ and DuQuant++, or their best Llama-3-8B perplexity with 11x less quantization time.
  • llama.cpp output supports 2-, 3- and 4-bit GGUF; Diffusers, PyTorch and vLLM use W4A4, and vLLM also offers weight-only.
  • W4A4 matmul kernels for Huawei Ascend are in CANN pull requests ops-nn #12568 and ops-tilelang #44; the code is Apache 2.0.

Why it matters

Quantization decides whether a large model fits on local hardware, and the usual trade-off is between quality loss and how long calibration takes. MACaQ's pitch is both at once: lower perplexity than MR-GPTQ and DuQuant++, or the same Llama-3-8B perplexity in 11x less quantization time. It also reaches past text, covering image, video and audio generators in the same toolchain, and it ships GGUF files that run on a standard llama.cpp install.

Who it affects

People running models locally with llama.cpp, teams serving text models through vLLM, and users of Diffusers for image, video or audio generation are the direct audience. Researchers working on low-precision weights and activations get a paper and a reference implementation. Developers on Huawei Ascend hardware get W4A4 kernels in CANN, though full-model execution there needs integration with an Ascend inference runtime.

How to use it

Pick the backend that matches your model and follow its guide, using Python 3.12 and a separate environment for each backend. For text, the llama.cpp path quantizes to 2-, 3- or 4-bit GGUF and runs with the bundled runtime; llama.cpp also runs on CPU. If you already have a quantized MoE GGUF, the MACaQ-R refine command takes the GGUF, the original Hugging Face model, an FP16 GGUF as teacher and a calibration text file, and writes a refined GGUF that runs with a standard llama.cpp installation. A ready-made Qwen3.6-35B-A3B MACaQ-R GGUF is on Hugging Face. For serving, the vLLM path defaults to W4A4. The code is Apache 2.0.

How solid is it

The numbers come from the project's own README, which points to the accompanying paper (arXiv 2611.10649) for the details. The 64-minute, 32.5 GB figure is for a single 122B-parameter example, and the source does not say which hardware produced it. The comparisons are against MR-GPTQ and DuQuant++ only, and no comparison with any other quantization product is stated. The KL divergence and quality charts are not reproduced in the text.

Risks and caveats

The headline claims are the authors' own and no independent reproduction is mentioned. W4A4 is a low-precision mode, so output quality should be checked on your own model and task. The Ascend kernels are in pull requests rather than a finished runtime. Each backend needs its own environment. The GitHub organization is huawei-csl, but the README text itself does not name Huawei or the authors' employers.

“MACaQ can quantize a 122B-parameter model in 64 minutes using just 32.5 GB of device memory.”

— MACaQ README