Glonce

Hugging Face Transformers adds llama.cpp GGUF support

toolsThe Glonce editorsHugging Face🔥61Glonce rating61Impact60Novelty45Relevance85Surprise35Players60Credibility90
Hugging Face Transformers adds llama.cpp GGUF support

Hugging Face has added the ability to load llama.cpp's GGUF quantized model checkpoints directly into its Transformers library, through the standard from_pretrained call. The feature reuses llama.cpp's underlying ggml kernels via the kernels library and trims overhead in generate, so that running a GGUF model in Transformers gets close to llama.cpp's own performance rather than merely being compatible with the file format. The initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture. GGUF, built by the llama.cpp team, packages a model's weights and metadata, including tokenizer information and an optional chat template, in a single file, and supports quantization levels such as Q4_K_M, which mixes mostly 4-bit weights with higher precision kept for sensitive tensors. GGUF checkpoints from the Hub, published by teams such as Unsloth, LM Studio Community and bartowski under names like ggml-org, have been downloaded millions of times, and llama.cpp itself powers local tools including Ollama, LM Studio and Jan. To load a GGUF checkpoint, a user installs the main branch of transformers plus the kernels package, then passes the Hub model_id and the GGUF filename as gguf_file to AutoModelForCausalLM.from_pretrained and AutoTokenizer.from_pretrained; no further configuration is required, since transformers automatically pulls the matching ggml/Metal kernel build and uses it as the attention implementation. If that kernel cannot be fetched, the model falls back to the standard sdpa attention implementation with a warning, and sdpa can also be forced explicitly. Without a compatible quantization kernel at all, the loader instead dequantizes the model, which uses more memory. The same checkpoints can be served through transformers serve, which exposes an OpenAI-compatible API and lets a client such as Jan or Pi connect to it as a custom provider; a reasoning flag controls whether chat-template thinking is switched on, off, or left to the template's default. For fine-tuning, GgufConfig(dequantize=True) unpacks a GGUF checkpoint back into a standard transformers training workflow. Hugging Face benchmarked the new path against llama.cpp's own llama-bench tool across three GGUF checkpoints, a small dense model, a larger dense model and a mixture-of-experts model, on a MacBook Pro M2 Max with 32GB of unified memory, running macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0. The llama.cpp figure is tg128, the token-generation rate over 128 decoded tokens, averaged across three repetitions with prompt processing excluded, measured with llama-bench build 5f55650a7, release b10200, on ggml 0.18.0's Metal backend; the transformers figure is generate producing the same 128 tokens from a 12-token prompt, the best of three warmed runs, and it does include prefill. Hugging Face reports that transformers came close to llama.cpp across all three checkpoints, while cautioning that the two numbers are not measuring identical conditions because of that prefill difference. The benchmark script itself pauses 90 seconds between runs to let the machine cool, since back-to-back runs without a break decayed by 10% or more. Hugging Face frames llama.cpp as remaining its recommended engine when raw local-inference efficiency is the priority, and positions the new GGUF path instead as a way to experiment with, evaluate, validate and fine-tune the same checkpoints inside the ordinary PyTorch and transformers tooling. The post also points past GGUF itself: because a ggml kernel operates on tensors rather than requiring a full llama.cpp implementation of a model, the same kernels could in principle accelerate architectures llama.cpp does not support, and extend beyond text generation into vision, audio and multimodal models, though each architecture would still need its own integration and validation. Illustrating what local inference already feels like, the post embeds a tweet from Julien Chaumond (@julien_c), dated April 24, 2026, describing a Qwen3.6 27B model running inside the Pi coding agent via llama.cpp on a MacBook Pro as feeling close to using Anthropic's Claude Opus on non-trivial Hugging Face codebase tasks.

Key facts

  • Transformers can now load llama.cpp's GGUF quantized checkpoints directly via from_pretrained, reusing llama.cpp's ggml Metal kernels through the kernels library rather than just reading the file format.
  • The initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture; GGUF checkpoints on the Hub, from publishers such as Unsloth, LM Studio Community and bartowski, have been downloaded millions of times.
  • Hugging Face benchmarked it against llama.cpp's llama-bench tool (tg128, 128 decoded tokens averaged over three runs) on a MacBook Pro M2 Max with 32GB unified memory, and reports transformers came close to llama.cpp across a small dense, a larger dense and a mixture-of-experts checkpoint, though the transformers figure includes prefill and llama-bench's does not.
  • Loading needs only a gguf_file argument to from_pretrained; the same checkpoints can also be served through transformers serve's OpenAI-compatible API, or dequantized with GgufConfig(dequantize=True) for fine-tuning.
  • Without a compatible ggml/Metal kernel, the loader falls back to the sdpa attention implementation with a warning, or dequantizes the model outright, using more memory.

Why it matters

llama.cpp is the engine behind most popular local-AI tools, including Ollama, LM Studio and Jan, and GGUF, its quantization format, has become the default way people share and run open-weight models on their own machines, with checkpoints downloaded millions of times. Until now, using a GGUF checkpoint meant stepping outside the Python and PyTorch tooling that Transformers users already know. By loading GGUF files directly and reusing llama.cpp's own ggml Metal kernels through the kernels library, Hugging Face is closing that gap: the same checkpoint can now be run, inspected and modified with the ordinary transformers API while staying close to llama.cpp's own speed, instead of forcing a choice between compatibility and performance.

Who it affects

Developers running open-weight models locally on Apple Silicon Macs are the immediate audience, since that is the only supported hardware so far and the initial architecture is Qwen3.5. It also matters to publishers of GGUF checkpoints such as Unsloth, LM Studio Community, bartowski and the llama.cpp team's own ggml-org releases, whose files become directly loadable without conversion. Within Hugging Face's own workflows, the integration is aimed at people who want to evaluate GGUF checkpoints with existing transformers tooling, validate that a GGUF conversion matches the original weights, experiment with hooks and custom decoding logic, or dequantize a GGUF checkpoint to continue fine-tuning it.

How to use it

The requirements are an Apple Silicon Mac, one of the two most recent PyTorch releases supported by the published ggml-quantization kernel builds, and the main-branch version of transformers together with a compatible kernels package, installed with pip install -U "git+https://github.com/huggingface/transformers.git" kernels. Loading a checkpoint takes one extra argument: passing the Hub model_id and the GGUF filename as gguf_file to both AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained. No further setup is needed, since transformers automatically fetches the matching ggml/Metal kernel and uses it as the attention implementation, with sdpa available as an explicit fallback via attn_implementation="sdpa". The same checkpoint can be served through transformers serve, which installs with the [serving] extra and exposes an OpenAI-compatible endpoint that clients such as Jan or Pi can connect to as a custom provider; a --reasoning flag turns chat-template thinking on, off, or leaves it at the template's default. For fine-tuning, GgufConfig(dequantize=True) unpacks the GGUF weights back into a standard transformers training workflow. The source gives no pricing or licensing terms beyond the software itself being open source and installed from GitHub.

How solid is it

Hugging Face's own comparison used llama.cpp's llama-bench tool as the reference, running three GGUF checkpoints: a small dense model, a larger dense model, and a mixture-of-experts model. Both sides were measured on the same MacBook Pro M2 Max with 32GB of unified memory, on macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0, plugged in. The llama.cpp number is tg128, the decode-only token-generation rate over 128 tokens averaged across three repetitions, from llama-bench build 5f55650a7, release b10200, on ggml 0.18.0's Metal backend; the transformers number is generate producing the same 128 tokens from a 12-token prompt, the best of three warmed runs, and it includes prefill, which llama-bench's figure excludes. Hugging Face reports the two came close across all three checkpoints but is explicit that the comparison is not apples to apples because of that prefill difference, and gives no percentage or ratio for how close the numbers actually landed. The benchmark script also builds in a 90-second cooldown between runs after finding that back-to-back runs without one decayed by 10% or more, which points to a deliberately careful methodology rather than a one-shot number.

Risks and caveats

Support is currently limited to Apple Silicon; the source makes no claim about Windows, Linux or non-Apple GPUs. It also requires transformers' main branch rather than a released version, plus a PyTorch release recent enough to have a published ggml-quantization kernel build, so it is not yet a drop-in feature for an existing stable install. Where the matching ggml/Metal kernel cannot be fetched, the model silently degrades to the sdpa attention path with only a warning, and where no compatible quantization kernel exists at all, the loader dequantizes the whole model, which increases memory use. Hugging Face itself still names llama.cpp as the recommended engine when efficient local inference, rather than Python-side flexibility, is the priority, and states only qualitatively that transformers performance is 'close' to llama.cpp's, without a quantified gap.

“This is where we are right now. And i’m not gonna lie it feels pretty magical 🧙‍♀️”

— Julien Chaumond (@julien_c)