Glonce

AirLLM runs the 2.8T-parameter Kimi K3 on under 4GB of VRAM

toolsThe Glonce editorsHacker News🔥76Glonce rating76Impact72Novelty68Relevance100Surprise70Players35Credibility82
AirLLM runs the 2.8T-parameter Kimi K3 on under 4GB of VRAM

AirLLM is an open-source Python library that lets large language models run on GPUs far smaller than the model would normally need, without quantization, distillation, or pruning. Instead of loading an entire model into VRAM, it keeps only one layer on the GPU at a time, so the memory required tracks a model's layer size rather than its total parameter count. For models built as sparse mixtures of experts, it goes further and streams a single expert per token rather than a whole layer, which is why very large MoE models fit in unusually little VRAM.

The project states several configurations directly: a 70B model runs on a single 4GB GPU, Llama 3.1 405B runs on 8GB, DeepSeek-V3 (671B parameters) runs on about 12GB, and Qwen3-235B runs on about 3GB. The July 2026 release added support for Kimi K3, a 2.8-trillion-parameter model the project describes as the largest open-source model released to date, running it on under 4GB of VRAM. The README reports a measured figure for that configuration: 3.72GB of VRAM end to end on a single RTX 6000 Ada card.

Running Kimi K3 carries specific setup requirements: the compressed-tensors and flash-attn packages, since the model's code mandates flash attention regardless of what the user requests; a CUDA 12 build of torch, because no prebuilt flash-attn wheel exists yet for CUDA 13; and transformers version 4.56.x, since the model's remote code does not load on the 5.x branch.

Usage is a single AutoModel.from_pretrained() call that accepts a Hugging Face repository ID, working across major model families including Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. Optional 4-bit or 8-bit block-wise quantization can additionally speed up inference by up to 3x, which the project describes as having almost ignorable accuracy loss; the library also runs on Apple silicon Macs via mlx. The project credits Gavin Li as author and says much of the underlying approach builds on GitHub user SimJeg's work in a Kaggle competition. The source gives no tokens-per-second or latency figures for any configuration, only VRAM footprint, and does not say who built Kimi K3 or when it was first released.

Key facts

  • AirLLM runs a 70B model on a single 4GB GPU without quantization, distillation, or pruning by keeping only one layer on the GPU at a time.
  • The same approach scales further: Llama 3.1 405B fits in 8GB, DeepSeek-V3 (671B) in about 12GB, and Qwen3-235B in about 3GB.
  • The July 2026 release added support for Kimi K3, a 2.8-trillion-parameter model billed as the largest open-source model released to date, measured at 3.72GB of VRAM end to end on a single RTX 6000 Ada card.
  • Running Kimi K3 requires the compressed-tensors and flash-attn packages, a CUDA 12 build of torch, and transformers 4.56.x.
  • Optional 4-bit or 8-bit block-wise quantization can speed inference up to 3x with what the project calls almost ignorable accuracy loss.

Why it matters

AirLLM turns hardware that could not previously hold a large model into hardware that can run one, without the accuracy tradeoffs of quantization, distillation, or pruning. A 2.8-trillion-parameter model running on a single consumer or workstation GPU card, using well under 4GB of VRAM, is the kind of gap between model size and required hardware that normally does not exist; it is only possible because the library never loads more than one layer, or one MoE expert, into memory at a time.

Who it affects

Individual developers, researchers, and hobbyists who want to run very large open models locally but do not have access to multi-GPU servers or data-center-grade cards. It also affects anyone maintaining or contributing to the growing list of model families AirLLM supports, since each new architecture (most recently Kimi K3) needs its own compatibility work, such as the specific flash-attn, CUDA, and transformers version requirements the README lists.

How to use it

Install with pip install airllm, then call AutoModel.from_pretrained() with a Hugging Face repository ID or a local model path; the same one-line call works across Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. Optional settings include compression ('4bit' or '8bit') for up to 3x faster inference, a custom layer_shards_saving_path, an hf_token for gated models, and prefetching to overlap loading with compute. MacOS is supported on Apple silicon via mlx. Kimi K3 specifically needs pip install compressed-tensors flash-attn, a CUDA 12 build of torch, and transformers 4.56.x.

How solid is it

The project has a multi-year public history, from an initial 2023 release through incremental version updates adding compression, safetensors, CPU inference, and support for new model families as they are released, most recently Kimi K3 in July 2026. The core technique traces back to work by GitHub user SimJeg in a Kaggle competition. The source gives a concrete measured figure for the Kimi K3 case, 3.72GB on one RTX 6000 Ada, but no tokens-per-second or latency numbers for any configuration, so the loading-time cost of streaming layers from disk is not documented.

Risks and caveats

Because AirLLM streams model weights from disk layer by layer, the tradeoff for fitting huge models into little VRAM is not stated in the source: no inference speed or latency figures are given for any configuration, only memory footprint. The accuracy impact of block-wise quantization is described only in vague terms, as 'almost ignorable.' The source also does not identify who built or released Kimi K3, or when it originally came out, only that AirLLM added support for running it.

“sparse MoE models stream one expert at a time rather than a whole layer”

— AirLLM project README