Hugging Face's TRL v1.14 adds adapter-only LoRA sync, letting Hugging Face Jobs swap adapters through a Storage Bucket instead of NCCL
Hugging Face's TRL, its library for reinforcement-learning fine-tuning, has added LoRA support to its AsyncGRPOTrainer. The change, delivered through PR #7017 and shipping in TRL v1.14, lets the trainer train a LoRA adapter instead of the full model and sync only that adapter to vLLM, the inference engine that generates rollouts. A Hugging Face blog post walks through a real project built on the new path, where the trainer and two vLLM replicas run as separate Hugging Face Jobs on separate machines, with no NCCL connection between them at all.
The motivation for training a LoRA adapter instead of the full model, in reinforcement learning specifically, comes from Thinking Machines's blog post "LoRA Without Regret," which the Hugging Face authors cite as showing that LoRA can match full fine-tuning for policy-gradient RL even at rank 1. Their explanation: an episode's advantage function carries only about O(1) bits of information, so there is not much for the model to learn from any single step, and a rank-1 adapter already has enough capacity to hold it. That has a systems payoff too. A rank-1 adapter for the 1.5B-parameter model used in this project is a few megabytes, against around 3 GB for the full model, so after every update the trainer can ship just the adapter rather than the whole policy. Because vLLM can keep several adapters loaded at once, rollouts already in flight keep running under the policy version they started with while new rollouts pick up the newest one.
AsyncGRPOTrainer already lets training and generation run as separate processes on different machines, which is straightforward on a single node or a cluster where both share a filesystem or can form an NCCL group. Hugging Face Jobs offer neither: one Job is a single container on a single VM, capped at 8xH200 GPUs per node, that cannot itself span multiple nodes to hold a trainer plus a fleet of vLLM servers, and separate Jobs have no shared disk and cannot form an NCCL group with each other. A full-weight sync would need to move gigabytes between machines with nothing available to move it over. A LoRA sync only needs to move a few megabytes, small enough to travel through a Storage Bucket that Hugging Face Jobs can mount as a volume at the same path in every Job, using hf-mount to expose the bucket as a POSIX filesystem inside each container. The authors say nothing in TRL or vLLM itself had to change to support this: the trainer just writes adapter files to the mounted path and the vLLM servers read them from the same path. The same bucket also holds checkpoints and the final adapter, so a Job that gets preempted, and Hugging Face Jobs are ephemeral, can resume training instead of losing progress.
The finished setup combines: one trainer Job running AsyncGRPOTrainer with LoRA (and FSDP, though the post does not explain that part beyond naming it); two vLLM Jobs, each serving the base model plus whichever adapter the trainer most recently published; a Storage Bucket mounted at the same path in all three; and a small proxy server in front of the two vLLM replicas. Every few optimizer steps, the trainer saves the adapter under a versioned path, publishes it with an atomic rename, and calls vLLM's /v1/load_lora_adapter endpoint with that path; vLLM loads the adapter from disk and can then serve it under a matching model name. Each vLLM replica runs the stock vllm/vllm-openai image on one GPU with runtime LoRA loading turned on, and the whole recipe is pinned to vLLM v0.27.1, because the exact flags and endpoints it relies on are tied to that version. The replicas reserve six adapter slots. With max_staleness set to 4, a rollout generated under an older policy version must be allowed to finish under that same version even after the trainer has moved several versions ahead, so vLLM needs to keep the current policy plus the four before it loaded, plus one more slot to hold an incoming version while the oldest is swapped out; five slots instead of six would let vLLM evict a policy that still has rollouts in flight. During startup, TRL checks the vLLM server's /server_info endpoint and switches to this adapter-only sync automatically when it finds a LoRA config, logging "Adapter-only vLLM sync enabled"; a configuration vLLM cannot serve directly, such as DoRA, modules_to_save, or a LoRA rank above the server's maximum, falls back to the older merged-weight sync with a warning instead.
The team explicitly rejected a simpler design where the trainer always publishes the adapter under one fixed name, because vLLM keys its prefix cache by adapter name. Reusing a single name would let cached prefix blocks computed under old weights match again after an update, so a rollout could have its prompt processed under one policy version and its output decoded under the next, without the trainer being able to tell; the only symptom would be an importance-sampling ratio drifting away from 1. Giving every version its own name rules that out, since a cached prefix can never match a version other than the one it was computed under.
A proxy sits in front of the two vLLM replicas for two reasons. First, every exposed Job endpoint requires an Authorization: Bearer
To validate the whole pipeline, the team trained on sail/Sanity-Test-R1D-1.5B, a dataset built for the paper "Defeating the Training-Inference Mismatch via FP16" (Qi et al., 2025; reproduction code in sail-sg/Precision-RL). Its authors generated 40 answers to each MATH problem with DeepSeek-R1-Distill-Qwen-1.5B and kept the 1,460 questions where that model's success rate fell between 20% and 80%, so the set is neither already solved nor hopeless, giving a model an early signal to learn from. It is also small enough to cycle through in under two hours, and sensitive enough that if one vLLM replica ever silently served the base model under an adapter's name, the training curve would show it within a few dozen steps. Training used Qwen/Qwen2.5-Math-1.5B with a rank-1, alpha-2 LoRA adapter on all linear layers, a learning rate of 4e-5, 8 samples per prompt, 128 completions per step, up to 3,000 generated tokens and a 4,096-token context; the trainer published a new adapter every 4 optimizer steps and checkpointed every 50 steps. Running that recipe for 500 steps, the AsyncGRPO metrics let the team see where the bottlenecks sat: wall-clock time across five runs of the same recipe fell from 3 hours 27 minutes down to 53 minutes.
Key facts
- TRL v1.14 (from PR #7017) lets AsyncGRPOTrainer train a LoRA adapter instead of the full model and sync only that adapter to vLLM.
- A rank-1 adapter for the 1.5B-parameter model used here is a few megabytes against around 3 GB for the full model, small enough to move through a Storage Bucket mounted in every Hugging Face Job instead of over NCCL.
- The setup runs one trainer Job and two single-GPU vLLM Jobs plus a proxy that adds the auth header, routes each prompt's eight rollouts to the replica already holding its cached prefix, and broadcasts adapter loads to both replicas; each replica reserves six adapter slots for a max_staleness of 4.
- Adapters publish under versioned names like trl-policy-v{N} rather than one fixed name, because vLLM's prefix cache is keyed by adapter name and a shared name could let a rollout's prefill and decode silently come from two different policy versions.
- Tested on the 1,460-question Sanity dataset (Qi et al., 2025) with Qwen2.5-Math-1.5B, rank-1 alpha-2 LoRA and a 4e-5 learning rate, the built-in AsyncGRPO metrics helped cut wall-clock time for 500 steps of the same recipe from 3 hours 27 minutes to 53 minutes across five runs.
Why it matters
Reinforcement-learning post-training is normally built around dense GPU clusters that share a filesystem or an NCCL interconnect, and serverless-style compute, one container per Job that cannot form an NCCL group or share a disk with another, is a poor fit for that. Hugging Face's answer here is less a new algorithm than an observation about scale: a rank-1 LoRA adapter is small enough, a few megabytes against around 3 GB for the full model, that ordinary cloud object storage can stand in for the specialized networking RL training normally needs. That turns a class of infrastructure TRL could not previously use for this, Hugging Face Jobs that cannot form an NCCL group with each other, into a workable place to run asynchronous RL training, at the cost of building a proxy and a careful adapter-versioning scheme to replace what a shared node or an NCCL group would otherwise guarantee for free.
Who it affects
Teams doing reinforcement-learning post-training with Hugging Face's TRL, in particular AsyncGRPOTrainer, especially those without a dense, NCCL-capable GPU cluster who provision compute one Hugging Face Job, or a comparable serverless GPU job, at a time. The benefit is specific to plain LoRA configurations vLLM can serve as-is: anyone who needs to update the full model weights, or whose adapter setup vLLM cannot serve directly, such as DoRA, modules_to_save, or a LoRA rank above the server's configured maximum, still falls back to the older, heavier merged-weight sync that this feature was built to avoid.
How to use it
The feature ships in TRL v1.14, originally landing as PR #7017. It requires configuring an AsyncGRPOTrainer with a LoraConfig, rank 1 and alpha 2 on all linear layers in the post's own example. TRL checks the vLLM server's /server_info endpoint at startup and switches to adapter-only sync automatically when it finds a LoRA config, logging "Adapter-only vLLM sync enabled"; unsupported configurations fall back to merged-weight sync with a warning instead. On Hugging Face Jobs specifically, the post's own recipe is to mount one Storage Bucket at the same path in the trainer Job and every vLLM Job, pin vLLM to the exact version the setup was tested against (v0.27.1 here), size --max-loras to two more than max_staleness, and run a small proxy on the trainer Job that adds the required Authorization header, spreads rollouts across replicas, and rebroadcasts every adapter load and pause or resume call to all of them. The post gives a Jobs cost, about $20 per hour for the trainer's h200x2 Job plus the two single-h200 vLLM Jobs, but no price for Storage Buckets.
How solid is it
Wall-clock time is one measured outcome: the same 500-step recipe fell from 3 hours 27 minutes to 53 minutes across five runs, using AsyncGRPO's own metrics to find the bottlenecks in between. Those metrics also cover pipeline health across all 126 syncs: the importance-sampling ratio stays at 1.000 on every step, mean staleness holds at 1.5 policy versions against a maximum of 4, all 252 adapter loads succeed, and the proxy routes 84.5% of requests as affinity hits. Training reward is reported too, rising from a mean of 0.145 over the first 20 steps to 0.438 over the last 20; no held-out accuracy or benchmark score is given for the resulting model. The validation dataset, 1,460 MATH problems where a 1.5B model succeeds 20% to 80% of the time, was chosen specifically because a broken setup, such as a replica silently serving the base model under an adapter's name, would show up in the training curve within a few dozen steps, so the test's own sensitivity is part of the evidence offered. Even so, this is one team's account of one project, on one small model size (1.5 billion parameters), credited to four named authors (Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul and Sergio Paniego), and it does not say whether the approach has been tried at larger scale.
Risks and caveats
The design leans on details that could silently break: it pins an exact vLLM version, v0.27.1, because the flags and runtime-LoRA endpoints it relies on are tied to that release, and it needs enough adapter slots, six here for a max_staleness of 4, or vLLM can evict a policy version that still has rollouts depending on it in flight. Versioning every adapter by name is a deliberate defense against one specific failure mode, a rollout whose prefill and decode silently come from two different policy versions, which the trainer could otherwise only detect indirectly, as an importance-sampling ratio drifting away from 1. The post also leaves one of its own promises unfulfilled: it mentions FSDP by name without ever explaining it. No results beyond a single 1.5B-parameter model are given.