Hugging Face trains RL harness adapters for Qwen3.8-27B in three agent harnesses

Hugging Face trains RL harness adapters for Qwen3.8-27B in three agent harnesses

Hugging Face published a blog post introducing "harness adapters": small LoRA adapters trained with reinforcement learning inside a specific agent harness, so a general model fits the harness and the tasks you actually run. The post's central claim is that combining RL on a custom task distribution with your favorite agent harness yields more accurate and efficient models for that target harness and task distribution.

The headline example uses Qwen3.8-27B under the pi harness. The base model scores 39% on Terminal-Bench 2.1 and uses on average 2.9K tokens per task. Freeze those weights, add a custom RL-trained LoRA adapter, and the post reports 55% pass@1 with average token usage of 1.5K. The authors say most of the gain comes from the model reasoning less, calling fewer tools and performing better first time. The work extends earlier Hugging Face experiments (multi-harness RL and harness adapters) that covered data-analysis and code-building tasks; this time the team tested whether the result holds at 27B, on harder real-world software engineering tasks, across three harnesses: Pi, OpenCode and Hermes Agent. The final docker images ship Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6.

One LoRA adapter was trained per harness, with RL inside the harness itself. Datasets, training and agent rollouts all run on the Hugging Face Hub in Jobs, using TRL's AsyncGRPOTrainer, OpenEnv and harbor. The post says each harness adapter beats the base model under its own harness on both benchmarks, for both performance and efficiency, and that the three LoRAs are also merged into a single "super adapter" that runs unchanged under pi, Hermes Agent and OpenCode. Adapters, GGUFs, RL tasks and a safeagents plugin are collected in a Hugging Face collection.

A design point the authors stress: many RL frameworks train a re-implementation of the agent loop (they name verl's AgentLoop, SkyRL-Gym, rLLM, slime and ART). Here the harness runs exactly as it ships, and the trainer learns from the exact tokens it produced, recorded by a proxy between the harness and vLLM. They note that Agent Lightning v1.0 and SkyRL's Harbor integration move in a similar direction.

Much of the post is about building the task set. Tasks use the Harbor format, which Terminal-Bench 2.1 also uses. A broad Hub search returned about 59,000 Harbor tasks, including duplicates, evaluation sets and tasks marked not for training. The team kept two Hub sources built for training and generated a third. SWE-smith (Python, Go) injects bugs into real repositories, via an LLM rewrite or a mutation such as a flipped operator, and adds volume. SWE-Gym, packaged as Harbor tasks in TaskTrove, turns real GitHub issues and merged fixes into tasks. Tasksmith, a pipeline in Repo2RLEnv, generates tasks from merged pull requests: an LLM agent writes the instruction and tests, and the reference and a valid alternative must pass while the unfixed code and a known-wrong solution must fail. The team used it to target skills tested by Terminal-Bench 2.1, the hard subset of SWE-bench Pro and DeepSWE, such as HTTP streaming, filesystem operations and serialization. From 30 merged pull requests it produced 24 tasks in 10 Python repositories that passed its checks, 17 of them on a second run.

Every task then goes through a metadata screen (permissive licences, training splits, no duplicates, no tasks marked not for training, none of the 253 tasks in the target benchmarks or from repositories those benchmarks use) and through openenv validate in fresh trials, each a separate Hugging Face Job. The checks: the reference solution earns maximum reward, doing nothing scores clearly lower, no canary beats doing nothing, and a replay of the reference gives the same reward. 858 of 1,127 Hub candidates passed. Wrong-answer controls caught real defects, including a Tasksmith task that accepted an incomplete implementation and a Terminal-Bench 2.1 verifier (regex-log) that accepted known-wrong answers. Most failures were infrastructure problems such as images that did not start.

The difficulty check follows from how GRPO works: it scores each attempt against others at the same task, so a task where every attempt passes or every attempt fails gives no learning signal. Qwen3.8-27B attempted each task 4 times, or 8 times for a stratified sample of six tasks per repository. Of 442 scored tasks, it solved 151 every time and 120 never. The 171 it solved only sometimes form the qualified set. The model solves most Go tasks every time, fails every attempt on more than a third of the SWE-smith Python tasks, and lands in the middle on 73% of SWE-Gym tasks. In one example, 1,127 tasks were filtered and validated down to 278 included directly in the final dataset; the pipeline then took 522 rejected tasks, repaired and complexified them with Tasksmith and validated again. The published dataset, harness-hub/agentic-coding-envs, holds 871 tasks from 35 repositories, with Hub tasks capped at 40 per repository; it has a view of every accepted task and a qualified subset for GRPO, and rejected attempts stay in a separate config.

The authors also ran four short training runs of up to 50 steps, each on 32 tasks from a different pipeline stage (unvalidated, validated, unqualified, qualified), to test whether the filters change what the model learns. All used OpenCode with up to 48 model calls on one 8xH200 Hugging Face Job, and were compared against base on Terminal-Bench 2.1.

Training setup: LoRA adapters attached with the all-linear flag on Qwen/Qwen3.8-27B, trained with AsyncGRPOTrainer, whose main advantage is that it does not wait for all rollouts to finish before the next optimization step. Thinking mode was turned off during rollouts for efficiency, and the final adapter was evaluated with and without thinking. The reward is verifier correctness plus a small tool-efficiency bonus of 0.1, chosen because it worked best in the previous iteration. A 50-step learning-rate sweep on Termgrade environments found 1e-5 barely moved training correctness (0.52 to 0.50), 1e-4 collapsed rollouts (turns per rollout fell from 18 to 6, entropy rising), and 5e-5 improved from 0.49 to 0.60 by step 40 but looked unstable in longer runs. The final runs used 2e-5. Each final adapter trained on 2xB200 GPUs, one for the vLLM server and one for training.

Key facts

  • Hugging Face trained one RL LoRA adapter for Qwen3.8-27B per agent harness (Pi, OpenCode, Hermes Agent), with training running inside the unmodified harness; the three LoRAs are also merged into one super adapter.
  • Headline example: base Qwen3.8-27B under pi scores 39% on Terminal-Bench 2.1 at 2.9K tokens per task on average; with a custom RL-trained LoRA the post reports 55% pass@1 at 1.5K tokens.
  • The task set is built from SWE-smith, SWE-Gym and Tasksmith; 858 of 1,127 Hub candidates passed openenv validate, and 171 of 442 scored tasks were solved only sometimes by the base model and so kept as the qualified set.
  • The published dataset harness-hub/agentic-coding-envs holds 871 tasks from 35 repositories, and training runs on Hugging Face Jobs with TRL's AsyncGRPOTrainer, OpenEnv and harbor.
  • Final runs used a learning rate of 2e-5, a 0.1 tool-efficiency bonus on top of verifier correctness, and 2xB200 GPUs per adapter.

Why it matters

Models are tuned for general performance, so pairing one with a given harness and your own tasks leaves accuracy and cost on the table, the authors argue. The 39% to 55% and 2.9K to 1.5K token figures, if they hold for your setup, mean both a better score and roughly half the tokens per task. The method also differs from many RL frameworks: it trains the harness as shipped rather than a re-implementation of the agent loop, learning from the exact tokens recorded by a proxy between harness and vLLM. The post builds on earlier Hugging Face multi-harness and harness-adapter work and pushes it to 27B, harder software engineering tasks and three harnesses.

Who it affects

Teams that run an open model inside a particular coding-agent harness such as Pi, OpenCode or Hermes Agent and care about pass rate and token spend on their own task mix. Anyone building RL environments will find the task pipeline relevant: retrieval from the Hub, generation with Repo2RLEnv's Tasksmith, validation with OpenEnv and difficulty filtering. Users of Qwen3.8-27B are the direct target, since the adapters are built for that model.

How to use it

The post offers a reusable recipe meant to be reproduced on your own task, harness and model combination. Datasets, training and rollouts all run on the Hugging Face Hub in Jobs using TRL's AsyncGRPOTrainer, OpenEnv and harbor. Any Harbor task can be checked with openenv validate path/to/task --sandbox hf-sandbox, which runs trials in Hugging Face Jobs. The dataset harness-hub/agentic-coding-envs has an all-accepted view and a qualified subset for GRPO. Adapters, GGUFs, RL tasks and the safeagents plugin are in a Hugging Face collection. Practical settings from the post: LoRA with the all-linear flag, learning rate 2e-5, a 0.1 tool-efficiency bonus, thinking off during rollouts, and two GPUs per run (one for vLLM, one for training). The post gives no price or licence terms for any of this.

How solid is it

This is a first-party Hugging Face post, so the benchmark numbers are the authors' own and there is no independent replication in it. The method details are specific and checkable: task counts, validation pass rates, difficulty splits, sweep results and tool versions are all stated. The headline 55% pass@1 is given for the pi harness context; the passage that states it does not say whether it comes from the single-harness adapter or the merged super adapter, nor whether thinking was on or off. The results section with per-harness and super-adapter numbers is not part of the text this retelling is based on, so those figures are not reported here.

Risks and caveats

The claim is scoped to the specific harness and task distribution the adapter was trained on; the authors say the gains apply to that target, not to models in general. The gain is attributed to the model reasoning less, calling fewer tools and getting more right first time, and the tool-efficiency bonus in the reward may favour that behaviour by design. Validation found real flaws in existing tasks, including a Terminal-Bench 2.1 verifier (regex-log) that accepted known-wrong answers, which shows task quality is a risk in RL training. Learning rate was sensitive: 1e-4 collapsed rollouts and longer 5e-5 runs looked unstable. Do not mix the setups: the short filter-test runs used one 8xH200 Job with OpenCode, while the final adapters used 2xB200 GPUs each. The outcome of the four filter-test runs and the post's open question on what the model really learned are not covered here.

“Combining RL on a custom task distribution with your favorite agent harness yields more accurate and efficient models for that target harness and task distribution.”

— Hugging Face blog post, Introducing harness adapters