RepoFactory builds 10,159 coding-agent training environments from real repos

RepoFactory builds 10,159 coding-agent training environments from real repos

A paper on Hugging Face Papers tackles a bottleneck in training coding agents for long-horizon tasks: such agents need environments that give verifiable feedback. The authors argue that existing environment-construction pipelines often follow a linear process that ends with validation. That establishes an environment is usable but gives little guidance on how to improve its training value. They also point out that recursive self-improvement often requires model rollouts, or model training followed by downstream evaluation, and that the costs and feedback delays are amplified on long-horizon tasks. Their question is how to use timely, reliable proxy signals to improve training value and synthesize environments at scale.

Their answer is RepoFactory, an automated pipeline that converts real repositories into verifiable environments. In each one, an agent builds a complete project from scratch given a language specification. A component called the Environment Evolution Loop then optimizes the environments through proxy feedback. It optimizes a multiplicative objective that combines three things: validity, structural coupling, and implementation scope.

RepoFactory produces 10,159 environments across 17 software domains. Training Qwen3.5-35B-A3B on these environments yields a model the authors call RepoHorizen-35B-A3B. It achieves gains of 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks. The five cover long-horizon coding and agentic tasks in real-world production settings. Without environment evolution, the corresponding gains are 17.5 and 6.4 points. So the Environment Evolution Loop accounts for the difference: 5.1 points on NL2Repo-Bench and 2.1 points on the five-benchmark average. The authors say the results highlight the training value of RepoFactory environments and the additional gains from the loop.

Key facts

  • RepoFactory is an automated pipeline that converts real repositories into verifiable environments where agents build complete projects from scratch given a language specification.
  • Its Environment Evolution Loop optimizes a multiplicative objective combining validity, structural coupling, and implementation scope, using proxy feedback instead of model rollouts or training runs.
  • The pipeline produces 10,159 environments across 17 software domains.
  • Training Qwen3.5-35B-A3B on them yields RepoHorizen-35B-A3B, with gains of 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks.
  • Without environment evolution the gains are 17.5 and 6.4 points, so the loop adds 5.1 and 2.1 points respectively.

Why it matters

Long-horizon coding agents need environments that can check their work, and making those environments at scale is hard. The authors say current pipelines stop at validation, which shows an environment works but not how useful it is for training. RepoFactory's idea is to improve environments using cheap proxy signals, so there is no need to run model rollouts or train a model and evaluate it downstream, which the authors say is slow and costly on long-horizon tasks. The reported result is that the evolution step adds measurable gains on top of the environments alone.

Who it affects

Mainly researchers and teams that train coding agents and need large sets of verifiable tasks. The environments cover 17 software domains, and the tasks ask an agent to build a whole project from a language specification, not to fix a single bug or finish a snippet. The model in the paper is built on Qwen3.5-35B-A3B.

How to use it

The abstract describes the method, not a product. It does not say whether the code, environments or model weights are released. It does not name the 17 software domains or the programming languages involved. For a team building its own training setup, the transferable idea is the design: turn real repositories into build-from-spec tasks, and tune them with a multiplicative objective over validity, structural coupling and implementation scope.

How solid is it

This is a research preprint, and everything above comes from its abstract. The headline numbers are gains in points, and the source says "on average" without specifying mean or median. The abstract gives no absolute benchmark scores, baseline or final. It does not say which benchmarks make up the five besides NL2Repo-Bench. It also does not compare against other models or other environment-synthesis pipelines, only against training without environment evolution. The abstract names no authors or institutions.

Risks and caveats

Because no absolute scores are given, the size of the gains is hard to place against other systems. The only comparison is the ablation: 22.6 versus 17.5 points on NL2Repo-Bench and 8.5 versus 6.4 on the five-benchmark average. The abstract does not give training cost, compute or the number of rollouts. The claims about the value of the environments and of the evolution loop are the authors' own conclusions.

“How can we use timely, reliable proxy signals to improve training value and synthesize environments at scale?”

— RepoFactory paper abstract