Glonce

MindForge fine-tunes Qwen3.6-27B to 49.51% on ProgramBench

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact65Novelty75Relevance90Surprise55Players38Credibility85
MindForge fine-tunes Qwen3.6-27B to 49.51% on ProgramBench

Researchers introduce MindForge, an automated pipeline that converts open-source command-line programs into training environments for software engineering agents. The paper's starting point is a specific gap: coding agents have gotten good at modifying existing codebases (bug fixing, feature implementation), but building a complete program from scratch remains largely unsolved, with even frontier models fully resolving fewer than 1% of ProgramBench's from-scratch tasks. The authors argue this is partly because no scalable training environments exist for the full software engineering life cycle; prior environment-construction frameworks only cover a single phase of development.

MindForge addresses this by taking an open-source command-line program and turning it into a 'source-free' environment: the resulting environment exposes only a compiled reference executable and its documentation, not the original source code. Using this pipeline, the authors built training environments from repositories that are disjoint from those used in ProgramBench, then curated a data recipe of program synthesis trajectories generated with GLM-5.2 acting as the teacher agent.

They fine-tuned Qwen3.6-27B on these trajectories. On ProgramBench, the model's average test pass rate rose from 37.98% to 49.51%, which the authors describe as comparable to substantially larger frontier models (the text does not name which models this comparison refers to). The fine-tuned model also improved over its own base version on all seven additional, unseen software engineering benchmarks the authors tested, covering long-horizon repository generation and translation, bug fixing, feature implementation, and cross-language issue resolution. The absolute point gains were 31.00 on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70 on NL2Repo-Bench with tests and 4.56 without tests, 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench.

Key facts

  • Even frontier models fully resolve fewer than 1% of ProgramBench's from-scratch program construction tasks.
  • MindForge converts open-source command-line programs into source-free training environments exposing only a compiled executable and its documentation.
  • Training environments were built from repositories disjoint from ProgramBench, with program synthesis trajectories generated by GLM-5.2 as the teacher agent.
  • Fine-tuning Qwen3.6-27B on these trajectories raised its ProgramBench average pass rate from 37.98% to 49.51%.
  • The fine-tuned model improved over its base version on all seven unseen benchmarks tested, with absolute gains from 4.56 points (NL2Repo-Bench, without tests) to 31.00 points (RepoZero-C2Rust).

Why it matters

Coding agents have made real progress editing existing codebases, but building a program from nothing is a different and much harder problem: frontier models solve under 1% of ProgramBench's from-scratch tasks. The paper's argument is that this gap persists partly because no training setup covers the whole software engineering life cycle end to end; existing environment-construction tools only touch one phase. MindForge is an attempt to close that training-data gap by automatically generating whole-life-cycle environments from ordinary open-source command-line tools, rather than hand-curating them.

Who it affects

The direct audience is researchers building or fine-tuning coding models, especially those working on small or mid-sized language models for software engineering, where matching frontier-model performance without frontier-model scale is the goal. It also speaks to teams building benchmarks and training environments for coding agents, since MindForge's source-free conversion method is itself a reusable technique, not just a one-off dataset.

How to use it

This is a research pipeline and training recipe, not a shipped product: the text gives no pricing, license, or release details for MindForge itself, its generated environments, or the fine-tuned Qwen3.6-27B checkpoint. The practical takeaway for practitioners is the recipe: convert command-line programs into source-free environments, generate program synthesis trajectories with a strong teacher model such as GLM-5.2, and fine-tune a smaller target model on the result.

How solid is it

The result rests on fine-tuning one model, Qwen3.6-27B, and evaluating it on ProgramBench plus seven additional, unseen benchmarks (RepoZero-C2Rust, DeepSWE, NL2Repo-Bench with and without tests, SWE-bench Verified, SWE-bench Pro, SWE-bench Multilingual, and FeatBench), where it improved consistently over its own base version. The training repositories were kept disjoint from ProgramBench's, which guards against direct data leakage on that specific benchmark. The gains are self-reported by the authors in a single paper; the text does not name the specific 'substantially larger frontier models' Qwen3.6-27B is said to match, nor does it give the number of training environments, dataset size, or compute cost involved.

Risks and caveats

The work is scoped narrowly to software engineering and, within that, to command-line programs specifically, so how far the source-free environment-generation method generalizes beyond this domain is untested here. The abstract text does not name institutions, publication date, or the frontier models used as the performance comparison point, which limits independent verification of some of its framing claims. All reported numbers are absolute point or percentage gains as stated in the abstract; no independent replication is described.