Glonce

CodeMidas builds RL environments for coding agents from raw source code

researchThe Glonce editorsHugging Face🔥69Glonce rating69Impact60Novelty70Relevance95Surprise45Players35Credibility85
CodeMidas builds RL environments for coding agents from raw source code

Training a capable coding agent with reinforcement learning needs a large supply of tasks paired with reliable checks on whether a solution is correct. Existing pipelines for generating such tasks typically mine development artifacts, issues and commits from open-source repositories, which caps how many usable tasks can be extracted. CodeMidas is an agentic pipeline that instead builds reinforcement-learning environments directly from a codebase's source code, with no issues, commits or other development history required as input.

CodeMidas spreads agentic compute across every stage of environment construction. Agents first explore a codebase's implemented functionality to formulate behavioral specifications, then construct tests grounded in actually running the original code, and finally validate and filter the candidate tasks through execution checks and repeated solution rollouts. Run at scale, the pipeline produced 5,545 training tasks drawn from 3,185 open-source codebases, spanning 23 programming languages and 15 technical domains.

The authors trained the MiMo-V2.5 model on this task set using GRPO and evaluated it on five separate coding benchmarks; performance improved on all five. Only three of the five are named in the paper: issue repair (DeepSWE, up 11.7%), whole-program construction (ProgramBench, up 17%) and terminal work (Terminal-Bench v2.1, up 8.5%). Ablation experiments showed that simply adding more high-quality training tasks kept improving performance, and a trajectory analysis found the RL-trained agent exploring codebases more thoroughly and using more diverse self-verification than before training. The authors present this as evidence that source code alone, without any accompanying development history, is a scalable foundation for building the RL environments that coding agents are trained on.

Key facts

  • CodeMidas builds reinforcement-learning environments for coding agents directly from a codebase's source code, without relying on issues or commits as earlier methods did.
  • The pipeline produced 5,545 training tasks from 3,185 open-source codebases, spanning 23 programming languages and 15 technical domains.
  • Training MiMo-V2.5 with GRPO on these tasks improved performance on all five tested benchmarks, including DeepSWE (+11.7%), ProgramBench (+17%) and Terminal-Bench v2.1 (+8.5%); the other two benchmarks are not named in the paper.
  • Ablation experiments found that adding more high-quality training tasks kept improving performance.
  • Trajectory analysis of the RL-trained agent showed increased codebase exploration and more diverse self-verification behavior compared with before training.

Why it matters

RL training for coding agents is bottlenecked by the supply of tasks with reliable verifiers, and prior pipelines pulled those tasks from issues and commits, a narrower slice of what a codebase actually contains. CodeMidas argues that a codebase's source code alone, explored and tested by agents rather than mined for its history, can serve as a much larger and more scalable source of such tasks, which is why it was able to produce over five thousand tasks from more than three thousand codebases.

Who it affects

The immediate audience is researchers and labs building RL training pipelines for coding agents, including whoever trains models like MiMo-V2.5. Indirectly, it affects anyone who benefits from stronger coding assistants and agentic coding tools, since the reported gains show up on issue repair, whole-program construction and terminal-use benchmarks, the kinds of tasks such tools are judged on.

How to use it

CodeMidas is described as a pipeline: agents explore a codebase's implemented functionality to write behavioral specifications, build tests grounded in running the original code, then filter the resulting candidate tasks through execution checks and repeated solution rollouts. The output is a training dataset, here 5,545 tasks across 23 languages and 15 domains, that other teams could in principle use the same way, training a coding agent on it with a method such as GRPO. The source text gives no information on whether the pipeline, code or dataset itself has been released.

How solid is it

The evidence is the authors' own: five benchmarks were run, and the paper reports gains on all of them, with specific relative numbers given for three (DeepSWE, ProgramBench, Terminal-Bench v2.1) but not the remaining two. An ablation isolating task count and a trajectory analysis of agent behavior both back the central claim, adding more high-quality tasks helps, and the trained agent behaves differently, not just scores differently. What is missing is any absolute benchmark score, any head-to-head comparison against other RL-environment-construction methods beyond the general statement that they rely on issues and commits, and any account of the compute cost or time the pipeline itself took to run.

Risks and caveats

The paper as described names no authors or institutional affiliations, gives no publication or submission date, and reports only relative improvements rather than absolute scores on three of the five benchmarks, leaving the other two unnamed entirely. Without absolute numbers or a comparison to competing methods on matched terms, it is hard to judge how large the described gains are in practice, or how the approach fares against alternatives on equal footing.