Glonce

Essay argues open models have no posttraining data commons

researchThe Glonce editorsHacker News🔥52Glonce rating52Impact40Novelty40Relevance90Surprise35Players15Credibility60

A blog essay on openathena.ai argues that the AI field got "extraordinarily lucky" with pretraining data and has no equivalent for posttraining. For about 50 years, the author says, people built a public record of knowledge: Wikipedia, open source code (a couple of petabytes), Stack Exchange, ArXiv, Project Gutenberg, the Internet Archive. Common Crawl has archived snapshots of a sizable part of the web since 2007 and now holds something like 10 petabytes. GPT-3 was trained on about 60% Common Crawl-derived data, and the original Llama mix was almost entirely drawn from the internet commons. The author notes that incumbent large players have seemingly moved on to their own crawlers (Cloudflare reports that over half of crawler traffic is for AI), but open LLMs such as Nemotron and the author's own still rely on this commons for the vast majority of pretraining data. Without it, the essay says, truly open LLMs cannot exist.

Frontier research has since shifted to posttraining, especially agentic AI, which also needs vast compute and data. But there is no ready-made trove. The core difference, in the author's words: pretraining consumes documents, while posttraining consumes tasks. A human task is situated in a moment and cannot recur exactly; the author compares it to an oral story, which is an event as much as an object. An RL task is frozen, schematized and legible. It needs at least a world, an intent and a set of tools, plus a grader that scores how well the intent was met.

Software is the exception, the essay says, because engineers have spent decades writing work down: version control, continuous integration, unit tests, virtual machines and containers. Determinism suits reinforcement learning, which works best when an agent can try a task, discard the changes and restart from the same point. Even so the author notes that software is imperfect (Debian has worked on reproducible builds since 2013 and still does not call the distribution fully reproducible), and that the missing information may never have been recorded, so turning repositories into training data is closer to manufacturing than curation.

The essay walks through SWE-bench as the evidence. Its framing maps the world to the repository and its build machinery, the tools to the shell and compiler, the intent to GitHub issues or pull requests, and the grader to whether tests pass. The original team crawled 12 Python repositories and got a bit over 90,000 PRs. Only about 1 in 8 mapped readily to that frame, because the rest were unmerged, not linked to an issue, or did not change a test. Of the remaining 11,407, about 80% could not be turned into executable tasks, leaving the 2,294 tasks of the original SWE-bench. A year later OpenAI hired 93 experienced Python developers to review 1,699 randomly selected tasks. More than two thirds were flagged for removal for underspecified prompts, unfair tests or other major issues, leaving a pool of 500 "Verified" tasks. A later audit of a problematic subset found that another sixth or so of the 500 were also broken.

The author says this is not a dig at SWE-bench, which is one of the better constructed public coding datasets. Looking at TaskTrove, the author's group found that in one subset of 5,000 converted Java tasks, more than 90% had to be discarded during repair, since tests did not parse, checked nothing, or depended on missing project context and external services. In another Python subset, almost 98% of inspected verifier bundles could not start their tests because dependencies were missing. Executability is only the start: an audit by Vals found that two thirds of the 2,698 coding environments in Xiaomi's MiMo v2.6 public release still contained reference fixes recoverable from Git. In one demonstrated case the agent copied the fix and passed all five hidden tests.

If software is this hard, the author asks, how can frontier labs encode the rest of "economically valuable human activity", the work of lawyers, scientists, salespeople and doctors, as repeatable tasks? There is not even the beginning of a public task commons for it. Labs instead pay such professionals to fit their work into the RL task frame, creating tasks, environments, rubrics, verifiers and simulators, which is the basis of the rapidly expanding posttraining data industry.

For the open models community, the author's answer is that a posttraining commons must begin with large-scale, systematic inscription: finding human tasks and transmuting them into RL tasks (or finding a better paradigm, which would be preferable). It is a hard lift because much of the pretraining commons was already produced in roughly the right form, while the goals, decisions, revisions and false starts behind it are mostly gone. The author sees reasons for hope: the open source commons is a treasure trove of partially inscribed tasks, other fields leave traces such as documents with tracked changes, logs and procedures, and agents are increasingly able to help reconstruct tasks from outputs and traces. The view that "we just need to make a whole lot of RL environments" is true on the surface, the author says, but the deeper job is deciding which human activity to inscribe and building the machinery and institutions to do it. The approach used to build GDPval points at one way to do that. Automation of legible tasks seems inevitable, the essay says, and the open question is whether the resulting inscriptions become private assets held by a few companies or a commons everyone can build on. The visible text ends mid-sentence, as the author begins to discuss the ability to develop and direct AI capability.

Key facts

  • The essay's thesis: pretraining consumes documents and had Common Crawl (since 2007, about 10 petabytes, about 60% of GPT-3's data), while posttraining consumes RL tasks and has no equivalent public commons.
  • An RL task needs a world, an intent and tools, plus a grader; the author says turning software repositories into such tasks is closer to manufacturing than curation.
  • Original SWE-bench: 12 Python repositories, a bit over 90,000 PRs, only about 1 in 8 mappable, and 2,294 tasks left after about 80% of the 11,407 survivors failed to become executable.
  • SWE-bench Verified: 93 developers reviewed 1,699 tasks, more than two thirds were flagged, 500 remained, and a later audit found another sixth or so of the 500 broken.
  • The author calls for the open models community to build a posttraining commons through systematic inscription of human tasks into RL tasks, with agents helping reconstruct tasks from traces.

Why it matters

Open model builders have leaned on free web data for pretraining, and the essay says that route does not carry over to the agentic posttraining where frontier research now focuses. If the author is right, the data needed for the next stage is being produced by paid professionals inside labs and a growing posttraining data industry, and could end up as private assets. The essay frames the stake as whether a meaningful share of those task inscriptions becomes a commons.

Who it affects

Mainly the open models community, including projects like Nemotron and the author's own, which the essay says still depend on Common Crawl for the vast majority of pretraining data. It also touches the makers and users of public coding benchmarks and datasets such as SWE-bench, SWE-bench Verified, TaskTrove and Xiaomi's MiMo v2.6 environments, and the frontier labs and data vendors that pay lawyers, scientists, salespeople and doctors to write tasks.

How to use it

The essay is an argument, not a tool, so there is nothing to install. For people building open posttraining data, its practical message is to look for human work that leaves traces (open source repositories, documents with tracked changes, logs, procedures), use agents to reconstruct tasks from those outputs, and check environments hard: leaked reference fixes, missing dependencies and weak tests are the failures it documents. It names the approach taken in constructing GDPval as one way of achieving systematic inscription.

How solid is it

It is an opinion essay on the openathena.ai blog, with the main thesis stated as the author's view. The concrete numbers on SWE-bench, SWE-bench Verified, the Vals audit of Xiaomi's MiMo v2.6 environments and Cloudflare crawler traffic are reported inside the essay rather than shown with their underlying data, and the TaskTrove figures come from the author's own examination. The text available ends mid-sentence, so the closing argument is not visible.

Risks and caveats

The author belongs to an open-models effort and has a stake in the argument. The essay does not describe how that organisation plans to build a commons, and gives no project, funding or timescale. It also concedes that its own proposal is hard and that a better paradigm than RL tasks would be preferable. The SWE-bench critique is not a verdict on the benchmark: the author calls it one of the better constructed public coding datasets, and the audit findings concern specific subsets.

“Pretraining consumes documents, while posttraining consumes tasks.”

— From the openathena.ai essay