Glonce

h1, a simple Lean 4 agent loop, tops PutnamBench and FATE-X

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact50Novelty60Relevance92Surprise45Players15Credibility72
h1, a simple Lean 4 agent loop, tops PutnamBench and FATE-X

A new paper asks a bitter-lesson question about formal theorem proving: can a simple agentic loop that scales test-time compute match the elaborate workflows built around models? The authors note that progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, while formal theorem proving has increasingly leaned on human-designed workflows such as planning, decomposition, and orchestrating specialized provers. They argue the question matters because formal verification has to scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it: their gains are often confounded with stronger models or more compute.

To test it, the authors build h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection. It imposes no decomposition or search structure on the model.

The reported results: h1 reaches state-of-the-art on PutnamBench, FATE-X, and FormalConjectures100-Solved, and it settles eight open problems in FormalConjectures. At matched models, the authors say it matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus. They describe one finding as surprising: a single attempt (pass@1) can often scale to solve open problems.

Through ablations and matched-model comparisons with Claude Code and AxProverBase, the authors conclude that simplicity alone is not enough. An efficient Lean interface largely determines cost and solve rate. Their recommendation is that future work evaluate frontier models with efficient execution feedback before prescribing workflows, "at the risk of bitter lessons."

Key facts

  • h1 is a simple harness that gives a model an efficient interface to Lean 4 and keeps it going through verifier feedback and self-reflection, with no imposed decomposition or search structure.
  • The authors report state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, and say h1 settles eight open problems in FormalConjectures.
  • At matched models, h1 matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, according to the authors.
  • Ablations and matched-model comparisons with Claude Code and AxProverBase lead the authors to conclude that an efficient Lean interface largely determines cost and solve rate.
  • The authors recommend evaluating frontier models with efficient execution feedback before prescribing workflows.

Why it matters

Formal theorem proving has increasingly relied on human-designed scaffolding around models: planning, decomposition, and orchestrating specialized provers. The paper tests whether the bitter lesson applies here, that is, whether a plain loop with more test-time compute can match that scaffolding. The authors say yes, at matched models, and that a single attempt (pass@1) can often scale to solve open problems. They also point to a methodological problem: gains reported for new workflows are often confounded with stronger models or more compute.

Who it affects

Researchers building automated provers and Lean 4 tooling are the direct audience, especially those designing workflows with planning, decomposition, or multiple specialized provers. It also bears on anyone comparing proving systems, since the authors recommend evaluating frontier models with efficient execution feedback before prescribing workflows.

How to use it

The abstract does not describe a release, so there is nothing to install from it. The practical lesson the authors draw is about design: give the model an efficient interface to Lean 4, feed it verifier feedback, let it reflect, and check how far that goes before adding decomposition or search structure. When comparing systems, hold the model fixed, as the authors did in their matched-model comparisons.

How solid is it

This is a preprint, and the claims are the authors' own. The abstract names three benchmarks with state-of-the-art claims and a count of eight settled open problems, and it backs its main conclusion with ablations and matched-model comparisons. The source does not give numeric scores, pass rates, costs or benchmark margins. It does not name the base models used with h1, and it does not say which eight FormalConjectures problems were settled or whether they are independently verified or peer reviewed.

Risks and caveats

The source gives no figures for how far h1 beats the previous state of the art, so the size of the advantage is unknown. The results of the Claude Code and AxProverBase comparisons are not given beyond the general conclusion about the Lean interface. The authors also stress that simplicity alone is not enough: the result depends on an efficient Lean interface, so a naive loop should not be expected to match it. No authors or institutions are named in the source.

“simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate”

— From the h1 paper's abstract