Glonce

StochBench benchmark scores Opus 4.8 agent at 34.9% proof rate

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact65Novelty75Relevance95Surprise35Players50Credibility85
StochBench benchmark scores Opus 4.8 agent at 34.9% proof rate

A research team has introduced StochBench, a benchmark for formal theorem proving in the Lean 4 proof assistant built specifically around stochastic processes, a graduate-level branch of applied mathematics. The authors argue that existing benchmarks for testing large language models on formal proofs draw mainly from competition math, such as problems from the International Mathematical Olympiad and the Putnam competition, and that this small, contest-style pool does not represent how the field is actually applied. StochBench consists of 450 problems spanning finite and countable Markov chains, renewal processes, random walks, martingales, stopping times, queues, Brownian motion, stochastic calculus, weak convergence, and Poisson and continuous-time Markov processes, each paired with its original natural-language statement alongside the formal Lean 4 version. The authors note that stochastic processes are a field underrepresented in Mathlib, the community-maintained library of formalized mathematics that Lean-based provers rely on, meaning models have less existing formal material to draw on when attempting these proofs. To test the benchmark, the team ran an agent based on Opus 4.8 against all 450 problems under a 15-minute limit per problem, and the agent completed formal proofs for 157 of them, a 34.9% success rate. The authors describe StochBench as better representing domain-specific applied mathematics than prior benchmarks while still remaining difficult for advanced proving systems.

Key facts

  • StochBench is a Lean 4 benchmark of 450 graduate-level stochastic-processes problems, each paired with its natural-language source statement.
  • It covers finite and countable Markov chains, renewal processes, random walks, martingales, stopping times, queues, Brownian motion, stochastic calculus, weak convergence, and Poisson and continuous-time Markov processes.
  • The authors say it targets a field underrepresented in Mathlib, the formal-math library most Lean provers build on.
  • An Opus 4.8-based proving agent solved 157 of the 450 problems, a 34.9% proof rate, under a 15-minute per-problem limit.
  • The authors frame existing formal-proof benchmarks, drawn from competition math like the IMO and Putnam, as too narrow to represent field-specific applied mathematics.

Why it matters

Most formal theorem-proving benchmarks for large language models are built from competition mathematics: small, curated problem sets like the IMO and Putnam that reward contest-style tricks rather than the sustained, domain-specific reasoning that applied fields actually require. StochBench targets that gap directly by building a benchmark out of graduate stochastic processes, a subject the authors say is underrepresented in Mathlib, the formal library Lean provers depend on. That matters because a model's performance on olympiad puzzles says little about whether it can formalize the kind of probability and stochastic-calculus reasoning that shows up in real applied math and statistics work.

Who it affects

The benchmark is aimed at researchers building and evaluating formal theorem provers and proof assistants based on large language models, as well as contributors to Mathlib and the wider Lean ecosystem who work on extending formal-math coverage into applied probability. It also gives a concrete measuring stick to anyone comparing how well different LLM-based provers handle domain-specific mathematics rather than contest problems.

How to use it

StochBench is used by running a proving agent against its 450 Lean 4 problems, each with a matching natural-language statement, and checking how many the agent can formally prove within a fixed time budget. The authors tested it with an Opus 4.8-based agent under a 15-minute limit per problem, which produced formal proofs for 157 of the 450, a 34.9% success rate. No pricing, licensing terms or public release details for the benchmark are given in the source text.

How solid is it

The result comes from a single model family, an Opus 4.8-based agent, tested against the benchmark's full 450 problems, giving a 34.9% proof rate. The source text does not provide comparison proof rates for other benchmarks or other models, nor a human or expert baseline to weigh that 34.9% figure against; it offers only the authors' qualitative claim that prior competition-math benchmarks poorly represent field-specific applications. The scale of the benchmark, 450 problems across ten distinct areas of stochastic processes, is the main evidence of its breadth described in the text.

Risks and caveats

The source text does not name the authors' institution or affiliation, give a publication or release date for StochBench, or break down which specific problem categories the agent succeeded or failed on. With results reported for only one proving agent and no baseline from competing systems or human experts, the 34.9% figure should be read as a first data point on a new benchmark rather than a settled measure of how hard the field is for formal provers in general.

“Leading benchmarks for formal theorem proving with large language models are small collections drawn from competition math, such as the IMO and Putnam, that poorly represent field-specific applications.”

— StochBench paper