TB-MathModel benchmark tests AI agents on 18 real math-modeling contest problems
A paper on Hugging Face introduces TB-MathModel, the Terminal-style Mathematical Modeling Bench. Its starting point is a complaint about how mathematical modeling is measured. Modeling turns an open-ended real-world problem into assumptions, a model, a solution and a defended recommendation, and the answers matter only if they are both close to a credible reference and physically feasible. The authors say existing benchmarks either score solver fragments or grade reports with an LLM judge, which leaves the numbers unchecked.
The benchmark covers 18 real problems from the CUMCM and MCM contests, drawn from 2023 to 2025. Each task is delivered in a containerized terminal with the verbatim problem statement, the official attachments and the figures. The agent works there and returns a single JSON artifact. That artifact is scored on the numeric core of the final question, measured against a reproduction of the Outstanding-award paper for that problem. The authors call it the first benchmark to make complete contest-style modeling problems agentic, verifiable and terminal at once.
Two design choices set it apart. The first is direction-aware, one-sided scoring, which never penalises an answer for beating the anchor. The second is two-layer feasibility verification: a programmatic gate over six decidable invariant forms, plus 624 Lean 4 obligations. A hard gate zeroes any infeasible answer.
Because one reference paper is a narrow target, each task is scored against a panel of human outstanding papers, so a model has to be close to many accepted answers. The paper evaluates seven multimodal entries, with three seeds each. The strongest reach a multi-reference O-Eval of 64.58%, while a cheap open model holds 59.81% at a small fraction of the cost. The abstract's price text for that model is garbled, so no figure is given here.
The authors position TB-MathModel as a reproducible, machine-checkable signal for measuring long-horizon modeling competence and for collecting agent trajectories. The source code is on GitHub at github.com/wxj630/terminal-bench-math-modeling.
Key facts
- TB-MathModel covers 18 real CUMCM and MCM modeling problems from 2023 to 2025; each runs in a containerized terminal and the agent returns one JSON artifact.
- Answers are scored on the numeric core of the final question against reproductions of Outstanding-award papers, using a panel of human papers rather than a single reference.
- Feasibility is checked in two layers: a programmatic gate over six decidable invariant forms and 624 Lean 4 obligations; infeasible answers are zeroed.
- Across seven multimodal entries (three seeds each), the strongest reach a multi-reference O-Eval of 64.58%, and a cheap open model holds 59.81%.
- Scoring is direction-aware and one-sided, so beating the anchor is never penalised; code is available on GitHub.
Why it matters
The paper's argument is that modeling benchmarks have measured the wrong things: either isolated solver fragments, or whole reports graded by an LLM judge whose numbers nobody checks. TB-MathModel tries to score complete contest problems end to end while keeping the result machine-checkable. The authors describe it as the first benchmark to be agentic, verifiable and terminal at once. That is their own claim. The one-sided scoring is a notable choice: an agent that does better than the human reference is not marked down for it.
Who it affects
Researchers building and evaluating agents for long-horizon quantitative work are the direct audience, since the authors pitch the benchmark for measuring long-horizon modeling competence. It also targets anyone who wants to collect agent trajectories on realistic modeling tasks, which the paper names as a second use. The tasks come from the CUMCM and MCM contests, so the setting is contest-style modeling.
How to use it
The source code is published at https://github.com/wxj630/terminal-bench-math-modeling. Per the abstract, each task ships its verbatim statement, official attachments and figures inside a containerized terminal, and the agent must return a single JSON artifact. Scoring then compares the numeric core of the final answer with the panel of Outstanding-award reproductions and applies the feasibility gate.
How solid is it
This is a benchmark paper and the account here rests on its abstract. The headline numbers are 64.58% for the strongest entries and 59.81% for the cheap open model, on seven multimodal entries with three seeds each. The abstract does not name the models, say which entry holds the top score, or give the exact definition of O-Eval beyond calling it multi-reference. The price of the open model is garbled in the available text, so the cost comparison cannot be stated precisely. The abstract also gives no result on how many answers the feasibility gate zeroed.
Risks and caveats
With only 18 problems, the benchmark is small, and each score depends on how well the Outstanding-award papers were reproduced as anchors. The scoring covers the numeric core of the final question, not the whole report, so the assumptions and the defended recommendation are not what is graded. The narrow gap between the top score (64.58%) and the cheap open model (59.81%) is hard to interpret without knowing which models are involved. The claim to be first is the authors' own.
“the first benchmark to make complete contest-style modeling problems agentic, verifiable, and terminal at once”
— TB-MathModel paper abstract