Glonce

gpt-oss-120b carries exact state across 196 chained tool calls to compute MD5

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact65Novelty72Relevance95Surprise62Players52Credibility85

A new evaluation isolates a question that agentic benchmarks have not answered cleanly: can an LLM carry exact intermediate state across a long chain of dependent tool calls at all? Long-horizon tasks are rare in LLM evaluation for a reason, the authors argue: when each step depends on the last, per-step accuracy that looks excellent in isolation decays catastrophically, as errors cascade and the end-to-end failure probability grows sharply with length. Existing agentic benchmarks report only end-to-end success, confound the state-tracking difficulty with instruction interpretation, give no control group that isolates it, and are vulnerable to shortcuts such as a hallucinated final answer, so they cannot say why a long run fails. To test state-tracking cleanly, the authors have a model compute a cryptographic hash, MD5, step by step: a sequence of 196 dependent tool calls over 64 rounds, with the model carrying four 32-bit words (a, b, c, d) in its own context from one call to the next. Interpretation is trivial, and because the authors implement MD5 from scratch following RFC 1321, they can align every call to a ground-truth trace and check the final digest to the bit, so any failure is pure bookkeeping rather than a misunderstood instruction. gpt-oss-120b, a mixture-of-experts model with only about 5.5B active parameters per token, tested at temperature 0 with a short fixed prompt, carries the full state across all 196 calls and returns the correct digest on a majority of completed runs (the text does not give an exact percentage). In the strongest setting tested, the authors replace every primitive tool with a second LLM, so a driver model and a worker model compute the whole hash from scratch between them with no exact-arithmetic oracle in the loop. Two ingredients decide success, and neither one involves changing the model's weights: keeping the model's own reasoning in its context on every turn, and voting over a thinking-enabled worker to remove its modular-arithmetic slips. The authors localize the residual failures by origin, separating state-carrying failures from arithmetic failures and from serving failures.

Key facts

  • Researchers had gpt-oss-120b compute an MD5 hash step by step across 196 dependent tool calls over 64 rounds, implementing MD5 from scratch per RFC 1321 so every call could be checked against a ground-truth trace and the final digest verified to the bit.
  • gpt-oss-120b, a mixture-of-experts model with only about 5.5B active parameters per token, ran at temperature 0 with a short fixed prompt and carried its four 32-bit state words (a, b, c, d) correctly across all 196 calls, returning the correct digest on a majority of completed runs.
  • In the strongest setting, every primitive tool was itself replaced with a second LLM: a driver and a worker compute the whole hash from scratch together with no exact-arithmetic oracle in the loop.
  • Two factors decided success without any change to the model's weights: keeping the model's own reasoning in context on each turn, and voting over a thinking-enabled worker to catch its modular-arithmetic mistakes.
  • The authors say existing agentic benchmarks confound state-tracking difficulty with instruction interpretation and can be gamed by a hallucinated final answer, so this design instead traces every call against ground truth and separates residual failures by origin: state-carrying, arithmetic, or serving.

Why it matters

Long-horizon agentic tasks chain many dependent steps, and per-step accuracy that looks fine in isolation can decay catastrophically as errors cascade, so end-to-end failure probability grows sharply with length. Whether an LLM can carry exact intermediate state across many tool calls at all was not well established, because existing agentic benchmarks only report whether a run succeeded overall and cannot separate that failure from a misunderstood instruction or a lucky guess. This test isolates the state-tracking question by making the model compute a hash it cannot bluff its way through, MD5, checked bit for bit against a ground-truth trace. Correct digests on a majority of completed 196-call runs is direct evidence that state-tracking failure is not an automatic consequence of a long tool-calling chain.

Who it affects

Anyone building or evaluating long-horizon agentic systems, pipelines that chain dozens or hundreds of dependent tool calls, such as multi-step coding agents, workflow automations, or research assistants that must keep running state correct over many turns. It also matters to people designing agentic benchmarks: the paper argues that current ones confound state tracking with instruction interpretation and can be gamed by a hallucinated final answer, so this design offers a cleaner template.

How to use it

The source names no product, price, or release; it is a research evaluation, not a shipped tool. What it does offer are two mechanism-level, not model-level, changes for anyone building tool-calling pipelines: keep the model's own reasoning in context from one call to the next rather than resetting it, and where a step involves arithmetic, run a thinking-enabled worker and vote over several of its outputs to catch modular-arithmetic slips. Neither requires retraining or fine-tuning the underlying model.

How solid is it

The test is designed so interpretation is trivial, computing a hash, and success is checked to the bit against a ground-truth trace built from an RFC 1321 implementation, which the authors say removes the ambiguity of benchmarks that only report end-to-end success. The paper reports correct digests on a majority, not all, of completed runs and does not state an exact success percentage or compare gpt-oss-120b's performance to other models on the same task.

Risks and caveats

The headline result is majority success on completed runs, not a guarantee, and the source gives no exact percentage or a full breakdown by failure type beyond noting that residual failures are localized by origin: state-carrying, arithmetic, or serving. The evaluation covers a single model, gpt-oss-120b, on a single synthetic task, computing MD5, so how the finding generalizes to other long-horizon tasks or other models is not established in the text. No author names, institutional affiliation, or publication date appear in the source.