Glonce

StateBench tests long-context state tracking across four difficulty levels

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact45Novelty55Relevance95Surprise35Players45Credibility75
StateBench tests long-context state tracking across four difficulty levels

A Hugging Face blog post introduces StateBench, a benchmark for long-context state tracking: how well a language model keeps track of information about entities in a long document as that information changes. The post links a tech report, data and code. The motivating examples range from reading code, where variables are assigned and reassigned, to following a story in which characters move and objects change hands. In a mystery novel, "John gave Mary the knife" right before a murder changes the state of each character. A harder variant would have John give Mary the item in his backpack, after Edward is known to have put a knife in that backpack. Some problems only need the latest update; harder ones need a chain of changes, where one missed step throws off the answer.

The authors say transformers have well-documented theoretical limits on hard state tracking with long dependency chains, which helped motivate recurrent and hybrid architectures such as Gated DeltaNet that carry a running memory forward. Existing tests tend to focus on the hardest cases. The authors want to know which kinds of state tracking each architecture can handle, including the simpler and potentially more common kinds.

StateBench has four difficulty levels, which can be read as four kinds of variable assignment. R-trivial problems use immutable assignment: a value is set once and never changes, like a birth date, so the model only needs the first value. Aperiodic problems allow reassignment by value, like a thermostat, where only the most recent value matters. Soluble problems add self-referential updates such as x = x + 5, like a game score, so the model must carry changes forward. General finite problems add assignment by reference, such as x = y or swapping two variables. The example is a magician hiding a ball under one of five cups and repeatedly swapping them; this is the S₅ problem, and it sits at the hardest level. The levels are also grounded in Krohn-Rhodes theory, an area of algebra that decomposes state-tracking problems, viewed as word problems over finite monoids, into atomic primitives. To compare the levels fairly, the authors start from one general finite sequence of assignments and derive three easier versions. All four use the same number of variables and distinct values, with assignments at the same points; only the type of assignment changes.

First the authors tested open pretrained models on the r-trivial, aperiodic and general finite levels. The set included two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller models with 275 million and 2.7 billion parameters. All were pretrained on the same data, none had seen contexts longer than 64,000 tokens in training, and most were tested well beyond that. Under standard theoretical assumptions, transformers should handle r-trivial robustly, and soluble and aperiodic to some degree, but fail on general problems, while Gated DeltaNet with a more flexible memory update can in principle handle all four levels. In practice, almost all models did well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Format mattered: the 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits. That is one reason StateBench tests the same problem in several formats.

Next came four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. They were asked to read each sequence of assignments and produce the final value of every variable. All four did well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs grew, dropping to 13% or lower on the longest general finite problems. The authors name one likely factor, sensitivity: at the harder levels many more steps can affect the final answer, leaving less room to miss an update.

The third experiment asked whether the failures come from the architecture or from pretraining not teaching the skill. The authors trained several 275-million-parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. Training sequences went up to 256 assignments; tests went as long as 64,000 assignments, or 250 times the maximum training length. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble problems out to the longest sequences tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences and never learned general finite problems. On the hardest level, the Gated DeltaNet variant with the more flexible memory update generalized farthest, staying near perfect out to roughly 10 times the longest sequences it saw in training. For Gated DeltaNet and hybrid models, the authors read the gap between pretrained and directly trained results as pointing to training rather than architecture. Transformers were different: even trained directly, they never learned the hardest level.

Some questions remain open. A new proof in the paper shows that Gated DeltaNet with a more limited memory update (non-negative eigenvalues) can solve all aperiodic tasks and certain periodic tasks, such as tracking whether a repeated input has occurred an even or odd number of times, under the paper's fixed-precision arithmetic with rounding. Its exact placement in the hierarchy is open, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers changes things. StateBench also assumes a fixed number of variables, while real tasks, such as a long conversation, often introduce new entities over time; the current framework does not cover that.

Key facts

  • StateBench has four difficulty levels (r-trivial, aperiodic, soluble, general finite), built from one general finite sequence so that only the type of assignment changes between levels.
  • Open pretrained models, including the 7B Olmo 3 transformer and Olmo Hybrid, mostly handled r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems.
  • Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna did well on r-trivial and aperiodic problems, but accuracy fell sharply on soluble and general finite ones, to 13% or lower on the longest general finite problems.
  • Among 275M-parameter models trained from scratch on sequences up to 256 assignments and tested up to 64,000, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, while transformers never learned general finite problems.
  • StateBench assumes a fixed number of variables, so it does not yet cover real settings where new entities appear over time.

Why it matters

State tracking underlies reading code and following long stories or conversations, and the post argues that existing tests focus on the hardest cases. StateBench separates the problem into four levels grounded in Krohn-Rhodes theory, so researchers can see which kinds of tracking an architecture handles, not only whether it survives the worst case. The headline finding is that, trained directly on the task, recurrent and hybrid models generalize much farther than transformers, while the four named frontier models still break down on harder forms as inputs grow.

Who it affects

Researchers building or comparing architectures are the main audience: the authors hope StateBench gives them a finer-grained way to find where models break down, and to build training and evaluation data for capabilities that still fall short. They also call it a "North Star" for developing architectures with more robust state tracking. Anyone relying on frontier models to follow long chains of changes, such as running values through a long sequence of updates, should note the sharp accuracy drop on the soluble and general finite levels.

How to use it

The post links a tech report, a dataset and code, so the benchmark can be run on other models. Its design suggests testing the same underlying problem in several formats, since the 275M hybrid and Gated DeltaNet models scored near ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits. The post gives no licence or pricing details.

How solid is it

This is the authors' own write-up of their benchmark and experiments, with a tech report, data and code linked. The post gives mostly qualitative results: the one hard figure for the frontier models is 13% or lower on the longest general finite problems, and no per-model scores are given. The sensitivity explanation for the frontier failures is described only as one likely factor. The conclusion that the pretrained-versus-trained gap points to training rather than architecture is stated for Gated DeltaNet and hybrid models, not for transformers.

Risks and caveats

The authors flag open questions. The exact placement of Gated DeltaNet with a limited memory update in the hierarchy is unresolved, including whether it can solve all soluble tasks, and its proof assumes fixed-precision arithmetic with rounding. StateBench assumes a fixed number of variables, while real tasks often introduce new entities over time, and the current framework does not cover that setting. Results also depend on format, as the spreadsheet-edit versus RULER-style gap shows, and the from-scratch results come from 275M-parameter models trained on the task itself, not from general-purpose pretraining.

“we hope it serves as a North Star for researchers to develop new architectures with more robust state-tracking abilities compared to current models.”

— StateBench blog post