Paper proves matching bounds on state needed for long-memory forecasting

The paper, "The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models", asks a resource question: for a given law of predictive memory, how much state, context or dynamical criticality does a sequence model need to forecast accurately? The authors study this directly in forecasting risk.

For algebraically decaying predictive memory, they prove matching upper and lower approximation bounds for exponential and finite-state modes. The best r-mode forecast error decays as e^{-Θ(√r)}. Turned around, reaching forecast error τ needs r = Θ(log²(1/τ)) states or modes, a square-log law. Both bounds are asymptotic orders. The authors position this against earlier curse-of-memory results, which established broad limitations of stable recurrent models under different approximation notions. Here the two sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target.

The paper then turns to genuine fractional long memory, which it says changes the geometry itself. Forecast error is measured after fractional integration. Prediction from a finite context of length L has an exact 1/L leading order. A fixed fractional strength d keeps the square-log state-complexity law. Near the short-memory boundary the authors identify the relevant d² and d⁴ scales, and in the intermediate regime they give a uniform constructive law.

For nonlinear contextual recurrences with uniformly contractive state dynamics, the authors derive an exponential first-chaos envelope and an explicit necessary condition linking forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality. That condition is necessary and not by itself sufficient. Finite-sample Kullback-Leibler calculations further connect the predictive geometry to statistical information.

Finally, the authors report theorem-matched experiments with contractive state-space, gated recurrent and attention models, which they say reproduce the state and stability predictions.

Key facts

For algebraically decaying predictive memory, the paper proves matching upper and lower approximation bounds for exponential and finite-state modes.
The best r-mode forecast error decays as e^{-Θ(√r)}, so reaching error τ needs r = Θ(log²(1/τ)) states or modes.
In the fractional long-memory setting, prediction from a finite context of length L has an exact 1/L leading order, and a fixed fractional strength d keeps the square-log state-complexity law.
For contractive nonlinear recurrences, vanishing forecast error on an algebraic target forces the recurrence toward criticality; this is necessary but not by itself sufficient.
Theorem-matched experiments with contractive state-space, gated recurrent and attention models are said to reproduce the state and stability predictions.

Why it matters

Long-range dependence is a basic difficulty for sequence models, and this paper tries to put a price on it. Earlier curse-of-memory results showed broad limitations of stable recurrent models under different approximation notions. This work closes the gap for one canonical predictive target in forecast risk: the upper and lower bounds match, which fixes the optimal resource exponent for that target. The headline result is a square-log law: reaching forecast error τ takes r = Θ(log²(1/τ)) states or modes.

Who it affects

Mainly researchers working on the theory of sequence models, including state-space, gated recurrent and attention architectures, and those studying long-memory or fractional processes in forecasting. The abstract does not tie the results to any specific product or company.

How to use it

This is a theory result, not a tool. It is most useful as a reference point: it tells a theorist what state or context scaling to expect for algebraically decaying memory, and it gives a necessary condition tying forecast accuracy to the contraction margin in contractive nonlinear recurrences. No practical recommendation for model design or comparison of architectures' performance is stated.

How solid is it

The core claims are theorems with matching upper and lower bounds, which is a strong form of result, though the abstract is all that was available here and the proofs were not seen. The Theta bounds are asymptotic orders; no explicit constants are given. The experiments are only said to reproduce the predictions; no concrete numeric experimental results (accuracies, model sizes, datasets) are given. The abstract names no authors or institutions.

Risks and caveats

The matching bounds hold for one canonical predictive target in forecast risk, not for every task or approximation notion, and the abstract says earlier results used different notions. The criticality condition is explicitly necessary and not by itself sufficient. The experiments cover contractive models and are theorem-matched, so they check the theory rather than test performance on real workloads. The abstract gives no submission date or version information.