SKL teaches AI agents to predict from state, not trajectories
A new paper argues that the common way LLM agents learn from their own experience, by reflecting on entire trajectories after the fact, is flawed. The authors say this trajectory-level reflection operates on episodic hindsight rather than predictive foresight, which yields brittle, path-dependent heuristics that agents cannot easily transfer to new situations.
To fix this, the paper proposes Stateful Knowledge Learning (SKL), which shifts an agent's focus away from summarizing whole trajectories and toward maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to individual states. Rather than only compressing what happened along one path, the agent builds and keeps predictions tied to specific states it encounters, which the authors say provides finer granularity, improves generalization, and lets the agent bootstrap further knowledge from what it already holds.
To scale the idea beyond a single demonstration, the authors introduce two training algorithms: SKL-SD, based on self-distillation, and SKL-RL, based on reinforcement learning. Both train agents to autonomously extract state-grounded predictive knowledge from their experience and to use that knowledge when choosing actions, that is, in policy making. The approach is evaluated on two interactive environments, WebShop and ScienceWorld, plus a complex reasoning task, ChessPuzzles. Across these benchmarks, the authors report that agents equipped with the ability to learn stateful predictive knowledge significantly outpace current reflection-based training paradigms. The paper does not give specific accuracy numbers, success rates, or margins for this improvement, and it names no authors, institutions, or submission date.
Key facts
- The paper argues that trajectory-level reflection, the common way LLM agents learn from experience, operates on episodic hindsight rather than predictive foresight and produces brittle, path-dependent heuristics.
- Stateful Knowledge Learning (SKL) has agents maintain explicit, declarative predictive assessments anchored to state instead of summarizing whole trajectories.
- Two training algorithms are introduced: SKL-SD, based on self-distillation, and SKL-RL, based on reinforcement learning, both teaching agents to extract state-grounded knowledge and use it for policy making.
- The method is tested on the interactive environments WebShop and ScienceWorld and on the complex reasoning task ChessPuzzles.
- Agents equipped with SKL are reported to significantly outpace current reflection-based training paradigms, though the paper gives no specific accuracy or success-rate numbers.
Why it matters
Most LLM agents that improve from experience today rely on trajectory-level reflection, summarizing an entire episode after it ends. The authors argue this approach operates on episodic hindsight rather than predictive foresight, which produces brittle, path-dependent heuristics that do not transfer well to new situations. Stateful Knowledge Learning reframes the problem: instead of compressing what happened along one path, the agent maintains explicit, declarative predictive assessments anchored to specific states it encounters. The authors say this state-anchored form of knowledge is more granular, generalizes better, and lets an agent bootstrap further knowledge from what it already holds.
Who it affects
The paper targets people building LLM agents that learn from their own experience across multi-step tasks. It tests the method on interactive environments (WebShop, ScienceWorld) and on ChessPuzzles, a complex reasoning task, so the direct audience is anyone training agents for this kind of interactive or reasoning work through reflection-based self-improvement, the paradigm SKL is proposed to replace.
How to use it
SKL is introduced as two training algorithms. SKL-SD trains agents using self-distillation, and SKL-RL trains them using reinforcement learning. Both are designed to make an agent autonomously extract state-grounded predictive knowledge from its experience and then use that knowledge for policy making, meaning choosing its next action. The text gives no code or model release, no license terms, no pricing, and no author names, institutional affiliations, or submission date.
How solid is it
The central claim rests on experiments across WebShop, ScienceWorld and ChessPuzzles. The authors report that equipping models with the ability to learn stateful predictive knowledge significantly outpaces current reflection-based training paradigms. The source text gives no accuracy figures, success rates, or numeric margins for that improvement, so the size of the gain cannot be checked from what is stated. This is an arXiv preprint and has not gone through peer review.
Risks and caveats
The paper's main performance claim, that SKL significantly outpaces reflection-based training, is not quantified anywhere in the text, so readers cannot verify how large the advantage is. No author names or institutional affiliations are given, and no submission or publication date is stated. The text also does not explain mechanically how SKL-SD and SKL-RL differ beyond one using self-distillation and the other reinforcement learning.
“SKL shifts the agent's focus from trajectory-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state.”
— the paper