Glonce

PAST-Bench tests whether AI agents actually learn from past sessions

researchThe Glonce editorsHugging Face🔥55Glonce rating55Impact60Novelty72Relevance100Surprise45Players30Credibility87
PAST-Bench tests whether AI agents actually learn from past sessions

Researchers introduced PAST-Bench, a benchmark built to test a specific question: personal AI agents that retain preferences, task histories, tool routines, and learned skills across sessions are supposed to improve as they accumulate experience, but whether that retained experience actually makes them better has not been systematically tested until now. PAST-Bench runs each agent through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off, so the same agent can be compared with and without access to its own history. The benchmark spans 26 scenarios and 204 episodes across four capability areas: memory, procedural reuse, information gathering, and update. For each agent the researchers measure two separate things: whether performance on later tasks actually improves, and whether that improvement follows the intended pathway of saving, retrieving, and updating stored experience, rather than arising some other way. Testing across seven base models and four agent frameworks, the authors found that improvement from retained experience is real but uneven across capabilities, and that two agents can show the same headline performance gain while differing sharply in whether that gain is actually backed by evidence of the intended save-retrieve-update pathway. Guided by these findings, the team built Hermes+, an extension of an existing agent framework called Hermes, adding five targeted interventions across different stages of the agent loop. Hermes+ raises the average gain from retained experience compared with the baseline and produces clearer evidence that the gain follows the intended pathway. Its strongest improvement shows up on tasks that require replacing outdated stored state with current information, though the authors note the effect still varies by capability and by the underlying model. The code for PAST-Bench and Hermes+ has been released on GitHub.

Key facts

  • PAST-Bench spans 26 scenarios and 204 episodes across four capability areas: memory, procedural reuse, information gathering, and update.
  • The benchmark isolates the effect of retained cross-session experience by running matched task sequences with that experience turned on and off.
  • Across seven base models and four agent frameworks, improvement from retained experience is real but uneven across capabilities.
  • Agents can show the same headline performance gain while differing markedly in whether it is backed by evidence of the intended save-retrieve-update pathway.
  • Hermes+, an extension of Hermes with five targeted interventions across the agent loop, raises the average gain from retained experience and shows its strongest effect on tasks that require replacing outdated stored state.

Why it matters

Personal AI agents are marketed on the promise that they get better the longer you use them, learning your preferences and routines across sessions. PAST-Bench is one of the first attempts to systematically check whether that promise holds, rather than assuming that agents with memory automatically improve. It separates the question of whether an agent gets better from the question of whether it gets better for the right reason, which is a distinction most benchmarks skip.

Who it affects

The work is aimed at researchers and developers building or evaluating personal AI agents that carry memory or state across sessions, including anyone designing agent frameworks meant to accumulate experience over time. It is less directly relevant to end users, though it bears on any product claiming an agent 'learns from you.'

How to use it

PAST-Bench and the code for Hermes+ are released on GitHub, so developers can run their own agents through the 26 scenarios and 204 episodes to check both performance gains and whether those gains follow the intended memory pathway. The paper does not state a license or pricing for the code.

How solid is it

The evaluation covers seven base models and four agent frameworks, which gives the benchmark's findings on unevenness across capabilities reasonable breadth. The abstract does not report specific accuracy or percentage-point figures for Hermes+ against the Hermes baseline or other frameworks, so the size of the reported gains cannot be verified from the source text alone, and the abstract does not name its authors or their institutions.

Risks and caveats

The abstract does not detail the five specific interventions that make up Hermes+, describing them only as spanning 'stages of the agent loop,' so the mechanism behind its improvements is not fully specified here. The authors themselves caveat that Hermes+'s strongest effect, on tasks requiring outdated state to be replaced, remains dependent on the specific capability and underlying model rather than holding uniformly.

“Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway.”

— PAST-Bench paper