Glonce

UserArena benchmark offers 64,729 real shopping sessions to test user simulators

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact45Novelty60Relevance95Surprise25Players12Credibility75
UserArena benchmark offers 64,729 real shopping sessions to test user simulators

The paper starts from a problem in agentic recommendation: shopping agents pursue a shopper's goals through sequences of context-dependent decisions. To develop and evaluate such policies, the authors say, you need faithful recommendation world models that capture both the interactive shopping environment and how users respond over long interactions. Existing resources, they argue, rarely provide long-horizon, fine-grained real-user trajectories, and they rarely support trajectory-level evaluation or offline validation against production A/B results.

To fill that gap the authors introduce UserArena, a real-world e-commerce interactive benchmark. It contains 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. The trajectories come paired with an interactive environment. The benchmark also supports offline validation against outcomes from a controlled production A/B test.

That design enables five-level evaluation of a simulator, running from output validity and local decisions up through sequential behavior, personalization, and outcomes.

The second contribution is a training method called On-Policy Trajectory Distillation (OPTD), meant to improve simulator fidelity over long-horizon interactions. OPTD uses trajectory-level intent to provide dense teacher targets on student-generated rollouts.

The authors report that in their experiments OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol. They also say it most closely reproduces the production policy effect in offline simulation. The abstract gives no numeric scores or margins for these results.

Key facts

  • UserArena is a real-world e-commerce benchmark with 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types.
  • It pairs the trajectories with an interactive environment and supports offline validation against a controlled production A/B test.
  • It enables five-level evaluation: output validity, local decisions, sequential behavior, personalization, and outcomes.
  • The authors propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to give dense teacher targets on student-generated rollouts.
  • The authors report OPTD has the strongest overall fidelity among the evaluated simulators and most closely reproduces the production policy effect offline.

Why it matters

Testing a shopping agent on live customers is slow and risky, so teams lean on simulated users. A simulator is only useful if it behaves like real shoppers over long sessions. The authors argue that existing resources rarely offer long-horizon, fine-grained real-user trajectories or let researchers check results against production A/B outcomes. UserArena is pitched as a way to do both: real sessions, an interactive environment, and a link to an actual A/B test.

Who it affects

Researchers building user simulators and recommendation world models for e-commerce are the direct audience, along with teams developing agentic recommendation policies that need an offline way to evaluate them. The users behind the data are 11,010 shoppers whose sessions make up the trajectories.

How to use it

The abstract describes a benchmark with an interactive environment and a five-level evaluation protocol, from output validity and local decisions to sequential behavior, personalization, and outcomes. A simulator can be scored at each level and checked against the outcomes of a controlled production A/B test. OPTD is offered as a training recipe for simulators: a teacher gives dense targets on rollouts that the student itself generates, guided by trajectory-level intent.

How solid is it

This is a preprint, and the claims here come from the authors' abstract. The headline result is that OPTD has the strongest overall fidelity among the evaluated simulators under UserArena's own evaluation protocol. The abstract gives no numeric results or margins. It does not name the simulators OPTD was compared against, and it does not state the size or duration of the A/B test or its measured effect.

Risks and caveats

The abstract names no authors or institutions and does not name the e-commerce platform that supplied the data. It does not give the time period the trajectories cover, and it mentions no release of data or code. The base models used for the teacher and student are not named. Because OPTD is judged under the benchmark its authors built, the result should be read as their own evaluation.