MerchantBench: LLM agents reach only 27.3% of human e-commerce net assets
Researchers introduced MerchantBench, a benchmark built to test whether large language model agents can stay coherent over long stretches of time rather than just complete short, bounded tasks. Most existing agent benchmarks judge success on isolated actions with immediate feedback. MerchantBench instead simulates 365 days of running a seller account in e-commerce, grounded in 98,843 real product records, and gives each agent 26 tools to work with. Agents have to manage product sourcing, listing and pricing, cash flow, and a mix of fast and slow feedback: supplier events show up right away, but the outcome of an individual order only becomes clear after a delay, so an agent has to track many open order lifecycles at once and revisit earlier pricing and sourcing decisions as new information arrives.
The researchers evaluated eight LLMs, each run under two different agent frameworks, for a total of 48 runs, with every run spanning the full simulated 365 days. They then compared each agent's final net assets against the net assets human participants reached running the same simulated business over the same period.
The gap was large: the best-performing LLM configuration ended the simulation with only 27.3% of the mean final net assets that human participants achieved. The paper frames this as evidence of a substantial gap between current LLMs and human performance specifically on long-horizon, delayed-feedback decision making, even though these are the same models that already score well on many short, bounded benchmark tasks.
Key facts
- MerchantBench simulates 365 days of running an e-commerce seller account, grounded in 98,843 real product records, with 26 tools available to each agent.
- It combines quickly observable supplier events with delayed order outcomes, so agents must track individual order lifecycles and revisit earlier decisions.
- Eight LLMs were tested under two different agent frameworks, for 48 total runs, each covering the full 365 simulated days.
- The best LLM configuration reached only 27.3% of the mean final net assets that human participants achieved over the same simulated period.
- The paper positions the gap as evidence that current agents struggle specifically with long-term coherence, not just with bounded, immediate-feedback tasks.
Why it matters
Most agent benchmarks measure whether a model can complete a single bounded task with fast, clear feedback. MerchantBench argues that real deployments look different: decisions made today constrain what an agent can do later, feedback on those decisions can take a long time to arrive, and small incoherent choices compound over months. By running agents through a full simulated year of running a business, the benchmark tries to measure exactly the kind of sustained, self-consistent decision making that short-task benchmarks cannot capture, and it finds today's models fall well short of it.
Who it affects
The direct audience is AI researchers building and evaluating agentic systems, since MerchantBench gives them a new way to test long-horizon coherence rather than one-shot task completion. It is also relevant to anyone considering LLM agents for seller-side e-commerce operations such as sourcing, listing, pricing or cash-flow management, since the results speak directly to whether current models can be trusted to run that kind of operation unsupervised over an extended period.
How to use it
MerchantBench is a research benchmark rather than a product: it provides a 365-day simulated environment built on 98,843 real product records, with 26 tools an agent can call to source products, manage listings and pricing, and handle cash flow. The abstract does not state whether the benchmark code or environment has been released for others to run, and it gives no cost, pricing or hardware information, so those details are not covered here.
How solid is it
The evaluation is reasonably broad for a single paper: eight different LLMs were tested under two separate agent frameworks, adding up to 48 runs, and each run covered the full 365 simulated days rather than a short slice. The simulation is also grounded in a large set of real product records rather than synthetic or toy data. That said, the abstract does not name the authors, their institution, the specific LLMs evaluated, or a publication date, which limits how far the result can be checked without reading the full paper.
Risks and caveats
The source available here is the paper's abstract, not the full text, so details on methodology, the identity of the eight LLMs, and how human participants were recruited and scored are not available. Results from one simulated e-commerce economy may not generalize to other kinds of long-horizon agent deployments, and no cost or compute information is given for either the LLM runs or the human baseline.
“Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence.”
— MerchantBench paper