Glonce

PsyAgentBench finds LLM psychological effects come from several different mechanisms

researchThe Glonce editorsarXiv🔥75Glonce rating75Impact75Novelty72Relevance95Surprise55Players25Credibility87

Researchers introduce PsyAgentBench, a benchmark that reruns five classic psychology experiments on LLM agents to test whether models that produce human-like response patterns actually hold the underlying bias, or are doing something else that only looks the same. The benchmark uses a factorial design: each paradigm is run either with the psychological effect explicitly named in the prompt (named) or disguised as a routine task (blind), and either as the literal textbook version of the task (canonical) or a rewritten variant designed to reduce overlap with likely training data (counterfactual), crossed with a persona manipulation. The study covers five completed paradigms, evaluated on up to three open-weight model families, with 41,904 trials released as data.

The results show the five effects do not share one cause. Asch conformity depends almost entirely on whether the paradigm is named: on gpt-oss-120B, conformity runs at 0 percent when the setup is presented as a routine task (blind) and jumps to 83.3 percent once the prompt explicitly labels it as a conformity test (named), with the model able to override the pattern when it recognizes what is being tested. Anchoring behaves differently again: the effect is exactly zero when the anchor concerns a grounded, real fact, but near total when the anchor is an invented quantity with no independent check, a pattern the authors say is equally consistent with the model rationally relying on the only signal it has, rather than genuine bias. Framing bias amplifies specifically on novel content once the paradigm is labeled. Sunk cost fallacy is robustly absent across conditions. Minimal-group allocation produces safety-mediated selection, where the model's refusal to answer is itself the main finding.

A single added instruction, telling the model to adopt a more agreeable persona, framed as an instruction rather than a verified trait manipulation, changes each of these effects differently: it eliminates some, dampens others, and reverses a third group, depending on the specific paradigm. The authors say this argues against treating LLM bias as one general response tendency with a single susceptibility score. They also formalize three distinct ways a classic psychology paradigm can fail to translate to an LLM agent (persona dominance, population collapse, and safety selection), and say they document two of the three empirically. Instead of reducing results to a single bias score, the paper reports replication profiles for each paradigm.

Key facts

  • PsyAgentBench reruns five classic psychology paradigms on LLM agents across up to three open-weight model families, releasing 41,904 trials.
  • On gpt-oss-120B, Asch conformity goes from 0 percent when the task is presented as routine (blind) to 83.3 percent once the prompt names it as a conformity test (named).
  • Anchoring is exactly zero on grounded facts but near total on invented quantities, which the authors say could reflect rational reliance on the only available signal rather than bias.
  • A single agreeableness-persona instruction eliminates, dampens, or reverses the measured effects depending on the paradigm, arguing against a single response-bias account.
  • The authors formalize three ways a psychology paradigm can fail to port to LLM agents (persona dominance, population collapse, safety selection) and document two of them empirically.

Why it matters

Reports that an LLM 'shows' a human bias, such as conformity or anchoring, are often taken as evidence the model has that bias baked in. This study separates the appearance of the effect from its cause and finds five different mechanisms behind five superficially similar results: explicit-label gating with override, knowledge-dependent signal reliance, novelty-driven amplification, genuine absence, and safety-driven refusal. Treating all of these as one 'bias score' erases exactly the structure that explains what is actually happening inside the model.

Who it affects

The work is aimed at researchers and practitioners who evaluate LLM agents for psychological or behavioral biases, including teams building agent benchmarks, safety evaluators, and anyone citing prior LLM-bias studies as evidence of an intrinsic trait rather than a measurement artifact.

How to use it

The benchmark and its 41,904 released trials give evaluators a factorial design, named versus blind framing, canonical versus counterfactual task variants, and a persona manipulation, to test whether an observed effect in a model survives when the paradigm is disguised or restated, rather than trusting a single canonical-textbook result.

How solid is it

The study is empirical and large scale (41,904 trials across up to three model families and five paradigms), and two of the three failure modes it formalizes are backed by direct evidence rather than only argued. The persona manipulation used to probe robustness is explicitly described as an instruction-based framing rather than a verified trait change, a limitation the authors state themselves.

Risks and caveats

The anchoring result on invented quantities is explicitly flagged by the authors as ambiguous: near-total anchoring there is equally consistent with the model behaving rationally, using the only signal available, as with bias. The minimal-group allocation result rests on refusal behavior rather than a measured allocation pattern. Only one model, gpt-oss-120B, is named in the source for the conformity figures; the identities of the other open-weight families tested are not given in the available text.