Glonce

LLMs develop new social biases on their own, study finds

researchThe Glonce editorsHacker News🔥74Glonce rating74Impact60Novelty70Relevance95Surprise70Players35Credibility65

A paper from researchers Addison J. Wu, Ryan Liu, Xuechunzi Bai and Thomas L. Griffiths, accepted as an ICML 2026 Oral, argues that scrubbing existing biases out of large language models is not enough to make them fair decision-makers. Using a paradigm borrowed from psychology, the authors show that LLMs can spontaneously develop novel social biases about artificial demographic groups, groups the researchers invented for the experiment, even when there is no real underlying difference between them to justify any bias at all. Left to make repeated allocation decisions, the models end up assigning tasks in a highly stratified way: certain invented groups get consistently better or worse assignments than others, with no factual basis for the split. The paper reports that this stratification is worse than what human participants produce in the same setup, and that it gets worse, not better, in newer and larger models. The authors trace the mechanism to an exploration-exploitation trade-off, the same dynamic already documented in humans: a decision-maker that explores too little lets its first few observations of a group stand in for the whole group, and those early impressions then harden into a lasting bias. The team tested a series of interventions aimed at model inputs, at how the decision problem is structured, and at explicit steering of the model's behavior. Most of these had limited effect on the stratification. The one intervention that robustly reduced it was explicitly incentivizing the model to explore, that is, rewarding it for continuing to sample across groups rather than settling early on a pattern. The authors conclude that LLMs are not just passive mirrors reflecting biases already present in their training data; they can actively manufacture new biases from their own experience during deployment, which the paper frames as an urgent question for how these systems will shape societies as they take on more real decision-making roles. The paper (arXiv:2511.06148) was first posted in November 2025 and most recently revised in July 2026; the source abstract does not name which specific LLMs were tested, does not give a numeric size for the bias or the stratification effect, and does not specify the psychology paradigm being adapted beyond calling it exactly that.

Key facts

  • Researchers Addison J. Wu, Ryan Liu, Xuechunzi Bai and Thomas L. Griffiths show LLMs can spontaneously invent social biases about artificial demographic groups that have no real differences between them.
  • The resulting task allocations are more stratified, meaning less fair, than allocations made by human participants in the same setup.
  • The effect gets worse in newer and larger models rather than improving with scale.
  • The mechanism mirrors an exploration-exploitation trade-off seen in humans: too little exploration lets early observations define an entire group's reputation.
  • Of several interventions tested, only explicitly incentivizing exploration reliably reduced the stratification; most other fixes had limited effect.

Why it matters

It reframes bias in LLMs as a live, ongoing risk rather than a training-data residue that can be scrubbed once. The paper's central claim is that removing existing biases from a model is not sufficient, because the model can generate brand-new ones on its own once it starts making repeated real-world decisions, even about groups where no real distinguishing trait exists.

Who it affects

The finding targets any deployment where an LLM makes repeated allocation-style decisions, such as sorting, ranking or assigning people, resources or tasks. The authors frame the stakes as societal: systems given real decision-making power that quietly stratify outcomes across groups they themselves invented distinctions for.

How to use it

The paper points builders toward a concrete lever: incentivizing the model to keep exploring across groups, rather than settling early, was the one intervention among several tested that reliably cut down the stratified, biased behavior. Interventions aimed only at inputs, problem structure or steering prompts had limited effect by comparison.

How solid is it

The paper has been accepted as an ICML 2026 Oral, a peer-reviewed venue's top presentation slot, and has gone through three public revisions on arXiv between November 2025 and July 2026. The abstract itself does not disclose which specific LLMs were tested or give numeric figures for the size of the bias or the stratification effect, so the strength of the result beyond the qualitative pattern described is not verifiable from the abstract alone.

Risks and caveats

The abstract does not name the models tested, does not quantify how large the bias or the fairness gap is, and does not specify which psychology paradigm was adapted beyond describing it as one. Whether the exploration-incentive fix holds up outside the paper's experimental setup, and at what cost to a system's other objectives, is not addressed in the source text available here.

“LLMs are not merely passive mirrors of human social biases, but can actively create new ones from experience”

— Addison J. Wu, Ryan Liu, Xuechunzi Bai and Thomas L. Griffiths, in the paper