LLMs predict harsher social punishment than humans do, study finds
Most AI alignment work checks whether a model knows a norm was broken, such as recognizing that stealing is wrong. A new paper argues that is only half of social intelligence: knowing a rule was violated does not mean knowing how other people will respond to the violation, whether with a shrug, a scolding, or a call to the police. The paper calls these second-order expectations "metanorms" and introduces a framework for testing whether LLMs can reason about them along two dimensions: emotional appraisal (how the violator and observers are expected to feel) and behavioral response (what they are expected to do about it). Two classification tasks follow from this: predicting self-regulation, meaning how a violator responds to their own transgression, and other-regulation, meaning how an observer responds to someone else's.
To test this, the researchers built and released NormReact, a dataset of 450 norm violation scenarios. Each scenario is hand-annotated for the emotions and behavioral responses involved, and the annotations vary the violator's gender and the observer's social closeness to the violator, so the dataset can capture whether reactions shift depending on who is involved and how close the observer is to them.
Running six LLMs against these scenarios, the researchers found a consistent pattern: the models portray a harsher social world than humans do. Across all six, the models overpredict negative sanctions such as punishment or public shaming in situations where human respondents expected people to simply let the violation go. The mismatch gets worse, not better, as social distance increases: the further removed the observer is from the violator, the less the models' predictions track what humans actually expect. The paper does not name the six models tested, nor does it give a numeric size for either the overprediction or the growing misalignment, reporting only the direction of both effects.
Key facts
- The paper defines "metanorms": second-order expectations about who enforces a broken social norm and how, as distinct from simply recognizing that a norm was broken
- A new framework evaluates this along two dimensions, emotional appraisal and behavioral response, via two tasks: predicting a violator's self-regulation and an observer's other-regulation
- The released NormReact dataset contains 450 hand-annotated norm violation scenarios, varying violator gender and observer social closeness
- Across six (unnamed) LLMs, the models consistently overpredict negative sanctions in cases where human respondents expected inaction
- Model-human alignment gets worse as social distance between observer and violator increases, with no numeric size given for either effect
Why it matters
AI safety work has mostly asked whether a model can tell right from wrong. This paper points out that deploying a model in any role that touches conflict, moderation, or social judgment also requires it to predict how people will actually react to a violation, and that this is a separate skill the model can get wrong even after correctly identifying that a norm was broken.
Who it affects
The paper names conflict mediation and policy simulation as domains where this matters, and the same reasoning extends to any AI system asked to moderate, arbitrate, or advise on interpersonal or social disputes, since a system that consistently expects harsher consequences than people actually deliver will misjudge those situations.
How to use it
The NormReact dataset and the underlying framework are released for evaluating metanorm reasoning, giving researchers and developers a way to test whether a given model overpredicts punishment or fails to track how expected reactions shift with social distance, before deploying it in a norm-sensitive setting.
How solid is it
The evaluation spans six LLMs and a 450-scenario hand-annotated dataset with an explicit design to vary gender and social closeness, which supports the direction of the finding. The abstract does not disclose which six models were tested, how the annotations were produced beyond being hand-annotated, or the numeric magnitude of the overprediction and the alignment decline, so the size of the effect cannot be judged from the available text.
Risks and caveats
The authors themselves frame this as a risk rather than a settled harm: AI systems used in norm-sensitive domains may produce a distorted picture of social regulation, one that over-represents punishment and under-represents the tolerance, restraint, and relational calibration that characterize how norm enforcement actually works in the real world.
“Current LLMs portray a harsher social world: across six models, they overpredict negative sanctions where humans would expect inaction”
— the paper's findings