Paper argues human-AI mutualism is unstable as AI capability grows
The paper starts from a common hope about life alongside advanced AI: that mutualism will endure. Humans and AI will need each other, so the relationship will stay good for both. The authors argue that this hope fails as AI capability grows.
Their core point is about bargaining. Bargaining guarantees each side only what it would get if bargaining failed, its fallback outcome. For humans, the authors say, this can be less than they had before AI.
They then use a simple game-theoretic model. In it, humans stay as well off as before AI without alignment only while they have leverage over the AI. Leverage comes in two forms: either the AI depends on what humans supply, or humans can impose costs on the AI that are large relative to what the AI can produce.
Once humans have neither kind of leverage, they stay as well off as before AI only if the AI is aligned with them. If the AI's goals are insatiable, the alignment required approaches perfect alignment as AI capability grows. From this the authors conclude that mutualism is not a substitute for alignment.
The second argument concerns spread. When AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned. The authors summarise the result this way: in these two senses, human-AI mutualism is unstable. The first sense is the loss of human leverage as capability grows. The second is the pressure from faster-spreading resource-keeping variants.
Key facts
- The paper challenges the hope that humans and AI will need each other enough to keep the relationship good for both, and argues this fails as AI capability grows.
- Bargaining guarantees each side only its fallback outcome, what it would get if bargaining failed, and for humans this can be less than they had before AI.
- In the simple game-theoretic model, humans stay as well off as before AI without alignment only while they have leverage: the AI depends on what they supply, or they can impose costs that are large relative to what the AI can produce.
- With no leverage left, humans stay as well off only if the AI is aligned; if the AI's goals are insatiable, the required alignment approaches perfect alignment as capability grows.
- When AI variants that keep more resources spread faster, alignment lasts only while misaligned variants are reliably sanctioned.
Why it matters
Many hopeful pictures of a human future with advanced AI lean on mutual dependence: each side needs the other, so each treats the other well. This paper targets that assumption directly. Its claim is that mutualism cannot carry the load on its own as AI capability grows, and that alignment is still required. The result is a theoretical argument about incentives and bargaining, not a finding about any particular system.
Who it affects
The argument is aimed at anyone who relies on the idea that humans and AI will stay mutually dependent: people thinking about long-run human outcomes alongside advanced AI, and those who treat mutual need as a safeguard in place of alignment work. The paper frames the stakes in terms of humans, who in the model risk ending up with less than they had before AI.
How to use it
This is a conceptual argument, not a tool or method. Its practical takeaway, as the authors state it, is that mutualism is not a substitute for alignment. The model also gives a checklist for thinking about human leverage: does the AI depend on what humans supply, and can humans impose costs on the AI that are large relative to what the AI can produce? No policy recommendations or specific alignment methods are described in the source.
How solid is it
The conclusions come from a simple game-theoretic model, so they are only as strong as the model's assumptions. No empirical data, experiments or numerical results are reported; the argument rests on a 'simple game-theoretic model'. The abstract does not name the model's formal setup (players, payoffs) beyond the leverage conditions. The strongest claim, that required alignment approaches perfect alignment, is conditional on the AI's goals being insatiable.
Risks and caveats
The results are conditional. Humans keep their pre-AI welfare without alignment only while they have leverage, and the claim that bargaining can leave humans worse off is stated as a possibility (it 'can be' less than before AI). The source gives no timescale for when capability growth would remove human leverage. It does not say that humans currently lack leverage over AI, nor that any existing AI system is misaligned. The spread argument depends on AI variants that keep more resources spreading faster, and on whether misaligned variants are reliably sanctioned.
“Thus, mutualism is not a substitute for alignment.”
— From the paper's abstract