Glonce

GPT-5.2 survives only 42% of a new interactive-story consistency test

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact60Novelty70Relevance100Surprise60Players45Credibility80
GPT-5.2 survives only 42% of a new interactive-story consistency test

Researchers introduced NCP-Bench, a benchmark aimed at a problem they say prior work has largely ignored: whether an LLM agent can maintain long-horizon logical consistency and narrative integrity in interactive storytelling once a user starts pushing back or steering the story off-script. They call this challenge Narrative Commitment Preservation (NCP) and use interactive narrative as the testbed. NCP-Bench consists of 100 narrative environments built from movie synopses. Each environment carries a structured narrative specification, made up of a trajectory, a set of commitments, and a set of initial facts, that can be checked automatically throughout the interaction between a player agent and a narrator agent.

Across state-of-the-art LLMs, the results show a substantial long-horizon consistency gap. High linguistic quality does not guarantee commitment preservation: even strong models frequently generate content that logically conflicts with the story's established facts once a player attempts an adversarial intervention. The best-performing model tested, GPT-5.2, achieved only a 42% survival rate after 20 turns. Fact conflict rates ranged from 40% to 68% across the models tested. Within the benchmark's 100-turn limit, only isolated runs satisfied all of a story's achievement commitments.

Key facts

  • NCP-Bench comprises 100 narrative environments derived from movie synopses, each with a structured specification (trajectory, commitments, initial facts) checked automatically during play.
  • The benchmark runs a player agent against a narrator agent under unconstrained, adversarial user interventions.
  • GPT-5.2, the best-performing model tested, survived only 42% of runs after 20 turns.
  • Fact conflict rates across the tested models ranged from 40% to 68%.
  • Only isolated runs satisfied all achievement commitments within the benchmark's 100-turn limit.

Why it matters

LLMs are increasingly used to drive open-ended, fluid interactive storytelling in games, but most prior research has focused on generating fluent text rather than testing whether an agent can hold a story together once a player pushes back or tries to derail it. NCP-Bench formalizes this gap as Narrative Commitment Preservation and shows that even a leading model breaks its own plot commitments more often than it keeps them once a player interferes.

Who it affects

Anyone building LLM-powered games, interactive fiction, or narrator agents where a coherent, evolving storyline has to survive unpredictable player input, plus researchers evaluating long-horizon consistency in LLM agents generally. Since GPT-5.2 was the best performer among the models tested and still failed most runs after 20 turns, the finding describes a gap across current state-of-the-art models rather than a flaw specific to one product.

How to use it

NCP-Bench works by checking a narrative specification, its trajectory, commitments, and initial facts, automatically throughout play as a player agent interacts with a narrator agent, then scoring survival rate and fact conflict rate over the course of up to 100 turns. The source text does not state whether the benchmark's code or the 100 narrative environments are publicly released.

How solid is it

The benchmark is built from 100 narrative environments derived from movie synopses, each with a structured specification that is checked automatically rather than judged subjectively, and results are reported across what the authors describe as state-of-the-art LLMs. The source text names only one model specifically, GPT-5.2, and does not identify the paper's authors, institution, or publication venue.

Risks and caveats

The source text gives no author names, institution, or affiliation, and no release date, code availability, or publication venue. It also does not define how 'fact conflict rate' or 'survival rate' are computed beyond the summary description, and it does not list which LLMs besides GPT-5.2 were tested, so the 40% to 68% conflict-rate range cannot be attributed to specific models from this text alone.

“High linguistic quality does not guarantee commitment preservation.”

— NCP-Bench paper