Glonce

LongNovel benchmark tests AI hallucinations in novel summaries

researchThe Glonce editorsarXiv🔥71Glonce rating71Impact62Novelty72Relevance100Surprise48Players32Credibility88

Context windows in large language models have grown a lot in recent years, but hallucination during long-context summarization has not gone away. The authors argue that novels are a better testing ground for this problem than news articles or academic papers, because novels carry far more intrinsic detail and long chains of events and dialogue for a summarizer to get wrong. They say that current research lacks a multi-scale benchmark for hallucination detection in long-context novel summarization, and that no prior work has closely examined how hallucination behavior changes as the context grows longer.

To address both gaps, the authors built LongNovel, a multi-scale, bilingual benchmark covering Chinese and English. The Chinese side is constructed from 29 full novels ranging from 16,000 to 100,000 tokens; the English side uses chapter-level data taken from the existing BookSum dataset rather than full novels. The benchmark defines eight distinct hallucination types, though the abstract does not name or describe what those eight types are.

To generate hallucinated content for the benchmark, the team combined two techniques: Multi-Model Arbitration and Entity-Referenced Hallucination Generation. Together, these are meant to keep the generated hallucinations realistic while keeping the eight categories evenly represented, rather than skewed toward whichever type is easiest to produce; the abstract does not explain the mechanics of either technique. The authors say they then manually revised the content of the test set to guarantee data reliability.

The authors report that extensive experiments show LongNovel is a challenging benchmark. The abstract does not give the accuracy, detection-rate or other quantitative results behind that claim, and it does not name which summarization systems, if any, were evaluated against the benchmark. It also names no authors or institutions. LongNovel and its code are released publicly on GitHub, at BDML-lab/LongNovel, for other researchers to use.

Key facts

  • LongNovel is a multi-scale, bilingual (Chinese and English) benchmark for detecting hallucinations in long-context novel summarization.
  • The Chinese side draws on 29 full novels ranging from 16,000 to 100,000 tokens; the English side uses chapter-level data from the existing BookSum dataset.
  • The benchmark defines eight hallucination types, generated through a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation to keep the categories realistic and balanced.
  • The authors manually revised the test-set content afterward to guarantee data reliability.
  • The authors report that extensive experiments show LongNovel is challenging, but the abstract gives no accuracy or detection-rate figures and names no evaluated systems or authors; the code is released on GitHub at BDML-lab/LongNovel.

Why it matters

Context windows in large language models have grown a lot, but hallucination during long-context summarization has not gone away. The authors argue that novels are a better testing ground for this problem than news articles or academic papers, because novels carry far more intrinsic detail and long chains of events and dialogue for a summarizer to get wrong. They also say that current research lacks a multi-scale benchmark for hallucination detection in long-context novel summarization, and that no prior work has closely tracked how hallucination behavior changes as the context grows longer. LongNovel is built to close both of those gaps.

Who it affects

The benchmark is aimed at researchers building or evaluating long-context summarization systems, particularly teams working on hallucination detection and faithfulness in large language models. Because it is bilingual, it lets a team test the same underlying problem across Chinese and English text rather than in one language alone. It is a research instrument released on GitHub, not a product aimed at end users of a summarization tool.

How to use it

LongNovel and its code are released publicly on GitHub, at BDML-lab/LongNovel, for other researchers to build on. The abstract states only that the benchmark is released for future research; it gives no price or licence terms, so none should be assumed.

How solid is it

The methodology has real safeguards built in: the eight hallucination types are generated through a combination of Multi-Model Arbitration and Entity-Referenced Hallucination Generation, meant to keep the generated hallucinations realistic and the eight categories evenly represented rather than skewed toward whichever is easiest to produce, and the authors say they manually revised the test-set content afterward to guarantee reliability. Beyond that, the abstract is thin on verification detail: it does not name the eight hallucination types, explain how the two generation techniques work, give any quantitative result such as accuracy or detection rate behind the claim that the benchmark is challenging, or say which summarization systems, if any, were tested against it. The abstract also names no authors or institutions.

Risks and caveats

The bilingual claim is not entirely symmetric: the Chinese side is built from 29 full novels of 16,000 to 100,000 tokens, while the English side draws only on chapter-level data from the existing BookSum dataset, not full novels of comparable scale. The paper is currently only a preprint abstract, and its central claims rest on the authors' own description rather than on numbers a reader can check: that the hallucination categories are balanced, that the test set is reliable, and that the benchmark is challenging.