NarrativeBench finds 14 LLMs fall short on diverse multilingual news narratives

A preprint listed on Hugging Face Papers starts from a simple observation: large language models increasingly mediate how people seek and consume information, including accounts that differ across languages and regions, especially on contested topics. Supporting diverse information seeking, the paper says, requires models to help users navigate competing narratives across languages and information streams.
The authors define six capabilities they consider necessary for diverse multilingual information seeking. Three are foundational: narrative inclusion, narrative exclusion, and missing narrative identification. Three are higher-level: presenting diverse narratives, preserving narrative coverage under confirmatory queries, and maintaining information parity across languages.
To guide work on these capabilities, they introduce NarrativeBench. It comprises 1,333 news articles in 23 publication languages, spread across 23 contested topics, plus 460 Reddit-grounded queries.
The authors evaluated 14 LLMs across 24 query languages. Their finding is that current models lack the capabilities needed to reliably navigate diverse narratives. In their words, this limits users' control over the perspectives they receive and leaves substantial gaps in exposure to competing accounts, even when diversity is explicitly requested.
Information access also differs across query framings and languages. The authors say this risks echo chambers, linguistic filter bubbles and information disparities that weaken the common ground needed for democratic discourse. They present NarrativeBench as a foundation for building information systems that give users across languages and regions agency to navigate diverse narratives.
Key facts
- The paper defines six capabilities for diverse multilingual information seeking: three foundational (narrative inclusion, narrative exclusion, missing narrative identification) and three higher-level (presenting diverse narratives, preserving coverage under confirmatory queries, information parity across languages).
- NarrativeBench contains 1,333 news articles in 23 publication languages across 23 contested topics, plus 460 Reddit-grounded queries.
- The authors evaluated 14 LLMs across 24 query languages and report that current models lack the capabilities needed to reliably navigate diverse narratives.
- Even when diversity is explicitly requested, models leave substantial gaps in exposure to competing accounts, and information access differs across query framings and languages.
Why it matters
As people turn to LLMs for news and information, the way a model presents competing accounts of a contested topic shapes what users see. The paper frames the problem as more than answer accuracy: it asks whether a model can include a narrative, exclude one on request, spot a missing one, present several, keep coverage when a user asks a leading question, and give the same information in different languages. The authors warn that failures here risk echo chambers, linguistic filter bubbles and information disparities that weaken the common ground needed for democratic discourse.
Who it affects
The authors point to users across languages and regions who rely on LLMs for information on contested topics, and to people building information systems meant to give those users agency over which narratives they see. Researchers working on multilingual evaluation are the direct audience for the benchmark.
How to use it
NarrativeBench is meant to guide the development of the six capabilities, giving builders of LLM-based information systems a way to test them across languages and query framings. No release, code or dataset availability is mentioned.
How solid is it
This is a research preprint evaluating 14 LLMs across 24 query languages on a benchmark of 1,333 articles and 460 queries. The abstract names no authors or institutions. It does not name which 14 LLMs were evaluated. It gives no per-model scores, rankings or numeric results. The headline conclusion is the authors' own reading of their evaluation.
Risks and caveats
The abstract does not say which 23 contested topics or which languages are covered, so the breadth of the benchmark is hard to judge from it alone. The claim that current models lack the capabilities is stated in general terms without per-model results. The wider risks to democratic discourse are presented by the authors as risks, not as measured outcomes.
“NarrativeBench provides a foundation for developing information systems that give users across languages and regions agency to navigate diverse narratives.”
— Paper abstract, NarrativeBench