Glonce

YANAMI introduces 400K multimodal dialogues to test stateful interaction

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact45Novelty55Relevance100Surprise25Players15Credibility75
YANAMI introduces 400K multimodal dialogues to test stateful interaction

The authors start from a gap they see in unified multimodal models (UMMs), which combine visual understanding, generation and editing in one framework. In their account, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even recent multi-turn datasets often let many turns be solved independently, without tracking information from one turn to the next. That gives limited supervision and evaluation for what they call multimodal stateful interaction: understanding, maintaining, updating and faithfully realizing an evolving dialogue state across turns.

To fill the gap they introduce YANAMI, which they describe as the first large-scale multimodal dialogue dataset and benchmark for stateful interaction. It has two parts. YANAMI-400K is the training set: 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues cover four scenarios, Multi-Image Fusion, Image Editing, World Modeling and Visual Reasoning, and require models to keep track of, preserve and revise the changing dialogue state over the course of a conversation.

YANAMI-Bench is the evaluation set, with 400 multi-turn dialogues. It scores two things together: Dialogue State Understanding and Visual Realization. The aim is to measure the gap between a model's ability to understand an evolving dialogue state and its ability to faithfully render that state in visual outputs.

The reported results come in two parts. Fine-tuning BAGEL on YANAMI-400K improves image editing, generation and multimodal stateful interaction by 10.3%, 2.0% and 13.23% respectively, while preserving the model's general visual understanding. And evaluations on YANAMI-Bench show that existing UMMs still struggle to maintain, update and visually realize evolving multimodal dialogue states.

Key facts

  • YANAMI is a dataset and benchmark for multimodal stateful interaction, where a model must track and update dialogue state across turns; the authors call it the first large-scale one.
  • YANAMI-400K, the training set, has 400K multi-turn multimodal dialogues, more than 2M images and 2.6M dialogue turns, spanning four scenarios: Multi-Image Fusion, Image Editing, World Modeling and Visual Reasoning.
  • YANAMI-Bench has 400 multi-turn dialogues and scores Dialogue State Understanding and Visual Realization.
  • Fine-tuning BAGEL on YANAMI-400K improves image editing by 10.3%, generation by 2.0% and multimodal stateful interaction by 13.23%, with general visual understanding preserved.
  • On YANAMI-Bench, existing unified multimodal models still struggle to maintain, update and visually realize evolving dialogue states.

Why it matters

Unified multimodal models are meant to understand, generate and edit images in one system, but the authors argue the data and tests for them are mostly single-turn. Real conversations about images build up changes, so a model has to remember what was edited, fused or decided earlier. YANAMI is aimed at that gap, with training data and a benchmark built so that turns depend on one another rather than being solvable in isolation.

Who it affects

Mainly researchers and teams building or evaluating unified multimodal models that handle image understanding, generation and editing. The reported experiments use BAGEL as the model being fine-tuned, and the benchmark is meant to be run on UMMs more broadly.

How to use it

The intended use is twofold: YANAMI-400K as fine-tuning data for multi-turn multimodal dialogue, and YANAMI-Bench as a 400-dialogue test of both state understanding and visual realization. No information on dataset availability, license, download link or code release is given in the source.

How solid is it

This is the authors' own description of their work, and the headline results are self-reported. The 10.3%, 2.0% and 13.23% gains come with no stated metric or baseline score, and the source does not say whether they are percentage points or relative improvements. The source does not say which models other than BAGEL were evaluated on YANAMI-Bench, nor how they scored. The claim to be the first large-scale dataset of this kind is the authors' own.

Risks and caveats

No details are given on how the dialogues and images were constructed or generated, and no BAGEL model size or version is given. The 2.0% generation gain is small next to the editing and stateful-interaction gains. The finding that existing models struggle is reported without per-model numbers in the source, so the size of the gap cannot be judged from it.

“existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states”

— YANAMI paper abstract