IntReal: a dataset and benchmark for interleaved text-image generation

Humans routinely consume content where text and images are interleaved. Unified Multimodal Models (UMMs) can generate both text and images, but the paper argues that multimodal interleaved generation, producing the two together in one output, remains largely underexplored. It names two obstacles: there are no well-defined interleaved generation tasks with matching training data, and there are no fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions.

To address both, the authors present IntReal, which has two parts. The first is IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories. The second is the IntReal Evaluation Suite, which scores model performance across seven complementary dimensions using specialized expert models, so that a model can be diagnosed from coarse to fine detail.

The authors then test the suite itself. They report low redundancy between metrics, with the maximum pairwise absolute Spearman's rho at 0.42 or below. They report 83.3% accuracy in agreement with human judgments. And they report strong consistency across different expert evaluators, with Spearman's rho above 0.9. In the authors' words, this supports the complementarity and reliability of the metrics.

The authors benchmark four families of UMMs and say the suite reveals distinct strengths and weaknesses across models. Post-training experiments with the dataset show that IntReal-100K is useful for improving interleaved generation. They also show that training unimodal generation can benefit interleaved generation. From this the authors conclude that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.

Key facts

IntReal has two parts: IntReal-100K, a real-world dataset of multimodal interleaved samples in four major categories, and the IntReal Evaluation Suite.
The evaluation suite scores models on seven complementary dimensions using specialized expert models, allowing coarse-to-fine diagnosis.
The suite is validated with a maximum pairwise absolute Spearman's rho of 0.42 or below between metrics, 83.3% accuracy in agreement with human judgments, and Spearman's rho above 0.9 across expert evaluators.
Four families of UMMs were benchmarked, and the suite is said to expose distinct strengths and weaknesses across them.
Post-training experiments show IntReal-100K helps interleaved generation, and training unimodal generation also benefits it, which suggests modality-specific optimization can work.

Why it matters

Unified Multimodal Models can jointly generate text and images, yet the authors say interleaved generation is largely underexplored. They point to missing task definitions, missing training data and missing fine-grained evaluation as the reasons. IntReal tries to supply the data and the yardstick together. The finding that unimodal training can also help interleaved output, if it holds up, hints that teams need not always train text and image generation jointly at every stage.

Who it affects

Researchers building or evaluating Unified Multimodal Models are the main audience, since the paper offers a dataset for post-training and a suite for diagnosing capability differences. Teams comparing UMM families may find the seven-dimension breakdown more informative than a single score.

How to use it

The paper positions IntReal-100K as post-training data for improving interleaved generation, and the evaluation suite as a way to diagnose a model across seven dimensions, from coarse to fine. The source does not say whether the dataset, code or benchmark are publicly released, or under what licence.

How solid is it

The claims below are the authors' own. They validate the evaluation suite with three checks: low redundancy between metrics (max pairwise absolute Spearman's rho of 0.42 or below), 83.3% agreement with human judgments, and Spearman's rho above 0.9 across different expert evaluators. The source does not say how the 83.3% human agreement was measured, or on how many samples. It also names no benchmarked models, gives no per-model scores, and does not quantify the post-training improvement.

Risks and caveats

The authors say their findings suggest, rather than prove, that interleaved generation can benefit from modality-specific optimization. The four dataset categories and seven evaluation dimensions are not named in the source, and neither are the four UMM families tested, so the scope of the conclusions is hard to judge from this text alone. The source gives no author names or institutions.

“These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.”

— IntReal paper abstract