Glonce

LongBanana benchmark tests image generators on 90 multi-reference tasks

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact40Novelty60Relevance100Surprise25Players5Credibility75
LongBanana benchmark tests image generators on 90 multi-reference tasks

The authors introduce LongBanana, which they describe as, to their knowledge, the first expert-designed and expert-verified benchmark for long-context multi-reference image synthesis. The task asks a system to retrieve dispersed visual evidence from a large set of reference images, reject distractors, bind entities and relations correctly, and compose one coherent final image.

The benchmark has 90 carefully designed tasks organized by a 3-by-3 taxonomy. Each task includes about 32 reference images, 20 constraints and 50 evaluation items, and the benchmark totals 4,670 expert-specified atomic evaluation items. The per-task figures are approximate.

Scoring is graph-grounded. Each task is instantiated with reference images and an entity-relation graph. Node, edge and whole-graph obligations are then converted into checklist items under three headings: Content Fidelity, Relational Binding and Global Constraint Satisfaction.

To show the labels are reliable, the authors report that the reviewed expert labels include a 15% double-labeled subset, with 89.4% exact agreement and a quadratic weighted kappa of 0.83.

The authors also introduce LongBanana-Gen, an agent-harness framework for long-horizon image generation. In their experiments, minimal agent harnesses improve over direct generation. A ReAct-Plan-Exec-Reflect (RPR) workflow yields harness-dependent gains by strengthening evidence grounding and repair. The authors present LongBanana as an auditable testbed for measuring dense visual evidence use and as a shared target for building stronger agent-driven image-generation systems. Code and data are available at the project page, https://long-banana.com/.

Key facts

  • LongBanana has 90 tasks organized by a 3-by-3 taxonomy, each with about 32 reference images, 20 constraints and 50 evaluation items.
  • The benchmark totals 4,670 expert-specified atomic evaluation items, scored under Content Fidelity, Relational Binding and Global Constraint Satisfaction.
  • On a 15% double-labeled subset, expert labels show 89.4% exact agreement and a quadratic weighted kappa of 0.83.
  • The companion LongBanana-Gen agent harness improves over direct generation; a ReAct-Plan-Exec-Reflect (RPR) workflow gives harness-dependent gains.
  • Code and data are released at https://long-banana.com/.

Why it matters

Most image-generation tests give a model a prompt or a handful of references. LongBanana targets a harder setting: dozens of reference images, some of them distractors, from which a system must pull the right evidence and combine it into one image that respects entity relations and global constraints. The authors say it is, to their knowledge, the first expert-designed and expert-verified benchmark for this kind of long-context multi-reference synthesis, and they frame it as a shared target for agent-driven image-generation systems.

Who it affects

Researchers building or evaluating agent-driven image-generation systems are the direct audience, since the benchmark and the LongBanana-Gen harness are meant as a testbed and a starting point. Teams working on multi-reference or long-context image synthesis can use the checklist-style scoring to see where a system fails: fidelity to content, binding of relations, or satisfaction of global constraints.

How to use it

The authors state that code and data are available at the project page, https://long-banana.com/. The scoring is a checklist: each task's entity-relation graph is turned into items covering nodes, edges and the whole graph, grouped into Content Fidelity, Relational Binding and Global Constraint Satisfaction. LongBanana-Gen offers a harness to compare against direct generation, including the ReAct-Plan-Exec-Reflect (RPR) workflow.

How solid is it

This is a preprint listed on Hugging Face Papers, and the claims come from the authors' own summary. The expert labels have reported agreement figures: 89.4% exact agreement and a quadratic weighted kappa of 0.83 on a 15% double-labeled subset. The claim of being the first expert-verified benchmark of this type is hedged by the authors with "to our knowledge". No model names, scores or benchmark results are given for any image generator; the size of the gains from agent harnesses or the RPR workflow is not quantified.

Risks and caveats

The abstract does not say which models were tested or which harnesses count as "minimal", so the claimed improvement over direct generation cannot be sized. The per-task figures (about 32 images, 20 constraints, 50 items) are approximate, and the abstract does not reconcile them with the 4,670 total; 90 tasks at exactly 50 items would be 4,500. The contents of the 3-by-3 taxonomy are not described. It does not say how many experts took part or how they were recruited, and no license for the code and data is stated.

“LongBanana provides an auditable testbed for measuring dense visual evidence use and a shared target for developing stronger agent-driven image-generation systems”

— LongBanana abstract