MPIE-Bench exposes anatomy errors that VLM judges miss in group photo edits

Text-to-image and personalized editing models can now render a single subject with high fidelity, but placing several named people into shared contact actions such as an embrace, a carry, or a grapple still breaks down: limbs fuse together, extra limbs appear, and bodies interpenetrate. Jiajia Lin and co-authors argue that existing evaluations largely miss these anatomical and geometric failures, and that the common practice of using a vision-language model as a judge saturates on an 'Interaction' score even when the errors are obvious to a human looking at the same image. To fix this, the authors built MPIE-Bench, a 2,500-sample benchmark of editing triplets mined from video, covering 405 scenes, 14 interaction categories, and four contact densities labeled C0 through C3. Alongside it they propose MPIE-Eval, an evaluation method built on two new axes that score contact-time geometry using a frozen, publicly available multi-person mesh reconstruction model rather than a VLM. The Anatomy axis checks whether every human-like mass visible in the edited image is accounted for by a complete set of reconstructed bodies, catching fused or invented limbs. The Interaction axis checks whether the penetration and surface distance between the reconstructed bodies match the contact the editing instruction actually asked for. Testing ten image editors against these axes, the authors found that mesh-based Anatomy scores top out at 0.65 and mesh-based Interaction scores top out at 0.72, achieved by two different models, meaning no single editor scores well on both axes at once. Over the same images, VLM checklists hand out scores above 0.95, a gap the paper treats as evidence that VLM-as-judge overstates how well multi-person edits actually work. A study with five human raters found that both new mesh-based axes track human judgment of the images more closely than a zero-shot VLM judge does, and the ranking of the ten editors held up even after the authors ablated every weight and threshold in the scoring method.

Key facts

MPIE-Bench is a 2,500-sample benchmark of video-mined editing triplets spanning 405 scenes, 14 interaction categories, and four contact densities (C0-C3)
MPIE-Eval scores contact-time geometry with two new axes, Anatomy and Interaction, computed from a frozen public multi-person mesh reconstruction model instead of a VLM judge
Across ten tested image editors, mesh Anatomy tops out at 0.65 and mesh Interaction tops out at 0.72, on two different models, so no editor is strong on both
VLM-as-judge checklists rate the same images above 0.95, far higher than the mesh-based scores
A five-rater human study found both mesh-based axes track human judgment more closely than a zero-shot VLM judge, and the editor rankings held under ablation of every weight and threshold

Why it matters

Multi-person image editing, putting two or more named people into a shared physical action like a hug or a struggle, is a common request but a weak point for current models: bodies fuse, limbs multiply, and figures pass through each other. The paper's core claim is that the field has been grading itself on the wrong test. VLM-as-judge checklists, the default way many editing papers report quality, saturate near-perfect scores on interaction quality even when a human looking at the same image immediately spots fused limbs or interpenetrating bodies. MPIE-Bench and MPIE-Eval are built to close that gap with a geometry-grounded score instead of an image-level VLM opinion.

Who it affects

Researchers and labs building or benchmarking text-to-image and personalized editing models, especially anyone shipping features for group photos or multi-subject scenes, are the direct audience. It also affects anyone relying on VLM-as-judge scores to compare editing systems, since the paper's results suggest those scores can overstate real quality on multi-person contact scenes specifically.

How to use it

The paper positions MPIE-Bench as a benchmark dataset, 2,500 editing triplets across 405 scenes, 14 interaction categories, and four contact densities, that a model developer can run their own editor against, then score the outputs with MPIE-Eval's two axes. Anatomy checks whether all visible human-like mass is explained by a complete set of reconstructed bodies; Interaction checks whether the penetration and surface distance between bodies match what the edit instruction asked for. Both axes are computed from a frozen, publicly available multi-person mesh reconstruction model rather than requiring a VLM call. The source text does not state pricing, license terms, or a public release location for the benchmark or evaluation code.

How solid is it

The authors tested ten editors and report mesh Anatomy topping out at 0.65 and mesh Interaction topping out at 0.72, each achieved by a different one of the ten, so the benchmark is discriminating enough to show that no editor excels at both axes at once. They validated the metric itself with a five-rater human study, finding both mesh-based axes correlate with human judgment better than a zero-shot VLM judge does, and they checked robustness by ablating every weight and threshold in the scoring pipeline, reporting that the resulting editor rankings held.

Risks and caveats

The evaluation leans on a single frozen public multi-person mesh reconstruction model as its geometric ground truth, so any blind spot or bias in that reconstruction model could propagate into both the Anatomy and Interaction scores. The human validation study used five raters, a small panel for establishing that a metric tracks human judgment. The headline finding, that no editor scores well on both axes, also means the benchmark currently has no strong baseline to calibrate against beyond the ten editors tested.

“Across ten editors, mesh Anatomy tops out at 0.65 and mesh Interaction at 0.72 on two different models, so no single editor is strong on both, while VLM checklists rate the same images above 0.95.”

— Jiajia Lin and co-authors, MPIE-Bench paper