Glonce

ReactHuman benchmark finds MLLMs mishandle one in three hazards

researchThe Glonce editorsHugging Face🔥76Glonce rating76Impact70Novelty78Relevance95Surprise60Players40Credibility87
ReactHuman benchmark finds MLLMs mishandle one in three hazards

Researchers introduced ReactHuman, described as the first physics-grounded benchmark for human-like reactive decision-making in embodied multimodal LLMs (MLLMs). In it, the evaluated model acts as the decision core of a simulated humanoid that is suddenly confronted with a household hazard, the kind a person reacts to instantly: a slipping plate, a falling knife. The authors argue that existing evaluations do not measure this at all, since they either probe intuitive physics passively through question answering over video, or target slow, deliberate, long-horizon tasks such as navigation and rearrangement, rather than immediate, safety-critical action.

The benchmark spans 17 event families and more than 1,000 bit-for-bit reproducible scenes, with exact, annotation-free ground truth derived from a 240 Hz rigid-body physics simulation. Some scenes use adversarial objects whose appearance contradicts their actual physics, such as a foam anvil or a steel apple, specifically to test whether a model reasons from an object's real physical behavior or is fooled by how it looks. Each model's reaction is scored by a five-metric suite along three axes: whether the reaction is reasonable, safe, and physically grounded. Every committed plan is then physically executed in the simulation, so a model's decision has observable consequences rather than being scored in the abstract.

Seven representative MLLMs were evaluated on the benchmark. The results show reactive safety is far from solved: the models mishandle roughly one hazard in three. They tend to act from fixed dispositions rather than from the scene actually in front of them, and they trust an object's appearance over its motion. Even when a model picks the correct action, it can miss the interception point by a meter or so. None of these failure patterns shrink as model scale increases. The authors present ReactHuman as both a fine-grained diagnostic tool and a scalable training signal for building physically grounded, safety-aware embodied agents, and have published the benchmark as a dataset on Hugging Face.

Key facts

  • ReactHuman spans 17 event families and over 1,000 bit-for-bit reproducible scenes, with ground truth derived from 240 Hz rigid-body simulation.
  • Seven representative MLLMs were evaluated as the decision core of a simulated humanoid reacting to sudden hazards like a falling knife or a slipping plate.
  • Models mishandle roughly one hazard in three, trust appearance over motion, and miss interception points at meter scale even when they choose the right action.
  • None of these failure patterns shrink with model scale, so larger models are not proportionately safer at this task.
  • Adversarial objects such as a foam anvil and a steel apple test whether a model reasons from an object's real physics rather than its look.

Why it matters

Deploying MLLMs as the decision core of household robots requires converting physical understanding into split-second, safety-critical action, and the authors say no prior evaluation measured that directly: existing benchmarks either quiz models about physics passively through video question answering, or test slow, deliberate tasks like navigation and rearrangement. ReactHuman is presented as the first benchmark built specifically to test immediate reactive response to a sudden hazard, with a simulated humanoid standing in for the robot.

Who it affects

Anyone building or evaluating MLLMs intended to control physical or household robots, and researchers designing safety evaluations for embodied AI more broadly, since the benchmark is framed as a diagnostic and training signal rather than a product.

How to use it

The benchmark is published as a dataset on Hugging Face, linked from the paper, for running the same reproducible scenes against other models. The source text does not give pricing or licensing terms.

How solid is it

Ground truth is annotation-free and derived from 240 Hz rigid-body simulation, and the 1,000-plus scenes are bit-for-bit reproducible, which removes labeling subjectivity from scoring. Each committed plan is physically executed rather than judged in the abstract, so a model's decision has a real, observable outcome in the simulation. Seven models were evaluated across 17 event families using a five-metric suite scored along three axes: reasonable, safe, and physically grounded.

Risks and caveats

The source text does not name the seven evaluated models or give numeric per-model accuracy scores, only qualitative failure descriptions; it names no authors or institutional affiliations and gives no publication or release date; and it does not state whether the reported failures were tested for statistical significance. Separately, the core finding is itself a caveat for deployment: models trusting an object's appearance over its actual motion, and falling back on fixed dispositions instead of reading the scene, are exactly the failure modes that matter most in a real household hazard.