SmartMage routes modalities per query for 3D scene understanding
Researchers describe SmartMage, a multimodal large language model built for 3D scene understanding, the kind of joint reasoning over visual and geometric cues that embodied AI systems need to interpret a physical space. Their starting complaint is about existing multimodal LLMs: they typically process a fixed combination of modalities for every query, regardless of what the query actually needs. The authors argue this rigid setup lets semantic noise from irrelevant modalities leak in, underuses the modalities that would actually help, and ends up wasting computation on diluted reasoning.
SmartMage addresses this with two components. A Semantic-guided Modality Adaptive RouTng module, SMART, selects which modalities are relevant to a given task using semantic priors, alignment between text and modality, and a measure of modality quality. A second module, Modality-Aware Gating Expert (MAGE), uses modality priors to decide which experts in the model get activated, so the system can specialize adaptively across different kinds of multimodal reasoning rather than routing every query the same way.
On the results side, the authors report that SmartMage achieves state of the art performance across five 3D scene understanding benchmarks and also posts competitive results on benchmarks that use RGB only video, without geometric input. To probe why the dynamic routing helps, they built a diagnostic benchmark called ScanFacet that splits tasks into fine grained semantic categories, which lets them examine which modality combinations each type of task actually prefers. The patterns they observe there are offered as further evidence that the dynamic, query dependent routing is doing real work rather than just adding complexity.
Key facts
- SmartMage dynamically selects and weights modalities such as visual and geometric cues on a per query basis, instead of relying on one fixed combination for every task.
- A SMART module (Semantic-guided Modality Adaptive RouTng) picks relevant modalities using semantic priors, text-modality alignment, and modality quality.
- A MAGE module (Modality-Aware Gating Expert) uses modality priors to guide which experts activate, enabling adaptive specialization in multimodal reasoning.
- The authors report state of the art results on five 3D scene understanding benchmarks and competitive results on RGB-only video understanding benchmarks.
- A diagnostic benchmark, ScanFacet, breaks tasks into fine-grained semantic categories to show which modality combinations each type of task prefers.
Why it matters
The paper targets a specific inefficiency in multimodal LLMs used for 3D scene understanding: feeding every query the same fixed bundle of modalities. The authors argue that approach lets irrelevant modalities introduce semantic noise while leaving more informative ones underused, which wastes computation and dilutes reasoning. SmartMage's pitch is that deciding what to attend to on a per-query basis, rather than by a fixed schedule, fixes that mismatch directly.
Who it affects
This is aimed at researchers and engineers building embodied AI and 3D scene understanding systems, robotics perception stacks, and multimodal LLMs more broadly, anywhere a model has to combine visual and geometric information and the relevant mix genuinely changes from query to query.
How to use it
The paper points to a project page for SmartMage (https://yuecheong.github.io/SmartMage/); the source text does not give release details, licensing terms, or a code repository, so nothing beyond that link can be stated here.
How solid is it
The claims come from the paper's own text on Hugging Face Papers: SmartMage is reported to beat prior methods on five 3D scene understanding benchmarks and to be competitive on RGB-only video benchmarks, backed by the ScanFacet diagnostic analysis. The available text does not name the specific benchmarks, give numeric scores, or list authors and affiliations, so those specifics cannot be checked from what is here.
Risks and caveats
Because the source material does not disclose concrete benchmark names, scores, model size, training data, or authorship, the state-of-the-art claim and the modality-routing results rest on the paper's own framing rather than independently verifiable figures in the available text.