Glonce

Lucida pipeline lifts real-to-sim scene detection mAP by 69%

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact65Novelty75Relevance95Surprise50Players30Credibility85
Lucida pipeline lifts real-to-sim scene detection mAP by 69%

Composable scene modeling tries to turn a real indoor scene into a set of complete, editable object assets arranged exactly as they were observed, so a robot simulator or an embodied AI system gets a simulation-ready replica of the room whose objects can be moved and manipulated one by one. The authors describe the standard approach as a three-step pipeline: parse the observations into separate object instances, generate a 3D asset for each instance, then place each asset back into the scene. Their claim is that every one of those three steps quietly assumes an input that a cluttered, real-world capture rarely provides: accurate instance-level geometry, views that are not blocked by other objects, and generated assets that actually match what the camera saw.

The paper proposes Lucida, which keeps the same three-step order but changes what each step is asked to rely on, so precision is reached progressively rather than demanded upfront. Lucida first parses an input video into a scene graph in which each node holds multi-view evidence for one object instance. From that evidence it generates a complete asset for each instance. It then places every asset using a new component called GizmoAct: a vision-language-model policy that treats placement as a multi-turn graphical interaction, manipulating the object's on-screen gizmo (the standard 3D move and rotate handle) in a closed loop and deciding for itself when the object is correctly aligned, rather than stopping after a fixed number of steps or a preset threshold.

The paper reports results across three separate evaluation tasks. On scene-level 3D object detection, measured on the R2S-Scene benchmark, Lucida improves mean average precision (mAP) over a baseline called Boxer by 69%. On object pose estimation, measured on CA-1M with the ADD-SB@0.05 metric, Lucida raises the score from 57.8% to 83.4%. On scene reconstruction, measured by scene F-Score, Lucida increases the result from 0.794, achieved by a method called SAM3D, to 0.924.

Key facts

  • Lucida is a real-to-sim pipeline that turns a captured video of an indoor scene into complete, individually editable 3D object assets arranged as observed.
  • It keeps the standard parse, generate, place structure but redistributes what each step requires, so precision builds up rather than being assumed from the start.
  • Placement is handled by GizmoAct, a VLM policy that manipulates each object's gizmo through multi-turn GUI interaction and decides on its own when alignment is reached.
  • On R2S-Scene, Lucida improves mAP over the Boxer baseline by 69%; on CA-1M, it raises ADD-SB@0.05 from 57.8% to 83.4%; on scene reconstruction, it raises scene F-Score from 0.794 (SAM3D) to 0.924.

Why it matters

Embodied AI and robot simulation need scenes that are not just visually accurate but functionally editable: individual objects that can be picked up, moved, or swapped out in simulation the way they would be in the real room. Prior pipelines split this into parsing, generating, and placing assets, but each step assumed clean inputs: accurate instance geometry, unobstructed views, and assets that match observations, all of which a cluttered real capture rarely delivers. Lucida's contribution is redesigning the same three-step structure so it tolerates the messier evidence a real capture actually provides, pushing precision to the end of the pipeline instead of requiring it at the start.

Who it affects

The result is aimed at researchers and engineers building robot simulation environments and embodied AI systems that need simulation-ready digital replicas of real physical spaces, where individual objects must remain separately manipulable rather than baked into a single static reconstruction.

How to use it

The source text describes the method and its benchmark results but does not state a release date, code availability, or whether the pipeline or its models have been published for others to use.

How solid is it

The paper reports quantitative improvements on three distinct, named benchmarks: a 69% relative gain in mAP over the Boxer baseline on R2S-Scene for scene-level 3D object detection, a jump in ADD-SB@0.05 from 57.8% to 83.4% on CA-1M for object pose estimation, and a rise in scene F-Score from 0.794 (SAM3D) to 0.924 for scene reconstruction. The source does not define these metrics beyond naming them, nor does it describe the compared baselines (Boxer, CA-1M, SAM3D) in further detail.

Risks and caveats

The source text does not name individual authors or their institutions, and gives no information on code, dataset, or model release. The reported gains come from the paper's own benchmarks and baselines; independent replication is not addressed in the available text.