Microsoft's MindTopo finds VLMs fail at topology planning
Microsoft Research has introduced MindTopo, a benchmark that tests whether multimodal AI models understand 3D topology: structural relationships that persist as objects bend, stretch, or move, rather than exact distances, angles, or shapes. The blog post frames the idea with concrete questions: can a model tell whether two rooms stay connected after a wall goes up, whether an animal is inside or outside a fence, whether a rope is truly knotted rather than just tangled, or rearrange several ropes without letting them pass through one another.
MindTopo organizes its tasks around five categories, inspired by Piaget and other cognitive-science classifications of topological ability: continuity (does a path or object stay unbroken), separation (do nearby elements form one structure or distinct parts), order (how elements are arranged along a path or transformation), enclosure (does a boundary create an inside and an outside), and knots (is a rope actually knotted or linked, as opposed to merely tangled in appearance).
Each category is evaluated at two cognitive levels. Reasoning tasks show a model one or more rendered scenes and ask a question about their topological structure, such as whether two maze points are connected or a rope is truly knotted. Planning tasks put the model inside a simulated environment where it must choose actions that create, preserve, or remove a relation, for example rotating pipe segments, drawing a separating path, rearranging blocks, trapping a moving agent, or untangling ropes. The environments enforce legal actions, so a model cannot solve a rope puzzle by passing one strand through another. All scenes come from controlled simulators that provide exact ground truth and adjustable difficulty, which lets the researchers separate two different failure modes: a model failing because a scene looks visually complex, versus a model failing because it cannot keep track of the underlying relationship as objects move.
Across a broad set of proprietary and open-weight models, performance was consistently stronger on static reasoning than on interactive planning, and both remained well below human performance, a gap that widened when success depended on preserving a relationship across many actions. The error patterns point to where things break down: static mistakes usually started with perception, such as missing a wall, an opening, or a crossing, while planning mistakes showed up after the scene had already been understood correctly. Models would follow a locally plausible move without tracking its later consequences, lose track of the task over multiple turns, or propose an action that violated the environment's own physical rules.
The team also tested whether image and video generation could help models hold on to a topological relationship. Image generation sometimes helped when the relevant relation was visible within a single frame, but it stayed unreliable across a sequence of crossings or moves, and video rollouts frequently altered the topology or broke the task's dynamics outright. The researchers frame MindTopo as a diagnostic for a specific gap: robots, accessibility tools, and interactive assistants need to understand not just where objects are, but what stays connected, enclosed, ordered, or knotted as actions unfold, and closing that gap may require models that carry an explicit topological state, or world models whose predictions preserve topology by construction.
Key facts
- MindTopo organizes its tasks around five topological categories inspired by Piaget's classification of topological ability: continuity, separation, order, enclosure, and knots.
- Each category is tested at two cognitive levels: static reasoning over rendered scenes, and interactive planning inside simulated environments that enforce legal actions, so a model cannot solve a rope puzzle by passing one strand through another.
- Across a broad set of proprietary and open-weight models, performance was consistently stronger on static topological reasoning than on interactive planning, and both stayed well below human performance.
- Planning failures typically emerged after a scene was already understood correctly: models made locally plausible moves without tracking later consequences, lost track over multiple turns, or proposed actions that violated the environment's physical constraints.
- Generative tools gave limited help: image generation sometimes preserved topology when the relation fit in one frame, but video rollouts frequently altered topology or broke task dynamics.
Why it matters
Most spatial benchmarks for multimodal models test Euclidean properties such as distance, direction, size, and relative position. MindTopo targets a different layer: topological properties like connectivity, enclosure, order, and knottedness, which cognitive science treats as foundational to human spatial understanding but which AI evaluation has largely ignored. The benchmark's core finding, that recognizing topology in a still image is not the same as maintaining it while acting, has direct stakes for any system that has to operate in a changing physical or simulated scene rather than just describe one.
Who it affects
Researchers building and evaluating multimodal large language models, both proprietary and open-weight, are the immediate audience. The stated motivation reaches further, to developers of robots, accessibility tools, and interactive assistants, systems that must track what stays connected, enclosed, ordered, or knotted as their actions change a scene, rather than just perceive a single frame correctly.
How to use it
MindTopo is introduced as a diagnostic benchmark in a Microsoft Research blog post, covering five categories (continuity, separation, order, enclosure, knots), each tested at two levels: reasoning, where a model answers questions about static rendered scenes, and planning, where it selects actions inside a simulated environment, such as rotating pipe segments, drawing a separating path, rearranging blocks, trapping a moving agent, or untangling ropes. Scenes are generated from controlled simulators that provide exact ground truth and adjustable difficulty. The source does not give pricing or licence terms, a roster of the specific tested models, or release and availability details for the benchmark's code or data.
How solid is it
The findings come from a Microsoft Research blog post presenting the team's own study; the article does not disclose specific numeric scores or percentage gaps between static and interactive performance, nor does it name individual researchers or the particular models tested, referring only to a broad set of unnamed proprietary and open-weight models. The methodology has a built-in safeguard against a common confound: because environments enforce legal actions and simulators supply exact ground truth, the authors say they can separate failures caused by visually complex scenes from failures to maintain a relationship as objects move.
Risks and caveats
No publication date, dataset size, task or scene count, or details on releasing the benchmark's code or data appear in the source text. The results are self-reported by the team that built the benchmark, and the comparison rests on an unnamed set of proprietary and open-weight models rather than a listed, independently checkable roster.
“Robots, accessibility tools, and interactive assistants must understand not only where objects are, but also what remains connected, enclosed, ordered, or knotted as actions unfold.”
— Microsoft Research, MindTopo blog post