MaP-WAM turns robot memory into plans, hits 83.3% on RMBench

Most robotic manipulation policies are built on a Markovian assumption: the next action depends only on the current observation. The authors argue that many real manipulation tasks are not like that: they are inherently non-Markovian and need memory reaching back beyond the current frame. Existing fixes, such as condensing history into language summaries, letting the visual context window grow, or combining the two, may lose fine-grained visual evidence or force a trade-off between how much history a policy retains and how efficiently it can execute.

MaP-WAM (Memory-as-Plans, World-Action Modeling) splits the problem into two separate jobs instead of forcing one model to keep replaying the full history. Completed parts of an episode are stored as segment records, each pairing a language instruction with a sparse set of visual context. A planning stage reads this episodic memory and compresses it into a compact plan for what comes next: a segment-level language instruction plus matching visual guidance. A separate World-Action-Progress (WAP) model then carries out that one plan, running for however long it takes. At each inference step it predicts both the next chunk of actions and how far along the current segment the robot has gotten; that progress estimate is checked against what the robot is actually observing, and the comparison is what triggers the move to the next segment and refreshes the closed-loop context, rather than the executor ever seeing the entire task history at once.

Because the executor only ever works from one compact plan, its context length stays fixed regardless of how long the episode has run, and the framework's structured attention design additionally allows key-value caching during both planning and execution. The authors report MaP-WAM reaching state-of-the-art performance on the RMBench benchmark, an 83.3% success rate, and separately attaining 78.0% success on real-robot tasks, while executor inference latency stays roughly constant even as the task history grows longer.

Key facts

MaP-WAM splits long-horizon robotic manipulation into a memory-grounded planning stage and a separate plan-conditioned execution stage, instead of feeding the executor the full task history.
Completed episode segments are stored as records pairing a language instruction with sparse visual context, then compressed into a compact plan: the next segment-level instruction plus matching visual guidance.
The World-Action-Progress (WAP) model jointly predicts action chunks and execution progress, calibrating progress against what the robot observes to trigger segment transitions and closed-loop context updates.
MaP-WAM reaches state-of-the-art performance on the RMBench benchmark with an 83.3% success rate, and separately attains 78.0% success on real-robot tasks.
Because the executor's context length stays fixed and structured attention enables key-value caching, inference latency stays roughly constant as task history grows.

Why it matters

Most robotic manipulation policies assume a Markovian world: the next action depends only on the current observation. The authors argue this breaks down for many real tasks, which are inherently non-Markovian and require memory reaching back beyond the current frame. Existing fixes, such as language summaries of the past, growing visual context windows, or a mix of the two, may lose fine-grained visual evidence or force a trade-off between how much history a policy retains and how efficiently it can execute. MaP-WAM's contribution is a different split: instead of making one model repeatedly digest the full history, it separates memory-grounded planning (reading long-term multimodal episodic context to produce a plan) from plan-conditioned execution (carrying out just that one plan). That decomposition is what lets the executor's context length stay fixed no matter how long a task runs.

Who it affects

The framework targets robotics and embodied-AI researchers building manipulation policies for long-horizon tasks, where a robot has to remember instructions or events from earlier in an episode rather than react only to what it currently sees. It is also relevant to anyone dealing with the compute cost of memory-augmented robot policies: approaches that keep growing the executor's context as an episode lengthens pay for it in inference latency, and MaP-WAM's fixed-context, structured-attention design is aimed directly at that cost.

How to use it

MaP-WAM works in two stages. Completed parts of an episode are stored as segment records, a language instruction paired with sparse visual context, and a planning step compresses that episodic memory into a compact plan: the next segment-level language instruction plus matching visual guidance. A World-Action-Progress (WAP) model then executes one plan at a time, jointly predicting action chunks and how far the current segment has progressed. That progress estimate is calibrated against what the robot actually observes, which is what triggers the move to the next segment and a closed-loop context update. Structured attention allows key-value caching during both planning and execution, while the executor's context length stays fixed regardless of episode length.

How solid is it

The two headline numbers, an 83.3% success rate on RMBench and 78.0% success on real-robot tasks, are the results given in the text. RMBench is named as a benchmark but not defined or described, and no prior state-of-the-art figure or named competing method is given, so the 83.3% figure has no visible baseline in the source to compare against. The real-robot evaluation is described even less: the text does not say what robot was used, what the tasks were, or how many trials were run. The claim that executor inference latency stays roughly constant as task history grows is stated as a result rather than shown with supporting numbers in the text.

Risks and caveats

The text does not name any authors or institutions, and no submission or publication date is given. There is also no mention of releasing code, model weights, or a dataset. Combined with the missing real-robot task details and the absence of a named baseline for the RMBench comparison, that leaves little in the text itself to check the two headline success rates against.