Glonce

Routing Policy Optimization uses attention routing for diffusion RL, claims nearly 4x training efficiency

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact45Novelty60Relevance95Surprise30Players10Credibility75
Routing Policy Optimization uses attention routing for diffusion RL, claims nearly 4x training efficiency

Reinforcement learning can align visual generators with reward signals that go beyond their pretraining objectives. The paper's authors say the existing score-function methods usually get the randomness they need for exploration by perturbing the reverse denoising trajectory, and that this leads to costly rollout-based optimization. Their question is whether stochastic exploration can instead be introduced inside the generator while its original sampler stays untouched.

Their answer is Routing Policy Optimization (RPO), which turns attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition. A Gaussian likelihood around the dense velocity then scores the perturbation with a score-function objective.

RPO also uses what the paper calls a controlled two-rollout estimator. Two rollouts share the same prompt, initial latent, routed denoising levels and dense anchor layers, and differ only in their sampled attention routes. The difference between their rewards provides the advantage signal for the routed transitions.

Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and keeps the original deterministic sampler, so the trained model is used the way it was before.

The abstract says experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. It points to a project page at https://velox-world.com/rpo for visual results.

Key facts

  • RPO gets stochastic exploration for RL by randomly routing attention inside the generator, rather than perturbing the reverse denoising trajectory.
  • Routing is applied at only a few selected denoising levels: each query block keeps its own block and samples further key blocks in proportion to native attention mass, or keeps dense attention.
  • A controlled two-rollout estimator uses two rollouts that share prompt, initial latent, routed denoising levels and dense anchor layers; their reward difference is the advantage signal.
  • At inference RPO restores dense attention and retains the original deterministic sampler.
  • The abstract reports improved quality on text-to-image and text-to-video and nearly 4x higher training efficiency than Flow-GRPO-style methods.

Why it matters

Aligning image and video generators with reward signals through reinforcement learning is costly, and the authors point to rollout-based optimization as the expensive part. Existing score-function methods typically create randomness by perturbing the reverse denoising trajectory. RPO moves the randomness into the attention layers instead, and leaves the generator's sampler as it was. If the claimed nearly 4x gain in training efficiency over Flow-GRPO-style methods holds up, that would make reward-based tuning of visual generators cheaper.

Who it affects

The work is aimed at researchers and engineers who tune diffusion or flow-based visual generators, for text-to-image and text-to-video, with reinforcement learning. Because RPO restores dense attention and the original deterministic sampler at inference, the claim is that the trained generator is used at inference as before.

How to use it

The abstract describes the method in enough detail to follow the recipe: pick a few denoising levels, route attention there by sampling key blocks in proportion to native attention mass (or keep dense attention), run two rollouts that differ only in their routes, and use their reward difference as the advantage. It points to a project page at https://velox-world.com/rpo for visual results. The abstract does not say whether code or weights are released.

How solid is it

This is a paper abstract. It states that RPO improves generation quality on text-to-image and text-to-video generation, but it gives no numeric quality scores or improvements. No benchmark names, datasets, reward models or base generator models are given. The nearly 4x figure is a ratio against Flow-GRPO-style methods, and the abstract does not say how training efficiency is measured (wall-clock, number of rollouts, GPU hours) or which Flow-GRPO-style methods are compared. Results are the authors' own claims.

Risks and caveats

The abstract mentions no limitations, failure cases or compute costs. Without the measurement basis for the efficiency claim and without quality numbers, the size of the real gain is unclear. The claim that learned updates stay effective after routing is removed rests on the authors' argument that the likelihood differentiates the dense velocity itself; the abstract offers no figures to test it.

“At inference, RPO restores dense attention and retains the original deterministic sampler.”

— Abstract of the Routing Policy Optimization paper