Fixed steering vectors recover 90-96% of RL gains in image diffusion models

Fixed steering vectors recover 90-96% of RL gains in image diffusion models

A paper listed on Hugging Face Papers asks how much of the improvement from reinforcement learning (RL) post-training of diffusion transformers can be explained by something very simple. RL post-training is known to substantially improve visual quality, text rendering and compositional accuracy. Because rendering requested text or composing several objects depends on the prompt, the natural hypothesis, as the abstract puts it, is that improving these abilities requires learning complex, input-dependent transformations.

To test that, the authors probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. The vectors are identical for every prompt, token and sampling seed. They are fit to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. On SD3.5-M, the vectors hold 65x fewer parameters than the RL-tuned LoRA: 0.29M against 18.8M.

Despite being input-independent, the vectors recover 90.0-96.3% of the reward gains of six RL-tuned image models. On two RL-tuned video models they recover 89.8% and 111.9% of the gains, so in one case the figure is above the RL-tuned model's own gain.

The authors also train the vectors directly. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M.

Their conclusion: a constant shift suffices for most of the gain of policy-gradient RL post-training. Without new input-dependent transformations, RL steers the pretrained generator toward what it can already compute.

Key facts

  • A fixed set of steering vectors added to the block outputs of a frozen base diffusion transformer recovers 90.0-96.3% of the reward gains of six RL-tuned image models.
  • On two RL-tuned video models the vectors recover 89.8% and 111.9% of the gains.
  • On SD3.5-M the vectors have 0.29M parameters against 18.8M for the RL-tuned LoRA, 65x fewer.
  • The vectors are fit to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective, and are identical for every prompt, token and seed.
  • Trained from zero with online RL, the vectors alone exceed weight-based RL on OCR and PickScore on SD3.5-M.

Why it matters

The common assumption is that capabilities such as text rendering and multi-object composition need prompt-dependent changes to the model. This paper tests that assumption with a constant, input-independent shift and finds it explains most of the gain. The authors conclude that policy-gradient RL post-training mostly steers the pretrained generator toward what it can already compute, rather than adding new input-dependent transformations.

Who it affects

The result is aimed at people who study or run RL post-training of diffusion transformers for image and video generation. It bears on how they read what RL fine-tuning actually changes in a model, and on how large the learned update needs to be.

How to use it

The source presents the steering vectors as a probe to isolate how much of the RL gain an input-independent shift explains. It describes two ways of obtaining them: fitting them to public RL-tuned checkpoints without any reward, or initializing them at zero and training with online RL. No code release, compute cost or training time is mentioned.

How solid is it

The claims come from the paper's abstract as listed on Hugging Face Papers, so the detailed experiments are not visible here. The numbers are specific: 90.0-96.3% across six image models, 89.8% and 111.9% on two video models, and 0.29M against 18.8M parameters on SD3.5-M. These percentages are shares of each RL-tuned model's reward gain, not absolute scores.

Risks and caveats

The parameter comparison is stated for SD3.5-M only. The names of the other five image models and the two video models are not given, nor are the reward metrics behind the 90.0-96.3% range, other than OCR and PickScore in the online-RL experiment. The size by which the vectors beat weight-based RL on those two metrics is not given. No authors or institutions are named, and no publication date. Recovering most of a reward gain is not the same as matching the RL-tuned model on every measure.

“A constant shift thus suffices for most of the gain of policy-gradient RL post-training”

— Paper abstract, Hugging Face Papers