Glonce

Paper adapts pretrained LlamaGen and RAR image models for inpainting at 13-33% extra compute

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance100Surprise30Players10Credibility75
Paper adapts pretrained LlamaGen and RAR image models for inpainting at 13-33% extra compute

Autoregressive image generators produce an image token by token in a fixed raster-scan order. The paper's authors say this hardcodes a single factorization of the image distribution into the model, so it cannot inpaint, outpaint or edit at inference time.

Existing fixes, including the modern discrete-diffusion family, train from scratch on a mixture of token orderings. According to the authors, that costs roughly 3x the compute of standard raster-scan training. Discrete diffusion also gives up compatibility with the autoregressive inference ecosystem (KV caching, vLLM) by construction. The authors argue this cost is unnecessary: a pretrained raster-scan model has already learned the joint distribution over image tokens, and what it lacks is the ability to express that knowledge through non-sequential conditional pathways.

They propose a recipe to adapt pretrained raster-scan models into any-order generators, with two components. The first is future-aware positional embeddings, which resolve a target-position identifiability problem at negligible overhead. The second is an ordering curriculum, which traces a stable optimization path from the pretrained solution to a good any-order basin.

The recipe was applied to four base models across two architectures and two scales: LlamaGen-L, LlamaGen-XL, RAR-L and RAR-XL. The authors report that it substantially outperforms MaskGIT on ImageNet inpainting (FID 6.21 versus 10.44, lower is better) and on outpainting, and that it is competitive with MAR-L. The flexibility has a price in image quality: a 1.2-1.9 increase in class-conditional FID. It also costs 13-33% additional compute beyond pretraining, while leaving the standard AR inference stack untouched.

The authors conclude that inpainting, outpainting and editing need not be added to autoregressive image models; they can be unlocked from the ones that exist. Code is available at https://github.com/Ordered-AR/unlockedadaptation.

Key facts

  • The recipe adapts pretrained raster-scan autoregressive image models into any-order generators, enabling inpainting, outpainting and editing.
  • It has two parts: future-aware positional embeddings (negligible overhead) and an ordering curriculum that moves the model from the pretrained solution to an any-order one.
  • It costs 13-33% additional compute beyond pretraining, against roughly 3x the compute of raster-scan training for existing from-scratch approaches, according to the authors.
  • On ImageNet inpainting it reaches FID 6.21 versus 10.44 for MaskGIT, at the price of a 1.2-1.9 rise in class-conditional FID.
  • Tested on LlamaGen-L/XL and RAR-L/XL; the standard AR inference stack is left untouched and code is public.

Why it matters

Autoregressive image models fit well with the inference tooling built around language models, such as KV caching and vLLM, but a fixed raster-scan order stops them from filling in or extending parts of an image. The usual remedies retrain from scratch on mixed token orderings at roughly 3x the compute, and discrete diffusion gives up AR tooling compatibility. This paper argues that existing pretrained models can be converted instead, for 13-33% extra compute.

Who it affects

Teams that already hold pretrained raster-scan AR image generators, such as LlamaGen or RAR checkpoints, and want inpainting, outpainting or editing without moving to discrete diffusion or retraining from scratch. It also concerns people who rely on the AR inference stack and want to keep it.

How to use it

The authors release code at https://github.com/Ordered-AR/unlockedadaptation. The method starts from a pretrained raster-scan model, adds future-aware positional embeddings, and trains with an ordering curriculum. It was evaluated on LlamaGen-L/XL and RAR-L/XL.

How solid is it

The results are the authors' own, reported on four base models across two architectures and two scales, with ImageNet inpainting compared against MaskGIT (FID 6.21 versus 10.44) and a competitive showing against MAR-L. The source does not say the results have been peer-reviewed. The source names no authors or institutions.

Risks and caveats

The flexibility is not free: class-conditional FID rises by 1.2-1.9, and the extra compute is 13-33% on top of pretraining. The source gives no FID values for outpainting or for MAR-L, and does not say which base model produced the 6.21 figure. It gives no absolute compute figures and describes no qualitative image examples or editing benchmarks. The curriculum and embeddings are described only in one-line summaries.