Glonce

SpecSlot splits image detail into DCT slots to cut attention cost

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact42Novelty55Relevance100Surprise25Players10Credibility75
SpecSlot splits image detail into DCT slots to cut attention cost

Pixel-space generative Transformers usually use one token per image patch. That means finer detail requires more tokens, and global attention grows increasingly expensive. The paper starts from an observation: finer detail does not necessarily require more global communication. Low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally.

Building on that, the paper introduces SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. It expands each spatial anchor into several slots, and each slot is assigned a disjoint band of DCT coefficients. This separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor takes part in global attention, while all slots interact locally. The model can therefore add spectral detail without enlarging the global sequence. A spatial head adds a local correction to the spectral prediction, and the generative state, target and loss all stay in pixel space.

The reported results are against JiT baselines. On ImageNet-256, SpecSlot reaches a comparable final FID to a FLOPs-matched JiT control and converges faster. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within 0.69 FID of JiT-B/16 at 38.2% lower cost. On text-to-image generation, it surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.

Key facts

  • SpecSlot turns each spatial anchor into multiple slots, each covering a disjoint band of DCT coefficients; only the lowest-frequency slot per anchor joins global attention.
  • On ImageNet-512 it improves FID by 2.84 over a FLOPs-matched JiT control while using 3.9% fewer FLOPs.
  • On ImageNet-512 it comes within 0.69 FID of JiT-B/16 at 38.2% lower cost.
  • On text-to-image generation it surpasses JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.
  • On ImageNet-256 the final FID is comparable to the FLOPs-matched JiT control, with faster convergence.

Why it matters

In pixel-space generators, adding detail normally means adding tokens, and global attention gets more expensive with every token. SpecSlot argues the two can be separated: low-frequency structure needs global exchange, while high-frequency detail can be handled locally. By letting only the lowest-frequency slot of each anchor join global attention, it raises spectral detail without lengthening the global sequence. The results are framed as quality at lower cost against JiT baselines.

Who it affects

Mainly researchers working on pixel-space generative Transformers and rectified-flow models, who face the trade-off between detail and attention cost. The paper's comparisons are with JiT variants on ImageNet class-conditional generation and on text-to-image generation.

How to use it

This is a method described in a paper abstract. No code, model weights or release is mentioned in the source, so there is nothing to download or run from this text. Practitioners can take the design idea: split each anchor into frequency-band slots, keep only the lowest band in global attention, and add a spatial head for local correction.

How solid is it

The claims rest on the paper's own benchmark results: FID on ImageNet-256 and ImageNet-512, and DPG-Bench for text-to-image. The source gives differences rather than absolute FID values, and the ImageNet-256 comparison has no numeric values or convergence speedup figure. The 38.2% figure is described only as lower cost, while the 40.6% figure is forward cost and the 3.9% figure is FLOPs.

Risks and caveats

The source gives only the abstract, so model sizes, parameter counts, training budget and hardware are unknown. The unit or scale of the DPG-Bench score is not stated beyond the 1.06 margin. The source mentions no comparison to latent-space diffusion models. On ImageNet-512, SpecSlot is still 0.69 FID behind JiT-B/16, and on ImageNet-256 it only matches the control on final FID.

“finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally”

— SpecSlot paper abstract