MoE-ViE's largest model matches a SOTA encoder's performance at 76% of the latency

Vision encoders are a core building block of vision-language models, and scaling their capacity generally improves performance. But dense scaling, the standard approach, increases both compute cost and inference latency. Mixture-of-Experts (MoE) architectures already scale large language models efficiently, but according to the authors, the MoE design space for CLIP-style vision encoders had not been explored at state-of-the-art (SOTA) levels before this work.

The authors systematically studied MoE designs for vision encoder scaling and found that fine-grained MoE topologies yield substantial gains over both dense encoders and standard MoE encoders. On top of that finding, they built two supporting pieces: an auxiliary-loss-free balancing variant for better expert utilization, and a specialized MoE kernel to reduce inference latency overhead. To extend the encoder to video while preserving what it already knows about images, they also introduce frame-level distillation paired with a new freezing mechanism.

The team pretrained a series of MoE-ViE models across a range of sizes, and every size consistently outperformed its dense counterpart. The largest MoE-ViE model matches the zero-shot performance of a state-of-the-art encoder 1.7 times its size, while needing only 76% of that encoder's latency. When paired with a language model, MoE-ViE surpasses every compared encoder on image and video benchmarks, including some with up to 5 times more activated parameters.

The paper, titled "MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding," was posted to Hugging Face's Papers listing; its abstract names no individual authors, identifies its institution as Meta only through its code URL rather than by name, gives no publication or submission date, and does not name the SOTA encoder or the benchmarks used for comparison. Beyond the three relative figures above (1.7x, 76%, 5x), no absolute accuracy, throughput, or parameter-count numbers, nor any training data or compute details, appear in the abstract. The authors say the code is available on GitHub, at facebookresearch/moe_vie.

Key facts

MoE-ViE is a new family of Mixture-of-Experts vision encoders, pretrained across a range of sizes; every size outperforms its dense counterpart.
The largest MoE-ViE model matches the zero-shot performance of a state-of-the-art (SOTA) vision encoder 1.7 times its size, while using only 76% of that encoder's latency.
When paired with a language model, MoE-ViE beats every compared encoder on image and video benchmarks, including some with up to 5 times more activated parameters.
The design combines fine-grained MoE topologies, an auxiliary-loss-free method for balancing load across experts, and a specialized MoE inference kernel built to cut the latency overhead MoE routing normally adds.
Video understanding was added through frame-level distillation paired with a new freezing mechanism, meant to build video capability without degrading the encoder's existing image performance; code is public on GitHub at facebookresearch/moe_vie.

Why it matters

Dense vision encoders get better as they scale up, but scaling them densely also raises compute cost and inference latency. Mixture-of-Experts already solved a version of that trade-off for large language models, but the authors say the MoE design space for CLIP-style vision encoders, the kind that connect images to language models, had not been pushed to match state-of-the-art dense encoders before this work. That gap matters because a vision encoder's cost is paid on every image or video frame a vision-language system processes. The paper's central result, a MoE encoder matching a much larger dense encoder's performance while using well under its full latency, is direct evidence that MoE is a viable alternative to dense scaling for vision encoders, not just for language models.

Who it affects

The immediate audience is researchers and engineers who build or fine-tune vision-language models and currently rely on CLIP-style dense vision encoders as the image or video front end. Teams that care most about inference latency and serving cost, since encoder cost is paid on every image or video frame a system processes, stand to benefit directly if the approach holds up outside the paper's own benchmarks.

How to use it

This is a research contribution rather than a shipped product. The authors say the code is available on GitHub, at facebookresearch/moe_vie, which gives teams building their own vision-language models something concrete to review or build on: the MoE topology, the balancing method, and the training recipe described in the paper.

How solid is it

The claims rest entirely on the paper's own abstract, posted on Hugging Face's Papers listing under the id 2608.17402. The abstract names no individual authors and carries no publication or submission date. It does not name an institution in prose either, but its code URL identifies Meta as the lab, so there is a track record to check the claims against beyond the text itself. The three headline figures, 1.7x, 76%, and up to 5x, are all relative comparisons: no absolute accuracy, throughput, or parameter-count numbers appear anywhere in the abstract, and none of the results have been verified outside the authors' own testing. The code being public on GitHub, at facebookresearch/moe_vie, is the clearest path to independent verification: reproducing the pretraining runs and rechecking the comparisons is possible in principle, even though the abstract alone does not confirm them.

Risks and caveats

The abstract does not name the SOTA encoder MoE-ViE's largest model is compared against, the other compared encoders, or the specific image and video benchmarks used, so there is no way to judge from this text alone whether those were the strongest available baselines. The 76% latency figure is a single number without hardware or serving-setup details attached, and latency can vary with both, so it may not transfer directly to a different deployment. No training data, compute budget, or hardware details are given at all, which leaves the total cost of reproducing or adopting the approach, as distinct from its claimed inference latency, unclear from the abstract.