Glonce

OmniPack keeps 98% of the original performance at 16.7% of the FLOPs

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact65Novelty65Relevance95Surprise45Players50Credibility85
OmniPack keeps 98% of the original performance at 16.7% of the FLOPs

Omni-modal large language models, or Omni-LLMs, that reason jointly over audio, video and text have gotten good at audio-visual understanding tasks, but processing the long, highly redundant token sequences that video and audio produce is computationally expensive. Deploying these models efficiently requires aggressive token compression, and the authors argue that existing compression methods break down at low token budgets in two different ways. Compressing tokens before they reach the LLM can discard evidence that is structurally important but scattered across the input, since that stage has no way to know what the query will later ask about. Compressing inside the LLM, by contrast, tends to underuse the collaboration between audio and visual signals that is conditioned on the specific query.

OmniPack is a training-free framework built to avoid both failure modes by combining the two stages instead of relying on just one. Before the LLM, it strips out structural redundancy using three criteria: modality-specific importance, global coverage, and similarity-aware merging. After the model has had enough multimodal interaction to engage with the query, OmniPack runs a second pass inside the LLM that consolidates the remaining, task-relevant representations, guided by the text of the query and by collaboration between the audio and visual streams. Because it requires no additional training, it is designed to be applied directly to an already-trained Omni-LLM.

The authors tested OmniPack on five benchmarks across three Omni-LLM backbones, though only one of those backbones, Qwen2.5-Omni-7B, is named in the text available here. On Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the model's original, uncompressed performance while cutting FLOPs to 16.7% of the original; pushed to a more aggressive setting, it still retains 92.9% of original performance while running on just 6.8% of the original FLOPs. Across the five benchmarks and three backbones, the authors report that OmniPack consistently delivers the best performance-efficiency trade-off among the methods tested, at every retention ratio tried, outperforming all existing methods they compared it against.

Key facts

  • OmniPack is a training-free framework that compresses the audio and visual tokens omni-modal LLMs (Omni-LLMs) process, aiming to cut inference cost without retraining the model.
  • It combines structural compression before the LLM (modality-specific importance, global coverage, similarity-aware merging) with task-relevant semantic refinement inside the LLM (textual guidance and audio-visual collaboration).
  • On Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while cutting FLOPs to 16.7%, and still retains 92.9% of performance at just 6.8% of the original FLOPs.
  • Across five benchmarks and three Omni-LLM backbones, the authors report OmniPack consistently gives the best performance-efficiency trade-off among tested methods, at every retention ratio tried.
  • Existing compression methods fail differently at low token budgets: doing it before the LLM can discard structurally important, widely distributed evidence, while doing it inside the LLM often underuses audio-visual signals tied to the specific query.

Why it matters

Token compression is the main lever for making omni-modal LLMs cheap enough to run in production, but pushing it too far usually costs accuracy. OmniPack's contribution is running compression in two coordinated stages, structural cuts before the LLM and query-aware semantic consolidation inside it, rather than committing to only one approach the way prior methods did. Because it needs no retraining, it is presented as something that could be layered onto an existing Omni-LLM rather than requiring a new model to be built from scratch. The claimed payoff, staying within a couple of points of original performance while running on a fraction of the compute, would matter to anyone trying to make audio-visual AI systems fast and cheap enough to deploy at scale.

Who it affects

The immediate audience is researchers and engineering teams building or deploying omni-modal LLMs, models that process audio, video and text together, particularly where inference cost, latency or hardware budget is a constraint. It is also directly relevant to researchers working on token compression and efficient multimodal inference, since the paper positions OmniPack as beating existing methods on that specific problem. The text available here does not name the authors or their institution, so it is not possible to say which lab or company is behind the work.

How to use it

OmniPack is training-free, meaning it is applied to an already-trained Omni-LLM at inference time rather than requiring retraining or fine-tuning: a structural compression pass before the LLM removes redundant audio and visual tokens, then a second pass inside the LLM further consolidates what is left, guided by the query text and by audio-visual collaboration. The text available here does not say whether code, model weights or the evaluation datasets have been or will be released, and gives no publication date, so it is not possible to say from this alone whether or when the method can be tried directly.

How solid is it

The results come from the authors' own experiments: five benchmarks and three Omni-LLM backbones, though the text available here names only one of the backbones, Qwen2.5-Omni-7B, and does not identify the five benchmarks or the specific baseline methods OmniPack was compared against. The two headline figures, 98.0% of performance at 16.7% of the FLOPs, and 92.9% of performance at 6.8% of the FLOPs, are both reported specifically for Qwen2.5-Omni-7B; results on the other two backbones are not given in the text available here. The broader claim that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods, is the authors' own summary of their results, not an independently verified comparison.

Risks and caveats

Even the better-preserved setting still gives up 2.0% of the model's original performance, and the more aggressive one gives up 7.1%, in exchange for a much larger compute saving; whether that trade-off is worth it depends on the task and how much accuracy loss is tolerable, and the text available here does not break that down beyond the two headline numbers. No absolute latency, memory or dollar-cost figures are given, only performance and FLOPs as percentages of the uncompressed baseline, so the real-world speed or cost benefit is not quantified here. With no authors, institution, release plan or publication date stated in the text available, the work cannot currently be traced back or independently checked beyond what is in this abstract.