Survey maps ways to cut inference costs in VideoLLMs

Video large language models, or VideoLLMs, pair video representations with a pretrained language model and generate output conditioned on a text prompt. They already do well at captioning, question answering, retrieval and temporal grounding, but that performance comes at a computation and memory cost that grows with the number of frames processed and the length of the context. That growth is what keeps VideoLLMs out of real-time systems, mobile devices and other resource-constrained deployments.

A new survey pulls together the mechanisms researchers have proposed to bring that cost down. It only counts methods that report a concrete reduction in parameter count, FLOPs per input, latency, memory, or the number of visual and audio tokens the model has to process, rather than techniques that merely claim efficiency. The authors trace bottlenecks through four stages of the pipeline: how frames are sampled from the video, how video and audio are encoded into features, how a connector module reduces those features to a smaller set of tokens before they reach the language model, and how the language model itself handles prefilling and decoding.

Methods are grouped by which of those four stages they act on. The survey covers VideoLLMs built since late 2022, and it also includes older frame-sampling and vision-encoder techniques that predate that wave of models but are still used inside current pipelines. Where the literature allows it, the authors line up accuracy against cost for methods tested on the same host model and the same input setup, and they keep that evidence apart from comparisons pulled from different papers that used different models and protocols. They also point to two open gaps: audiovisual efficiency is underexplored, and the field lacks standardized evaluation.

Alongside the survey, the authors maintain a public repository at github.com/momentslab/awesome-efficient-videollm.

Key facts

The survey only counts methods with a concrete, reported reduction in parameter count, FLOPs per input, latency, memory, or visual and audio token count, not general efficiency claims.
It analyzes four pipeline bottlenecks: frame sampling, video and audio encoding, connector-level token reduction, and the language model's prefilling and decoding.
Coverage spans VideoLLMs built since late 2022, plus earlier frame-sampling and vision-encoder techniques that are still used in current pipelines.
Where possible, the authors compare accuracy against cost for methods tested on the same host model and input setup, keeping that apart from evidence pulled across different papers with different models and protocols.
The authors maintain a companion GitHub repository, momentslab/awesome-efficient-videollm, and name audiovisual efficiency and standardized evaluation as open gaps.

Why it matters

VideoLLMs already perform well at captioning, question answering, retrieval and temporal grounding, but every added frame and every extra token of context pushes up their compute and memory cost, and that cost is what keeps them out of real-time systems, mobile devices and other resource-constrained settings. Rather than proposing one more technique, this survey collects and organizes the efficiency work that already exists in the literature by where in the pipeline it acts, turning scattered papers into a single map of what has already been tried.

Who it affects

Anyone building or deploying a VideoLLM. Researchers designing new frame-sampling, encoding or token-reduction methods can see where their work fits against the rest of the field, and engineers trying to fit video understanding into real-time, mobile or otherwise resource-constrained products get a map of the pipeline stage where they are hitting a bottleneck, rather than a shelf of individual papers to work through.

How to use it

There is no product or price here: the entry point is the survey itself, read for whichever pipeline stage matches the bottleneck a project is hitting, whether that is frame sampling, encoding, connector-level token reduction, or the language model's prefilling and decoding, and the companion repository at github.com/momentslab/awesome-efficient-videollm, which the authors maintain.

How solid is it

The survey is a literature review, not new experiments, and it says so plainly: its accuracy-cost comparisons are literature-reported, and the authors separate the more reliable ones, methods tested on the same host model and input setup, from ones pulled across papers that used different models and protocols. The abstract names no authors or institutions, and it gives no concrete benchmark numbers, accuracy or speed figures for any individual method: it describes categories of reported gains rather than their size, so anyone wanting the actual numbers has to go to the individual papers it draws on.

Risks and caveats

The efficiency numbers a given method reports were measured under whatever setup that method's own paper used, not one shared benchmark: the survey lines up apples-to-apples comparisons where a shared host model and input protocol exist, but by its own account much of the evidence is still heterogeneous, gathered across papers that differ in model and setup. Audiovisual efficiency work is thin and the field lacks standardized evaluation, two gaps the survey names without resolving, so any specific speedup or memory-saving figure is best read next to the setup that produced it rather than treated as directly comparable across methods.

“Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings.”

— the paper's authors