Frame selection, not compression, is the real bottleneck in long-video AI
Long-video multimodal language models cannot examine every frame of a video: sampling once per second turns a single hour into 3,600 images, and any real system keeps only a small, fixed slice of that pool for the model to look at. Researchers behind this study argue that which frames make that cut is usually treated as a minor preprocessing detail, and set out to test whether it should be. Comparing published frame selectors directly is hard because each one changes several things at once, the frame scorer, the prompt boundary, the resolution policy and the answering model, so the study holds each of these fixed and varies one decision at a time: which frames get selected, how much each selected frame is spatially compressed, and whether the tokens freed by compression are reinvested into more frames. The comparison spans six training-free selection rules, three long-video benchmarks and two answering models, all run inside one controlled harness. Selection turns out to be the largest single lever of the three: on the hour-long bin of the LongVideoBench benchmark, picking eight frames chosen for relevance to the query beats sampling sixteen frames evenly spaced through the video by 6.9 points. One of the six selectors, Orthogonal Matching Pursuit, an unmodified, decades-old sparse-approximation algorithm with no video-specific design, matches or comes within one point of every purpose-built selector tested against it, across all three benchmarks. Spatial compression, by contrast, is nearly free on its own: halving each frame's spatial token budget at fixed timestamps costs at most 0.44 points of accuracy. That saved budget only becomes useful once it is reinvested: spending it on twice as many compressed frames, at a total token cost no higher than the original eight full frames, recovers a further two to three points of accuracy. The authors note that compression only pays off when handled this way, as a source of extra frames rather than a saving kept on its own. The controlled harness also surfaced two data points about how noisy cross-paper comparisons in this area can be: the authors found an implementation bug in their own reimplementation of the AKS baseline selector, and measured a gap of 0.07 to 3.74 points between two different harnesses running the same published selection rules at the same token budget.
Key facts
- Sampling one hour of video once per second produces 3,600 frames, far more than any system can feed to a model, making frame selection unavoidable.
- On LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced frames by 6.9 points of accuracy.
- Orthogonal Matching Pursuit, a decades-old, unmodified sparse-approximation algorithm, matches or comes within one point of every purpose-built frame selector tested, across all three benchmarks.
- Halving each frame's spatial token budget costs at most 0.44 points of accuracy, but reinvesting the freed tokens into twice as many compressed frames recovers a further two to three points.
- Running published selection rules inside one harness versus another produced a 0.07 to 3.74 point gap at the same token budget, and exposed a bug in the study's own AKS baseline reimplementation.
Why it matters
Most work on long-video multimodal models has treated frame selection as a preprocessing detail and focused engineering effort on compressing or encoding the frames a model does see. This study isolates frame selection as its own variable and finds it is the largest lever of the three tested, larger than spatial compression. That reorders where effort should go: a system built around a strong selector plus cheap compression can beat one that spends its complexity budget on compression alone.
Who it affects
The result matters to teams building or evaluating long-video multimodal models and video question-answering systems, where a fixed token or frame budget forces a choice about which parts of a video the model actually sees. It also matters to researchers benchmarking frame selectors against each other, since the study shows that comparisons run across different papers and harnesses can disagree by up to 3.74 points on identical rules at the same budget.
How to use it
The practical recipe the study supports is to prioritize a relevance-based frame selector over uniform sampling, treat spatial compression of selected frames as nearly free, and reinvest the tokens freed by compression into additional compressed frames rather than banking the saving. The authors report that Orthogonal Matching Pursuit, a generic sparse-approximation algorithm with no video-specific engineering, is competitive with selectors purpose-built for this task, which lowers the bar for teams without the resources to build a custom selector.
How solid is it
The claims rest on a controlled comparison across six training-free selection rules, three long-video benchmarks and two answering models, with each design decision, selection, compression and reinvestment, varied one at a time rather than bundled together as prior published selectors did. The study also reports its own harness-versus-harness gap of 0.07 to 3.74 points on identical published rules, which is evidence the authors use to argue for exactly this kind of single-harness comparison rather than trusting numbers reported across separate papers.
Risks and caveats
The source text names no authors or institutional affiliations, no publication venue or date, and does not name the two harnesses compared for the 0.07 to 3.74 point gap, the two answering models used, or the benchmarks beyond LongVideoBench. All reported figures are point-margin differences in accuracy rather than absolute accuracy percentages, so they describe relative gaps between conditions rather than how well any system performs in absolute terms. The study also disclosed an implementation bug in its own reimplementation of the AKS baseline, a reminder that even a controlled harness built to fix cross-paper noise can introduce its own errors.