OmniXRef edits video from mixed text, image and video references
OmniXRef is a framework for editing an existing video using several kinds of guidance at once. Creative intent, the authors note, can be expressed as a text instruction (for example, "remove the people"), as an image reference (for example, a subject's appearance), as a video reference (for example, camera movement), or as a combination. OmniXRef lets a user combine variable numbers of text, image and video references to control different attributes of a source video.
The paper has several parts. The input design uses a reference-bank interface, and the authors introduce R^2-RoPE to support generalization to more references. The model, called OmniXRef-Model, uses a single-stream diffusion transformer. It processes the source video together with multimodal context from an MLLM, arranged in one interleaved sequence.
Training data for video-reference editing is scarce, so the authors built a dataset, OmniXRef-Data, using synthesis pipelines for different editing tasks and combinations of references. Training uses curriculum learning, which raises task complexity and resolution step by step.
To evaluate video as a control signal, the authors also introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer. It covers five categories: Camera, Motion, Content, Style, and Visual Effects (VFX). The authors report that comprehensive evaluations show OmniXRef-Model outperforms task-specific baselines and commercial models.
Key facts
- OmniXRef is a unified framework for video editing that accepts variable numbers of text, image and video references to control attributes of a source video.
- OmniXRef-Model is a single-stream diffusion transformer that processes the source video and MLLM-derived multimodal context in one interleaved sequence; R^2-RoPE is introduced to generalize to more references.
- OmniXRef-Data is built with synthesis pipelines for different editing tasks and reference combinations, and training uses curriculum learning over task complexity and resolution.
- OmniXRef-Bench evaluates video-reference attribute transfer across five categories: Camera, Motion, Content, Style and VFX.
- The authors report the model outperforms task-specific baselines and commercial models.
Why it matters
The authors note that creative intent in video editing can be expressed through text, image or video references, or a combination of them. This work tries to unify text, image and video references in a single model, so a user could say what to change in words, show a subject with an image, and show a camera move with a clip. It also targets video as a control signal, which the authors say lacks systematic evaluation, and adds a benchmark for it.
Who it affects
The work is aimed at researchers building controllable video editing and generation systems, and potentially at creators who want to mix several reference types when editing footage. The paper frames the problem around users expressing creative intent through text, images and video.
How to use it
The source gives no information on code, weights, dataset release or a usable product, so there is nothing to run yet. The practical takeaways are design ideas: a reference-bank input interface, R^2-RoPE for scaling the number of references, synthesized training data, and curriculum learning on complexity and resolution.
How solid is it
This is a preprint abstract listed on Hugging Face Papers. The headline claim, that OmniXRef-Model beats task-specific baselines and commercial models, is the authors' own. The source does not name the baselines or the commercial models, and it gives no scores or metrics, so the size of any advantage cannot be judged from the text.
Risks and caveats
The comparison results are unquantified and the competitors are not named. Model size, dataset size and training compute are not stated. The training data is synthesized, and the source does not say how well that transfers to real-world editing. The benchmark was introduced by the same authors as the model.