OmniVChat-RL training improves Qwen3-Omni-Instruct's dialogue skills
Researchers introduce OmniVChat, native audio-visual dialogue in which an omni model receives a user's audio and video feed directly and replies in text, with no separate typed question, no external captioning step and no speech-recognition step in between. The user's request is carried entirely in what the model sees and hears.
Two problems have held this task back: there is little real-world footage of people talking to their own devices to train on, and a good reply depends on context such as the user's surroundings, facial expressions and nearby objects, which can be phrased correctly in many different ways, so grading replies by keyword matching does not work. To get around both, the team built OmniVChat-Studio, a multi-agent data engine that synthesizes single-turn and multi-turn audio-visual dialogues. Synthesized dialogues from the engine are used to build OmniVChat-Bench, an evaluation benchmark that scores an omni model's basic dialogue ability across five separate ability categories; the paper's abstract does not spell out what those five categories are.
The team also designed OmniVChat-RL, a reinforcement-learning reward that optimizes for three things at once: whether a reply is correct, how efficient it is, and its style. They trained Qwen3-Omni-Instruct with OmniVChat-RL on the synthesized dialogues and report improved performance both on OmniVChat-Bench and on a separate, human-recorded companion benchmark, OmniVChat-Bench-Human. The paper treats the transfer from synthetic training and evaluation data to human-recorded results as validation that the reward design works, though it does not state the size of the score gains.
Key facts
- OmniVChat defines native audio-visual dialogue: a model takes a user's audio and video directly and answers in text, skipping a separate text question, captioning or speech recognition.
- OmniVChat-Studio is a multi-agent engine built to synthesize single- and multi-turn audio-visual dialogues, addressing the scarcity of real recordings of people talking to their devices.
- OmniVChat-Bench, built from the synthesized dialogues, scores omni models' basic dialogue ability across five ability categories.
- OmniVChat-RL is a reinforcement-learning reward that jointly targets reply correctness, efficiency and style.
- Training Qwen3-Omni-Instruct with OmniVChat-RL improved results on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human, without the paper stating the size of the gain.
Why it matters
Most voice or video assistants today still run a pipeline underneath: transcribe the user's speech to text, maybe caption the video, then hand a text prompt to a language model. OmniVChat targets a model that skips that pipeline and reasons over the raw audio and video feed directly, which should cut the latency and compute that the intermediate steps add while keeping cues, like tone of voice or a glance at an object, that get lost when everything is flattened into a transcript.
Who it affects
The immediate audience is researchers and teams building omni models that take live audio and video input and reply conversationally, plus anyone building or evaluating a talking assistant that has to react to what it sees as well as what it hears. The specific training and benchmark results concern Qwen3-Omni-Instruct, the model the OmniVChat-RL method was applied to in the paper.
How to use it
The paper describes a benchmark (OmniVChat-Bench, plus the human-recorded OmniVChat-Bench-Human) and a training method (OmniVChat-RL) rather than a shipped product. Teams working on omni models could, in principle, use the data engine to synthesize training dialogues and the RL reward to fine-tune a model the way the authors fine-tuned Qwen3-Omni-Instruct, but the source gives no code, model weights, license or pricing details.
How solid is it
The central claim, that OmniVChat-RL training improves Qwen3-Omni-Instruct on both the synthetic OmniVChat-Bench and the separately human-recorded OmniVChat-Bench-Human, is a meaningful check because the second benchmark was not itself built from synthesized data. That said, the abstract gives no numeric scores, so the size of the improvement cannot be judged from the source text alone, and no author names or affiliations appear in the text to weigh against.
Risks and caveats
The abstract does not say what the five ability categories in OmniVChat-Bench actually measure, how large the reported gains are, or how OmniVChat-Studio's multi-agent data engine works internally. Because the training and much of the evaluation data is synthesized rather than recorded from real users, how well results generalize beyond the paper's own human-recorded benchmark is not established here.
“Direct audio-visual input reduces external latency and computation while preserving perceptual cues.”
— the paper's abstract