Omni-Streaming Thinking curbs cross-modal hallucinations in streaming AI

Researchers describe a failure mode in streaming omni-modal models: when a system watches video and listens to audio at the same time, a visual cue can suggest an interpretation before the matching utterance or sound event has actually finished. If that early guess is written into the model's memory as a settled fact, later reasoning keeps repeating it even after the audio goes on to contradict it. The authors name this failure premature cross-modal commitment. To fix it, they propose Omni-Streaming Thinking (OST). Instead of committing to an answer immediately, OST generates structured outputs that record the evidence observed so far, a forecast of evidence still to come, and claims built on that evidence. Each claim starts out marked as pending and is tied to a future verification interval. Audio and visual evidence are stored separately, and at the end of the verification interval OST checks the claim against evidence from the specific modality it depends on. When the check finds contradictory evidence, a refutation process lowers the influence of that claim and anything that depended on it, then updates the model's state using the new evidence instead. A separate answer gate decides whether the claims a response actually depends on are solid enough before the system answers at all. OST is built on top of a frozen Qwen3-Omni-30B-A3B-Instruct backbone with only lightweight adaptation added, rather than retraining the underlying model. Tested on five streaming and audio-visual benchmarks, OST beats the strongest open baselines by more than 10% relative on average. The authors also introduce a diagnostic benchmark, OST-DiagBench, which keeps the video fixed and edits only the audio track to probe five conditions: agreement, absence, contradiction, coexistence, and subtitle-speech conflict. On this benchmark OST reaches a d-prime score of 2.95, against at most 1.38 for the open baselines tested, while also reducing vision-induced auditory hallucinations.

Key facts

Omni-Streaming Thinking (OST) targets premature cross-modal commitment: a streaming model locking in an early visual interpretation as fact before the audio that should confirm or contradict it has finished playing.
OST marks claims as pending, ties each to a future verification interval, checks it against evidence from the specific modality it depends on, and runs a refutation process that lowers a claim's influence and updates the state when the evidence contradicts it.
An answer gate withholds a response until the claims it depends on meet the verification conditions.
Built as a lightweight adaptation on a frozen Qwen3-Omni-30B-A3B-Instruct backbone, OST outperforms the strongest open baselines by more than 10% relative on average across five streaming and audio-visual benchmarks.
On the authors' new OST-DiagBench, which edits only the audio track against a fixed video to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict, OST scores a d-prime of 2.95 versus at most 1.38 for open baselines.

Why it matters

Streaming systems that watch video and listen to audio together have to decide what to say and when, before the clip has finished playing. The paper's core finding is that these systems can jump to a visual interpretation early, write it into memory as settled, and keep repeating it even once the audio contradicts it. Naming that failure premature cross-modal commitment gives the problem a shape other builders of streaming multimodal systems can check their own models against, and OST is offered as a direct fix rather than a general robustness tweak.

Who it affects

The work targets people building or evaluating streaming omni-modal AI, systems that reason over live or ongoing video and audio together rather than a single static input. That includes real-time assistants, video-call or meeting tools, and any pipeline that has to commit to an answer before all the evidence has arrived. The text does not name any company or product shipping this technique.

How to use it

OST is not a standalone model but an adaptation layer added on top of a frozen Qwen3-Omni-30B-A3B-Instruct backbone, described as lightweight. That means the underlying model is not retrained; the structured evidence tracking, pending-claim bookkeeping, and answer gate sit on top of it. The source does not state whether code, model weights, or the OST-DiagBench dataset will be released, so builders currently have the method described in the paper without a confirmed public implementation to run.

How solid is it

OST is evaluated on five streaming and audio-visual benchmarks, where it beats the strongest open baselines by more than 10% relative on average; the paper does not name which five benchmarks these are. The authors also built a dedicated diagnostic benchmark, OST-DiagBench, that holds the video constant and edits only the audio to create five specific conditions (agreement, absence, contradiction, coexistence, and subtitle-speech conflict). On that benchmark OST reaches a d-prime of 2.95 against at most 1.38 for the open baselines tested, a wide margin on a metric designed specifically to catch this failure mode.

Risks and caveats

The abstract does not name the five main benchmarks, give the size of OST-DiagBench, state author affiliations, or say whether code, weights, or the dataset will be released, which limits independent verification of the claims for now. The strong OST-DiagBench result comes from a benchmark the same authors designed to test exactly the failure their method targets, so it is not yet clear how the improvement transfers beyond the conditions the diagnostic set was built to probe.

“If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment.”

— the authors