Glonce

StreamDuo gives a live game commentator a wiki research agent

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact45Novelty60Relevance100Surprise30Players15Credibility75
StreamDuo gives a live game commentator a wiki research agent

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. A new paper studies a harder setting, cross-game streaming commentary, where a model trained on one game has to commentate games it has never seen. The authors find that current commentators transfer their timing but not their knowledge of the game world: they know when to speak, but not what the things on screen are called.

Their answer is StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames to the research agent. The agent returns a few wiki entries two seconds later, while the stream continues.

Only the commentator is trained, on a single game, in two stages. In the first, information-gain labels teach it to ask only where an answer would help it say what the human caster said. In the second, on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers, since what the research agent returns will not always be perfect.

The model was trained on Baldur's Gate 3 and tested on ten unseen games. Against Proact-VL trained on the same data, StreamDuo raises entity recall from 0.082 to 0.116 and cuts source-game leakage from 9.9% to 1.0%. The authors also report that it achieves the best timing and the highest judged liveliness and overall quality among streaming models.

Key facts

  • StreamDuo pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played.
  • When the commentator sees something it cannot name, it sends the current frames, and the agent returns a few wiki entries two seconds later while the stream continues.
  • Trained only on Baldur's Gate 3 and tested on ten unseen games, it raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data.
  • Source-game leakage falls from 9.9% to 1.0%.
  • Training has two stages: information-gain labels for when to ask, and on-policy self-distillation for using noisy answers.

Why it matters

The paper separates two things a live commentator needs: timing and knowledge of the game world. The authors find that current commentators carry over the first to unseen games but not the second. StreamDuo handles the second by looking facts up in the game's wiki instead of expecting a model trained on one game to know them all. Source-game leakage, meaning the model talking about the game it was trained on while watching another, drops from 9.9% to 1.0%.

Who it affects

The work is aimed at researchers building streaming video LLMs and live commentary systems, where a model trained on one title has to cope with others. The test setup was one training game, Baldur's Gate 3, and ten unseen games.

How to use it

The abstract describes a method rather than a product. The design is a commentator that asks for help only when it meets something it cannot name, plus a frozen research agent that reads the game's wiki and answers within two seconds. Only the commentator needs training, and a single game was enough for the reported results. The abstract does not say whether code or data are released.

How solid is it

The headline comparison is against Proact-VL trained on the same data, which makes it like for like. Entity recall rose from 0.082 to 0.116, which are absolute scores; both are low in absolute terms. The claim of best timing and highest judged liveliness and overall quality is limited to streaming models. The abstract does not say who the judges were, human or model, and gives no timing values or scores for those judgments. No comparison against models other than Proact-VL is quantified. The ten unseen games are not named.

Risks and caveats

The abstract is the only source here, so the results are the authors' own and unreviewed in this account. Absolute entity recall stays low even after the gain. The system depends on a wiki existing for each game, and the research agent's answers can be noisy, which is why the second training stage exists. The two-second delay is part of the design, so the commentator must keep talking while it waits.

“current commentators transfer their timing but not their knowledge of the game world”

— StreamDuo paper abstract