AVA-Encoder converts films into knowledge graphs for creative agents
Creative agents still lack an effective way to learn from high-quality human films, which limits their ability to produce cinematic-grade video. The authors of a new paper identify the core problem as the absence of a structured video representation that stays faithful to a film's content while remaining directly usable for an agent's reasoning and editing.
To close that gap, they propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning what they call agent-native video representations through agentic auto-encoding. AVA-Encoder converts a video into a knowledge graph (KG), then reconstructs the video back from that graph. Hierarchy and state nodes in the graph hold structured text; a linked asset layer holds the generated images, audio and video; typed edges connect the text descriptions to those assets in a form agents can understand, query and edit.
Training relies on the differences between a video and its reconstruction. Those differences drive a textual-gradient optimization framework that turns evaluation feedback into natural-language update directions. In an outer loop, this produces what the authors call Data-Independent Encoding Policy Pseudo-Training; at test time, an optional inner loop adds Data-Dependent KG Representation Refinement.
In the paper's experiments, AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. The paper does not name that baseline or the metric behind the figure. In the controlled policy-only setting, the pseudo-trained shot-level Agentic Video Encoder policy also beats a carefully human-tuned policy while using 74.3% fewer system-prompt tokens.
The authors say they are releasing the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and what they describe as the first dataset of high-quality film knowledge graph representations. The paper gives no release date, license, or download location for any of the three, and does not state the dataset's size.
Key facts
- AVA-Encoder converts a video into a knowledge graph, with structured text in hierarchy and state nodes and generated images, audio and video in a linked asset layer, then reconstructs the video from that graph.
- Reconstruction differences drive a textual-gradient optimization framework that turns evaluation feedback into natural-language update directions, used for outer-loop pseudo-training and an optional inner-loop refinement step at test time.
- AVA-Encoder improves by 20.7 percentage points over the strongest external baseline in the paper's experiments; the baseline and the metric behind the figure are not named.
- In the controlled policy-only setting, the pseudo-trained shot-level Agentic Video Encoder policy also beats a carefully human-tuned policy while using 74.3% fewer system-prompt tokens.
- The authors are releasing the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and what they call the first dataset of high-quality film knowledge graph representations.
Why it matters
The authors frame the problem as a gap: creative agents have no structured way to learn from high-quality human films, which limits their ability to produce cinematic-grade video. Their fix is a knowledge graph whose hierarchy and state nodes hold a film's structure as text an agent can read, while a linked asset layer keeps the generated images, audio and video attached as nodes the agent can still query and edit. That targets the specific problem they describe: no existing representation was both faithful to real footage and directly usable for an agent's own reasoning and edits.
Who it affects
The work targets researchers and engineers building AI agents that generate, edit or reason about video, since AVA-Encoder proposes a representation layer for exactly that use case. The released benchmark and dataset also matter to anyone trying to reproduce or compare against agentic video-encoding methods; the dataset in particular is described as the first of its kind for film knowledge graph representations.
How to use it
The authors say they are releasing the complete AVA-Encoder framework, the agentic video reconstruction benchmark, and the film knowledge graph dataset described in the paper. The paper gives no release date, license, or download location for any of the three.
How solid is it
The headline 20.7-percentage-point gain and the claim of using 74.3% fewer system-prompt tokens both come from the authors' own experiments, including their own human-tuned comparison policy. The paper does not name the 'strongest external baseline' behind that gain, nor the metric it is measured in, which limits how independently the figure can be checked. The authors do say they are releasing the framework, benchmark and dataset together, which would let outside researchers rerun the comparison once the materials are available.
Risks and caveats
The text does not name any authors or institutions, so there is no way from this document alone to check the team's track record. No definition or example is given for the 'controlled policy-only setting' beyond its name, so it is unclear exactly what conditions produced the 74.3% figure. The paper also does not state the size, in videos, hours or films, of the released film knowledge graph dataset. Until the promised release lands, all of these numbers rest on the authors' own description of their own setup.
“Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos.”
— the paper