EgoCaptioner trains video captioner on 50,000 examples, beats Gemini 3.1 Pro on PhysCapBench

The paper, listed on Hugging Face, takes on egocentric video captioning: describing footage filmed from the wearer's point of view. The authors say this needs fine-grained detail and accurate timing, yet current multimodal large language models (MLLMs) often miss brief actions, confuse their order, or misinterpret the wearer's motion. They add that existing caption formats offer limited support for combining detailed narration with precise timing.
Their answer is EgoCaptioner. Each caption pairs a video-level overview (the setting, the participants and the wearer's overall activity) with timed steps. The narration in those steps adapts to varying perceptual demands and marks key moments at specific timestamps.
To train it, the authors built a coarse-to-fine annotation pipeline. It first establishes the event structure, then expands the step descriptions. The pipeline produced 50,000 training examples from eight egocentric video sources.
Training uses Group Relative Policy Optimization (GRPO) with complementary rewards. One assesses factual accuracy and coverage, one measures step-level temporal alignment, and one penalizes fragmented or redundant narration. To make the factual supervision more reliable, the authors supplement caption references with pose-derived motion facts when sensor data are available.
Across five egocentric benchmarks, the authors report that EgoCaptioner achieves leading performance among evaluated open-source models in caption quality, caption-conditioned question answering, and temporal alignment. They also report that it surpasses Gemini 3.1 Pro on PhysCapBench, and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol. Ablations support the complementary roles of the factual and temporal rewards, and show that coarse-to-fine annotation gives more effective supervision than single-pass annotation.
Key facts
- EgoCaptioner writes each egocentric video caption as a video-level overview plus timed steps that mark key moments at specific timestamps.
- A coarse-to-fine annotation pipeline yielded 50,000 training examples from eight egocentric video sources.
- GRPO training uses three rewards: factual accuracy and coverage, step-level temporal alignment, and a penalty for fragmented or redundant narration.
- Across five egocentric benchmarks, the authors report leading performance among evaluated open-source models.
- The model is reported to surpass Gemini 3.1 Pro on PhysCapBench, and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol.
Why it matters
Egocentric footage, filmed from the wearer's point of view, is hard to caption well. The paper says current MLLMs often miss brief actions, confuse their order, or misread the wearer's motion. EgoCaptioner targets that gap by combining detailed narration with precise timing in one caption format, and by training on rewards that check facts, timing and redundancy separately.
Who it affects
The work is aimed at researchers building video understanding and captioning systems, especially for first-person video. The abstract does not name any product or deployment.
How to use it
The abstract mentions no code, data or weights release, so there is nothing to run yet. The transferable ideas are the caption format (overview plus timed steps), the coarse-to-fine annotation order (event structure first, then step detail) and the three-part GRPO reward. Pose-derived motion facts can supplement caption references when sensor data are available.
How solid is it
The results are the authors' own claims, taken from the abstract. Ablations are reported to support the complementary roles of factual and temporal rewards and the value of coarse-to-fine over single-pass annotation. The abstract gives no numeric scores or margins, so the size of the gains cannot be judged from it.
Risks and caveats
The leading-performance claim covers only the evaluated open-source models, not all open-source models. The win over Gemini 3.1 Pro is stated for PhysCapBench only, and the win over all evaluated proprietary baselines is stated for Minerva-Ego only under the caption-to-QA protocol. Only PhysCapBench and Minerva-Ego are named among the five benchmarks. The base model and its size are not stated.