GPT-4o, Gemini 2.5 Pro score under 10% on new benchmark

Researchers have introduced The Unwritten Benchmark, a test built around a task they call acousto-kinematic word inference. A model or a person is given only the audio of a pen scratching across paper and video of a hand moving as it writes; the ink trace itself is never shown. The task is to decipher the word being written, across 3 different writing styles, from those indirect cues alone.

The results show a wide gap between humans and machines. Human participants reach over 80% ordered letter accuracy on the task. Leading multimodal models, including GPT-4o and Gemini 2.5 Pro, fail to surpass 10% accuracy on the same test.

The authors also report what they call a paradoxical fusion effect: giving the models both the audio and the video together often makes their performance worse, not better, compared with using either signal alone. They read this as a sign of a more basic problem, that these models cannot properly combine complementary perceptual cues for this kind of task, rather than a simple gap in raw perception.

The authors frame the findings as exposing limits in two related abilities: cross-modal causal reasoning, working out what caused a sound or a motion from indirect evidence, and understanding of the fine-grained micro-kinematics involved in an action like handwriting.

Key facts

The Unwritten Benchmark tests acousto-kinematic word inference: models must decipher words, across 3 different writing styles, from only the audio of pen scratches and video of hand movement, with no ink trace ever shown.
Human participants reach over 80% ordered letter accuracy on the task.
Leading multimodal models, including GPT-4o and Gemini 2.5 Pro, fail to surpass 10% accuracy on the same task.
Giving models both modalities together, audio plus video, often degrades their performance rather than improving it, a paradoxical fusion effect.
The authors say the results point to a fundamental breakdown in synthesizing complementary perceptual cues, and expose limits in cross-modal causal reasoning and in understanding fine-grained micro-kinematics.

Why it matters

The Unwritten Benchmark targets an ability most multimodal tests skip: inferring content that was never shown directly, only implied by how a dynamic process unfolds. Current models are strong at recognizing what is plainly in front of them, a photo, a spoken word, but this task asks them to reconstruct a word from indirect physical cues alone, the sound of a pen on paper and the sight of a hand moving, without the ink trace it left. The reported gap, over 80% for humans against under 10% for GPT-4o and Gemini 2.5 Pro, is wide enough to mark this as a distinct, currently missing capability rather than a minor edge case.

Who it affects

It matters most to researchers building or evaluating multimodal models, since the benchmark gives them a concrete way to test whether combining modalities actually helps reasoning, a question the paper answers with a counterintuitive result. It is also relevant to anyone designing systems that pair video and audio understanding, such as interpreting a physical action from footage and sound together, since the paper suggests that pairing is not automatically an advantage.

How to use it

The paper describes the benchmark's design, words in 3 writing styles judged solely from pen-scratch audio and hand-movement video, but does not mention a public dataset, code release, leaderboard or any pricing or access terms. At this stage it exists as a described method and a set of evaluation results rather than a released tool that others can run against.

How solid is it

The findings come from a single arXiv paper reporting concrete comparative numbers, over 80% human accuracy against under 10% for both GPT-4o and Gemini 2.5 Pro, which gives the core claim a clear, checkable shape. What is missing from the available text is any sample size, the number of words or trials involved, and any author names or institutional affiliation, which limits how far the results can be independently assessed at this point.

Risks and caveats

No sample size, no description of the three writing styles, and no per-model breakdown below the 10% ceiling are given, so the exact size and consistency of the gap is easier to state than to fully verify from the text alone. The paradoxical fusion effect, models doing worse with more modalities, is a striking claim that cuts against the usual assumption that additional input signal helps, and is worth treating as specific to this benchmark until it is tested on other tasks.

“We identify a paradoxical fusion effect in the models, where providing both modalities often degrades performance rather than improving it.”

— the paper's abstract