Google Research: GPT-5, Gemini-3-Pro know facts they can't recall

Google Research scientists Nitay Calderon and Gal Yona published a blog post on August 12, 2026, introducing knowledge profiling, a behavioral framework for diagnosing why large language models get facts wrong. The framework separates 'empty shelves,' facts never encoded in a model's parameters, from 'lost keys,' facts that are encoded but cannot be produced on demand: a distinction that ordinary accuracy scores collapse into a single number, even though the two failures call for different fixes. Encoding failures argue for scaling model size or broadening data coverage; recall failures argue for post-training or inference-time methods that help a model use what it already stores.

To operationalize the framework, the team built WikiProfile, a benchmark of 2,150 Wikipedia-derived facts, each paired with ten questions: two probing encoding, four probing recall, and four multiple-choice variants probing recognition. The benchmark was assembled through an automated pipeline built on Gemini-2.5-Pro with thinking, generating direct and reverse questions that were then filtered through a search engine and a final manual validation pass. Each fact is classified into one of five profiles: encoding failure, recall failure, direct recall, recall with thinking, or inference without encoding. The researchers evaluated 13 LLMs, each with and without thinking, sampling eight responses per model, fact and task, for roughly 4.5 million responses in total, graded by prompted LLM autoraters.

Across frontier models, including Gemini-2.5-Pro, Gemini-3-Pro, Gemini-3 Flash and GPT-5, encoding is close to saturation. For Gemini-3-Pro and GPT-5 specifically, 95 to 98 percent of facts are encoded, yet these same models still fail to directly recall 26 to 34 percent of them, and even when allowed to think first, they still fail on 11 to 12 percent. The authors read this as a sign that factual errors in frontier models increasingly come from knowledge that is stored but not reliably accessible, rather than from knowledge that was never acquired. Scaling reinforces the pattern: within the Gemma 3 family, larger models show far fewer encoding failures, but recall failures persist and make up a growing share of what remains, meaning scale improves what a model stores more than what it can retrieve.

The same access gap shows up in two specific places. Rare, low-popularity facts turn out to be encoded at rates close to popular facts, so the encoding gap between them is modest, but the recall gap is much larger, reframing the long-tail problem as one of access rather than capacity. The team also revisited the 'reversal curse,' where a model that knows 'A is B' cannot answer 'what is B?' In open-ended recall, reverse questions are consistently harder than direct ones. But in multiple-choice recognition, reverse questions are no harder, and are often easier. Since the fact is recognizable among alternatives even when it cannot be generated in reverse, the researchers conclude that the reversal curse is a recall problem rather than evidence that bidirectional knowledge is missing.

Thinking helps most exactly where direct recall is weakest. In thinking-optimized models, it recovers roughly 40 to 65 percent of facts that are encoded but not directly recallable, with the largest gains on rare facts and reverse questions, and it does much less for facts that were never encoded in the first place. The authors note that thinking carries a computational cost and that it remains unclear when a model should invoke it. Their broader conclusion: since encoding in frontier models is already near saturation, the next gains in factuality may come less from scaling model size or data and more from methods that improve how a model retrieves knowledge it already holds.

Key facts

WikiProfile benchmark: 2,150 Wikipedia-derived facts, ten questions each, tested across 13 LLMs with about 4.5 million graded responses total
Gemini-3-Pro and GPT-5 encode 95 to 98 percent of facts but fail to directly recall 26 to 34 percent of them, and still fail on 11 to 12 percent even with thinking
Thinking recovers roughly 40 to 65 percent of encoded-but-not-recalled facts in thinking-optimized models, with the biggest gains on rare facts and reverse questions
The 'reversal curse' is reframed as a recall problem: reverse questions are harder to answer in open generation but no harder to recognize in multiple choice
In the Gemma 3 family, larger models show sharply fewer encoding failures, but recall failures persist and grow as a share of remaining errors

Why it matters

The default assumption about LLM factual errors is that a missing fact means missing knowledge, which points toward scaling model size or training data as the fix. This work shows that in frontier models, encoding is already close to saturated, so much of what looks like a knowledge gap is really a retrieval gap. If that holds, further gains in factuality may come less from making models bigger and more from post-training or inference-time methods that help a model surface what it has already stored.

Who it affects

Anyone building or evaluating factuality and hallucination mitigations for LLMs: teams designing post-training or inference-time recall aids, and researchers benchmarking model reliability. The study itself covers 13 LLMs, with named examples spanning Google's Gemini-2.5-Pro, Gemini-3-Pro, Gemini-3 Flash and the Gemma 3 family, alongside OpenAI's GPT-5.

How to use it

This is a research framework and benchmark, not a shipped product, and the post gives no price or license because none applies. The practical takeaway for practitioners is diagnostic: knowledge profiling offers a way to tell whether a model's factual failure on a given case is an encoding problem, which calls for more scale or data, or a recall problem, which calls for techniques like eliciting intermediate reasoning before the answer. The post does not say when or whether the WikiProfile benchmark itself will be released.

How solid is it

The benchmark comprises 2,150 facts, each checked with ten questions across encoding, recall and recognition, filtered through a search engine and a manual validation pass. Evaluation covered 13 LLMs, each run with and without thinking, sampling eight responses per model, fact and task, for about 4.5 million responses in total. Grading was done by prompted LLM autoraters rather than human reviewers, and the benchmark-construction pipeline itself relied on Gemini-2.5-Pro with thinking to generate and refine the questions.

Risks and caveats

Automated grading by LLM autoraters can introduce its own errors, and the benchmark draws exclusively on Wikipedia-derived facts, which may not generalize to other factual domains. The post does not name all 13 evaluated models, only representative examples; it gives no institutional affiliation for the two authors beyond Google Research, no publication venue for the paper, and no numeric comparison to prior encoding and recall benchmarks.

“Many factual errors in frontier LLMs are better understood as lost keys (recall failures), not empty shelves (encoding failures).”

— Nitay Calderon and Gal Yona, Google Research