Glonce

FinDendro benchmark shows LLM self-reports understate what models know

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact42Novelty62Relevance92Surprise55Players18Credibility72
FinDendro benchmark shows LLM self-reports understate what models know

Financial backtests of large language models are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners try to guard against that leakage in three ways: trusting a vendor's stated knowledge cutoff, asking the model what it knows, or instructing it to answer as of a past date. A new paper tests whether any of these safeguards can be trusted.

The authors introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026. The facts cover monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question; Recall and Predict questions that pair each candidate value with a claim of knowing or guessing, without a declared date; and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27.

The central result is that for most frontier models, self-reports understate knowledge. The example given is GPT-5: it answers 76% of controlled facts correctly but claims to know only 38%. It also answers 61% of the facts it says it does not know.

Two further findings follow. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters. And a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged, so telling a model to answer as of a past date changes what it says about itself but not what it can actually answer.

From this the authors conclude that backtests inside a model's training window cannot be certified leak-free by self-report, instruction or stated cutoff. They argue the findings motivate behavioral measurement of temporal knowledge boundaries, and offer FinDendro as a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.

Key facts

  • FinDendro is a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions.
  • GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and it answers 61% of the facts it says it does not know.
  • Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters.
  • A declared date removes most of the claims of prior knowledge while leaving accuracy unchanged.
  • The authors conclude that backtests inside a model's training window cannot be certified leak-free by self-report, instruction or stated cutoff.

Why it matters

A financial backtest of an LLM is only meaningful if the model did not already know the outcomes. The common safeguards are a vendor's stated cutoff, asking the model what it knows, and telling it to answer as of a past date. This paper reports that none of them can certify a backtest inside the model's training window as leak-free. The finding that models answer correctly on facts they say they do not know (61% for GPT-5) means a model's own account of its knowledge is an unreliable guide to leakage.

Who it affects

The paper frames the problem around practitioners who run backtests of LLMs on financial questions and who rely on stated cutoffs, self-reports or as-of-date instructions to avoid leakage. Anyone interpreting published LLM backtest results over periods the model may have been trained on is also touched by the conclusion.

How to use it

The authors offer FinDendro as a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge, rather than on what a model or vendor says it knows. In practice that means measuring behaviorally, by asking dated factual questions and checking the answers, where the model's temporal knowledge boundary actually lies.

How solid is it

The source is the paper's abstract as listed on Hugging Face. The design is concrete: 2,226 facts, four prompts per fact, and controls for persistence and dominant outcomes that keep chance accuracy near 0.27. The headline figures (76%, 38%, 61%, up to seven quarters) are stated plainly. The abstract gives per-model numbers only for GPT-5 and does not list the other frontier models.

Risks and caveats

Only GPT-5 is named, so the claim that self-reports understate knowledge for "most frontier models" rests on figures the abstract does not break out, and it does not say which models are exceptions. The 76% and 38% figures are for controlled facts; the abstract does not say whether the 61% is on the same subset. The abstract reports no actual backtest returns or trading results, and it does not quantify how many claims of prior knowledge a declared date removes beyond "most".