Stanford study: 44% of 50 widely cited LLM claims have expired

Stanford researchers Owen Queen and James Zou took 50 widely cited claims about large language models and tested them against 20 models, released between July 2024 and September 2026 by eight different developers. They found that 44 percent of the claims have already "expired", meaning the original effect has largely disappeared. Another 34 percent have weakened significantly. Only 16 percent still hold, and 6 percent have actually gotten stronger. Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months. After one year, 16 percent of claims had expired; after two years, 39 percent.

Claims about what models cannot do expire fastest. Five of nine reasoning-related findings are considered expired, and none of the nine have held up or gotten stronger. The expired weaknesses include counting letters in "strawberry", arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back. The researchers suspect these errors have a "correct" outcome, which gives developers a clear target to train toward.

Claims about preferences and tendencies last longer. Of the 11 findings that have held or gotten stronger, three involve LLMs acting as evaluators: they prefer their own answers, rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts. Three more involve sycophancy and agreement toward users. Other persistent findings are a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.

As a case study, Queen and Zou looked at fabricated scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. The Stanford team sent the same prompts to current models. GPT-6.1 Sol fabricated 0 percent and Claude Sonnet 5.5 came in at 1 percent. But Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent of its references in the same test.

The study warns against reading the results as an all-clear. On so-called two-hop questions, where a model must combine two known facts, Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions; the original paper reported 43 percent for ChatGPT. The "Reversal Curse" and contradictory answers to paired time-related questions have weakened too, but are still detectable. Social biases are a mixed case: the political bias persists, gender stereotypes and implicit association biases have weakened, and biases against speakers of African American English have flipped direction in the trend data.

Models differ widely even within one release window. Among models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider, nor model size reliably predicts the pattern.

Queen and Zou conclude that empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document the model, version and date, and publish prompts and analysis code so findings can be re-tested on future models. The researchers also point to a post by Andrej Karpathy on how perception gaps about LLMs' actual behavior affect trust in AI and how often people use it. The Decoder, which reports the study, describes Karpathy as an OpenAI co-founder who has since joined Anthropic, and notes he recently said currently available models are just a preview of future capabilities.

Key facts

Queen and Zou tested 50 widely cited LLM claims against 20 models from eight developers (released July 2024 to September 2026); 44 percent have expired and 34 percent have weakened significantly.
Only 16 percent of claims still hold and 6 percent have gotten stronger; among the 22 findings with an estimated expiration date, the average lifespan was 15.7 months.
Claims about what models cannot do expire fastest: five of nine reasoning-related findings are expired and none of the nine have held up or gotten stronger.
Citation fabrication fell from 55 percent for GPT-3.5 in the 2023 study to 0 percent for GPT-6.1 Sol and 1 percent for Claude Sonnet 5.5, yet Qwen3.7-Flash still fabricated 69 percent.
The authors argue LLM findings should be framed as observations about specific models at a specific time, with model, version and date documented and prompts and code published.

Why it matters

Much of what people believe about LLM behavior comes from papers written about models that no longer exist. This study puts numbers on how quickly such claims go stale: 44 percent of the 50 claims tested have expired, and for the 22 findings with an estimated expiration date the average lifespan was 15.7 months. The pattern is also informative. Weaknesses that can be pinned to a clear right answer, such as letter counting or reasoning errors, fade fastest, while tendencies such as self-preference, sycophancy and political lean tend to stay.

Who it affects

Researchers who publish findings about LLM behavior are the main audience, since the authors want their papers to carry model, version and date. It also touches anyone who relies on those findings: journalists, educators, and teams choosing or deploying models. The results differ by model, so users of particular systems are affected unevenly. Among models released between July and September 2026, expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol.

How to use it

Treat a published claim about LLM behavior as a statement about the models that were tested, not about LLMs in general. The authors recommend that papers document the model, version and date, and publish prompts and analysis code so the findings can be re-tested on future models. A reader can apply the same habit in reverse: check which models a claim was based on and when, and re-run the prompts on the model actually in use. The citation example shows why, since the same prompts gave 0 percent fabrication on GPT-6.1 Sol and 69 percent on Qwen3.7-Flash.

How solid is it

This is a secondary report by The Decoder on a Stanford study by Owen Queen and James Zou. The numbers are consistent across the article: 50 claims, 20 models, eight developers. The 15.7-month figure is an average over only the 22 findings with an estimated expiration date, not over all 50 claims. The article gives no title, venue, arXiv link or publication date for the study, and it does not say how a claim is judged expired or how the expiration date is estimated. No reactions from model developers or outside researchers are given.

Risks and caveats

The study itself warns against reading the results as an all-clear: Claude Sonnet 5.5 still fails in about 10 percent of two-hop cases where it answers both individual questions correctly, and several biases persist. The idea that errors with a "correct" outcome are easy to train away is the researchers' suspicion, not a tested mechanism. Results vary widely between models, and neither release date, provider nor model size reliably predicts the pattern, so one model's improvement says little about another. The article does not list all 50 claims, only examples.

“Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths.”

— Owen Queen and James Zou's conclusion, as reported by The Decoder