Qwen2.5-14B beats Watson, trails Claude Opus 4.8 on new Jeopardy clues
In 2011, IBM's Watson beat the strongest human Jeopardy! champions using its DeepQA system, a curated billion-document corpus running on a cluster of POWER7 servers. That knowledge base was frozen at build time and could not be moved or copied; it was, in effect, a sealed capsule of what its era could look up. A new paper argues the same kind of artifact, a snapshot of what a culture can answer, is now portable and essentially free, and tests the claim directly.
The authors evaluate a single 9GB open-weight model, Qwen2.5-14B running at 4-bit quantization, against the complete open Jeopardy! clue dataset: 529,939 clues across all 41 broadcast seasons, from 1984 to 2025. To their knowledge, this is the first time any model has been run over the full corpus rather than a sample. What the clues test is older than the show itself: the body of general knowledge a culture considers worth knowing, from ancient history and dead languages to science, literature and geography, each item with a verified correct answer.
Under a strict forced-response protocol using exact and fuzzy matching, the model answers 67.0% of all 529,939 clues, and its accuracy exceeds 85% on factoid categories specifically.
The more decisive test, the authors argue, is whether a model can answer clues that did not exist when it was trained. On clues aired after the local model's training cutoff, Qwen2.5-14B holds 65% accuracy. Claude Opus 4.8, tested on the same post-cutoff clues, holds 95%. Watson scores zero on this slice, not because its knowledge was weak but by construction: its corpus was assembled specifically to contain Jeopardy answers and tuned on past clues, so it had no way to answer anything outside that curated distribution.
The authors frame training-data exposure as something both a modern LLM and Watson share rather than a flaw unique to language models, and they call Watson's case the more extreme one of the two: it was built to the test in a way a general-purpose model is not. Their conclusion is about portability rather than raw accuracy: the same kind of capability that once needed a server room now fits in a 9GB file small enough, they write, to seal in a time capsule, and unlike Watson it is not frozen to the moment it was built.
Key facts
- A single 9GB open-weight model, Qwen2.5-14B (4-bit), was run against the complete open Jeopardy! clue dataset, 529,939 clues across all 41 broadcast seasons from 1984 to 2025, the first time any model has been tested on the full corpus.
- Under a strict forced-response protocol with exact and fuzzy matching, the model answers 67.0% of all clues and exceeds 85% on factoid categories.
- On clues aired after the local model's training cutoff, it holds 65% accuracy versus Claude Opus 4.8's 95%, while Watson scores zero on that slice by construction.
- Watson's 2011 corpus was built specifically to contain Jeopardy answers and tuned on past clues, so it could not answer anything outside that curated distribution.
- The authors frame the result as portability rather than a pure accuracy win: a capability that once required a server cluster now fits in a 9GB file.
Why it matters
Watson's 2011 Jeopardy win is the reference point most people still use for 'a computer that knows things.' This paper's point is not that a small open model beats that record on raw knowledge, since the source gives no single overall accuracy figure for Watson to compare against. Its point is portability: the kind of broad, verified general-knowledge capability that in 2011 required a curated billion-document corpus on a cluster of IBM POWER7 servers now runs from a single 9GB file, and, on genuinely new material the model was never trained on, does so far better than a system that could not step outside its own curated distribution at all.
Who it affects
The result speaks most directly to researchers and engineers who evaluate general knowledge and factual recall in language models, and to anyone weighing a small open-weight model against a frontier hosted one for that kind of task. It also gives a concrete, checkable benchmark, the full open Jeopardy! archive, to people skeptical that open models can match claims made about them.
How to use it
The evaluation uses the complete open Jeopardy! clue dataset, a public resource, run against a 9GB open-weight model rather than a server cluster. That makes the exact test reproducible by others: a full-corpus run against Qwen2.5-14B, or against another local model. The source gives no price, license or hardware-cost figures beyond the model's 9GB file size.
How solid is it
The text does not name the authors or their institution, nor state a publication date, so those cannot be verified from this dossier. The 67.0% and 85%+ figures come from a single run over the entire 529,939-clue corpus rather than a sample, which is a strength, but the text does not define exactly how the 'strict forced-response protocol' or the 'exact and fuzzy matching' scoring work beyond naming them, and it gives no per-category accuracy breakdown beyond the single factoid-category figure.
Risks and caveats
The headline post-cutoff comparison rests on a subset of clues, not the full 529,939-clue archive, and the source gives no training cutoff date for either Qwen2.5-14B or Claude Opus 4.8, so it is not possible to judge from this text how recent, or how large, that post-cutoff subset actually is. Only one local model, one quantization level, and one frontier model are compared; the paper does not test other open-weight models at similar or smaller sizes. And Watson's zero score on new clues is a structural artifact of how its corpus was built, not evidence about the ceiling of curated systems in general.
“The capability survives the move from a server room to a file you could seal in a time capsule, and unlike Watson it is not frozen to its own moment.”
— the paper's authors