ISOT fake-news corpus audit: 99% accuracy is metadata, not truth
Text classifiers trained on the ISOT/Kaggle "Fake and Real News" corpus routinely report accuracy and F1 above 0.98, a level of performance that sits uneasily beside how hard it actually is to judge whether an article is true. A new audit uses a transparent TF-IDF and linear-classifier pipeline as a measurement instrument to test that score along three leakage channels and two distribution-shift protocols, releasing all code and derived numbers.
The first finding is that the benchmark is partly degenerate. A classifier given only the subject metadata field, with the article text itself thrown away, still reaches F1 = 1.000, because the fake and real classes happen to have disjoint subject categories: the label can be read off the metadata alone. A second leakage channel is a newswire source tag present in 99.2% of real articles. A third is 6,251 duplicate documents that contaminate 19.4% of a naive test split. Yet removing all three channels together barely dents the score: F1 falls by only 1.21 points, from 0.9935 to 0.9814. Digging further, the authors find the residual signal is not a handful of giveaway words but diffuse editorial style: deleting the corpus's 1,000 highest-weight unigrams still leaves F1 at 0.926.
That style signal does not travel. Under a topic-disjoint protocol, where training and test articles cover different subjects, the linear model's average precision falls from 0.9995 to 0.9475 and its deployed F1 from 0.9905 to 0.8067; a prior-matched analysis confirms a genuine 5.2-point loss of discrimination. Temporal transfer, by contrast, is nearly lossless. A fine-tuned DistilBERT scores higher in-distribution (F1 = 0.9993) but degrades far more under the topic shift, losing 12.9 average-precision points against the linear model's 5.2. Transferred to the independent LIAR benchmark, all three models drop to near-chance ranking, with ROC-AUC between 0.54 and 0.57, and none beats a simple majority-class baseline.
The authors conclude that within-corpus scores on this benchmark measure source and topic separability, not veracity, and that giving a model more capacity only lets it exploit the shortcut harder rather than avoiding it. They recommend metadata-only, small-sample, and topic-disjoint baselines as cheap diagnostics that future fake-news detection work should run before trusting a headline accuracy number.
Key facts
- A classifier given only the subject metadata field, with the article text discarded, reaches F1 = 1.000, because fake and real articles fall into disjoint subject categories.
- Removing all three leakage channels, the metadata field, a newswire source tag present in 99.2% of real articles, and 6,251 duplicate documents contaminating 19.4% of a naive test split, drops F1 by just 1.21 points, from 0.9935 to 0.9814.
- Deleting the corpus's 1,000 highest-weight unigrams still leaves F1 at 0.926, showing the leftover signal is diffuse editorial style rather than a few giveaway tokens.
- Under a topic-disjoint transfer protocol, the linear model's deployed F1 falls from 0.9905 to 0.8067, and a fine-tuned DistilBERT loses 12.9 average-precision points against the linear model's 5.2.
- Transferred to the independent LIAR benchmark, all three models fall to near-chance ranking (ROC-AUC 0.54 to 0.57) and none beats a majority-class baseline.
Why it matters
The ISOT/Kaggle "Fake and Real News" corpus is a standard benchmark for fake-news text classifiers, and accuracy and F1 above 0.98 have long been cited as evidence that simple models can tell true articles from false ones. This audit shows those scores mostly reflect how easy it is to tell the two classes' sources and subjects apart, not whether a model can judge truth, which undercuts any claim of veracity detection built on this corpus alone.
Who it affects
Researchers and practitioners who train or benchmark fake-news detection models on the ISOT/Kaggle corpus, and anyone citing its near-100% accuracy as proof a model can spot misinformation. It is also a caution for reviewers and downstream users who take a single high in-distribution score at face value.
How to use it
The authors release their code and all derived numbers, so the audit's TF-IDF pipeline and ablations can be rerun on this or other corpora. They specifically recommend three cheap checks before trusting a fake-news classifier's headline score: a metadata-only baseline, a small-sample baseline, and a topic-disjoint evaluation split.
How solid is it
The method is transparent by design, a simple TF-IDF and linear-classifier pipeline used deliberately as a measurement instrument, tested across three separate leakage channels and two distribution-shift protocols, with code and numbers released for reproduction. The paper does not state the corpus's total size (only percentages and specific duplicate counts), names no authors or institutions in the text, and gives no submission, acceptance or venue date.
Risks and caveats
The findings are specific to the ISOT/Kaggle corpus and the LIAR benchmark used for the transfer test; they document that a fine-tuned DistilBERT degrades more than the linear model under topic shift but do not explain the mechanism behind that gap. Whether other widely used fake-news corpora hide the same kind of leakage is not something this paper tests.
“within-corpus scores here quantify source and topic separability rather than veracity”
— the study's authors