Glonce

Naive Bayes matches LLMs on labeled topic classification

researchThe Glonce editorsarXiv🔥73Glonce rating73Impact68Novelty71Relevance95Surprise58Players42Credibility85

A benchmark study pits Complement Naive Bayes (NB), a decades-old classical machine learning method, against zero-shot and few-shot large language models from four model families spanning a 37x range in scale, from 27 billion to a 1 trillion parameter mixture-of-experts model, on text classification tasks. It also compares both against a fine-tuned DistilBERT model, measuring accuracy alongside inference throughput and energy use.

In a zero-data regime, meaning no labeled training examples at all, LLMs dominate: on the Amazon Polarity sentiment dataset, a zero-shot LLM scores 98.0% accuracy against NB's 88.2%. But the authors find this win is contamination-prone, meaning benchmark data may have leaked into the LLM's training set. On a separate, low-contamination sentiment task, NB actually beats the zero-shot LLM, 81.7% to 73.0%.

Once labeled data becomes available, the picture flips. On the AG News topic classification dataset, NB reaches 89.1% accuracy, statistically indistinguishable from a zero-shot 27 billion parameter LLM's 89.0%, and ahead of a 397 billion parameter frontier model, which manages only 84.8%. A fine-tuned DistilBERT model edges out both at 90.6% accuracy, but at far lower throughput than NB running at batch size 1.

The efficiency gap is stark. Measured GPU throughput analysis shows small-LLM batched inference running 40 to 486 times slower than NB inference on a CPU, with the exact multiplier depending heavily on the host CPU used. The authors trace the gap to a structural bottleneck in memory bandwidth, and estimate NB uses roughly two orders of magnitude less energy per sample than the LLM setups measured. NB itself runs at thousands of samples per second on a commodity CPU.

The authors conclude that for resource-constrained HPC practitioners doing text classification with labeled data, NB remains the optimal choice. They find the right choice is task-dependent: for topic classification, NB reaches parity with LLM accuracy at around 10,000 labeled examples, while for zero-data sentiment classification the LLM wins at every amount of labeled data tested. To make the choice operational, the authors provide a Kubernetes Helm operator that automates model selection using configurable thresholds and Prometheus metrics.

Key facts

  • Across four LLM families spanning 27B to a 1 trillion parameter mixture-of-experts model (a 37x scale range), Naive Bayes matches or beats LLMs on topic classification once labeled data is available.
  • On AG News, NB scores 89.1% accuracy, statistically tying the zero-shot 27B LLM (89.0%) and beating a 397B frontier model (84.8%).
  • Zero-shot LLMs win only in the zero-data regime (98.0% vs NB's 88.2% on Amazon Polarity), but that edge is contamination-prone: NB wins 81.7% to 73.0% on a low-contamination sentiment task.
  • Small-LLM GPU inference runs 40 to 486 times slower than NB on a CPU, using roughly two orders of magnitude more energy per sample.
  • NB reaches LLM parity around 10,000 labeled examples for topic classification; the authors release a Kubernetes Helm operator that automates the NB-vs-LLM choice via configurable thresholds and Prometheus metrics.

Why it matters

The rise of large language models has pushed a recurring question in applied machine learning: should classical methods like Naive Bayes be retired in favor of LLMs for routine tasks such as text classification? This benchmark answers no, at least once labeled training data exists. That matters for any team building or maintaining text classification systems at scale, where LLM API costs and GPU inference latency are real operational constraints. The finding pushes back against the assumption that a bigger, newer model is automatically the better engineering choice.

Who it affects

The authors frame the result for resource-constrained HPC practitioners doing text classification with labeled data on hand. That covers teams running high-volume classification pipelines, such as content moderation, ticket routing or sentiment analysis, where inference happens at scale and cost per sample compounds quickly. It is less relevant to teams working with unlabeled data or one-off classification tasks, where the LLM's zero-shot advantage still holds.

How to use it

The decision the paper describes is task-dependent, not universal. For topic classification, NB reaches parity with LLM accuracy once about 10,000 labeled examples are available. For zero-data sentiment classification, the LLM wins at every amount of labeled data the authors tested, so NB is not a drop-in replacement everywhere. To make the choice operational rather than a one-off judgment call, the authors provide a Kubernetes Helm operator that automates model selection using configurable thresholds and Prometheus metrics teams can verify against their own workload.

How solid is it

The comparison spans four LLM model families across a 37x range in scale, from 27 billion parameters to a 1 trillion parameter mixture-of-experts model, plus a fine-tuned DistilBERT baseline, tested on Amazon Polarity sentiment, AG News, and a separate low-contamination sentiment task. The throughput and energy comparisons come from measured GPU inference benchmarks set against NB running on a CPU, not simulated figures. The source text does not name the four LLM families individually beyond their parameter counts, does not give author names, institutions, publication venue or peer-review status, and does not state a precise figure for NB's 'thousands of samples per second' throughput claim.

Risks and caveats

The authors themselves flag that the LLM's strongest win, 98.0% versus NB's 88.2% on zero-shot sentiment classification, is contamination-prone: it likely reflects benchmark data the LLM saw during training rather than genuine zero-shot skill, and on a low-contamination version of a similar task NB actually wins. The efficiency multiplier is not fixed either: the 40 to 486 times gap between LLM and NB inference speed depends heavily on the specific host CPU used, so the real-world gap for any given deployment could land anywhere in that range. Fine-tuned DistilBERT still edges out NB on raw accuracy, 90.6% versus 89.1%, so NB is the pragmatic choice under resource constraints, not the most accurate option available.

“For resource-constrained HPC practitioners performing text classification with labeled data, NB remains the optimal choice.”

— the authors