Glonce

Sebastian Raschka builds an AI text detector from scratch

toolsThe Glonce editorsSebastian Raschka🔥55Glonce rating55Impact45Novelty38Relevance90Surprise28Players55Credibility85
Sebastian Raschka builds an AI text detector from scratch

Sebastian Raschka published an end-to-end tutorial on building an AI-generated-text detector, prompted by two things at once: Substack recently added an AI-detector feature to its UI, and readers had been asking him for local, do-it-yourself LLM projects that show what small language models (SLMs) can do. The tutorial walks through dataset construction, model training, and local deployment, and then goes further: Raschka uses the finished detector as a verifier to train an SLM whose job is to write text that avoids detection, framed as reinforcement learning with verifiable rewards (RLVR), a technique he notes is usually applied to math and code reasoning rather than to writing style. He frames the practical case for such a detector two ways: filtering spammy content, and helping a writer polish their own prose without letting it drift into text that reads as AI-generated. His example: someone asks an AI checker to fix their grammar while making sure the result still scores 0% AI-generated, since heavy polishing by a general-purpose LLM can turn a person's own writing into something a detector flags anyway. The classifier he builds outputs a 0-100 score estimating how likely a text is AI-generated, produced by fine-tuning a DistilBERT model, an approach he says is similar to what Pangram models do, which he believes (explicitly hedging "as far as I know") sit behind Substack's own detection feature. The stated end product is an AI-detector API usable by both humans and agents, plus a user-facing UI. Raschka is upfront that AI detectors are inherently a cat-and-mouse game: a detector learns to spot a telltale pattern, the next generation of LLMs stops producing that pattern, and the detector has to be updated again, all while producing false positives along the way, flagging genuinely human-written text as machine-generated.

Key facts

  • Sebastian Raschka's tutorial builds an AI-text detector by fine-tuning a DistilBERT classifier that outputs a 0-100 AI-generation-likelihood score.
  • The finished detector doubles as a verifier: he uses it to train a small language model to write text that avoids detection, via RLVR, a technique he says is usually applied to math and code rather than writing.
  • The project was prompted by Substack's newly launched AI-detector UI feature and by reader requests for local DIY LLM demos built on small language models.
  • He describes his method as similar to Pangram models, which he believes, with an explicit hedge, power Substack's own detector.
  • The stated deliverable is an AI-detector API usable by humans and agents, plus a user-friendly UI, deployable locally.

Why it matters

The tutorial ties two live threads together: Substack just shipped an AI-detector feature in its UI, and there is active reader demand for concrete, local projects that show what small language models can actually do end to end. Raschka also uses it to demonstrate a verifier-based training setup, an SLM trained against a detector's judgment, outside the usual math-and-code reasoning domain where verifiers are normally applied.

Who it affects

Anyone building or evaluating AI-content detectors gets a worked, from-scratch reference implementation rather than a black box. Writers who lean on grammar checkers and general-purpose LLMs to polish their own prose get a concrete warning: heavy polishing can push genuinely human writing into a range that gets flagged as AI-generated.

How to use it

The tutorial covers dataset construction, fine-tuning a DistilBERT classifier to output a 0-100 score, local deployment, and wrapping the result in an API usable by both humans and agents plus a UI. The same detector is then reused as a verifier to train a small language model via RLVR to produce text that evades it.

How solid is it

This is an educational, single-author project, not a production-grade or benchmarked detector: no accuracy, precision, or false-positive-rate figures are given for the classifier it builds. The claimed link between the method and Pangram, and between Pangram and Substack's own detector, is Raschka's own hedged belief ("as far as I know"), not a confirmed fact.

Risks and caveats

Raschka is explicit that AI detectors are a cat-and-mouse game: a detector learns to catch one generation of LLMs, the next generation stops exhibiting that pattern, and the detector has to be updated again. He also flags that detectors are likely to produce false positives, flagging human-written text as AI-generated, including text a person wrote themselves but heavily polished with an LLM grammar checker.

“AI checkers are essentially a cat-and-mouse game.”

— Sebastian Raschka