Glonce

Header-based bot detection cut a blog's 'browser' traffic by 74.5%

securityThe Glonce editorsHacker News🔥36Glonce rating36Impact20Novelty30Relevance60Surprise15Players10Credibility75
Header-based bot detection cut a blog's 'browser' traffic by 74.5%

A blog owner built a request classifier into a Cloudflare Worker to separate readers from automated traffic without adding any JavaScript fingerprinting, then used its output to chase down why his own edge page-view counter disagreed with a separate script-based analytics counter on the same site. The classifier applies rules in order: it first checks named-client and Web Bot Auth signature matches, then reclassifies any browser-User-Agent request arriving from a curated list of hosting networks as cloud-browser regardless of its headers, then checks Fetch Metadata (Sec-Fetch-Mode and Sec-Fetch-Dest) against a claimed browser version, Chromium 76+, Firefox 90+, or Safari/iOS 16.4+, and finally checks whether the Accept header genuinely admits HTML. Over two complete UTC days, these rules moved 277 of 372 browser-User-Agent requests out of the Browsers category, a 74.5% share; of the remainder, 60 cloud-classified requests still carried the navigation headers the browser rule requires, and 95 Browser HTML observations remained against 14 page loads recorded by Cloudflare Web Analytics in the same window. The investigation started from an initial comparison saved on September 3: 578 daily client identifiers on the first-party side against 52 on the script-counter dashboard, a gap that looked like eleven times as many readers. The author's own note that night described the number as unbelievable and said something was not adding up. Once the comparison was made fairer, restricting to the same host, the same start-inclusive/end-exclusive UTC window, page events against page loads rather than daily identifiers against visits, and excluding the script dashboard's /stats route, the gap narrowed to 1,209 versus 113 page totals, still unresolved but no longer directly comparable to the original figures. On September 2, 100 of the 113 daily clients had loaded exactly one page each, 156 of 164 browser-User-Agent page observations carried no referrer, and one client classified as mobile fetched 31 distinct pages within the same timestamp second. A 72-hour log sample ending September 3 found that 430 of 844 successful page GETs were navigation-shaped traffic arriving from hosting networks, with the largest single cluster, 374 requests, attributed to one Google Cloud client whose User-Agent claimed to be Chrome Mobile 114; header checks alone could not separate that cluster from genuine browsing. Building the classifier also surfaced two bugs. The original Accept-header check read "Accept: text/html;q=0", which explicitly excludes HTML, as accepting it, while correctly admitting "Accept: text/*"; a September 6 rewrite now evaluates HTTP quality values, specificity and media-range rules per RFC 9110, and a 12-case local regression check turned 7 wrong acceptances into correct rejections, though the author says stored historical flags cannot show how many past requests the original bug misclassified. Separately, the site's stats page had been grouping Web Bot Auth-signed traffic and AI-assistant traffic together under one "AI agents" label, conflating a signature check with a client's actual role; that grouping is still pending a fix. The author also says an earlier version of the article's under-two validation threshold was unsupported and has been removed. On identity, the author draws a line between verifying a signer and knowing who is reading: in the tracked cohort since the signature verifier launched, through September 5 UTC, only nine requests carry a stored verified Web Bot Auth result, including crawlers and deliberate tests, which the author says corrects an earlier, wrong claim that no signed request had arrived. He notes a signed client can still be a crawler, citing Exa's own documentation of ExaSearchBot fetching and indexing pages, and stresses that a signature identifies a signer, not the requester's identity, delegation, or purpose. Separately, User-Agent matching alone, without signature verification, produced a 30-day snapshot of 33 ChatGPT-User observations across eight paths, 51 PerplexityBot observations across 31 paths, 37 Amazonbot observations across 36 paths, and 21 GPTBot observations across 21 paths; the author is explicit that these are recorded access patterns, not proof of why the crawl happened or of human reading behind any of it. He credits open-source bot classifiers, isbot for declared-client detection, Anubis for network classification (while noting Anubis makes access-control decisions where his own counter aims only to retain and explain access), and GoatCounter for the idea of attaching a stored reason to every classification rather than reporting a bare total. His stated motivation for building all this rather than simply blocking suspected bots is to keep and label every kind of automated reader, including AI agents fetching pages through tools like Playwright or Cypress, so that none of it is miscounted or hidden; he says he would like an eventual citation mechanism for his articles similar to how research papers cite each other, though the source describes that only as something he wants, not something that exists. The piece ends without resolving the original discrepancy: the author states plainly that neither a smaller Browser count nor agreement with the script counter by itself establishes how many actual readers the site has.

Key facts

  • Header and network rules moved 277 of 372 browser-User-Agent requests (74.5%) out of the Browsers category over two UTC days, leaving 95 Browser HTML observations against 14 recorded by Cloudflare Web Analytics.
  • An initial 578-vs-52 daily-client comparison saved September 3, which looked like an 11x gap, narrowed to a still-unresolved 1,209-vs-113 page-total gap once mismatched windows, daily identifiers and a stats-only route were accounted for.
  • In a 72-hour log sample ending September 3, 430 of 844 successful page GETs were navigation-shaped traffic from hosting networks, with 374 of those attributed to a single Google Cloud client claiming to be Chrome Mobile 114.
  • A bug that misread "Accept: text/html;q=0" as accepting HTML was fixed September 6; a 12-case regression check turned 7 wrong acceptances into correct rejections, but the historical scope of the bug is unknown.
  • Since the Web Bot Auth verifier launched through September 5, only nine requests carry a stored verified signature, while a separate 30-day snapshot logged 33 ChatGPT-User, 51 PerplexityBot, 37 Amazonbot and 21 GPTBot observations by User-Agent string alone.

Why it matters

As AI crawlers and AI assistants increasingly fetch pages on a reader's behalf, a bare User-Agent count of "browser" traffic stops meaning "human reader". This post is a from-scratch attempt to fix that without adding JavaScript fingerprinting: it layers network metadata (ASN), Fetch Metadata headers, Accept and Accept-Language checks, User-Agent matching, and Web Bot Auth signature verification into an explainable, per-request classifier running on a Cloudflare Worker, with a stored reason attached to every decision. The headline result, that three in four requests wearing a browser User-Agent were reclassified once header and network evidence were checked, is a concrete illustration of how far a declared browser identity has drifted from actual traffic composition on at least this one site.

Who it affects

The direct audience is website and blog operators who rely on first-party analytics and page-view dashboards to gauge readership and are seeing numbers that stop adding up, plus anyone building or evaluating similar bot classifiers, the author names isbot, Anubis and GoatCounter as prior art he drew on. It also touches the companies whose crawlers and assistants were separately logged by name: ChatGPT-User, PerplexityBot, Amazonbot and GPTBot each appear in the site's 30-day traffic snapshot, alongside Exa's ExaSearchBot, cited as an example of a signed client that is still a crawler rather than a human reader.

How to use it

The Worker applies its rules in a fixed order: signature and named-client checks run first, then a curated hosting-network list reclassifies matching browser-User-Agent requests as cloud-browser regardless of header shape, then a Fetch Metadata and browser-version check (Chromium 76+, Firefox 90+, Safari/iOS 16.4+) separates genuine navigations from clients missing Sec-Fetch-Mode, and an Accept-header check, rewritten September 6 to respect HTTP quality values and media-range rules under RFC 9110, filters on whether HTML is actually acceptable. Every request keeps its stored classification reason rather than a bare pass or fail flag, which is what let the author catch both the Accept-header bug and a stats page that had been grouping Web Bot Auth-signed traffic together with AI-assistant traffic under one "AI agents" label.

How solid is it

This is a single operator's own instrumentation of his own blog, described in his own post, not an independently audited system, and he repeatedly flags his own results as provisional. The 12-case check of the Accept-header fix is explicitly called a regression check rather than a production error-rate estimate, and he says stored historical flags cannot show how many past requests the original bug affected. The discrepancy the post set out to explain never gets resolved: the more comparable figures still show 95 Browser HTML observations against 14 Cloudflare Web Analytics page loads over the same window, and the author states plainly that neither a smaller Browser count nor agreement with the script counter by itself establishes how many actual readers the site has.

Risks and caveats

The post is explicit about what its own evidence cannot show. An ASN identifies the network a request arrived through, not the operator's purpose, and it can be defeated by routing through another network or proxy; a verified Web Bot Auth signature identifies a signer, not the initiating person's identity, delegation, or purpose. The curated hosting-network list is a deliberate trade-off: excluding shared-service and consumer-VPN networks avoids misclassifying legitimate browsing but also means some automation keeps arriving through networks the list does not cover. And the fix to the stats page that had conflated signed-agent traffic with AI-assistant traffic under one label is, by the author's own account, still pending.

“i am seeing daily clients as 113 for today, and it seems unbelievable to me, like which articles are they reading, where are they coming from and so on... i just published a new article and its not even coming up in the Top pages by views section... like whats going on...”

— the blog's author, in a note written the night he first saw the daily-client count