Crawler bots cost git.kernel.org more CPU than real traffic
In a 7th September 2026 link blog post, Simon Willison points to an account from Konstantin Ryabitsev describing how bad the "background radiation" of abusive crawlers has become, seen from the perspective of git.kernel.org, the official Git repository for the Linux kernel. Ryabitsev states that the site spends more CPU cycles rendering commits for scrapers than it spends on all other kinds of legitimate access put together, including git clones. Across git.kernel.org's 5 geo-distributed nodes, 14 CPU cores are doing nothing but rendering git commits as HTML for these crawlers at any one time. Willison adds that the situation worries him from the perspective of his own project, Datasette, which also serves a large number of crawlable web pages.
Key facts
- Konstantin Ryabitsev describes the crawler load on git.kernel.org, the official Git repository for the Linux kernel
- git.kernel.org spends more CPU cycles rendering commits for scrapers than on all other legitimate access combined, including git clones
- Across 5 geo-distributed nodes, 14 CPU cores are tied up at any one time doing nothing but rendering git commits as HTML for scrapers
- Simon Willison, sharing the account on his link blog, says it worries him for Datasette, which also serves many crawlable pages
Why it matters
This is a concrete, named-source data point on a cost that is usually only discussed in general terms: bot crawling is not a marginal nuisance on git.kernel.org, it consumes more CPU than the entire rest of legitimate traffic, git clones included. Because git.kernel.org sits at the center of Linux kernel development, the claim carries weight beyond one site: it suggests that any service rendering dynamic pages from structured data, git history in this case, can end up with scraper load that dwarfs its real user base.
Who it affects
Directly, the operators of git.kernel.org and, by extension, Linux kernel infrastructure that depends on the site staying responsive. Simon Willison flags the same risk for his own project, Datasette, which serves large numbers of crawlable pages, and the underlying pattern applies to any site or service that renders many pages dynamically from an underlying dataset or repository, not just Git hosts.
How to use it
There is no tool, fix or mitigation described in this account, so nothing here is directly actionable as a product or setting. The usable part is the benchmark itself: an operator running a similarly dynamic, crawlable service can treat the 14-cores-across-5-nodes figure as a reference point for how disproportionate scraper cost can get relative to genuine usage, and a prompt to check that split on their own infrastructure before it reaches the same state.
How solid is it
The claim comes as a direct, verbatim quote attributed by name to Konstantin Ryabitsev, describing git.kernel.org specifically, relayed by Simon Willison on his link blog. It is a single account rather than an independently corroborated measurement, and this post does not include a total baseline CPU or traffic figure against which to judge how large a share the 14 cores represent.
Risks and caveats
The source does not say who or what the abusive crawlers are, including whether they are AI-training scrapers specifically, nor does it describe any mitigation or response git.kernel.org has taken or plans to take. No total baseline CPU or traffic figure is given to put the 14-cores claim in proportion, and the post does not state when Ryabitsev's own discussion was originally published, only the date of Willison's post referencing it.
“we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones”
— Konstantin Ryabitsev