AI crawlers now consume 20% of git.kernel.org's CPU
Git.kernel.org, the site that hosts the canonical Linux kernel git repositories, has published hard numbers on how much load AI training crawlers put on its infrastructure. The headline figure: more CPU cycles now go to rendering individual commits as HTML for scrapers than to all other legitimate access combined, including ordinary git clones. At any given moment, across 5 geo-distributed nodes totaling 90 CPU cores, 14 to 16 cores are doing nothing but rendering commits for bots.
The site is an obvious target because Linux kernel history is fully open, cloneable, and guaranteed free of AI-generated text, which the author argues makes it valuable training data: training a model on text a model already wrote causes what the post calls a "digital prion disease." But instead of the efficient route, cloning the repositories once, scrapers instead fetch every commit as a rendered HTML page through cgit, which also serves patches, diffs, and range renders on request. linux.git alone holds about 1.48 million commits and has 922 forks on git.kernel.org that mostly share the same underlying objects; to a scraper hitting each rendered URL individually, that turns into a vastly larger crawl surface than the repository data itself.
Defenses escalated in stages. First came fail2ban blocking of bots that identified themselves through their user-agent string. When bots switched to spoofing ordinary browsers, the site moved to banning by IP, then by entire ASN, including ranges like Google Compute, accepting the occasional false positive against legitimate automation. Bots then shifted to routing through millions of residential and mobile IPs via what the post calls "proxy SDK monetization," each address making only 4 or 5 requests before disappearing, making IP-level blocking pointless.
About a year before this post, the site deployed Anubis, a proof-of-work challenge that forces a client's browser to find a string that, combined with its IP and a server secret, produces a sha256 hash with a set number of leading zeroes before it can pass through. It worked immediately: bots gave up and moved to easier targets for a few months. Then they came back solving difficulty level 4, so the site raised the difficulty to 5, a change that visibly warms up a phone while it computes the answer. That bought a few more months before bots started solving difficulty 5 as well.
Today git.kernel.org receives about 6 million daily requests for individual commits. Of those, 66% are still turned away by the Anubis challenge, but 33% now solve it and reach the site. Under what the author calls "a bunch of generous assumptions," only about 2% of total traffic is legitimate human or developer use; nearly everything else is scraping. The site says it is not currently at risk of falling over from this load alone, it stays responsive; the actual outages it has suffered come from poorly built CI systems doing simultaneous shallow clones from many nodes at once, a separate problem.
Rather than continue an unwinnable blocking arms race, kernel.org is now removing features and gating off the expensive rendering and diff endpoints that give scrapers so many URLs to hit, accepting that anonymous users will lose some functionality. It says it will still make all of its data available for download to anyone who asks, just with more steps involved. The post ends without a resolution: either the AI training boom eventually slows the number of crawlers, or scrapers start using the data efficiently instead of the wasteful way they do now.
Key facts
- Across 90 CPU cores on 5 geo-distributed nodes, 14 to 16 cores run constantly just rendering git commits as HTML for scrapers, about 20% of git.kernel.org's total capacity on average.
- The site gets about 6 million daily requests for commits; 66% are blocked by the Anubis proof-of-work challenge, but 33% now solve it and get through.
- Under generous assumptions only about 2% of git.kernel.org's traffic is legitimate; linux.git alone has 1.48 million commits and 922 forks, giving scrapers a huge crawl surface via rendered pages rather than a single clone.
- Defenses escalated from user-agent and IP bans to whole-ASN bans to an Anubis proof-of-work challenge raised from difficulty 4 to difficulty 5 after bots adapted each time; crawlers now also arrive through millions of residential and mobile IPs via proxy SDK monetization schemes.
- Instead of continuing to block, kernel.org is now removing crawlable URLs and gating expensive endpoints, while still promising to make all its data available for download to anyone who asks directly.
Why it matters
This is a first-party account, from the operators of the infrastructure that hosts the canonical Linux kernel source, of AI training crawlers imposing a real, measured cost on critical open-source infrastructure that was already freely cloneable. It shows that scrapers are not using the cheap, efficient path even when one is offered; they hit expensive per-commit rendering endpoints instead, and each round of defense the site deploys gets solved or routed around within months. The piece is evidence, not speculation, for a cost that open infrastructure projects across the ecosystem are absorbing.
Who it affects
Directly, the operators of git.kernel.org and the Linux kernel developers and mirror maintainers who depend on the site staying responsive. More broadly, any public git host or other open-source infrastructure project running cgit-style commit rendering, since the same combinatorics of forks times commits times renderable views applies to them too. Anonymous users of git.kernel.org are also affected, since the site is removing features and gating actions specifically for that access path.
How to use it
There is no product or pricing here, but the operational lesson is concrete: the load comes overwhelmingly from expensive render endpoints (per-commit HTML pages, diffs, patches, arbitrary-range comparisons), not from git clone or checkout traffic, so an admin running similar infrastructure should expect and can specifically restrict those endpoints rather than throttling clones. Anubis-style proof-of-work is a real but temporary brake, not a fix; kernel.org's own experience is that raising the difficulty buys a few months before bots catch up again.
How solid is it
The figures are first-party operational data from the people running git.kernel.org's own servers and logs, not a third-party estimate, which makes the CPU-core and request-volume figures solid. The one figure the author explicitly flags as soft is the roughly 2% legitimate-traffic estimate, described in the source as resting on "a bunch of generous assumptions" rather than a precise measurement, and the piece itself says it is impossible to know with certainty which requests are bots versus humans.
Risks and caveats
No specific AI company, lab, or model is named as responsible for the crawling; the post generalizes to "companies offering custom AI models" and app makers monetizing proxy traffic. No dollar cost is given for the wasted compute, only core-count and percentage-of-capacity figures. The article does not say what happens once bots also solve Anubis difficulty 5, nor which specific features are being disabled to cut crawlable URLs, so the endpoint of this arms race is left open.
“We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.”
— git.kernel.org operators, in the post "Creepy Crawlies"