Only 2% of kernel.org Traffic Is Legit. The Bots Do Math Now.

TL;DR

Konstantin Ryabitsev, who runs the git.kernel.org infrastructure, published the first hard numbers on what AI training scrapers actually cost the Linux kernel's hosting: about 6 million requests a day asking for rendered commit pages, roughly 2% of total traffic that is legitimate, and 14-16 of the fleet's 90 CPU cores doing nothing, around the clock, but rendering git commits as HTML for bots. The Anubis proof-of-work challenge that stopped the swarm cold a year ago is now routinely solved at difficulty 5. kernel.org's answer is to start turning features off. The era of anonymous, unmetered access to open-source infrastructure is ending, and the bots killed it.

The numbers, finally

Open-source operators have been complaining about AI crawler load for two years, usually in adjectives. Ryabitsev's August 29 post is different because it is an operator of one of the most-cloned code hosts on earth publishing measurements. git.kernel.org runs on 90 cores across 5 geo-distributed nodes. At any given moment, 14 to 16 of those cores are rendering commit pages for scrapers, about 20% of total capacity on average, and the load arrives in spiky waves, not a flat line.

His one-line summary: kernel.org now spends more CPU on rendering commits for scrapers than on every kind of legitimate access combined, git clones included.

Why is a git host this attractive? Because kernel history is guaranteed pre-AI text. Ryabitsev's phrasing: training an LLM on LLM output gives it "the equivalent of a digital prion disease," so a corpus that is provably human-written, like 1.48 million kernel commits, is worth its weight in gold to model trainers.

The absurd part is how they harvest it. Everything on kernel.org is a git clone away, the whole history in one efficient transfer. Instead, scrapers walk cgit's web interface commit by commit, and since git.kernel.org hosts about 922 forks of linux.git, each with the same 1.48 million commits plus per-commit patch, diff, and plain-text views, the crawlable URL space runs to the billions, almost all of it duplicates. The industry that talks the most about intelligence is consuming this data in the least intelligent way available.

An arms race in four escalations

The post doubles as a two-year war diary, and every defense bought months, not years.

First the bots identified themselves in their user-agent, and banning was trivial. Then they faked browser user-agents, so kernel.org banned IPs, then subnets, then entire ASNs. The counter-move was the ugly one: crawlers began arriving from millions of residential and mobile IPs, each making 4-5 requests and never appearing again, which makes IP-based banning worse than useless. That is the "proxy SDK monetization" economy, where app and smart-TV makers sell their users' idle connections as exit nodes. Ryabitsev's aside is that your TV is probably doing it.

So about a year ago kernel.org deployed Anubis, the proof-of-work wall built by Techaro's Xe Iaso that now has around 21,900 GitHub stars and sits in front of much of the FOSS web. Anubis is a toll booth that charges in pocket lint: each visitor must brute-force a string whose sha256 hash, combined with their IP and a server secret, starts with N zeroes. A human pays a few seconds of CPU once; a fleet re-running the route millions of times a day pays ruinously. For a few months, the bots simply gave up.

Then they started solving difficulty 4. kernel.org raised it to 5, which takes a few seconds on a phone and makes it uncomfortably warm. Months of peace. Now they solve difficulty 5 too: of the 6 million daily commit-page requests, 66% still bounce off the challenge, but 33% do the math and get through. Somebody ran the economics and decided pre-AI kernel history is worth the compute. The toll booth still stands. The fleet just budgeted for it.

This is a trend, not one grumpy sysadmin

kernel.org is the newest and best-quantified data point in a pattern the FOSS world has been documenting since early 2025. The Wikimedia Foundation reported a 50% jump in multimedia bandwidth since January 2024, driven by scrapers harvesting openly licensed images for training, and wrote a 20% crawler-request reduction target into its annual plan. GNOME's GitLab went behind Anubis. SourceHut spent 2025 fighting the same swarms. What kernel.org adds is precision: a named operator, a full request breakdown, and a CPU bill.

Note what is not the problem: actual outages at kernel.org come mostly from badly designed CI systems shallow-cloning stable.git from 20 nodes at once, not from scrapers. The scraper cost is quieter, a permanent background tax on capacity that someone else donated for a different purpose.

What breaks next

The part that should worry you if you run anything public: the proof-of-work assumption is dead. Anubis-style challenges work by making requests cost more than the content is worth, and the market just repriced the content. When training-data demand is high enough to fund solving sha256 puzzles at difficulty 5 from residential proxies, no challenge difficulty is both bot-proof and human-tolerable, because the legitimate mobile user hits the pain threshold long before the fleet does.

kernel.org's actual mitigation is therefore surface reduction: turning off cgit features to shrink the crawlable URL space and gating expensive operations away from anonymous access. Ryabitsev is explicit that functionality will be lost and that the data stays downloadable for anyone who asks, with more hoops. For self-hosters the transferable lesson is the same: challenges buy months, shrinking what an anonymous GET can make your CPU do buys years. If an endpoint can render arbitrary diffs between arbitrary commits, the swarm will find it.

There is also a takeaway for the AI side of the aisle, which is that this is a self-inflicted wound. Every one of these labs could get the identical corpus with one clone and local rendering at a fraction of their own cost, never mind kernel.org's. Until then, open infrastructure keeps sliding toward logins, tolls, and disabled features, and the open web that made the training corpora possible gets a little less open each quarter.

Key Takeaways

• kernel.org published operator-grade numbers: ~6M scraper requests/day, ~2% of traffic legitimate, and 14-16 of 90 cores permanently rendering commits for bots.
• Scrapers now solve Anubis proof-of-work at difficulty 5; 33% of challenged requests pay the toll and get through, ending the assumption that PoW walls hold.
• The swarm arrives via residential and mobile proxies, 4-5 requests per IP, making IP and ASN bans useless.
• Kernel history is prized precisely because it is guaranteed pre-AI, human-written text, and scrapers harvest it commit-by-commit over HTML instead of one git clone.
• kernel.org's response is surface reduction: fewer crawlable URLs and expensive operations gated away from anonymous users, with real functionality loss.
• With Wikimedia, GNOME, and SourceHut reporting the same pressure, unmetered anonymous access to open infrastructure is measurably closing down.

Sources: Konstantin Ryabitsev, "Creepy crawlies", LWN, Anubis (TecharoHQ), Wikimedia Diff