Scrapers Learned to Solve Anubis. The Fix Took a Year in Rust.

TL;DR

Anubis, the proof-of-work gate that GNOME, FFmpeg, Wine, FreeCAD, UNESCO and the Linux kernel's git server now sit behind, spent a year moving its challenge into Rust compiled to WebAssembly. The reason is not performance theater. The scrapers started solving the old puzzle.

At git.kernel.org, 33% of roughly 6 million daily requests now solve the challenge and walk in. The new challenge is memory-hard instead of CPU-hard, and it counts difficulty in bits instead of nibbles, so admins can finally turn the dial by 2x rather than 16x.

The gate stopped working

On August 29, Konstantin Ryabitsev, who runs kernel.org's git infrastructure, published the numbers behind the escalation in a post called Creepy crawlies. The arc is short and grim.

Anubis went in front of everything and worked immediately. The bots gave up. Months later they came back solving difficulty 4, so kernel.org raised it to 5. Difficulty 5 takes a few seconds on a phone and makes the phone uncomfortably warm, but it bought a few more months. Then the bots started solving difficulty 5 too.

We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.

Today git.kernel.org takes about 6 million daily requests demanding random commits. Two thirds are still batted away at the challenge. A third are paying the compute cost and getting through. With what Ryabitsev calls a bunch of generous assumptions, actual humans account for roughly 2% of the traffic.

The scale of what they are scraping explains the persistence. The linux.git repository holds about 1.48 million commits, and git.kernel.org carries roughly 922 forks of it. On the backend that is cheap, because the forks share objects. To a scraper following every patch and diff link, it is several billion valid URLs that resolve to 922 duplicates of the same 1.48 million commits. Picture a library that photocopied every book 922 times and shelved each copy in a different room, then hired someone who insists on reading all of them.

The cost lands on real hardware. Out of 90 cores across five geo-distributed nodes, 14 to 16 are permanently occupied rendering commits into HTML for bots. That is 20% of total capacity as a flat average, and the swarms arrive in waves, so the real graph is spikier than that.

Why "just raise the difficulty" ran out of road

Here is the design flaw that made the arms race so painful. The original fast challenge counted leading zero nibbles in a hash rather than leading zero bits. A nibble is four bits, so adding 1 to the difficulty made the puzzle sixteen times harder in the worst case.

That leaves an admin with a dial that has almost no usable positions. Difficulty 5 already cooks a mid-range phone. Difficulty 6 is sixteen times that. There was never a setting between "the bots are getting through" and "your visitors on a Moto G8 give up."

The v1.28.0-pre1 release notes publish a translation table for admins migrating settings. A fast difficulty of 4 maps to argon2id 4, sha256 16, and hashx 15. A fast 6 maps to sha256 24. If your config currently says 5, you now have every integer between 16 and 24 to choose from instead of a cliff.

Memory-hard is the part that actually bites

The bigger change is what the puzzle asks for. The WebAssembly path can use argon2id, a memory-hard function, instead of a pure CPU-hard SHA-256 grind. Anubis author Xe Iaso puts the goal bluntly in the release writeup: the "vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.

That distinction is the whole ballgame, so it is worth being precise. SHA-256 is arithmetic and nothing else, which is exactly what a GPU is built to do thousands of times in parallel, as a decade of cryptocurrency mining hardware demonstrates. A memory-hard function forces every single attempt to allocate a large block of memory and bounce around it unpredictably. Think of it as the difference between a mental-arithmetic test, where you can seat ten thousand candidates in a hall and they all work at once, and an open-book exam where each candidate needs their own desk stacked with reference volumes. The second one is capped by furniture, not by brainpower.

Scrapers can still solve it. They just cannot solve it cheaply by renting a GPU, which changes the economics of grinding 6 million challenges a day.

One binary, both sides of the gate

The other reason for WebAssembly is deduplication. The fast challenge currently exists twice: once in TypeScript for the browser, once in Go for the server. Every change has to land in both, and every assumption about challenge shape elsewhere in the codebase has to be updated too.

With WASM, the client and the server run the same binary, not merely the same algorithm. The challenge code is Rust, no-std, targeting wasm32-unknown-unknown, and it ships in two builds: a baseline one restricted to WebAssembly features present in Chrome 75, and a SIMD build for browsers from Chrome 91 onward, selected at runtime by feature detection.

Clients that disable WebAssembly but keep JavaScript, notably iOS Lockdown Mode and GrapheneOS' Vanadium browser, get a fallback produced by compiling the WASM itself to JavaScript with binaryen's wasm2js. It is slower, because those clients usually disable the JIT as well, and the progress bar does not currently update on that path. It is still one implementation, which was the point.

Where the year actually went

The interesting part of the writeup is that almost none of the year went into the proof-of-work code. It went into the boring compatibility floor.

• Chrome 75 is the floor, and it is not negotiable. Smart TVs and Android devices are marooned on that build with no upgrade path, so the JavaScript is compiled for Chrome 66 out of caution.
• Rust's precompiled standard library broke old Chrome. A stdlib function used WebAssembly reference types, which made Chrome 75 through 100 choke on a table index it did not expect. The fix was stripping features back with wasm-opt -mvp, with the working flag set found by an AI fuzzing loop.
• Build tools ship in the repo. wasm-opt and wasm2js are compiled to WASI and committed, then run under wasmtime in CI. As Iaso puts it, your build tools cannot differ from theirs if they ship you the build tools they use.
• Browser testing became its own product. A harness called chromesweep spawns many Chrome versions against a live Anubis, each pod wrapped in a Kata microVM behind a restrictive Kubernetes NetworkPolicy, because Chrome that old is a live security threat. The browser image library is open source as TecharoHQ/gubal.

And then there was the compiler. Builds of binaryen kept drifting by about 29 bytes each time. Disabling ASLR with setarch --addr-no-randomize made them reproducible within a single boot, which is the tell: something was iterating over machine pointers. It turned out to be an LLVM bug, llvm-project#204883, filed June 19 and closed June 27. Nondeterministic output because a WebAssembly backend pass used machine pointers as DenseMap keys. Iaso notes it is the first compiler bug of their career, which is a milestone most engineers get to skip.

What to do if you run Anubis

The WebAssembly challenges are off by default in v1.28.0. Iaso plans to make them the default in v1.29.0 once administrators have kicked the tires. Enable them per threshold or per bot rule with an algorithm and difficulty under the challenge block, documented on the WASM challenges page.

Two practical warnings. First, the new code is fast enough that you may need to raise your difficulty rather than lower it. Second, the mobile-fairness problem is not solved: the classifier meant to give phones grace, using IP reputation and TLS fingerprinting, is still a prototype and Iaso is openly asking for performance data.

One footnote worth savoring. The person building the wall against AI scrapers used a Claude Opus and GLM 5.2 loop to fuzz their compiler flags, then closed the post by noting that no AI touched the prose and linking the Google Doc revision history as proof. The tooling and the threat model are the same technology, and they are entirely comfortable saying so.

Key Takeaways

• Proof of work has a shelf life. At git.kernel.org the bots went from giving up, to solving difficulty 4, to solving difficulty 5, and now clear 33% of about 6 million daily challenges.
• The old difficulty scale was the real bug. Counting leading zero nibbles meant every step up was 16x harder. The WebAssembly challenges count bits, so a step is 2x and admins get a usable dial.
• Memory-hard beats CPU-hard against rented GPUs. Moving to argon2id targets the economics of scraping, not the possibility of it.
• One binary on both ends removes a class of bugs. Rust compiled to WASM replaces parallel TypeScript and Go implementations that had to be kept in sync by hand.
• Compatibility is the expensive part. A year went to Chrome 75 support, a wasm2js fallback, committed build tools, microVM browser testing, and an LLVM bug, not to the hashing.
• It is off by default. Test it in v1.28.0 before v1.29.0 turns it on for you, and expect to raise difficulty rather than lower it.

Sources: It took a year to ship WebAssembly in Anubis, Anubis v1.28.0-pre1 release notes, Creepy crawlies (Konstantin Ryabitsev, kernel.org), llvm-project issue 204883, Anubis WASM challenge docs, Anubis (software), Wikipedia