A Stanford PhD Student Just Made Tokenization 1,000x Faster. The Bottleneck Was the Regex Engine All Along.

TL;DR

Gigatoken, released Wednesday by Marcel Røed, a third-year Stanford PhD student working on ML and systems, is an MIT-licensed Rust tokenizer with Python bindings that positions itself as a drop-in replacement for HuggingFace's tokenizers library and OpenAI's tiktoken. On a dual-socket AMD EPYC server it tokenizes text at 24.53 GB/s with the GPT-2 tokenizer, which the README measures at 989x faster than HuggingFace and 681x faster than tiktoken, while producing byte-identical output in compatibility mode. It hit 400-plus points on Hacker News and 1,300-plus GitHub stars in a day, and installing it is one pip install gigatoken.

What shipped

This is a finished library, not a benchmark repo. Gigatoken supports 30-plus tokenizer families, including GPT-2, Llama 3, Phi-4, Qwen, and DeepSeek, and it ships a compatibility mode where, in the README's words, "a substantial amount of effort has been put into making sure the outputs match exactly" what HuggingFace or tiktoken would produce, plus validation tooling so you can check that claim on your own data instead of taking it on faith.

Røed says he spent the past year finding these optimizations during his PhD, with feedback from Percy Liang, Tatsunori Hashimoto, and Jure Leskovec. The core is about two-thirds Rust with a thin Python layer on top.

The numbers

The README benchmarks on the 11.9 GB OpenWebText corpus, chosen because it resembles what you actually get after extracting text from Common Crawl. Three machines, three very different results, all of them lopsided:

• Dual AMD EPYC 9565 (144 cores): 24.53 GB/s on GPT-2 (989x vs HuggingFace, 681x vs tiktoken), 24.00 GB/s on Phi-4, 22.15 GB/s on Llama 3.
• Apple M4 Max (16 cores): 8.79 GB/s on GPT-2 (1,268x vs HuggingFace, 140x vs tiktoken).
• AMD Ryzen 7 9800X3D (8 cores): 6.27 GB/s on GPT-2, a comparatively modest 106x vs HuggingFace.

At the EPYC machine's rate, the README claims you could tokenize the entirety of Common Crawl in just under 6.5 hours, on one machine. That reframes a job people schedule clusters for as something you run before lunch.

The spread between machines is itself informative: the multiplier depends heavily on how badly the baseline scales on your hardware, so an 8-core desktop sees two orders of magnitude while a laptop-class M4 Max sees more than three.

Where three orders of magnitude were hiding

BPE tokenization has two stages: pretokenization, which splits raw text into word-ish chunks using a pattern every lab defines as a gnarly regular expression, and the merge loop, which turns each chunk into token IDs. The industry consensus since roughly 2019 has been to hand stage one to a general-purpose regex engine and never think about it again. Gigatoken's core finding is that this outsourced, unexamined step is where most of the time goes.

The fix is to stop interpreting and start specializing. A regex engine is a universal tool: it walks a state machine byte by byte and can match any pattern you hand it, slowly. Gigatoken instead hand-writes the split logic in SIMD, so a single instruction classifies a whole lane of bytes against the one specific pattern that tokenizer will ever use. Think of it as replacing a Swiss Army knife with a blade forged in the exact shape of the cut, then making the cut 32 bytes at a time.

The second trick exploits how repetitive text is. Pretoken frequency is brutally long-tailed: the same words appear over and over, so Gigatoken aggressively caches the mapping from pretoken to token IDs, sized carefully against the CPU cache hierarchy, and only runs the actual BPE merge on cache misses. Add minimized branching and fewer Python-to-native round trips, and the GB/s numbers stop looking like a typo.

The Amdahl objection, answered honestly

The top pushback on Hacker News writes itself: tokenization is well under 0.1% of inference time, so a 1,000x speedup of almost nothing is still almost nothing. That is true, and worth saying out loud. Tokenization is 0.1% of your inference bill and 100% of your afternoon when you are the one babysitting the preprocessing job.

Because that is where this actually lands: data pipelines. Pretraining and fine-tuning runs tokenize terabytes before the GPUs do anything, and at HuggingFace's measured ~25 MB/s that stage is a genuine cluster job, while at 22-24 GB/s it collapses onto one machine. Røed also posted measurements showing 5-10% time-to-first-token improvements on 32K-token prompts, which is real money in latency-sensitive serving, and commenters flagged the unglamorous middle ground: token-counting for routing, rate limiting, and billing, where the tokenizer is the entire workload.

Caveats

The benchmarks are the author's own, one day old, and independent replication is just starting; the numbers above are the README's, not ours. WordPiece is unsupported, SentencePiece gets only minimal optimization, some slower tokenizer definitions see 10-20x rather than three digits, Windows is untested (WSL recommended), and there is no file sink yet, so writing tokenized shards straight to disk is on the roadmap rather than in the box. Exact compatibility mode also costs some throughput relative to the headline numbers. And the baseline being beaten, huggingface/tokenizers, is itself a Rust library with a decade of production use behind it; boring and battle-tested still counts for something in a pipeline you run once per training corpus.

Key Takeaways

• Gigatoken is an MIT-licensed, pip install-able Rust tokenizer claiming exact-match compatibility with HuggingFace tokenizers and tiktoken across 30-plus tokenizer families.
• Measured throughput: 24.53 GB/s on a dual-EPYC server and 8.79 GB/s on an M4 Max with the GPT-2 tokenizer, per the project's README.
• The speedup comes from replacing the regex engine with pattern-specialized SIMD, caching pretoken-to-ID mappings to exploit text's long-tailed repetition, and cutting Python round trips.
• The win is in data pipelines, token-counting services, and long-prompt TTFT, not steady-state inference, where tokenization was never the bottleneck.
• Caveats: author-run benchmarks, no WordPiece, minimal SentencePiece, machine-dependent multipliers (106x on an 8-core desktop vs 1,268x on an M4 Max), and no file sink yet.
• The meta-lesson for builders: the decade-old step everyone treats as free plumbing can be hiding three orders of magnitude. Profile the boring parts.

Sources: gigatoken on GitHub, Hacker News discussion, Marcel Røed's announcement, AI Weekly