The Stack v3 Ships 5 Trillion Tokens of Open Code Across 770 Languages. It Also Reveals Why v2 Was Quietly Too Small.

TL;DR

On July 23, Hugging Face's code research team published The Stack v3, the largest openly licensed corpus of source code ever assembled: 114 TB of raw code, roughly 5 trillion deduplicated and filtered tokens, spanning 770 programming languages across 224 million repositories and 44 billion files, crawled through August 2025. It ships as two datasets: stack-v3-full (the entire 114 TB) and stack-v3-train (the cleaned, PII-redacted, near-deduplicated split you actually train on). Scale is the headline, but the more interesting line is buried in the notes: while rebuilding deduplication, the team found a bug in The Stack v2's pipeline that had been silently over-deduplicating the data. Fix that, add two more years of GitHub, and the training-ready corpus jumps from about 550 billion tokens to roughly 5 trillion. That is about 9x the usable tokens off only 1.7x more raw data.

What actually dropped

The Stack is the open dataset that trains open code models. v1 and v2 came out of BigCode, the Hugging Face and ServiceNow collaboration behind the StarCoder series. v3 is the successor, and this time it comes in two flavors so you pick your own tradeoff. stack-v3-full is everything, unfiltered, for teams that want to run their own cleaning. stack-v3-train is the near-deduplicated split with quality heuristics and PII redaction applied, one representative kept per duplicate cluster, and rows grouped so each row is a whole repository. That train split is the roughly 5-trillion-token number.

The license matters as much as the size. The whole thing is released under Open Data Commons Attribution (ODC-By 1.0), and the maintainers say no restrictively licensed code is included. This is a corpus you can legally build on, not a scrape you have to pretend you never saw.

The self-contained part matters more than it sounds

v2 shipped you file IDs and object-store pointers. To get the actual code you had to go fetch every file yourself, which for a corpus this size is a small engineering project in its own right. It handed you a treasure map and wished you luck. v3 embeds the file contents inline, so you can start training the moment the download finishes. For anyone who ever tried to reconstitute v2 from its pointers, that change alone justifies the re-crawl.

The dedup bug that was starving v2

Here is the technically interesting part. At this scale you cannot spot near-duplicate files by comparing them byte for byte, so The Stack uses MinHash deduplication: it hashes overlapping shingles of tokens into compact fingerprints and clusters files whose fingerprints collide. v2's pipeline tokenized those shingles incorrectly, which made unrelated files look more alike than they were. The dedup step then binned originals it should have kept.

Picture a library weeding out duplicate books by comparing smudged call numbers. Smudge them enough and distinct titles start reading as copies, so the librarian throws out perfectly good originals. That is what happened to v2: it had been quietly on a diet. v3 fixes the tokenization, and a large share of the jump to roughly 5 trillion tokens is simply data that v2 wrongly discarded, on top of a fresh crawl through August 2025.

Where the growth landed

The new tokens are not spread evenly. The languages that gained the most are the systems and typed ones where open code models have always been thinnest, which is exactly where an agentic coding tool tends to fall over.

Why a builder should care

You may never train a foundation model, but the open coding models you download do. Better open code data feeds better open code models, which is the whole game if you want a StarCoder-class assistant you can self-host instead of renting from a frontier lab. Fine-tuners and domain adapters get a permissively licensed, self-contained base to start from, and researchers get a clean corpus with a documented pipeline instead of a mystery scrape.

The v2 story is the quiet lesson here. A single tokenization bug in a preprocessing step, not the model architecture, capped how much data an entire generation of open code models got to see. Data pipelines decide model quality just as much as attention layers do, and they fail in ways nobody notices until someone rebuilds them from scratch.

The caveats

• This is a dataset, not models. There is no StarCoder3 yet. v3 is the raw material, and someone still has to spend the compute to turn it into weights.
• "Fully open" is a license claim. The maintainers filtered for permissive licenses, but the broader fight over training on public code, opt-outs, and attribution is not settled by an ODC-By tag on the folder.
• Token counts are tokenizer-dependent. The roughly 5 trillion figure is measured with their tokenizer, so treat cross-corpus token comparisons as approximate rather than exact.
• You still need real infrastructure. The 114 TB full split is unfiltered; most people want the train split, and even that demands serious storage and pipeline muscle to use.

Key Takeaways

• Hugging Face's code team released The Stack v3 on July 23: 114 TB of raw code, roughly 5 trillion deduplicated and filtered tokens, 770 languages, 224M repositories, 44B files, crawled through August 2025.
• It comes as two datasets: a 114 TB full corpus and a cleaned, PII-redacted, near-deduplicated train split, both under the permissive ODC-By 1.0 license.
• A deduplication bug in The Stack v2 had incorrectly tokenized MinHash shingles and over-deduplicated the data, so v2's training set was smaller than it should have been.
• Fixing that plus a fresh crawl took training-ready tokens from about 550 billion to roughly 5 trillion, about 9x, off only 1.7x more raw data.
• Growth concentrated in systems and typed languages: C++ x15, TypeScript x7.5, Rust x7, Python x4.8, with 95 new languages added.
• Unlike v2, v3 embeds file contents inline, so you can train the moment the download finishes instead of fetching every file from pointers.

Sources: The Stack v3 (Hugging Face dataset), StarCoder2 and The Stack v2 (Hugging Face), The Stack v2 paper (arXiv), Latent Space AINews (July 24, 2026)