Pew Scanned 490,000 Pages. A Third of the New Web Is AI.

TL;DR

On August 20, Pew Research Center published a measurement of how much of the web is written with AI. The team sampled 490,000 English-language pages from Common Crawl, scored them with an open-weights detector, and found that 10% of a random July 2026 sample carries meaningful signs of AI authorship. Filter that sample down to pages actually published after ChatGPT shipped in November 2022, and the share climbs to over one third. On .com domains the rate went from 1.09% in 2021 to 9.35% in 2026. Pew also tracked the stylistic tells, and the em dash rate roughly doubled, which this blog will take as vindication of a house style rule.

What they actually measured

The methodology is unusually legible for this genre. Pew pulled 10,000 English-language pages from each of the 49 Common Crawl crawls created between January 2021 and July 2026, giving 490,000 documents spanning roughly two years before ChatGPT and three and a half years after.

Each page went through editlens_Llama-3.2-3B, the open-weights detector released by Pangram as part of Open Pangram. It returns a score from 0 (human) to 1 (AI). Pew classified anything at 0.2 or above as showing meaningful signs of AI authorship.

That threshold is the important design choice, and it is deliberately not a binary. The model is built to detect AI involvement, including light editing passes, not just fully generated text. A page where a human wrote the draft and a model tightened every paragraph scores as touched. So does a page a model wrote start to finish.

Pew validated the open model against Pangram's commercial 3.3 detector across 62,370 pages and reported 96% agreement with a Cohen's kappa of 0.61. That is moderate agreement, not near-perfect. Think of it as a good citywide weather forecast: reliable about the region, not a promise about your backyard. Pew says as much, warning that the findings apply to large collections of texts and not to individual documents.

10% is the wrong number. A third is the right one.

The headline stat everyone will repeat is 10%, and it is close to meaningless on its own. A random crawl sample is dominated by pages that have been sitting on the web since 2014. No model touched them, and no model ever will. Their presence dilutes the rate toward zero.

Restrict to pages with a detectable publication date after November 2022 and the number goes to over a third. That is the figure that describes what is being added to the web right now, which is the only number that matters if you are assembling a corpus, running retrieval, or trying to reason about what your next base model will eat.

The web splits along the TLD

The domain breakdown is the cleanest finding in the study, because the gap is enormous and the explanation is obvious. In the 2026 samples, around one in ten .com pages showed AI authorship signs. On .org it was 4.6%. On .edu and .gov it sat around 1%.

That is roughly ten times the rate on commercial domains versus institutional ones. Nobody is paid per page on a .gov site. Nobody publishes to a university domain to win a long-tail keyword. The incentive to generate volume lives almost entirely on .com, and the detector found it exactly where you would expect.

The tells, measured

Pew also counted four stylistic markers across the sampled web text, per 10,000 words, comparing 2023 with 2026. Every one of them rose.

• Oxford commas: 34.04 to 55.51, a 63% increase.
• AI-typical vocabulary: 11.94 to 26.02, more than double.
• Em dashes: 5.79 to 11.19, about double.
• Negative parallelism (the "it is not X, it is Y" construction): 0.87 to 2.36, nearly triple off a small base.

AI Bacon bans the em dash on sight as an editorial rule. Pew has now supplied the receipt, and we intend to be insufferable about it.

Read that chart the right way. An individual page with two em dashes and an Oxford comma tells you nothing at all, and using these markers to accuse a specific writer is how you end up apologizing to a human who simply reads a lot of Strunk and White. The signal only exists in aggregate, across hundreds of thousands of documents, which is exactly the scale Pew worked at.

Why this lands on your desk

Your pretraining corpus is downstream of this

Common Crawl is not an obscure archive. It is the substrate under nearly every open pretraining pipeline in existence. If a third of newly published pages carry model fingerprints, then every crawl from here forward is partly a recording of previous models talking. The feedback loop that researchers have been theorizing about for three years now has a measured rate attached to it.

Pre-2022 crawls just became an asset

The flip side of that same finding: the 2021 and early 2022 crawls in Common Crawl's archive are the last large-scale snapshots of the open web with essentially no generative contamination. At 1.09% on .com in 2021, they are as clean as a web-scale corpus is ever going to be again. Expect to see dataset cards start advertising their crawl cutoffs the way olive oil advertises a harvest year.

RAG grounds on the same web

If you run retrieval over general web content, your grounding documents come from the same distribution Pew just measured. "Grounded in sources" is doing less work than it used to when a growing share of those sources were themselves generated. This is not an argument against RAG. It is an argument for caring which corpus you point it at.

Detection policy needs to stay boring

A kappa of 0.61 is fine for population statistics and completely unfit for adjudicating whether one student, one applicant, or one contributor used a model. Pew is careful about this. Anyone quoting Pew in a disciplinary hearing will not be.

The caveats that matter

• Only 10% to 15% of sampled pages had a detectable publication date. The post-ChatGPT estimate describes dated content, not the web as a whole. That is the study's own framing, and it is the biggest limitation on the headline number.
• Paywalled and login-gated sites are underrepresented. Common Crawl sees what is publicly fetchable, which skews the sample toward open commercial content.
• English only. No claim is made about the rest of the web.
• Detectors are probabilistic. Pew states plainly that the model misclassifies in both directions and that results hold for large collections rather than single documents.
• "Signs of AI authorship" is not "written by AI." The 0.2 threshold catches editing assistance as well as generation, and the study does not separate the two.

Key Takeaways

• Pew scored 490,000 English-language Common Crawl pages from 49 crawls spanning January 2021 to July 2026 using the open-weights editlens_Llama-3.2-3B detector at a 0.2 threshold.
• 10% of a random July 2026 sample shows meaningful signs of AI authorship; restricted to pages published after ChatGPT's launch, it is over one third.
• .com pages hit 9.35% in 2026, up from 1.09% in 2021, versus 4.6% on .org and roughly 1% on .edu and .gov.
• Stylistic markers rose across the board from 2023 to 2026 per 10,000 words: em dashes 5.79 to 11.19, Oxford commas 34.04 to 55.51, AI-typical vocabulary 11.94 to 26.02, negative parallelism 0.87 to 2.36.
• Validation against Pangram's commercial model showed 96% agreement and a Cohen's kappa of 0.61, which is population-level evidence, not per-document proof.
• If you build corpora, run retrieval, or fine-tune on scraped text, the crawl date of your data is now a quality parameter.

Sources: Pew Research Center: How Much of the Internet Is Written With AI?, Pew Research Center: Methodology, TechCrunch, Pangram: Introducing Open Pangram, Common Crawl.