97.6% on 300-Year-Old Books. Then Came the Blank Pages.

TL;DR

On August 10, Hugging Face and EleutherAI published FineBooks, starting with a leaderboard that scores 14 open-weight OCR models on all 2,165 pages of an expert-transcribed corpus of 18th and 19th century natural history books. The top models read old print at roughly 97.6% character accuracy, and the whole run costs between $0.34 and $1.94 per 1,000 pages on rented GPUs. The interesting failure is not archaic type. It is that several models, handed a plate or a blank leaf, write paragraphs that were never there.

The benchmark nobody had

Measuring OCR needs pages whose correct transcription is already known, character for character. Almost nobody has that for historical books. FineBooks found a set: between 2011 and 2012, the EU's IMPACT digitisation programme and BHL-Europe hand-corrected six volumes from the Biodiversity Heritage Library, 2,165 pages in English, French, German and Latin, down to roughly one error in two thousand characters.

Both projects are long dead and their websites are gone. The transcriptions survived because they were released under CC-BY and preserved in a GitHub repository by IMPACT's technical manager, Clemens Neudecker. Somebody in 2012 picked an open license, which turns out to be the least glamorous form of time travel available to us: none of the models on this board existed then, and most of their architectures had not been invented.

The rebuilt ground truth ships as finebooks/bhl-impact-gt, with every scan matched to its transcription, and the scorer ships as bhl-ocr-eval. One methodology note deserves to be copied by every other leaderboard: they ran every model themselves under a pinned container image, pinned model revision, pinned script commit and a recorded job id. Scores from hosted inference routers are refused outright, because a router will not tell you which provider served the request, at what quantization, or under what serving config, and character error rate moves with all three.

The board

The headline metric is character error rate on the reading lane, which folds case, the long s and ligatures into their modern equivalents. dots.ocr and dots.mocr land at 2.35% and 2.37%, which the project's own results file calls statistically identical. OvisOCR2 takes third at 3.05%, PaddleOCR-VL-1.6 fourth at 3.92%.

Now read that chart again from the bottom. Tesseract 5, a non-neural engine whose lineage predates the transformer by decades, comes in at 6.42% and lands within a point and a half of Qwen3.5-9B, the largest model on the board. It also never once fell into a repetition loop, which is more than any neural row can say.

Cost splits the top differently than accuracy does. dots.mocr costs $1.94 per 1,000 pages on Hugging Face Jobs; PaddleOCR-VL-1.6 costs $0.34 for 96.1% accuracy. That is 5.7x the price for 1.5 points of characters, which is a trade you should make deliberately rather than by grabbing the top row.

Params are measured, not quoted

The board sums every weight in each repo's safetensors index, vision towers included, instead of trusting the name. GLM-OCR, usually listed at 0.9B, is 1.33B. olmOCR-2-7B is 8.29B. Winning a size class is easier when you get to pick the number on the label.

The blank-page problem

428 of the 2,165 pages are sparse: engraved plates, blanks, near-empty leaves whose entire correct transcription might be one line of credits. The board scores them as their own stratum, and this is where the models stop resembling each other.

Character error rate is a ratio: mistakes divided by the number of characters that should be on the page. When the denominator is tiny and the model invents text, the ratio blows past 1.0 and keeps going. A sparse CER of 26.93 means the model produced roughly 27 wrong characters for every real one. It is the OCR equivalent of an eyewitness giving a detailed description of a suspect at a crime scene where nobody was standing.

That contrast is the finding. GLM-OCR reads ordinary content pages at 2.36% CER, better than everything except the dots pair, and then produces 26.93 on sparse pages. LightOnOCR-2 posts 15.79, olmOCR-2 posts 13.64, Qwen3.5-9B posts 19.98. Meanwhile dots.ocr sits at 1.79 and DeepSeek-OCR-2 at 2.09.

If you are building a document pipeline, this is the column to weight, because real collections are full of plates, blanks, dividers and ads. An aggregate CER that averages over them will quietly hide a model that hallucinates whenever the page runs out of words.

Why this matters if you never touch a library

Public domain books are one of the largest pools of long-form text that anyone can legally train on, and the text quality is currently set by whatever OCR tool a library happened to run once, at scan time, years ago. FineBooks cites the Talkie project's measurement of what that costs: a language model trained on OCR-derived text learned at 30% of the efficiency of the same model trained on human transcriptions of the same books.

That reframes OCR from a digitisation chore into a training-data multiplier. When EleutherAI and collaborators shipped Common Pile, the largest openly licensed corpus to date, it carried around 300,000 public domain books whose text came from those earlier pipelines. Re-running them is, in the project's words, one of the highest-leverage improvements available to open training data.

The next phase is already stated. The Biodiversity Heritage Library holds more than 300,000 digitized items and more than 64 million pages; roughly 200,000 of those items carry public domain rights statements, and FineBooks intends to re-run OCR across them with a leading model and publish the result as its first dataset. Do the arithmetic on their own cost figures and 64 million pages runs about $22,000 of GPU time at $0.34 per thousand, or about $124,000 at $1.94. The public domain slice is smaller than that, which puts a full re-processing somewhere between a grant line item and a rounding error in a frontier training budget.

What the board does not tell you

The authors are unusually direct about scope, so take the caveats at full strength:

• Four languages, one typeface family. English, French, German and Latin, all antiqua. The board says nothing about German Fraktur, non-Latin scripts, non-European languages or handwriting.
• Books only. Single-column running text with consistent layout. Newspapers, magazines and archival material have different failure modes that are not measured here.
• The confidence intervals are wide on purpose. The bootstrap resamples the six volumes rather than individual pages, so it is honest about book-to-book variation. Where intervals overlap, the models are tied, and the top two overlap.
• Loop rate is a selection effect. Pages where a model repeats itself into its token cap are excluded from that row's other numbers, so heavy loopers look better than they are. PaddleOCR-VL-1.6 loops on 6.47% of pages; the top row is scored on 96.1% of the corpus.

The verdict splits by use case. Good enough for LLM training corpora: yes, comfortably better than the legacy text sitting in open corpora today. Good enough for libraries replacing legacy OCR: it depends, because legacy pipelines emit ALTO XML with word-level coordinates and these models emit Markdown or plain text, sometimes with region boxes but never word positions. Good enough for scholarly transcription: not quite, since the models silently modernise the long s and ligatures that a diplomatic transcription exists to preserve. SmolDocling at 19.65% CER is a reminder that the size floor is real, too.

Key Takeaways

• FineBooks, a Hugging Face and EleutherAI collaboration, scored 14 open-weight OCR models on all 2,165 pages of expert-transcribed 18th and 19th century books, with the ground truth, harness and per-run provenance published alongside.
• dots.ocr (2.35% CER) and dots.mocr (2.37%) lead and are statistically tied; the largest model on the board, Qwen3.5-9B at 9.65B parameters, finishes eighth at 5.07%.
• On the 428 sparse or blank pages, several models invent text wholesale: GLM-OCR scores 2.36% on content pages and 26.93 on sparse ones, with Qwen3.5-9B at 19.98 and olmOCR-2 at 13.64.
• Cost runs $0.34 to $1.94 per 1,000 pages on rented GPUs, which makes re-OCR of an entire collection a budgeting question rather than a research question.
• The motivation is training data: a model trained on OCR-derived text learned at 30% of the efficiency of one trained on human transcriptions, and Common Pile carries around 300,000 books produced by older pipelines.
• Scope is narrow by design: antiqua typefaces, four languages, single-column books, no Fraktur, no handwriting, no newspapers.

Sources: FineBooks: are open OCR models good enough to unlock historical knowledge?, BHL OCR Leaderboard, finebooks/bhl-ocr-eval, finebooks/bhl-impact-gt, The Decoder