Glonce

FineBooks tests 14 OCR models to fix AI training data quality

researchThe Glonce editorsThe Decoder🔥68Glonce rating68Impact62Novelty68Relevance87Surprise47Players65Credibility82
FineBooks tests 14 OCR models to fix AI training data quality

Training open AI language models on public-domain books runs into a quality problem: libraries scanned those books years ago and ran optical character recognition that is often full of errors. The Talkie project measured the cost of that error: a language model trained on OCR text learned at only 30 percent the efficiency of one trained on human transcriptions of the same books.

FineBooks, a joint project from Hugging Face and EleutherAI, set out to test whether today's open-source OCR models can close that gap. The team ran 14 open-weight OCR models, all of which run locally without an API key, on 2,165 pages from historical books and published the results as a public leaderboard. To score accuracy they used Character Error Rate (CER) against a ground-truth dataset: between 2011 and 2012, experts working on the IMPACT project and BHL-Europe transcribed six volumes from the Biodiversity Heritage Library in English, French, German, and Latin, with an error rate of about one character per 2,000. The leaderboard reports two variants: a "diplomatic" score that counts modernizing archaic characters like the long s as an error, and a "reading" score that tolerates it.

The best models reached over 97 percent character accuracy at a processing cost of less than two U.S. dollars per thousand pages. Model size did not predict quality. The top model, dots.mocr, uses just 3 billion parameters and beat Qwen3.5-9B, a model nearly three times as large. Second place went to OvisOCR2, at only 0.9 billion parameters and 46 cents per thousand pages.

The motivation is scale. When EleutherAI and its partners released Common Pile last year, the largest openly licensed training corpus to date, it included about 300,000 public-domain books carrying text from older, noisier OCR runs. The FineBooks authors argue that reprocessing those books with better models is one of the most effective ways to improve open AI training datasets. As a first target they picked the Biodiversity Heritage Library, which holds more than 300,000 digitized natural history documents totaling over 64 million pages and offers its collection as a bulk AWS download. The team plans to reprocess about 200,000 public-domain BHL documents with one of the top-scoring models and release the resulting text as an open dataset; new models are added to the leaderboard on a rolling basis.

The evaluation has defined limits: it covers only Antiqua typefaces in four languages, and does not test Fraktur, non-Latin scripts, handwriting, or multi-column layouts. The FineBooks authors judge the results as good enough for training language models, but not for scholarly or scientific use. The shortfall for scholars is not misread characters: the models silently modernize archaic characters and ligatures, such as swapping the long s for a modern s, which the team says targeted fine-tuning could fix. Libraries face a separate obstacle: their systems depend on the ALTO XML format for word-level coordinates, and the new OCR models output only Markdown or plain text without word positions, so they cannot plug into existing library infrastructure.

Key facts

  • FineBooks, a Hugging Face and EleutherAI collaboration, benchmarked 14 open-weight OCR models on 2,165 historical book pages.
  • The top model reached over 97 percent character accuracy at under two dollars per thousand pages.
  • The leading model, dots.mocr, uses only 3 billion parameters and outperformed the nearly three times larger Qwen3.5-9B; second-place OvisOCR2 runs at 0.9 billion parameters and 46 cents per thousand pages.
  • The team plans to reprocess about 200,000 public-domain documents from the Biodiversity Heritage Library, which holds over 300,000 documents totaling more than 64 million pages.
  • Results are judged good enough for AI training but not for scholarly use, since models silently modernize archaic characters and their output lacks the ALTO XML word-level coordinates libraries need.

Why it matters

Open AI training corpora like Common Pile lean on public-domain books whose text came from old, error-prone OCR runs; the Talkie project found this cuts training efficiency to 30 percent of what human transcriptions achieve. FineBooks is the first systematic attempt to measure which of today's open OCR models can fix that cheaply enough to reprocess collections at scale.

Who it affects

Groups building open training datasets, including EleutherAI and Hugging Face, benefit most directly, since better OCR feeds directly into corpora like Common Pile. The Biodiversity Heritage Library, chosen as the first reprocessing target, and similar archives stand to get cleaner digitized text. Libraries and scholars are explicitly not well served yet: the output format and residual errors don't meet their needs.

How to use it

All 14 tested models are freely available and run on local hardware without an API key, and the leaderboard and evaluation framework are public, with new models added on a rolling basis. dots.mocr (3B parameters, under $2 per thousand pages) leads on accuracy; OvisOCR2 (0.9B parameters, 46 cents per thousand pages) is the cheaper second-place option. The team plans to release reprocessed text from about 200,000 BHL documents as an open dataset.

How solid is it

The benchmark used a ground-truth dataset from the IMPACT project and BHL-Europe: expert transcriptions from 2011-2012 of six Biodiversity Heritage Library volumes in English, French, German, and Latin, with an error rate of about one character per 2,000, available under a CC-BY license. Accuracy was measured as Character Error Rate across 2,165 pages and 14 models, with separate scores for whether modernizing archaic characters counts as an error.

Risks and caveats

The evaluation covers only Antiqua typefaces in four languages and does not test Fraktur, non-Latin scripts, handwriting, or multi-column layouts, so the results may not generalize beyond that scope. Even the top models silently modernize archaic characters and ligatures rather than reproducing them faithfully, which the team says undermines scholarly use unless fixed through targeted fine-tuning. The models also output Markdown or plain text without word-level positions, so they cannot plug into library systems built on the ALTO XML format.