HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
getomni-ai
benchmark
TypeScript2025–2025
getomni-ai / benchmark

benchmark

OCR Benchmark

2Distinct people
10Mentions
0Explicit recommenders
639GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

OCR Benchmark

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 11:39:46 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by week10 dated mentions

Observed mentions: 2025-02-20 18:49:29 UTC to 2025-04-01 23:06:27 UTC. Weeks start Monday, UTC.

0123452025-02-17 00:00:00 UTC to 2025-02-24 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsFeb 172025-02-24 00:00:00 UTC to 2025-03-03 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-03-03 00:00:00 UTC to 2025-03-10 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2025-03-10 00:00:00 UTC to 2025-03-17 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsMar 102025-03-17 00:00:00 UTC to 2025-03-24 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-24 00:00:00 UTC to 2025-03-31 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-03-31 00:00:00 UTC to 2025-04-07 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 commentsMar 31

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Feb 172025-02-17 00:00:00 UTC2025-02-24 00:00:00 UTC110
Feb 242025-02-24 00:00:00 UTC2025-03-03 00:00:00 UTC101
Mar 32025-03-03 00:00:00 UTC2025-03-10 00:00:00 UTC505
Mar 102025-03-10 00:00:00 UTC2025-03-17 00:00:00 UTC000
Mar 172025-03-17 00:00:00 UTC2025-03-24 00:00:00 UTC000
Mar 242025-03-24 00:00:00 UTC2025-03-31 00:00:00 UTC101
Mar 312025-03-31 00:00:00 UTC2025-04-07 00:00:00 UTC211

What Hacker News says

5 sources
We also ran an OCR benchmark with LLM as judge using structured outputs. You can check out the full methodology on the repo [1]. But the general idea is: - Every document has ground truth text, a JSON schema, and the ground truth JSON. - Run OCR on each document and pass the result to GPT-4o along with the JSON Schema - Compare the predicted JSON against the ground truth JSON for accuracy. In our benchmark, the ground truth text => gpt-4o was 99.7%+ accuracy. Meaning whenever gpt-4o was given the correct text, it could extract the structured JSON values ~100% of the time. So if we pass in the OCR text from Mistral and it scores 70%, that means the inaccuracies are isolated to OCR errors. https://github.com/getomni-ai/benchmark
In Mistral OCR
Update: Just ran our benchmark on the Mistral model and results are.. surprisingly bad? Mistral OCR: - 72.2% accuracy - $1/1000 pages - 5.42s / page Which is pretty far cry from the 95% accuracy they were advertising from their private benchmark. The biggest thing I noticed is how it skips anything it classifies as an image/figure. So charts, infographics, some tables, etc. all get lifted out and returned as [image](image_002). Compared to the other VLMs that are able to interpret those images into a text representation. https://github.com/getomni-ai/benchmark https://huggingface.co/datasets/getomni-ai/ocr-benchmark https://getomni.ai/ocr-benchmark
In Mistral OCR
Update: Just ran our benchmark on the Mistral model and results are.. surprisingly bad? Mistral OCR: - 72.2% accuracy - $1/1000 pages - 5.42s / page Which is pretty far cry from the 95% accuracy they were advertising from their private benchmark. The biggest thing I noticed is how it skips anything it classifies as an image/figure. So charts, infographics, some tables, etc. all get lifted out and returned as [image](image_002). Compared to the other VLMs that are able to interpret those images into a text representation. https://github.com/getomni-ai/benchmark https://huggingface.co/datasets/getomni-ai/ocr-benchmark https://getomni.ai/ocr-benchmark
In Mistral OCR
Excited to test this our on our side as well. We recently built an OCR benchmarking framework specifically for VLMs[1][2], so we'll do a test run today. From our last benchmark run, some of these numbers from Mistral seem a little bit optimistic. Side by side of a few models: model | omni | mistral | gemini | 86% | 89% | azure | 85% | 89% | gpt-4o | 75% | 89% | google | 68% | 83% | Currently adding the Mistral API and we'll get results out today! [1] https://github.com/getomni-ai/benchmark [2] https://huggingface.co/datasets/getomni-ai/ocr-benchmark
In Mistral OCR

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.