marker

How does this differ from open source cli tools such as Marker (https://github.com/datalab-to/marker)? Also, it seems disingenuous to have “locally” in the title when your default is cloud inference providers and I don't see you naming a recommended LLM that could be hosted locally and is proven to work well for this task. You list “Your files” under “Stays on your machine” and then require an API key for a cloud provider.

We made Ruminate over the past few weeks because we were frustrated with how hard it is to read complex text with LLMs - research papers, novels, long articles, etc. We found ourselves copy-pasting the text/file into ChatGPT, having chats about different parts of the text in different tabs, and bummed we couldn't easily track the best insights from all those conversations in one place. We wanted one place where we could stay immersed in the text, dip into lengthy and generative rabbitholes when we wanted to (in an unobtrusive way), and have a 'work product' of your learning process (all your notes in one place). No more tab switching - one unified interface centered around the text itself. With Ruminate, you can: 1) Upload PDFs, EPUBs, or web articles and read them in our custom reader (PDF upload done with https://github.com/datalab-to/m

In our benchmarks, https://github.com/datalab-to/marker is the best if you need to deploy it on your own hardware.

Have you tried datalab-to/marker with the "Use LLM" option? They have a playground you can test it out on https://www.datalab.to/playground but I use their local CLI option: https://github.com/datalab-to/marker I just tried it on a fairly ugly TD Bank statement PDF I have and the markdown of the whole PDF (tables and all) is very accurate. Here is the config I use: marker_single --format_lines --use_llm --llm_service marker.services.gemini.GoogleGeminiService --gemini_model_name gemini-2.5-flash --disable_image_extraction --output_format markdown --output_dir "$OutDir" ` "$In" You might be able to tell the LLM to directly output the data in CSV format - granted it will still be in a .md file - using the `--block_correction_prompt` which apparently is "useful for custom formatting or logic that you want to apply to the output"

How does it compare to Datalab/Marker https://github.com/datalab-to/marker ? We evaluated many PDF->MD converters and this one performed the best, though it is not perfect.

How does this compare against marker[1]? 1: https://github.com/VikParuchuri/marker

I usually convert to markdown from PDF local laws when I need them as a reference for the functional specification. That way its easier to pinpoint to exact section of the thing. Its not easy to convert general law to markdown, it involved online converters and manual fixes. Currently experimenting with marker [1] on local LLM hardware and so far it is the best out there. [1]: https://github.com/VikParuchuri/marker

> with LLM as a judge For anyone else interested, prompt is here [0]. The model used was gemini-2.0-flash-001. Benchmarks are hard, and I understand the appeal of having something that seems vaguely deterministic rather than having a human in the loop, but I have a very hard time accepting any LLM-judged benchmarks at face value. This is doubly true when we're talking about something like OCR which, as you say, is a very hard problem for computers of any sort. I'm assuming you've given this some thought—how did you arrive at using an LLM to benchmark OCR vs other LLMs? What limitations with your benchmark have you seen/are you aware of? [0] https://github.com/VikParuchuri/marker/blob/master/benchmark...

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker . Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100. You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... . The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/benchmark... . Will run a full benchmark soon. Mistral OCR is an impressive model, but OCR is a hard problem, and there is a significant risk of hallucinations/missing text with LLMs.

I'm a fan of the team of Allen AI and their work. Unfortunately, the benchmarking of olmocr against marker (https://github.com/VikParuchuri/marker) is quite flawed. Throughput - they benchmarked marker API cost vs local inference cost for olmocr. In our testing, marker locally gets 20 - 120 pages per second on an H100 (without custom kernels, etc). Olmocr in our testing gets between .4 (unoptimized) and 4 (sglang) pages per second on the same machine. Accuracy - their quality benchmarks are based on win rate with only 75 samples - which are different between each tool pair. The samples were filtered down from a set of ~2000 based on opaque criteria. They then asked researchers at Allen AI to judge which output was better. When we benchmarked with our existing set and LLM as a judge, we got a 56% win rate for marker across 1,107 docu

Did you try marker? https://github.com/VikParuchuri/marker I haven't tried olmocr yet and I now realize my 8GB GPU probably won't cut it, as it used a 7B param VLM model under the hood.

How does this compare to Marker https://github.com/VikParuchuri/marker?

Marker (https://www.github.com/VikParuchuri/marker) works kind of like this. It uses a layout model to identify blocks and processes each one separately. The internal format is a tree of blocks, which have arbitrary fields, but can all render to html. It can write out to json, html, or markdown. I integrated gemini recently to improve accuracy in certain blocks like tables. (get initial text, then pass to gemini to refine) Marker alone works about as well as gemini alone, but together they benchmark much better.

I've been thinking a lot about how to accomplish various RAG things in Elixir (for LLM applications). PDF is one of the missing pieces, so glad to see work here. The really tricky part is not just parsing out the text (you can just call the pdftotext unix command line utility for that), but accurately pulling out things like complex tables, etc in a way that could be chunked/post processed in a useful way. I'd love to see something like Unstructured or Marker but in Rust (i.e., fast) that Elixir could NIF out to it. And maybe some kind of hybrid system that uses open llm models with vision capabilities. Ref: - https://github.com/Unstructured-IO/unstructured - https://github.com/VikParuchuri/marker