paperetl

Shell: GNU parallel, pdftotext Python: PyPdf2, PdfMiner.six, Grobid, PyMuPdf; pytesseract (C++) paperetl is built on grobid: https://github.com/neuml/paperetl annotateai: https://github.com/neuml/annotateai : > annotateai automatically annotates papers using Large Language Models (LLMs). While LLMs can summarize papers, search papers and build generative text about papers, this project focuses on providing human readers with context as they read. pdf.js-hypothes.is: https://github.com/hypothesis/pdf.js-hypothes.is: > This is a copy of Mozilla's PDF.js viewer with Hypothesis annotation tools added Hypothesis is built on the W3C Web Annotations spec. dokieli implements W3C Web Annotations and many other Linked Data Specs: https://github.com/dokieli/dokieli : > Implements versioning and has the notion of immutable resources. > Embedding da

Very interesting work! If biomedical research and paper analysis is of interest to you, I've been working on a set of open source projects that enable RAG over medical literature for a while. PaperAI: https://github.com/neuml/paperai PaperETL: https://github.com/neuml/paperetl There is also this tool that annotates papers inline. AnnotateAI: https://github.com/neuml/annotateai

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai (https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above. paperai (https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai) paperetl (https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.

One additional library to add, if you're working with scientific papers: https://github.com/kermitt2/grobid. I use this with paperetl (https://github.com/neuml/paperetl).

Nice project! I've spent quite a lot of time in the medical/scientific literature space. With regards to LLMs, specifically RAG, how the data is chunked is quite important. With that, I have a couple projects that might be beneficial additions. paperetl (https://github.com/neuml/paperetl) - supports parsing arXiv, PubMed and integrates with GROBID to handle parsing metadata and text from arbitrary papers. paperai (https://github.com/neuml/paperai) - builds embeddings databases of medical/scientific papers. Supports LLM prompting, semantic workflows and vector search. Built with txtai (https://github.com/neuml/txtai). While arbitrary chunking/splitting can work, I've found that integrating parsing that has knowledge of medical/scientific paper structure increases the overall accuracy and experience of downstream applications.