paperai

Very interesting work! If biomedical research and paper analysis is of interest to you, I've been working on a set of open source projects that enable RAG over medical literature for a while. PaperAI: https://github.com/neuml/paperai PaperETL: https://github.com/neuml/paperetl There is also this tool that annotates papers inline. AnnotateAI: https://github.com/neuml/annotateai

PaperAI is also an option if you prefer open-source: https://github.com/neuml/paperai Disclaimer: I'm the primary author of this project.

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai (https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above. paperai (https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai) paperetl (https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.

paper-qa: https://github.com/whitead/paper-qa : > LLM Chain querying documents with citations [e.g. a scientific Zotero library] > This is a minimal package for doing question and answering from PDFs or text files (which can be raw HTML). It strives to give very good answers, with no hallucinations, by grounding responses with in-text citations. pip install paper-qa > If you use Zotero to organize your personal bibliography, you can use the paperqa.contrib.ZoteroDB to query papers from your library, which relies on pyzotero. Install pyzotero to use this feature: pip install pyzotero > If you want to use [ paperqa with pyzotero ] in an jupyter notebook or colab, you need to run the following command: import nest_asyncio nest_asyncio.apply() ... there is also neuml/paper-ai w/ paperetl; https://news.ycombinator.com/item?id=39363

From "Artificial intelligence is ineffective and potentially harmful for fact checking" (2023) https://news.ycombinator.com/item?id=37226233 : pdfgpt, knowledge_gpt, elasticsearch : > Are LLM tools better or worse than e.g. meilisearch or elasticsearch for searching with snippets over a set of document resources? > How does search compare to generating things with citations? pdfGPT: https://github.com/bhaskatripathi/pdfGPT : > PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. GH "pdfgpt" topic: https://github.com/topics/pdfgpt knowledge_gpt: https://github.com/mmz-001/knowledge_gpt From https://news.ycombinator.com/item?id=39112014 : paperai neuml/paperai: https://github.com/neuml/paperai : > Semantic search and workflows for medical/scientific papers RAG: https://news.ycombinator.com/item?id=38370

What about citations, and indicating provenance of derived knowledge? neuml/txtai, neuml/paperai: https://github.com/neuml/paperai "Wikidata, with 12B facts, can ground LLMs to improve their factuality" (2023) https://news.ycombinator.com/context?id=38309408

Nice project! I've spent quite a lot of time in the medical/scientific literature space. With regards to LLMs, specifically RAG, how the data is chunked is quite important. With that, I have a couple projects that might be beneficial additions. paperetl (https://github.com/neuml/paperetl) - supports parsing arXiv, PubMed and integrates with GROBID to handle parsing metadata and text from arbitrary papers. paperai (https://github.com/neuml/paperai) - builds embeddings databases of medical/scientific papers. Supports LLM prompting, semantic workflows and vector search. Built with txtai (https://github.com/neuml/txtai). While arbitrary chunking/splitting can work, I've found that integrating parsing that has knowledge of medical/scientific paper structure increases the overall accuracy and experience of downstream applications.

txtai makes it easy to use Hugging Face embeddings and Faiss, all local and configurable. https://github.com/neuml/txtai paperai is a sub-project focused on processing medical/scientific papers. https://github.com/neuml/paperai Disclaimer: I am the author of both