open-rag-eval

One of the biggest challenges in RAG Evaluation is the assumption that you somehow can get the "source of truth" generated, specifically the set of "golden answers" (or golden chunks/documents). In practice that is extremely difficult and non scalable. Open-RAG-Eval is a new open source project that aims to address that via reference-free evaluation such as UMBRELA and AutoNuggetizer scores. Repo: https://github.com/vectara/open-rag-eval and a nice UI to use this with: openevaluation.ai Would love to hear feedback on this after you try it out and what you might want to see on the roadmap.

RAG Evaluation is difficult, primarily because it's hard to come up with "golden answers" (or golden chunks). We made Open-RAG-Eval to solve this - RAG Eval that only requires the question, yet provides great metrics for retrieval, generation, hallucination and citations for any RAG setup. This was in collaboration with Jimmy Lin and his students at UWaterloo. It has connectors to LangChain, LlamaIndex and Vectara, and hoping others can contribute more connectors to other RAG systems. repo: https://github.com/vectara/open-rag-eval UI for reviewing eval results: https://openevaluation.ai/ Papers: https://arxiv.org/pdf/2406.06519 and https://arxiv.org/abs/2504.15068