mteb

I wish they included the datasets they used for the evaluations. As far as I can tell, in appendix II they include some sample questions, answers, and golden chunks but they do not give the entire dataset or give an explicit information on exactly what the datasets are. Does anyone know if the datasets they used for the evaluation are publicly available or if they give more information on the datasets than what's in appendix II? There are standard publically available datasets for this type of evaluation, like MTEB (https://github.com/embeddings-benchmark/mteb). I wonder how this technique does on the MTEB dataset.

RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothetical embedding questions, meta data) - RAG will miserably fail with requests like "summarize the whole document" - to my knowledge, openAI embeddings aren't performing well, use a embedding that is optimized for question answering or information retrieval and supports multi language. SOTA textual embedding models can be found on the MTEB Leaderboard [2]. Also look into instructorEmbeddings - the LLM used for the Q&A using you

The corresponding paper: MTEB: Massive Text Embedding Benchmark https://arxiv.org/abs/2210.07316 And the repo: https://github.com/embeddings-benchmark/mteb