beir

It's always worth checking out the MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard There are some good open models there that have longer context limits and fewer dimensions. The benchmarks are just a guide. It's best to build a test dataset with your own data. This is a good example of that: https://github.com/beir-cellar/beir/wiki/Load-your-custom-da... Another benefit of having your own test dataset, is that it can grow as your data grows. And you can quickly test new models to see how it performs with YOUR data.

The dataset probably ranges from hundreds to tens of thousands of queries. The exact number is confidential and frankly, it changes over time and differs from product to product, so the order of magnitude is more indicative. This also matches the case of most public datasets. https://github.com/beir-cellar/beir?tab=readme-ov-file#beers... I guess the intuition is: if the dataset has less than 100 cases, it's arguably not diverse enough to cover all situations. On the other hand, the marginal gain of cases over 10,000 shrinks quickly. So O(1000) is probably a sweet spot if there is a way to automatically collect queries, e.g. from online traffic. If the dataset was hand-curated, it probably only makes sense to stay at O(100). It's also important to note that at Google there are ML trained automatic rating in addition to human raters. Ratin

The BEIR project might be what you're looking for: https://github.com/beir-cellar/beir/wiki/Leaderboard

> Meilisearch focuses on simplicity, relevancy, and performance. > excellent relevance out of the box > if ease of use, performance, and relevancy are important to you, Meilisearch was made for you Is there a benchmark that shows Meilisearch outperforming Elasticsearch in terms of relevance score? I couldn't find Meilisearch listed on https://github.com/beir-cellar/beir.

When the search performance is worse - may be no sense. Regarding Manticore, we conducted relevance tests and found it to be on par with Elasticsearch. In fact, the objective tests [1] showed that Manticore can even provide better relevance results than Elasticsearch, when using almost default settings. You can view the relevant pull request in the BEIR information retrieval benchmark [2]. [1] https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZ... [2] https://github.com/beir-cellar/beir/pull/92

You need a dataset and an evaluation metric. The usual evaluation metric is NDCG(Normalized Discounted Cumulative Gain): https://en.wikipedia.org/wiki/NDCG An example dataset is BEIR(BEnchmarking Information Retrieval), published in NIPS 2021: https://github.com/beir-cellar/beir

There are actually benchmarks that allow measuring search relevancy objectively, e.g. BEIR[1]. Manticore Search team did an effort to make a PR to include it to the list. The results are here [2]. Unfortunately the BEIR team seems to be too busy to review a whole pile of PRs including about Vespa. Nevertheless it would be nice to have both Meilisearch and Typesense there too since it's interesting what performance those non-tf-idf based search engines would show compared to BM25-based and vector search engines. [1] https://github.com/beir-cellar/beir [2] https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZ...

Here's a pull request to BEIR to compare Manticore with Elasticsearch in terms of relevance https://github.com/beir-cellar/beir/pull/92. In this test Manticore provides better relevance than Elasticsearch in average. In general I would say for most users the results quality in terms of full-text relevance is about the same in Elasticsearch and Manticore.

True! Here's a pull request to BEIR to compare Manticore with Elasticsearch in terms of relevance https://github.com/beir-cellar/beir/pull/92. Spoiler: in this test Manticore provides better relevance than Elasticsearch in average. Of course you can tune both further and Elasticsearch now has KNN which when combined with BM25 can give even better relevance. In general I would say for most users the results quality in terms of full-text relevance is about the same in Elasticseach and Manticore.