beir

You need a dataset and an evaluation metric. The usual evaluation metric is NDCG(Normalized Discounted Cumulative Gain): https://en.wikipedia.org/wiki/NDCG An example dataset is BEIR(BEnchmarking Information Retrieval), published in NIPS 2021: https://github.com/beir-cellar/beir

There are actually benchmarks that allow measuring search relevancy objectively, e.g. BEIR[1]. Manticore Search team did an effort to make a PR to include it to the list. The results are here [2]. Unfortunately the BEIR team seems to be too busy to review a whole pile of PRs including about Vespa. Nevertheless it would be nice to have both Meilisearch and Typesense there too since it's interesting what performance those non-tf-idf based search engines would show compared to BM25-based and vector search engines. [1] https://github.com/beir-cellar/beir [2] https://docs.google.com/spreadsheets/d/1_ZyYkPJ_K0st9FJBrjbZ...

Here's a pull request to BEIR to compare Manticore with Elasticsearch in terms of relevance https://github.com/beir-cellar/beir/pull/92. In this test Manticore provides better relevance than Elasticsearch in average. In general I would say for most users the results quality in terms of full-text relevance is about the same in Elasticsearch and Manticore.

True! Here's a pull request to BEIR to compare Manticore with Elasticsearch in terms of relevance https://github.com/beir-cellar/beir/pull/92. Spoiler: in this test Manticore provides better relevance than Elasticsearch in average. Of course you can tune both further and Elasticsearch now has KNN which when combined with BM25 can give even better relevance. In general I would say for most users the results quality in terms of full-text relevance is about the same in Elasticseach and Manticore.