tantivy

We are working on our two site-search search engines Monocle Search [0] and SearchCue [1]. The former is squarely aimed at people using Squarespace, whereas our newer offering SearchCue is aimed at anyone with a site that wants to add search. We used Meilisearch as the search backend in the beginning but have since replaced it with a quite sophisticated search stack built around Tantivy [2]. We now support crawling and indexing of pages, most common office documents and PDFs, run OCR and feature extraction of images you might have, offer typeahead search with the aim of giving you providing answers as fast as you can type, as well as more classic agentic/conversational ai search. There have been quite a number of interesting optimization challenges to solve in case anyone is interested. We have search nodes distributed around the globe to

Sorry, I don't know all the answers. However, I do know this (from the README): > Core Engine (Storage & Search): Tantivy[1] > Acts as both the primary storage for email content and the full-text search index. This unified approach ensures high performance and eliminates data redundancy. [1] https://github.com/quickwit-oss/tantivy

Hi, tantivy dev here. There are two recent performance improvements in tantivy, which should make term aggregations considerable faster. https://github.com/quickwit-oss/tantivy/pull/2740 https://github.com/quickwit-oss/tantivy/pull/2759

Meilisearch is great, used it for a quick demo However if you need a full-text search similar to Apache Lucene, my go-to options are based on Tantivy Tantivy https://github.com/quickwit-oss/tantivy Asian language, BM25 scoring, Natural query language, JSON fields indexing support are all must-have features for me Quickwit - https://github.com/quickwit-oss/quickwit - https://quickwit.io/docs/get-started/quickstart ParadeDB - https://github.com/paradedb/paradedb I'm still looking for a systematic approach to make a hybrid search (combined full-text with embedding vectors). Any thoughts on up-to-date hybrid search experience are greatly appreciated

Congratulations! The fact you and your team managed to built Tantivy is a huge contribution to open source. As someone who never managed to built a fond relationship with Apache Lucene based products (Solf, Elastic). I was extremely happy to see Tantivy in open source. BM25 scoring, proper asian language support, speed, memory foot prints, etc - amazing job! Thank you so much! https://github.com/quickwit-oss/tantivy IMHO Datadog made a smart move! If Tantivy itself just stays permanently under Apache2 licence and find a sustainable path to co exist with the rest of open source community - it's all good guys. You are more than deserve a commercial success.

This comment and an idea to work backwards deserve an article! Just finished a call a few mins, and we came to conclusion we do natural query language, BM25 scoring with Tantivy based code first https://github.com/quickwit-oss/tantivy In meanwhile we collect all questions to ask LLM so we can be more consious at Hybrid Search implementation phase

I don't believe there is just one answer to that, because it really depends on your needs, existing stack, and skillset. Disclaimer aside, I've heard very positive things about the search that ships with PostgreSQL, and similarly positive things for the few search extensions one can add (assuming extensions are available where you're running PG) https://github.com/quickwit-oss/tantivy#features seems to be a good starting place if you want to embed search, and its https://github.com/quickwit-oss/quickwit friend comes up a lot, but be sure to bear its AGPL license in mind Toshi is one of the ones which is striving for ES compatibility at the search level, making it a drop-in replacement https://github.com/toshi-search/Toshi#example-queries although last time I played with it, they were QUITE A WAYS away, depending on how deep one is in the

I believe the primary issue with using full-text search in PostgreSQL is its performance and scalability limitations. Elasticsearch/OpenSearch offers more than just search functionality; they come with extensive ecosystems, including ELK for logging and processors for ETL pipelines. These platforms are powerful and provide out-of-the-box solutions for developers. However, as the article mentioned, scaling them up or out can be costly. ParadeDB looks like is based on tantivy(https://github.com/quickwit-oss/tantivy) which is an impressive project. We leverage it to implement full-text indexing for GreptimeDB too (https://github.com/GreptimeTeam/greptimedb). Nevertheless, building full-text indexes with Tantivy is still resource-intensive. This is why Grafana Loki, which only indexes tags instead of building full-text indexes, is popular in

Fun fact: We've implemented binary embedding search [1] without the need for a specialized vector database. Instead, dimensional tokens like 'embedding_0_0', 'embedding_1_0' are created and being built into the tantivy index [2]. We're satisfied with the quality and performance this approach yields, while still keep Tabby embed everything into a single binary. [1] My binary vector search is better than your FP32 vectors: https://blog.pgvecto.rs/my-binary-vector-search-is-better-th... [2] Tantivy: https://github.com/quickwit-oss/tantivy

> Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config... Yes that's how you use tantivy normally, not sure which json config you mean. tantivy-cli is more like a showcase, https://github.com/quickwit-oss/tantivy is the actual project.

One workaround is to use the JSON field, see doc https://github.com/quickwit-oss/tantivy/blob/main/doc/src/js...

Beware, you still cannot add/remove fields: https://github.com/quickwit-oss/tantivy/issues/470 The only way to add fields is to reindex all data into a different search index.

The geocoder is built on top of tantivy which is fast and uses low resources too (https://github.com/quickwit-oss/tantivy). I'm curious about the comparison between those two.