quickwit

Nice... it's cool to see how different companies are putting together best fit solutions. I'm also glad that they at least started out with off the shelf apps instead of jumping to something like a bespoke solution early on. Quickwit[1] looks interesting, found via Tantivity reference. Kind of like ES w/ Lucene. 1. https://github.com/quickwit-oss/quickwit

"What are your main use cases for search or log indexing?" To me, storing and searching logs is quite different from most other search use-cases, and it's not obvious that they should be handled by the same piece of software. For example, tokenization, stemming, language support many other things and are basically useless in log search. Also, log search is often storing a lot of data, and rarely retrieving it (different usage pattern from many search use-cases which tend to be less write-heavy and more about reads). I know ElasticSearch has had success in both, but if I were Manticore/Typesense/Meilisearch I'd probably just skip logs altogether. Loki, QuickWit and other such tools are likely better suited for logs. - https://github.com/quickwit-oss/quickwit - https://github.com/grafana/loki

Meilisearch is great, used it for a quick demo However if you need a full-text search similar to Apache Lucene, my go-to options are based on Tantivy Tantivy https://github.com/quickwit-oss/tantivy Asian language, BM25 scoring, Natural query language, JSON fields indexing support are all must-have features for me Quickwit - https://github.com/quickwit-oss/quickwit - https://quickwit.io/docs/get-started/quickstart ParadeDB - https://github.com/paradedb/paradedb I'm still looking for a systematic approach to make a hybrid search (combined full-text with embedding vectors). Any thoughts on up-to-date hybrid search experience are greatly appreciated

I don't believe there is just one answer to that, because it really depends on your needs, existing stack, and skillset. Disclaimer aside, I've heard very positive things about the search that ships with PostgreSQL, and similarly positive things for the few search extensions one can add (assuming extensions are available where you're running PG) https://github.com/quickwit-oss/tantivy#features seems to be a good starting place if you want to embed search, and its https://github.com/quickwit-oss/quickwit friend comes up a lot, but be sure to bear its AGPL license in mind Toshi is one of the ones which is striving for ES compatibility at the search level, making it a drop-in replacement https://github.com/toshi-search/Toshi#example-queries although last time I played with it, they were QUITE A WAYS away, depending on how deep one is in the

> It’s in no way mainly for logs as they claim Where can I find more information on using it for user-facing search? The repository [0] starts with "Cloud-native search engine for observability (logs, traces, and soon metrics!)" and keeps talking about those. [0]: https://github.com/quickwit-oss/quickwit

If you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, it's OSS and works for append-only datasets (like logs, traces, ...) Repo: https://github.com/quickwit-oss/quickwit

Here's a sampling of stuff I've semi-recently encountered on Github that all spy on you by default. For some, it's undocumented, for others it's a minor subsection in the docs. (Some of them are source-available not free, I didn't keep track as I collected the URLs.) https://github.com/zed-industries/zed https://github.com/microsoft/vscode https://github.com/keystonejs/keystone https://github.com/meilisearch/meilisearch https://github.com/hasura/graphql-engine https://github.com/manticoresoftware/manticoresearch/ https://github.com/coder/code-server https://github.com/loft-sh/devpod https://github.com/quickwit-oss/quickwit https://github.com/rilldata/rill-developer/ https://github.com/hocus-dev/hocus https://github.com/firezone/firezone Here's a write-up of how bad it gets -- it's pretty obnoxious: https://www.roboleary.net/tools/2022/04/

Found recently Tantivy inside of ParadeDB (Postgres extension aiming to replace Elastic) https://github.com/paradedb/paradedb/blob/dev/pg_search/Carg... after listening Extending Postgres for High Performance Analytics (with Philippe Noël) https://www.youtube.com/watch?v=NbOAEJrsbaM And inside of the main thing - Quickwit(logs, traces, and soon metrics) https://github.com/quickwit-oss/quickwit Had a surprisingly good experience with combined power of Quickwit and Clickhouse for multilingual search pet project. Finally something usable for Chinese, Japanese, Korean https://quickwit.io/docs/guides/add-full-text-search-to-your... to_tsvector in PG never worked well for my use cases SELECT * FROM dump WHERE to_tsvector('english'::regconfig, hh_fullname) @@ to_tsquery('english'::regconfig, 'query'); Wish them to succeed. Will automatically up

Quickwit is an alternative with a strong focus on scalability (max we have seen is 40PB) with a decoupled compute and storage architecture. But we do only logs and traces for now. Repository: https://github.com/quickwit-oss/quickwit Latest release: https://quickwit.io/blog/quickwit-0.8

Hi everyone, During the last 3 months, I worked on the Quickwit search engine to adapt it for AWS Lambda runtime. I wrote a blog post to announce the beta release [0] and two tutorials to quickstart: - Indexing and searching 20 million logs entries dataset [1]. - Implementing an E2E use case where an application generates data, uploads it in batches to S3. Users can then search through an HTTP API authenticated with an API Key [2]. On the performance side, I observed on a 20 million log dataset that search is sub-second, and analytics queries take from 1 to 4 second. On the cost side, obviously the compute using Lambda is fairly expensive, but the huge win is that it scales to 0. Few other solutions do. For instance OpenSearch Serverless has a fixed cost around $700/months. On the search side, thanks to indexing, queries are usually way c

Hi folks, Quickwit cofounder here. We started Quickwit 3 years ago with a POC, "Searching the web for under $1000/month" (see HN discussions [0]), with the goal of making a robust OSS alternative to Elasticsearch / Splunk / Datadog. We have reached a significant milestone with our latest release (0.7) [1], as we have witnessed users of the nightly version of Quickwit deploy clusters with hundreds of nodes, ingest hundreds of terabytes of data daily, and enjoy considerable cost savings. To give you a concrete example, one company is ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit. They divided their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days. They also increased their durability, accuracy with exactly-once semantics thanks to the native Kafka support,

Hi HN, Quickwit cofounder here! Quickwit is a cloud-native search engine for observability, and we have just released a significant upgrade with performance improvements, extended Elasticsearch API compatibility, and better integration with OpenTelemetry tools, Grafana, and Jaeger. This release is also a significant milestone as we have been witnessing users of the nightly version of Quickwit deploy clusters with hundreds of nodes, ingest hundreds of terabytes of data daily, and enjoy additional cost savings. We are very excited to put it in the hands of all of our users. One report particularly compelling is coming from a company ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit, dividing their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days. They also incr

We tested S3 Express for our search engine quickwit [0] a couple of weeks ago. While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter. I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they have a lot of requests. [0] https://github.com/quickwit-oss/quickwit/

Hi HN, I’m one of the builders of Quickwit, a cloud-native OSS search engine for observability. As of 2023, we support logs and traces, metrics will come in 2024. You know the pitch: while software like Datadog or Splunk are great, they often comes with hefty price tags. Our mission is to offer an affordable alternative. So we’ve built Quickwit, we’ve made it compatible with the observabilty ecosystem (OpenTelemetry, Jaeger, Grafana) and above all, we’ve made it cost-efficient / “easy” to scale (well it’s never easy to scale to petabytes..). To give you a glance at the engine performance I made a benchmark on the GitHub Archive dataset, 23 TB of events, here are the main observations: Indexing: costs $2 per ingested TB. With 4CPU, throughput is at 20MBs However, you'll observe > 30MB throughput on simpler datasets, like logs and traces. S

Quickwit (https://quickwit.io/) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make exceptions for the right profiles. * Senior Software Engineer 80-110k€ + 0.25-1% equity based on experience. We’re looking for a senior software engineer to contribute to Quickwit(https://github.com/quickwit-oss/quickwit), our open-source search and analytics engine. We have an ambitious roadmap for the next 18 months (performance optimization, distributed storage, support for SQL, query optimizer, revamp of our execution engine, etc.), and this is a great opportunity to shape the future of Quickwit while tackling fun and challenging problems in the field of distributed databases. Full job description: https://quickwit.io/jo