pisa

More on latency and search: We benchmark against 10 other search APIs on fresh news queries. The median was around 1.2s; we came in around 166ms and scored highest on answer accuracy (89% vs 84% for the next cluster). Latency matters because agents loop. A 1.2s first call eats the budget for follow-ups — you get one shot at framing the query. At sub-250ms the agent can actually search, read, reformulate, and search again. Measuring this stuff carefully is something I've been at for a while. My ECIR 2019 paper (linked below) was an exhaustive study of 11 index compression methods across 5 query processing algorithms on standard collections — the codebase became PISA, which a lot of IR folks still use for research. Almost ten years later, the workload has changed completely (agents, not humans), but the benchmarking discipline is the same.

> We haven't discussed other solutions like Partitioned Elias-Fano indexes or Tree-Encoded Bitmaps. We simply did not investigate them. There are no efficient off-the-shelf implementations of these solutions, so we had to implement one. I guess they mean there are no implementations in Rust? The Partitioned Elias-Fano (PEF) paper links to the C++ implementation ([1], which I wrote) used for the experiments, it was efficient 9 years ago :) The library was later forked [2] and used in a few other papers, I believe it is somewhat maintained. The EF core algorithm implemented in folly [3] may be a bit faster, and implementing partitioning on top of that is relatively easy. It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps), PEF would likely be

FWIW... 1. Toshi https://github.com/toshi-search/Toshi (Rust, 3.1k stars) 2. Tantivy https://github.com/tantivy-search/tantivy (Rust, 4.5k stars) 3. PISA https://github.com/pisa-engine/pisa (C++, 486 stars) 4. Bleve https://github.com/blevesearch/bleve (Go, 7.4k stars) 5. Sonic https://github.com/valeriansaliou/sonic (Rust, 10.9k stars) 6. Partial comparison https://mosuka.github.io/search-benchmark-game/ (tantivy Vs lucene Vs pisa Vs bleve) 7. Bayard https://github.com/bayard-search/bayard (Rust, on top of Tantivy, 1.4k stars) 8. Blast https://github.com/mosuka/blast (Go, on top of Bleve, 930 stars) Algolia alternatives with some compatibility 1. MeiliSearch https://github.com/meilisearch/MeiliSearch (Rust, 12.4k stars) 2. typesense https://github.com/typesense/typesense (C++, 5.1k stars)