lucene

Here you go https://github.com/apache/lucene/pull/656 - no need to do anything from the user side to trigger it as far as I know.

"if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost." And if you need this in Java you still have resort to ugly hacks. https://github.com/apache/lucene/issues/7231

As add-on to this comment: There's another Lucene issue from 2 weeks ago that provides some more details on different approaches that were considered: https://github.com/apache/lucene/issues/12302

A little hard to understand why this is cool, but if I understand correctly: 1. Lucene is trying to get Approximate Nearest Neighbours (ANN) search working for semantic search purposes: https://issues.apache.org/jira/browse/LUCENE-9004 https://github.com/apache/lucene/issues/10047 2. The Panama Vector API allows CPU's that support it to accelerate vector operations: https://openjdk.org/jeps/438 So this allows fast ANN on Lucene for semantic search! How did people do this before Lucene supported it? Only through entirely different tools?

Are they forking Lucene or somehow getting the Lucene devs to increase that limit? Because this PR has been open for over a year now: https://github.com/apache/lucene/issues/11507

Oh! Damn it I forgot about manticore -- I had seen it before but forgot to include it. Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out: Search Engines AWS OpenSearch https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch-Dashboards https://github.com/opensearch-project/perftop https://github.com/go-ego/riot https://groonga.org/ https://github.com/groonga/groonga https://github.com/meilisearch/MeiliSearch https://github.com/mosuka/bayard https://github.com/nezaboodka/nevod https://github.com/searx/searx https://github.com/stryku/okon https://github.com/toshi-search/Toshi https://github.com/typesense/typesense https://github.com/valeriansaliou/s

Lucene's adaptation of Roaring uses the complement idea on a block-wise basis: https://github.com/apache/lucene/blob/84cae4f27cfd3feb3bb42d...

It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic. From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1] It's great to see folks that work for Elastic collaborating and building