nmslib

Various approaches based on LSH (locality sensitive hashing) and random projections. But "the best" really depends on your data profile, as the generic problem formulation, for completely generic vectors, is too… generic. See also the curse of dimensionality [0]. In practical terms, if you're looking for an actual implementation to use, here's a benchmark of some state-of-the-art implementations for high-dimensional vector search: http://radimrehurek.com/2013/12/performance-shootout-of-near... and also the updated https://github.com/erikbern/ann-benchmarks. In particular, for something good AND open source: - Leonid Boytsov's NMSLIB [1] - Erik Bernhardsson's Annoy [2] - Facebook's FAISS [3] For a commercial engine focused more on practical use (index management, transactions, versioning, support), see ScaleText [4] (disclaimer: our produc

> During the inference time, we first represent user input as a vector using query encoder; then iterate over all available products and compute the metric between the query vector and each of them; finally, sort the results. Depending on the stock size, the metric computation part could take a while. Fortunately, this process can be easily parallelized. An alternative is to precompute a search index over the item vectors if the dataset of items is very large and you’re OK with running an approximate search to trade a bit of recall for performance, using algorithms provided by libraries like the following. Nmslib: https://github.com/searchivarius/nmslib Faiss (Facebook): https://github.com/facebookresearch/faiss Annoy (Spotify): https://github.com/spotify/annoy

https://en.wikipedia.org/wiki/BK-tree A BK-tree is... specifically adapted to discrete metric spaces. There are others that have a similar principal for real valued metrics. • Ball Tree (also known as VP-Tree) (implemented in sklearn) • GH-Tree (rough description in this large PDF https://github.com/searchivarius/nmslib/raw/master/docs/manu... ) • Multi-Vantage Point Tree As for the rationale, I think grouping them together is an efficient way to exploit the triangle inequality for each subtree.

We have recently released a new version of our Non-Metric Space Library (NMSLIB). NMSLIB is an efficient and extendible toolkit for searching in generic non-metric spaces. Our killer feature is generality. Despite being general, some of the methods may outperform the celebrated LSH in the spaces where LSH methods seem to work best. For more details, please, check our repo and/or my reflections on the state of the library and future work https://github.com/searchivarius/nmslib/tree/v1.5: http://searchivarius.org/blog/what-non-metric-space-library