hnswlib

Hey HN! I built COSSgpt using videos from the Open Source Founder Podcast [1] and livestreams from COSS Office Hours [2][3] I transcribed the VODs using Whisper and vectorized fixed-size segments from the transcripts with MPNet on Replicate GPUs. I made these segments overlap a little to prevent semantic meaning being lost inbetween segments Then I indexed the vectors using HNSWLib in-memory vectorstore [4] and persisted the entire vectorstore into Tigris object storage [5] to cache multimedia and vectors across all Fly.io regions I built the app in Elixir, almost entirely server-side rendered with minimal diffs sent to the client over WebSockets using Phoenix LiveView. I also used Livebook [6] a ton when I was building the multimedia processing & ML pipeline. I'm super bullish on Elixir for building webapps and/or MLops! Let me know what

How does this compare to hsnwlib - is it faster? https://github.com/nmslib/hnswlib

Last week I released Moodflix (https://moodflix.streamlit.app), a movie recommendation engine based to find movies based on your mood. Moodflix was created on top of a movie dataset of 10k movies from The Movie Database. I vectorised the films using Hugging Face's T5 model (https://huggingface.co/docs/transformers/model_doc/t5) using the film's plot synopsis, genres and languages. Then I indexed the vectors using hnswlib (https://github.com/nmslib/hnswlib). LLMs can understand a movie's plot pretty well and distill the similarities between a user's query (mood) to the movie's plot and genres. I have got feedback from close friends around linking movies to other review sites like IMDB or Rotten Tomatoes, linking movies to sites to stream the movie and adding movie posters. I would also love to hear from the community what things you like,

Good suggestion! Actually the "ef" is not epsilon. It is a parameter of the HNSW index: https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md...

I totally agree and hnswlib is actually much faster than FAISS on CPU. I'm really happy to see `hnswlib` as a Python dependency since I'm the one who implemented PyPI support: https://github.com/nmslib/hnswlib/pull/140

hnswlib (https://github.com/nmslib/hnswlib) is a strong alternative to faiss that I have enjoyed using for multiple projects. It is simple and has great performance on CPU. After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat. https://github.com/jiggy-ai/hnsqlite

https://github.com/nmslib/hnswlib Used it to index 40M text snippets in the legal domain. Allows incremental adding. I love how it just works. You know, doesn’t ANNOY me or makes a FAISS. ;-)

hnswlib is in cpp and has python bindings (you should be able to make your own for other languages). Faiss, Annoy (by Spotify) should also provide similar functionality. https://github.com/nmslib/hnswlib

hnswlib[1] allows for incremental updates. And I believe in terms of accuracy it stacks up fairly well against alternatives like FAISS or ScaNN. [1]: https://github.com/nmslib/hnswlib/

There's also hnswlib[1], which has supposedly lower memory requirements and allows for adding new vectors to an existing index. [1]: https://github.com/nmslib/hnswlib/

great post and my favorite CS topic, LSH is particularly relevant to machine learning because of its use in indexing of embeddings, which are now omnipresent in ML (from word2vec to transformers to image and graph embeddings etc, etc.) it is now supported in ElasticSearch KNN index (they use HNSWLIB but you can call it a descendant of original LSH in a way) check out ANN benchmarks [0] for comparison of LSH performance to other state of the art methods like proximity graphs/HNSWLIB [1] and quantization/SCANN [2] As an introduction LSH (with MinHash) is also described in detail in the book "Mining Of Massive Datasets", ch.3, "Finding Similar items", highly recommended [3] if you want to play with LSH, python "annoy" library is the best place to start [4] [0] https://github.com/erikbern/ann-benchmarks [1] https://github.com/google-research/

https://github.com/nmslib/hnswlib for anybody else googling this library