hnsqlite

For anyone thinking about applications of langchain and pinecone but who are looking for something more turn-key check out https://jiggy.ai The core is actually open source as well, allowing you to take your data back out via sqlite and hnswlib (https://github.com/jiggy-ai/hnsqlite)

After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated open source hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat. https://github.com/jiggy-ai/hnsqlite

Hnswlib is a simple, performant vector search library that is a great choice in many of the same set of scenarios where sqlite is a great choice for other data. With hnsqlite I integrate these two powerhouses and add support for filtering metadata at embedding search time. While working through several LLM projects I found this to be very useful for self-hosted projects up to a reasonably large size (at least 5-10 million embeddings).

Thanks Leobg! For anyone else: you pass it directly in metadata see https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_col... https://github.com/jiggy-ai/hnsqlite/blob/main/test/test_col...

hnswlib (https://github.com/nmslib/hnswlib) is a strong alternative to faiss that I have enjoyed using for multiple projects. It is simple and has great performance on CPU. After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat. https://github.com/jiggy-ai/hnsqlite