vector-io

While building semantic search and LLM-powered apps, one needs to try out various vector databases because they differ widely in feature set, cost and other characteristics. The Vector-io library introduces a universal open format for storing vector datasets (Vectors, along with their metadata), along with import and export scripts for a wide range of vector databases: - Pinecone - Qdrant - Milvus - GCP Vertex AI Vector Search - KDB.AI - LanceDB - DataStax Astra DB - Chroma - Turbopuffer This will allow easier backup, snapshots, sharing of vector datasets and managing data across different vector DBs. I'm also curating a list of publicly available datasets in this format, which can be loaded directly from HuggingFace into your favorite VectorDB: https://huggingface.co/collections/aintech/vector-io-compati... If you have data in a vector D

for vector sinks, it would be good to integrate with Vector-io (https://github.com/ai-northstar-tech/vector-io) which provides import functions for ~10 vector databases already. disclosure: I'm the author of Vector-io

While building semantic search and LLM-powered apps, one needs to try out various vector databases because they differ widely in feature set, cost and other characteristics. The Vector-io library introduces a universal open format for storing vector datasets (Vectors, along with their metadata), along with import and export scripts for a wide range of vector databases (more to come!). This will allow easier backup, snapshots, sharing of vector datasets and managing data across different vector DBs. I'm also curating a list of publicly available datasets in this format, which can be loaded directly from HuggingFace into your favorite VectorDB: https://huggingface.co/collections/aintech/vector-datasets-v.... If you have data in a vector DB, please try it out and let me know if you have feedback. Thanks!

I also noticed, from same contributors, this repo with goal of enabling import/export of vector datasets, that can be migrated from one vectorDB vendor to another. The in-common format vectordb data can be stored/shared on HuggingFace datasets, using the in-built flag for it. Then anyone using another vectordb could download it. https://github.com/AI-Northstar-Tech/vector-io/tree/main?tab...