sage

While building our OSS chat-with-your-codebase system, we faced so many choices: chunking strategies, embeddings, retrieval algorithms, rerankers, etc. Since we don't want to make decisions based on vibes, and because academic benchmarks are somewhat contrived, we made our own. Our dataset consists of 1,000 questions about Hugging Face's Transformers library, where each question requires 1-3 Python files to be answered correctly. We started by comparing proprietary APIs for the various sub-tasks involved in an AI copilot. Here are our initial learnings: - OpenAI's text-embedding-3-small embeddings perform best. - NVIDIA's reranker outperforms Cohere, Voyage and Jina. - Sparse retrieval (e.g. BM25) is actively hurting code retrieval if you have natural language files in your index (e.g. Markdown). - Chunks of size 800 are ideal; going smal

Not sure if this is what you'd like, but I remember this repo from some time ago https://github.com/Storia-AI/sage Storia-AI/sage: Chat with any codebase with 2 commands

We're using an improvement over this exact blogpost actually. We started from there, but weren't happy that some of the chunks were really small (and they would undeservedly get surfaced to the top). So we added some extra logic to merge the siblings if they're small. https://github.com/Storia-AI/repo2vec/blob/1864102949e720320...

OP's cofounder here. Thanks for pointing out this test case. Surfaced that we weren't handling symlinks properly. With this fix, I was able to successfully embed and index most of the repo (though I stopped at 100 embedding jobs so that we don't burn through OpenAI credits). P.S. You'll see a bunch of warnings for e.g. binary files that are ignored. https://github.com/Storia-AI/repo2vec/commit/1864102949e7203...

Hi HN, We're excited to share repo2vec: a simple-to-use, modular library enabling you to chat with any public or private codebase. It's like Github Copilot but with the most up-to-date information about your repo. We made this because sometimes you just want to learn how a codebase works and how to integrate it, without spending hours sifting through the code itself. We tried to make it dead-simple to use. With two scripts, you can index and get a functional interface for your repo. Every generated response shows where in the code the context for the answer was pulled from. We also made it plug-and-play where every component from the embeddings, to the vector store, to the LLM is completely customizable. If you want to see a hosted version of the chat interface with its features, here's a link: https://www.youtube.com/watch?v=CNVzmqRXUCA

Since you didn't include the link I went ahead and searched for it. Is this it? https://github.com/Storia-AI/repo2vec There's been a lot of "chat with your x" projects and the value prop always eludes me. To use the example in the repo, if I want to know what image encoders are supported, I would do a repository search for the "Encoder" keyword to find where they're defined. Then I'd be able to see all the encoders that are supported. That takes me about 10 seconds - why would I want to use a chatbot to do this instead?