cs

Hah yes. Ill throw my own into the ring too https://github.com/boyter/cs although its a lot older than most other tools having been started 7 years ago. Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top. Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.

Tooting my own horn here for search but give https://github.com/boyter/cs a go. You can have a http server up if you like (even modify the templates) and have it sync your repos for you on a schedule you define. Let me know if it fails in some area and I’ll fix it.

you might be interested in https://github.com/boyter/cs pretty fast and neat project to search code interactively with a lot of optimizations on finding the right thing

Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly. I will have to add this as a comparison to https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it. This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of

Been working on https://searchcode.com/ again which I bought back, albeit as code search tool for LLMs. It solves the “should I use this library” by allowing the LLM to inspect search and analyse it before integration. Can use it to compare multiple repositories before downloading. It comes with a large amount of token savings and can be really useful when wanting to learn about a codebase. Since it does it anyway I added dossier pages to it as well https://searchcode.com/repo/github.com/rust-lang/rust Which is useful for humans, and shows what the system is creating. Best part is that I get to use the tools I have built, so https://github.com/boyter/scc and https://github.com/boyter/cs to improve it which benefits anyone using those tools.

Such a good read. I actually went back though it the other day to steal the searching for the least common byte idea out to speed up my search tool https://github.com/boyter/cs which when coupled with the simd upper lower search technique from fzf cut the wall clock runtime by a third. There was this post from cursor https://cursor.com/blog/fast-regex-search today about building an index for agents due to them hitting a limit on ripgrep, but I’m not sure what codebase they are hitting that warrants it. Especially since they would have to be at 100-200 GB to be getting to 15s of runtime. Unless it’s all matches that is.

I initially built cs (codespelunker) as a way to answer the question, can BM25 relevance search work without building an index? Turns out it can, and so I iterated on the idea, building it into a full CLI tool. Recently I wanted to improve it by adding relevance of tools like Sourcegraph or Zoekt but again without adding an index. cs uses scc https://github.com/boyter/scc to understand the structure of the file on the fly. As such it can filter searches to code, comments or strings. It also applies a weighted BM25 algorithm where matches in actual code rank higher than matches in comments (by default). I also added a complexity gravity weight using the cyclomatic complexity output from scc as it scans. So if you're searching for a function, the implementation should rank higher than the interface. cs "authenticate" --gravity=brain

That’s one of the reasons I made this actually https://github.com/boyter/cs I wanted and boolean syntax mixed with fzf instant search. It’s not as fast as ripgrep of course but it’s not solving the same problem.

And 'cs' for smaller repos and CLI use. [9]: https://github.com/boyter/cs

I'll add something I have been working on https://github.com/boyter/cs which is aimed at a smaller scale. It works fine for multiple repositories so long as they aren't too large.

Neat. This is similar to a tool I have been working on (but need to finish off) as I saw the same issue. Except rather than build an index I brute forced the search each time. For most repositories it’s fast enough even with ranking. https://github.com/boyter/cs For those interested it’s still very WIP with noticeable issues in TUI mode.

A few things. An implementation of bitfunnel search in Go which I plan to put into searchcode.com at some point once I get all the issues resolved and if performance is acceptable A command line search tool which brute forces with search ranking https://github.com/boyter/cs/ mostly for code but works pretty well for other things as well Atlassian Confluence Cloud plugins. Mostly out of personal interest and because there appears to be a good marketplace to produce mostly passive income there.

Not the point of the article, but the slow-down from using regular expressions can be avoided in this case by instead generating each case permutation and searching on that. Although its actually a bit more complex because you need to cut off doing this for long strings to avoid too many permutations but you get the idea. Posting just in case someone thinks they need to build an index in order to case-insensitive search a few gigabytes of text. Its actually how tools like ripgrep tend to work (assuming it can do literal extraction of the regular expression). The moment you get to 10's of gigabytes though an index is probably what you need. I actually have some Go code (since the example was in Go) which does this and assuming you are doing literal searches is a drop in replacement for re.FindAllIndex https://github.com/boyter/cs/blob/mast