swe-bump-bench

to the rest of the codebase. This requires bumpgen to understand how every function in a codebase interacts with each other and how a change to one would affect the others. To address this, we drew inspiration from CodePlan (https://arxiv.org/abs/2309.12499) that describes a theoretical(?) way to apply change consistently for an entire repository. In short, it combines a dependency graph and an oracle to understand the codebase then verify a change. If I were to summarize my learnings, it would be that applying codegen to this problem is much more a “refactoring” problem than it is a “coding” problem (if that makes sense). So to answer my own question, I’d say AI CAN keep your code up-to-date but ONLY if it is guided carefully through the codebase. We benchmarked bumpgen against our test suite (https://github.com/xeol-io/swe-bump-bench) and it currently stands at around 50% accuracy. The bench suite is a set of repos with human commits for version bumps. We run bumpgen on the prior commit then compare bumpgen’s code diffs to that of the human commit along with passing builds to determine success. There are a lot of improvements we want to test out in May to increase accuracy. Starting with improving and tightening our core CodePlan logic. We think getting this right will unlock automating far more complex breaking changes and codebases like upgrading a vue 2 to vue 3. We have been building bumpgen for a month and we would love to hear people’s thoughts, experiences, and suggestions!

ype errors into the LLM to increase its fix accuracy. Challenge [3] is definitely the trickiest. We built an AST of the codebase that we then use to create a “plan graph” (https://huggingface.co/papers/2309.12499) of the type error, the function it’s used in, and the functions calling it. This plan graph should tell us how a fix would need to be perpetuated throughout the codebase. At a high level bumpgen’s core loop is something like this: - build an AST to understand your code’s relationships - bump version and get type errors - combine the above to create a plan graph (https://huggingface.co/papers/2309.12499) of how to fix the error and apply it to the rest of the codebase - prompt the LLM to step through the plan graph and fix each breaking change - validate the fixes with a rebuild We also built our own benchmark suite (https://github.com/xeol-io/swe-bump-bench) to test bumpgen’s accuracy. It is a set of repos with human commits for version bumps. We would run bumpgen on the prior commit then compare bumpgen’s PR to that of the human commit to determine success. Our latest benchmark sits around ~50% accuracy. Up next we want to better address challenge [2] by building embeddings for different dependency versions to give the LLM more context on the changes across versions. After that we will be releasing a GitHub app with some qol features such as configuring update cadences, etc before moving onto C# and Golang support. We covered a lot of our architecture and thought process for bumpgen, we would love to hear your thoughts and feedback in the comments!