self-bench

Hey HN, Today, we're launching selfbench.dev, an open-source tool that lets you create and run evals automatically from your PRs. Every benchmark with sufficient trust eventually gets benchmaxxed (Goodhart's law) - the labs are incentivized to maximize their scores on that benchmark, which isn't predictive on whether it'll actually work within your setup. This has been a time-consuming process that only the largest companies can afford to do, so we built self-bench to fix this. Self-bench uses agents to author / review Harbor environments generated from your PRs. You then approve every eval that the agent created, and can model/harness evals concurrently on sandboxes. To save you money, we also allow you to connect your OpenAI and Claude subscriptions so you don't have to pay raw token costs. We ran evals on several large codebases like -

Hey HN! We all know that most public evals are saturated and are hard to trust. What matters is whether a model reliably works in your codebase in your actual day-to-day-work. To solve this, we built self-bench (https://github.com/mupt-ai/self-bench), an open-source repo that automatically turns your private, completed PRs into held-out coding-agent evals. It's fully open-source and you can run it on your own machine or on Modal sandboxes! With one CLI command, we give you a validated Harbor dataset based on real changes from your repository. We've used self-bench to build evals for popular OSS projects, including Next.js (https://huggingface.co/datasets/dari-ai/nextjs-selfbench) and Vite (https://huggingface.co/datasets/dari-ai/vite-selfbench). With self-bench, we want to make it easy to compare open-weight models, frontier models, and r