dirac

This sounds quite similar to dirac which made a stir a few months ago: https://github.com/dirac-run/dirac I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness. If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.

Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac (https://github.com/dirac-run/dirac).

This is a plug, but relevant. I recently added a 'build native tools on the fly' functionality to Dirac (https://github.com/dirac-run/dirac) that works like: 1. You can use the '/new-tool' and tell what kind of tool you want (including whether it should be task-scoped, workspace-scoped, or global), the model builds it, the harness runs validation and other tests until the tool is ready 2. The model decides that in such and such task, it would be helpful to have a tool like this, it can build a task-scoped tool. In either scenario, the tool catalog is rebuilt, and the new tool is instantly available in the next turn.

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks[1], grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac (https://github.com/dirac-run/dirac) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. [1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, prefera

It is not the raw prompt size that matters ultimately, otherwise Pi (and variants) would be the lowest costing agents. What matters is how efficient the prompt it. Prompt minimalism often gets conflated with efficiency. Having said that, CC does seem bloated for what it does. What matters even more is tooling quality. Bad/buggy tooling causes a lot more roundtrips that wipes out all gains from initial greedy approach. A few months ago, I did a full benchmark run of 7 agents over 8 tasks (https://github.com/dirac-run/dirac has the data and traces). I cannot claim neutrality because of the obvious connection to one of those, but the data should be reproducible and useful. Importantly, Dirac wins there significantly on those tasks because they are mostly refactoring related (which is where approaches like hash-anchoring and AST parsing tend

Dirac (https://github.com/dirac-run/dirac, https://dirac.run/) now supports gpt-5.6. This thing does now seem to be on the chatGPT/codex accounts yet. UPDATE: it is now available in chatGPT account also, they rolled it out

I have been experimenting with modifying Ghostty lately. It's a well attended codebase and a pleasure to work with, props to Mitchell. Since Ghostty is written in Zig, I ended up adding native Zig AST support in Dirac (https://github.com/dirac-run/dirac/blob/master/src/services/...) One thing the has been a little unintuitive is the pattern of all code and tests in single files, which makes the filesizes grow much larger. Also if you're coming from inheritance supported languages, Zig forces a different way of thinking

A new CLI for https://github.com/dirac-run/dirac and a paper that may or may not ever publish

For the most part, doing things right in the given language matters more than change of language. A lot of refactors in Rust (in the coding agent space) I see jump straight to Rust without considering what inefficiencies can be addressed before changing the language. Having said that, I considered a Go/Rust rewrite of Dirac (https://github.com/dirac-run/dirac) for some modules to support cases when someone wants to run like 30 agents, but it quickly became obvious that, a) while the node event loop is a bottleneck, it is not the sole bottleneck and b) if you have a VSCode extension, you can't totally get rid of TypeScript, so it just becomes the case of bi-lingual project and the maintenance burden that comes with it

Creator of Dirac here. I got something cooking https://github.com/dirac-run/dirac/issues/84

I just tried the Q4_K_M variant of this [] and this is one of the first models that run at ~20tps on my laptop. I also tried it with some "hard" maths questions and it clearly knows much. Can't wait to try some local coding agent harnesses with it (I recently discovered kon [1] and dirac [2] and wanted to try them out) The only thing I'm not sure about is if this model supports thinking or not. [1]: https://github.com/0xku/kon [2]: https://github.com/dirac-run/dirac

Saw this new Cline fork mentioned in another post: https://github.com/dirac-run/dirac Also heard of more and more people moving to Kilo Code or OpenChamber instead.

Currently, 14 most popular languages (https://github.com/dirac-run/dirac/tree/master/src/services/...). Easy to add more languages

It uses tree-sitter wasms. Currently, 14 languages are available (https://github.com/dirac-run/dirac/tree/master/src/services/...) The agent would work even without a language parser, just that the AST-based functionalities won't work

https://github.com/dirac-run/dirac#-evals README has eval of 8 tasks over 7 agents (including both pi and omp). Pi-mono costs second lowest across the 8 tasks (after Dirac) but occasionally misses produces incomplete changes. Interestingly, 2 tasks where pi missed some changes both were the tasks that benefitted from AST symbol understanding (e.g. find all instances of things that refer to this symbol and change those things). Since pi relies on bash type tooling, it missed some occurrences