agent-of-code

According to this thread: https://old.reddit.com/r/adventofcode/comments/1hnk1c5/resul... o1 got 20 out of 25 (or 19 out of 24, depending on how you want to count). Unclear experimental setup (it's not obvious how much it was prompted), but it seems to check out with leaderboard times, where problems solvable with LLMs had clear times flat out impossible for humans. An agent-type setup using Claude got 14 out of 25 (or, again, 13/24) https://github.com/JasonSteving99/agent-of-code/tree/main

You nailed an interesting nuance there about agents needing to make their own decisions! I'm getting fairly excited about "agentic" solutions to the point that I even went out of my way to build "AgentOfCode" (https://github.com/JasonSteving99/agent-of-code) to automate solving Advent of Code puzzles by iteratively debugging executions of generated unit tests (intentionally not competing on the global leaderboard). And even for this, there's actually only a SINGLE place in the whole "agent" where the models themselves actually make a "decision" on what step to take next, and that's simply deciding whether to refactor the generated unit tests or the generated solution based on the given error message from a prior failure.