terminal-bench-2-1

ard/backward scheduling > - P2P activation and gradient communication > - Scaled cross-entropy loss > - Required rank-specific return value > - Model structure restoration > > Validation: hook-based tests passed for world_size=1 and 2 (4 passed). Unless the user is tracking every line of reasoning, they wouldn't know that the agent deliberately found the solution online, as the agent withheld that information in it's final response. I had run thousands of tasks before seeing this behavior, the `torch-pipeline` task was only included in "full runs" as the majority of my runs were on a subset of commonly failing tasks, hence why the data is so low. And yes, feel free to rant on about the irony of this whole exercise, it certainly isn't lost on me! [0]https://github.com/jumploops/.com/commit/39b1791d3865a8566cb... [1]https://github.com/harbor-framework/terminal-bench-2-1/blob/...

he timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes [1] https://github.com/harbor-framework/terminal-bench-2-1/blob/... [2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...

because if the user were to run the same commands, they would conflict. - For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0]. - For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1]. These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively). The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice. [0] https://github.com/harbor-framework/terminal-bench-2-1/issue... [1] https://github.com/harbor-framework/terminal-bench-2-1/issue...