step_game

For people interested in these kinds of benchmarks, I have two multiplayer, multi-round games: - Elimination Game Benchmark: Social Reasoning, Strategy, and Deception in Multi-Agent LLM Dynamics at https://github.com/lechmazur/elimination_game/ - Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure at https://github.com/lechmazur/step_game/

Yes, predefined strategies are very interesting to examine. I have two simple ones in another multi-agent benchmark, https://github.com/lechmazur/step_game (SilentGreedyPlayer and SilentRandomPlayer), and it's fascinating to see LLMs detect and respond to them. The only issue with including them here is that the cost of running a large set of games isn't trivial. Another multi-agent benchmark I'm currently developing, which involves buying and selling, will also feature many predefined strategies.

Author here - I'm planning to create game versions of this benchmark, as well as my other multi-agent benchmarks (https://github.com/lechmazur/step_game, https://github.com/lechmazur/pgg_bench/, and a few others I'm developing). But I'm not sure if a leaderboard alone would be enough for comparing LLMs to top humans, since it would require playing so many games that it would be tedious. So I think it would be just for fun.

I ran three more of my independent benchmarks: - Improves upon GPT-4o's score on the Short Story Creative Writing Benchmark, but Claude Sonnets and DeepSeek R1 score higher. (https://github.com/lechmazur/writing/) - Improves upon GPT-4o's score on the Confabulations/Hallucinations on Provided Documents Benchmark, nearly matching Gemini 1.5 Pro (Sept) as the best-performing non-reasoning model. (https://github.com/lechmazur/confabulations) - Improves upon GPT-4o's score on the Thematic Generalization Benchmark, however, it doesn't match the scores of Claude 3.7 Sonnet or Gemini 2.0 Pro Exp. (https://github.com/lechmazur/generalization) I should have the results from the multi-agent collaboration, strategy, and deception benchmarks within a couple of days. (https://github.com/lechmazur/elimination_game/, https://github.com/lechmazur/step_ga

I have a set of independent benchmarks and most also show a difference between reasoning and non-reasoning models: LLM Confabulation (Hallucination): https://github.com/lechmazur/confabulations/ LLM Step Game: https://github.com/lechmazur/step_game LLM Thematic Generalization Benchmark: https://github.com/lechmazur/generalization LLM Creative Story-Writing Benchmark: https://github.com/lechmazur/writing Extended NYT Connections LLM Benchmark: https://github.com/lechmazur/nyt-connections/ and a couple more that I haven't updated very recently.