chess_gpt_eval

Relax. Anyone who's genuinely interested in the question will see with a few searches that LLMs can play chess fine, although the post-trained models mostly seem to be regressed. Problem is people are more interested in validating their own assumptions than anything else. https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2501.17186 https://github.com/adamkarvonen/chess_gpt_eval

>They still make these errors on anything that is out of distribution. There is literally a post in this thread linking to a chat where Sonnet failed a basic arithmetic puzzle: https://news.ycombinator.com/item?id=47051286 I thought we were talking about actual arithmetic not silly puzzles, and there are many human adults that would fail this, nevermind children. >LLMs can match an example at exactly that trivial level because it can be predicted from context. However, if you construct a more complex example with several rules, especially with rules that have contradictions and have specified logic to resolve conflicts, they fail badly. Even if that were true (Have you actually tried?), You do realize many humans would also fail once you did all that right ? >They can't even reliably play Chess or Poker without breaking the rules despite

>These things don't play at 1800 ELO Why are you saying 'these things'?. That statement is about a specific model which did play at that level and did not lose track of the pieces. There's no cheating or weirdness. https://github.com/adamkarvonen/chess_gpt_eval

LLMs playing chess isn't a big deal. You can train a model on chess games and it will play at a decent ELO and very rarely make illegal moves(i.e 99.8% legal move rate). There are a few such models around. I think post training messes with chess ability and Open ai et al just don't really care about that. But LLMs can play chess just fine. [0] https://arxiv.org/pdf/2403.15498v2 [1] https://github.com/adamkarvonen/chess_gpt_eval

LLMs can play chess fine. The best model you can play with is decent for a human - https://github.com/adamkarvonen/chess_gpt_eval SOTA models can't play it because these companies don't really care about it.

LLMs are capable of playing chess and 3.5 turbo instruct does so quite well (for a human) at 1800 ELO. Does this mean they can truly reason now ? https://github.com/adamkarvonen/chess_gpt_eval

3-turbo-instruct makes about 5 or less illegal moves in 8205. It's not here but turbo instruct has been evaled before. https://github.com/adamkarvonen/chess_gpt_eval

turbo instruct does play chess reasonably well. https://github.com/adamkarvonen/chess_gpt_eval Even the blog above says as much.

I suspect that as well, however, 3.5-turbo-instruct has been noted by other people to do much better at generating legal chess moves than the other models. https://github.com/adamkarvonen/chess_gpt_eval gave models "5 illegal moves before forced resignation of the round" and 3.5 had very few illegal moves, while 4 lost most games due to illegal moves.

You're right it's not in this blog but turbo-instruct's chess ability has been pretty thoroughly tested and it does play chess. https://github.com/adamkarvonen/chess_gpt_eval

It's not: 1. That would just be plain bizzare 2. It plays like what you'd expect from a LLM that could play chess. That is, level of play can be modulated by the prompt and doesn't manifest the same way shifting the level of stockfish etc does. Also the specific chess notation being prompted actually matters 3. It's sensitive to how the position came to be. Clearly not an existing chess engine. https://github.com/dpaleka/llm-chess-proofgame 4. It does make illegal moves. It's rare (~5 in 8205) but it happens. https://github.com/adamkarvonen/chess_gpt_eval 5. You can or well you used to be able to inspect the logprobs. I think Open AI have stopped doing this but the link in 4 does show the author inspecting it for Turbo instruct.

Gpt-3.5-turbo-instruct had something like 5(or less) illegal moves in 8205 https://github.com/adamkarvonen/chess_gpt_eval I expect the rest to be much worse if 4's performance is any indication

>I really think chess is just a terrible example though. You're really asking a lot of it but I'm honestly shocked it can do what it can. It seems to know some opening books, but falls down immediately. Which really makes sense because you'll find a lot of reading material on specific openings, but the problem space of the game is just too big to find texts about any given game state. Maybe if you have it reason about the board state and "think" about tactics you could push it farther. But we've already solved this. Just so you know, LLMs can play chess just fine. Like this isn't some inability. GPT-3.5-instruct-turbo plays at about 1800 Elo and makes legal moves consistently well past openings. https://github.com/adamkarvonen/chess_gpt_eval https://adamkarvonen.github.io/machine_learning/2024/01/03/c...

>What GPT3.5 does is not quite playing chess but creating realistic chess moves that a human might make. No it's playing games. And if you're not at about the level I spoke of, you will lose repeatedly over whatever number or stretch of games you imagine. https://github.com/adamkarvonen/chess_gpt_eval 3.5 Instruct (different model from regular 3.5 that can't play) can play chess. There's no trick. Any other framing seems like a meaningless distinction. The goal is to model the chess games and there's no better way to do that than to learn to play the game. >all it needs is grounding (by feeding it the current board state) The Model is already constructing a board state to play the game. https://www.neelnanda.io/mechanistic-interpretability/othell... >agency (RF to reward the model for winning games) Predict the next token loss is already