mithen

The promised recs for “videos about the reality of LLMs attempting to play chess” from the GothamChess channel.

• ChatGPT versus Stockfish (an actual, functional chess-playing program)
• ChatGPT versus GothamChess himself
• ChatGPT versus Google Bard (they both make absurd moves at each other)
• ChatGPT versus DeepSeek (it starts with valid moves, then deteriorates, and both boths keep giving nonsense explanations of their moves)
• The most recent ChatGPT versus Stockfish (this one has a twist at the end, I won’t even spoil it)
• He also did his own Chatbot Chess Championship (playlist is in reverse order, so I linked the last video in the list, which is the first game).

The host plays the games out on-screen for you, with explanations and commentary. These ones aren’t for serious chatbot-testing purposes, they’re for entertainment — so when the bots make up illegal moves, he usually just runs with them. Sometimes with narration like “and here ChatGPT summons an extra rook from another dimension” or “You might think this is just a pawn, but Grok knows it’s secretly a horse pawn!”

Once in a while, he’ll tell the bot its move is illegal. Some of them go into “yes, of course, you’re right, my mistake” sycophancy mode. Others just get weirder.

The bots teleport pieces through each other. Manifest already-taken pieces back from the Shadow Realm. Spawns more pieces than it had to start with. Move pieces in directions they don’t go. And just because it’s making up moves, doesn’t mean it’s making up good moves! Sometimes it takes its own pieces. Sometimes it puts itself in check!

Sometimes they also generate their opponent’s moves. Because “black moves 1” is typically followed by “white moves 2, black moves 3, white moves 4” — and the bots don’t actually have a meaningful sense of “stop auto-generating text at the end of move 1.”

I was curious if the LLM’s idea of moves included “making up whole new categories of pieces” or “moving to squares that aren’t on the 8×8 chess grid.” Haven’t seen either of those so far.

One thing I didn’t anticipate is, sometimes a bot tells the other player their move is illegal. Even when it’s not! Saying “there’s a piece in your way” (when there isn’t), or “the king can’t move to E7” (not for any rules-based reason, the bot was just gatekeeping E7).

The newer bots also give general paragraphs on “here’s the explanation for my move,” which are absolutely just LLM Word Salad(TM) made of chess words. As a person who knows Basic Chess Rules but doesn’t actively play the game, sometimes I need GothamChess’s breakdown to see why they’re nonsense. Other times it’s just the bot saying “I have put you in check!” when the other player is blatantly not in check.

The whole thing was very informative, and also really entertaining. (…And it doesn’t involve the chatbots doing anything consequential, so it’s a nice break from all the stories about LLMs putting someone’s life in danger.) Give it a look.