LLMs Fail a Simple Chess Puzzle: A Curious Test

The author tests various new LLMs with a simple chess puzzle featuring underpromotion and the 50-move rule. Despite guidance, most LLMs fail to solve it, highlighting limitations in logical reasoning and domain-specific knowledge. This quick test serves as a benchmark, sparking discussion on whether LLMs can ever reach master-level chess play without specialized training. The puzzle's simplicity and focus on a less common technique make it an insightful probe into LLM capabilities.