Building an Autonomous LLM Game Master with Small Models and Synthetic Data
2025-05-29
This post details the journey of building an autonomous LLM Game Master for TTRPGs. Initially aiming for an agentic approach, the author opted for a bottom-up strategy to gain deeper understanding of model development. Due to limited compute, a small Qwen3 model was chosen, trained on the Shadowdark RPG rulebook processed via OCR into markdown. A Shadowdark QA Bench was created for evaluation, comparing several metrics before settling on keyword-based matching. After pretraining and knowledge augmentation (creating multiple restatements of the rulebook text), the model achieved a 60% accuracy on the benchmark, meeting the author's goal. The next step is assistant tuning.
Development
Synthetic Data