The Exploration Bottleneck in LLMs: The Next Frontier of Experience Collection

The success of large language models (LLMs) relies on massive pre-training on vast text data, a resource that will eventually be depleted. The future of AI will shift towards an "Era of Experience," where efficient collection of the right kind of experience beneficial to learning will be crucial, rather than simply stacking parameters. This article explores how pre-training implicitly solves part of the exploration problem and how better exploration leads to better generalization. The author proposes that exploration consists of two axes: "world sampling" (choosing learning environments) and "path sampling" (gathering data within environments). Future AI scaling should optimize the information density on these two axes, efficiently allocating computational resources instead of simply pursuing parameter scale or data volume.