RL's GPT-3 Moment: The Rise of Replication Training

This article predicts a forthcoming 'GPT-3 moment' for reinforcement learning (RL), involving massive-scale training across thousands of diverse environments to achieve strong few-shot, task-agnostic abilities. This requires unprecedented scale and diversity in training environments, potentially equivalent to tens of thousands of years of 'model-facing task time'. The authors propose a new paradigm, 'replication training,' where AIs duplicate existing software products or features to create large-scale, automatically scoreable training tasks. While challenges exist, this approach offers a clear path to scaling RL, potentially enabling AIs to complete entire software projects autonomously.