Can AI Replace $1M in Freelance Software Engineering? OpenAI's Latest Research

OpenAI's new paper, SWE-Lancer, benchmarks frontier AI models on real-world software development tasks. Using over 1400 Upwork freelance jobs (totaling over $1 million), the study divided tasks into individual contributor tasks (bug fixing, feature building) and engineering manager tasks (selecting the best solution). Even the top performer, Claude 3.5 Sonnet, only completed 33.7% of tasks, earning roughly $403,000. AI excelled at selecting solutions over creating them, suggesting initial applications might focus on code review and architectural decisions. This benchmark offers a concrete way to measure AI progress, helping leaders understand and predict AI's capabilities and impact.